CIS5650-Fall-2024 · carlos-lopez-garces · Oct 24, 2024 · Oct 24, 2024 · Oct 24, 2024 · Oct 24, 2024
diff --git a/.gitignore b/.gitignore
@@ -24,4 +24,5 @@ dist-ssr
 *.sw?
 /.vite
 
-*/scenes
+*/scenes
+scenes/*
diff --git a/README.md b/README.md
@@ -1,26 +1,120 @@
 # Project5-WebGPU-Gaussian-Splat-Viewer
 
-**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4**
+**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* CARLOS LOPEZ GARCES
+  * [LinkedIn](https://www.linkedin.com/in/clopezgarces/)
+  * [Personal website](https://carlos-lopez-garces.github.io/)
+* Tested on: Windows 11, 13th Gen Intel(R) Core(TM) i9-13900HX @ 2.20 GHz, RAM 32GB, NVIDIA GeForce RTX 4060, personal laptop.
 
-### Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+### [Live Demo](http://carlos-lopez-garces.github.io/Penn-CIS-5650-Project5-WebGPU-Gaussian-Splat-Viewer)
+
+[![](images/bicycle_1.png)](http://carlos-lopez-garces.github.io/Penn-CIS-5650-Project5-WebGPU-Gaussian-Splat-Viewer)
 
 ### Demo Video/GIF
 
-[![](img/video.mp4)](TODO)
+| Bonsai | Bicycle |
+|----------|----------|
+| ![](images/bonsai_1.gif)   | ![](images/bicycle_1.gif)   |
+| **Truck** | **Train** |
+| ![](images/truck_1.gif)   | ![](images/train_1.gif)   |
+
+## WebGPU Gaussian Splat and Point Cloud Viewer
+
+This project implements a viewer and renderer for point clouds and Gaussian splats. The renderer implements the rasterizer of the paper "3D Gaussian Splatting
+for Real-Time Radiance Field Rendering" ([paper website](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/)).
+
+The viewer handles well many of the common scenes typically used by academic papers to present their results.
+
+| Bonsai | Bicycle |
+|----------|----------|
+| ![](images/bonsai_1.png)   | ![](images/bicycle_1.png)   |
+| **Truck** | **Train** |
+| ![](images/truck_1.png)   | ![](images/train_1.png)   |
+
+### Features
+
+- Real-Time Point Cloud Rendering: Visualize dense point cloud data from .ply files.
+
+- Gaussian Splatting: Reconstruct the radiance field of a scene from its point cloud using Gaussian Splatting.
+
+- Camera Controls: Pan, zoom, and rotate for detailed examination of Gaussian splats and point clouds.
+
+![](images/bonsai_3.gif)
+
+A point cloud is the main input to the Gaussian splatting rendering process.
+
+| Bonsai | Bicycle |
+|----------|----------|
+| ![](images/pc_bonsai_1.gif)   | ![](images/pc_bicycle_1.gif)   |
+| **Truck** | **Train** |
+| ![](images/pc_truck_1.gif)   | ![](images/pc_train_1.gif)   |
+
+### Gaussian Splat Preprocesing
+
+The preprocessing step transforms point data into splat representations for rendering by calculating 2D covariances, projecting spherical-harmonics-based color, and organizing/sorting splats for rendering based on depth.
+
+- Frustum culling checks if the splat falls within the visible area. Culling early reduces unnecessary computations for off-screen points.
+
+- The Gaussian’s scale and rotation are transformed into a 3D covariance matrix. This 3D covariance is projected to 2D to represent screen-space influence, helping to control the splat’s shape based on viewpoint.
+
+- Conic parameters are derived from the 2D covariance, resulting in a conic equation that approximates Gaussian distribution across x and y directions. Eigenvectors of the covariance provide major and minor axes, which are scaled to determine the Gaussian’s radius on the screen.
+
+- Spherical harmonics coefficients approximate lighting effects for each Gaussian based on its orientation.
+
+![](images/bicycle_2.gif)
+
+### Gaussian Splat Rendering
+
+For each splat, six vertices are generated to define a bounding quadrilateral in screen space. These vertices form a rectangle around each splat, with edges aligned based on the splat’s screen-space radius.
+
+In the fragment shader, the screen-space offset from the splat’s center to the fragment position is computed. A linear combination with the coefficients of a conic form is used to obtain the exponent by which the splat's opacity decays. Its color is blended with the splat's opacity, which finalizes the reconstruction of the radiance field.
+
+![](images/squares.png)
+
+### Performance Analysis
+
+The following table and chart show a few statistics about each of the scenes that were tested and the frames per second range achieved by the viewer for both the point cloud and the Gaussian splats. Because the process to render Gaussian splats is much more involved (preprocessing, data transfer, more complex computations per vertex and fragment), they consistently reach a much lower FPS. Also, FPS decreases with the number of points in the point cloud. FPS ranges are very wide and depend significantly on the camera angle; here they are shown from the default camera angle of the respective scene.
+
+Observe also that the size of the point cloud has an effect on FPS, but it's not the only factor to consider (note that the truck scene is twice the size of the train scene and they have comparable FPS ranges in the Gaussian splat case). In general, as the number of Gaussians increases, so does the FPS; the preprocessing step processes each of them (culled excluded), so FPS increases proportional. But as I said before, it's not the only factor; I investigate this phenomenon further below.
+
+| Scene  | Points  | Time (ms) | Point Cloud FPS | Gaussian Splat FPS |
+|--------|---------|-----------|-----------------|--------------|
+| Bonsai | 1,244,819 | 19,597    | 160         | 84       |
+| Bicycle| 1,063,091 | 25,271    | 150-181     | 54-85    |
+| Train  | 1,026,508 | 15,285    | 121-255     | 32-45    |
+| Truck  | 2,541,226 | 40,027    | 166-175     | 32-41    |
+
+![](images/Point%20Cloud%20vs%20Gaussian%20Splats%20FPS%20Across%20Scenes.png)
+
+Frustum culling is gives a very modest improvement in FPS, cutting down on the number of Gaussians processed per frame. In this example, where I've zoomed into Bicycle scene, leaving some points out of view, 15 fps was registered without frustum culling compared to 21 with frustum culling. Since the preprocessing compute shader runs simultaneously on the GPU, perhaps the work group size in this case (256) is such that Gaussians that are otherwise culled are processed simultaneously with the visible ones, so that both with and without culling the workgroups finish roughly in the same time.
+
+![](images/bicycle_closeup.png)
+
+![](images/bonsai_2.gif)
+
+Work group size has a two-fold effect: (1) render quality is optimal for a group size of 256 (sizes 64, 128, and 512 result in artifacts); (2) the FPS range reached with each of the sizes varies, with size 256 resulting in the highest FPS. A work group size of 256 allows for a high degree of parallelism, which is crucial for achieving high FPS. Smaller sizes might not leverage the full parallel processing power of the GPU, while larger sizes might introduce overhead that reduces performance.
+
+| Work Group Size | FPS Range |
+|-----------------|-----------|
+| 256             | 51 - 84   |
+| 128             | 11 - 71   |
+| 64              | 14 - 32   |
+
+### Artifacts During Development
+
+![](images/blooper_4.png)
+
+From not indexing the spherical harmonics coefficients correctly.
 
-### (TODO: Your README)
+![](images/blooper_1.png)
+![](images/blooper_2.png)
+![](images/blooper_3.png)
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+Flickering.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+![](images/bloopers_5.png)
 
 ### Credits
 

diff --git a/images/Point Cloud vs Gaussian Splats FPS Across Scenes.png b/images/Point Cloud vs Gaussian Splats FPS Across Scenes.png
diff --git a/images/bicycle_1.gif b/images/bicycle_1.gif
diff --git a/images/bicycle_1.png b/images/bicycle_1.png
diff --git a/images/bicycle_2.gif b/images/bicycle_2.gif
diff --git a/images/bicycle_closeup.png b/images/bicycle_closeup.png
diff --git a/images/blooper_1.png b/images/blooper_1.png
diff --git a/images/blooper_2.png b/images/blooper_2.png
diff --git a/images/blooper_3.png b/images/blooper_3.png
diff --git a/images/blooper_4.png b/images/blooper_4.png
diff --git a/images/bloopers_5.png b/images/bloopers_5.png
diff --git a/images/bonsai_1.gif b/images/bonsai_1.gif
diff --git a/images/bonsai_1.png b/images/bonsai_1.png
diff --git a/images/bonsai_2.gif b/images/bonsai_2.gif
diff --git a/images/bonsai_3.gif b/images/bonsai_3.gif
diff --git a/images/pc_bicycle_1.gif b/images/pc_bicycle_1.gif
diff --git a/images/pc_bonsai_1.gif b/images/pc_bonsai_1.gif
diff --git a/images/pc_train_1.gif b/images/pc_train_1.gif
diff --git a/images/pc_truck_1.gif b/images/pc_truck_1.gif
diff --git a/images/squares.png b/images/squares.png
diff --git a/images/train_1.gif b/images/train_1.gif
diff --git a/images/train_1.png b/images/train_1.png
diff --git a/images/truck_1.gif b/images/truck_1.gif
diff --git a/images/truck_1.png b/images/truck_1.png
diff --git a/src/renderers/gaussian-renderer.ts b/src/renderers/gaussian-renderer.ts
@@ -4,6 +4,8 @@ import renderWGSL from '../shaders/gaussian.wgsl';
 import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
+const c_size_splat = 4 * 2 * 3;
+
 export interface GaussianRenderer extends Renderer {
 
 }
@@ -35,6 +37,23 @@ export default function get_renderer(
   // ===============================================
 
   const nulling_data = new Uint32Array([0]);
+  const reset_buffer = createBuffer(
+		device, 'reset buffer', 4, GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST, nulling_data
+	);
+
+	const splat_buffer = createBuffer(
+		device, 'splat buffer', pc.num_points*c_size_splat, GPUBufferUsage.STORAGE
+	);
+
+  const indirect_render_buffer = createBuffer(
+		device, 'indirect buffer', 16, GPUBufferUsage.INDIRECT | GPUBufferUsage.COPY_DST,
+		new Uint32Array([6, 0, 0, 0])
+	);
+
+	const settings_buffer = createBuffer(
+		device, 'settings buffer', 8, GPUBufferUsage.COPY_DST | GPUBufferUsage.UNIFORM, 
+		new Float32Array([1.0, pc.sh_deg])
+	);
 
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
@@ -52,9 +71,21 @@ export default function get_renderer(
     },
   });
 
+  const preprocess_bind_group = device.createBindGroup({
+    label: 'preprocess bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } },
+      { binding: 1, resource: { buffer: settings_buffer } },
+      { binding: 2, resource: { buffer: pc.sh_buffer } },
+      { binding: 3, resource: { buffer: pc.gaussian_3d_buffer } },
+      { binding: 4, resource: { buffer: splat_buffer } }
+    ],
+  });
+
   const sort_bind_group = device.createBindGroup({
-    label: 'sort',
-    layout: preprocess_pipeline.getBindGroupLayout(2),
+    label: 'sort bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(1),
     entries: [
       { binding: 0, resource: { buffer: sorter.sort_info_buffer } },
       { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_depths_buffer } },
@@ -63,23 +94,83 @@ export default function get_renderer(
     ],
   });
 
-
   // ===============================================
   //    Create Render Pipeline and Bind Groups
   // ===============================================
-
+
+  const render_pipeline = device.createRenderPipeline({
+    label: 'render pipeline',
+    layout: 'auto',
+    vertex: {
+      entryPoint: 'vs_main',
+      module: device.createShaderModule({code: renderWGSL})
+    },
+    fragment: {
+      entryPoint: 'fs_main',
+      module: device.createShaderModule({code: renderWGSL}),
+      targets: [{
+        format: presentation_format,
+        blend: {
+          color: { srcFactor: 'one', dstFactor: 'one-minus-src-alpha' },
+          alpha: { srcFactor: 'one', dstFactor: 'one-minus-src-alpha' },
+        },
+      }]
+    },
+});
+
+  const render_bind_group = device.createBindGroup({
+    label: 'render bind group',
+    layout: render_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } },
+      { binding: 1, resource: { buffer: sorter.ping_pong[1].sort_indices_buffer } },
+      { binding: 2, resource: { buffer: splat_buffer } }
+    ]
+  });
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
-
+
+  const preprocess = (encoder: GPUCommandEncoder) => {
+    encoder.copyBufferToBuffer(reset_buffer, 0, sorter.sort_info_buffer, 0, 4);
+		encoder.copyBufferToBuffer(reset_buffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4);
+
+		const pass = encoder.beginComputePass({ label: 'preprocess pass' });
+		pass.setPipeline(preprocess_pipeline);
+		pass.setBindGroup(0, preprocess_bind_group);
+		pass.setBindGroup(1, sort_bind_group);
+		pass.dispatchWorkgroups(Math.ceil(pc.num_points / C.histogram_wg_size));
+		pass.end();
+  };
+
+  const render = (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+    encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, indirect_render_buffer, 4, 4);
+
+    const pass = encoder.beginRenderPass({
+      label: 'render pass',
+      colorAttachments: [{
+        view: texture_view,
+        loadOp: 'clear',
+        clearValue: { r: 0, g: 0, b: 0, a: 1 },
+        storeOp: 'store'
+      }],
+    });
+
+    pass.setPipeline(render_pipeline);
+    pass.setBindGroup(0, render_bind_group);
+    pass.drawIndirect(indirect_render_buffer, 0);
+    pass.end();
+  };
 
   // ===============================================
   //    Return Render Object
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+      preprocess(encoder);
       sorter.sort(encoder);
+      render(encoder, texture_view);
     },
     camera_buffer,
   };

diff --git a/src/shaders/gaussian.wgsl b/src/shaders/gaussian.wgsl
@@ -1,22 +1,76 @@
-struct VertexOutput {
-    @builtin(position) position: vec4<f32>,
-    //TODO: information passed from vertex shader to fragment shader
+struct CameraUniforms {
+    view: mat4x4<f32>,
+    view_inv: mat4x4<f32>,
+    proj: mat4x4<f32>,
+    proj_inv: mat4x4<f32>,
+    viewport: vec2<f32>,
+    focal: vec2<f32>
 };
 
 struct Splat {
-    //TODO: information defined in preprocess compute shader
+    mean_xy: u32,
+    radii: u32,
+    conic_xy: u32,
+    conic_z: u32,
+    rgb_rg: u32,
+    rgb_b_opacity: u32
+};
+
+@group(0) @binding(0)
+var<uniform> camera: CameraUniforms;
+@group(0) @binding(1)
+var<storage, read> sort_indices : array<u32>;
+@group(0) @binding(2)
+var<storage, read> splats: array<Splat>;
+
+const signs = array<vec2f, 6>(
+    vec2f(-1.0, 1.0), vec2f(-1.0, -1.0),
+    vec2f(1.0, -1.0), vec2f(1.0, -1.0),
+    vec2f(1.0, 1.0), vec2f(-1.0, 1.0)
+);
+
+struct VertexOutput {
+    @builtin(position) position: vec4<f32>,
+    @location(0) mean: vec2f,
+    @location(1) rgb_opacity: vec4f,
+    @location(2) conic: vec3f
 };
 
 @vertex
-fn vs_main(
-) -> VertexOutput {
-    //TODO: reconstruct 2D quad based on information from splat, pass 
-    var out: VertexOutput;
-    out.position = vec4<f32>(1. ,1. , 0., 1.);
-    return out;
+fn vs_main(@builtin(vertex_index) vertex_index: u32, @builtin(instance_index) instance_index: u32) -> VertexOutput {
+    let splat = splats[sort_indices[instance_index]];
+
+    let mean_xy = unpack2x16float(splat.mean_xy);
+    let diameter = 2.0 * unpack2x16float(splat.radii);
+    let rgb_rg = unpack2x16float(splat.rgb_rg);
+    let rgb_b_opacity = unpack2x16float(splat.rgb_b_opacity);;
+    let conic_xy = unpack2x16float(splat.conic_xy);
+    let conic_z = unpack2x16float(splat.conic_z);
+
+    let sign = signs[vertex_index];
+
+    return VertexOutput(
+        vec4(mean_xy.x + sign.x*diameter.x, mean_xy.y + sign.y*diameter.y, 0.0, 1.0),
+        vec2f(mean_xy.x, mean_xy.y),
+        vec4f(rgb_rg.x, rgb_rg.y, rgb_b_opacity.x, 1.0/(1.0+exp(-rgb_b_opacity.y))),
+        vec3f(conic_xy.x, conic_xy.y, conic_z.x),
+    );
 }
 
 @fragment
-fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
-    return vec4<f32>(1.);
+fn fs_main(input: VertexOutput) -> @location(0) vec4<f32> {
+    var pos_ndc = 2.0 * (input.position.xy / camera.viewport) - vec2(1.0, 1.0);
+    pos_ndc.y = -pos_ndc.y;
+
+    // Screen-space offset from the fragment position, with x-coordinate reversed.
+    var offset_screen = pos_ndc - input.mean;
+    offset_screen.x = -offset_screen.x;
+    offset_screen *= camera.viewport * 0.5;
+
+    var exponent = 
+        input.conic.x * offset_screen.x * offset_screen.x
+        + input.conic.z * offset_screen.y * offset_screen.y
+        + input.conic.y * offset_screen.x * offset_screen.y;
+
+    return vec4f(input.rgb_opacity.xyz, 1.0) * input.rgb_opacity.w * exp(-exponent/2.0);
 }