Project 5 - Yin Tang

.gitignore

@@ -24,4 +24,6 @@ dist-ssr
 *.sw?
 /.vite
 
-*/scenes
+*/scenes
+scene/
+*/package-lock.json

README.md

@@ -1,31 +1,37 @@
-# Project5-WebGPU-Gaussian-Splat-Viewer
+**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5 - WebGPU Gaussian Splat Viewer**
 
-**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4**
+![main](images/main.png)
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Yin Tang
+  * [Linkedin](https://www.linkedin.com/in/yin-tang-jackeyty/), [Github](https://github.com/JackeyTY), [Personal Website](https://jackeytang.com/)
+* Tested on: Google Chrome 130.0 on Windows 11 Pro, AMD Ryzen 9 7950X @ 5.00GHz 64GB, NVIDIA GeForce RTX 4090 24GB (personal desktop)
 
-### Live Demo
+<br>
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+### Overview
 
-### Demo Video/GIF
+In this project, I implemented a **3D Gaussian Splat Viewer** using **WebGPU**, which is based on the paper [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). Online interactive [demo](https://www.jackeytang.com/WebGPU-Gaussian-Splat-Viewer/), remember to download scene files first. 
 
-[![](img/video.mp4)](TODO)
+**Gaussian Splatting** is a novel technique in computer graphics and neural rendering that represents a scene or object using a collection of 3D Gaussian functions instead of traditional polygonal meshes or voxel grids. Each Gaussian, or "splat," has a position, size, orientation, and opacity, defining a smooth, continuous volume that approximates the underlying surface or appearance. This approach allows for efficient rendering, as the splats can be rasterized quickly, and their smooth nature helps reduce aliasing artifacts. Gaussian splatting is particularly well-suited for applications in real-time rendering, neural radiance fields (NeRFs), and point-based rendering, where it offers a compact and differentiable representation. By leveraging GPU acceleration, it enables high-quality rendering with fewer computational resources, making it a promising alternative for interactive graphics, virtual reality, and dynamic scene representations.
 
-### (TODO: Your README)
+![next](images/next.png)
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+<br>
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+### Performance Analysis
 
-### Credits
+- **Compare your results from point-cloud and gaussian renderer, what are the differences?**
 
-- [Vite](https://vitejs.dev/)
-- [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/)
-- [stats.js](https://github.com/mrdoob/stats.js)
-- [wgpu-matrix](https://github.com/greggman/wgpu-matrix)
-- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Guassian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization)
+​	Point-cloud is much easier to draw since only individual points are rendered, whereas gaussian splats render consists of more computation in the computer and render passes, which takes comparably large amount of time. On the other hand, the quality of the image rendered by the gaussian render is significantly better than that of point-cloud.
+
+- **For gaussian renderer, how does changing the workgroup-size affect performance? Why do you think this is?**
+
+​	Since the maximum total number of invocations per workgroup is 256, we can only decrease the workgroup size in the x-dimension, which results is worse performance since more workgroups need to be dispatched and less utilization.
+
+- **Does view-frustum culling give performance improvement? Why do you think this is?**
+
+​	It does improve the performance by around 5%, which is because we don't need to process out-of-bound splats so less computation and draw calls, but given the size of the scene, it is not evident.
+
+- **Does number of guassians affect performance? Why do you think this is?**
+
+​	As the number of gaussians, which is the upper bound for draw invocations, the performance decreases as more computation and draw calls need to be processed.

src/renderers/gaussian-renderer.ts

@@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
 export interface GaussianRenderer extends Renderer {
-
+  scaling_buffer: GPUBuffer
 }
 
 // Utility to create GPU buffers
@@ -34,53 +34,218 @@ export default function get_renderer(
   //            Initialize GPU Buffers
   // ===============================================
 
+  const scaling_buffer = createBuffer(
+    device, 
+    'render_settings_buffer', 
+    4,
+    GPUBufferUsage.COPY_DST | GPUBufferUsage.UNIFORM, 
+    new Float32Array([1.0])
+  );
+
   const nulling_data = new Uint32Array([0]);
 
+  const nulling_buffer = createBuffer(
+    device, 
+    'nulling buffer', 
+    4,
+    GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST, 
+    nulling_data
+  );
+
+  const splat_size = 24;
+
+  const splat_buffer = createBuffer(
+    device, 
+    'splat buffer', 
+    splat_size * pc.num_points, 
+    GPUBufferUsage.STORAGE, 
+    null
+  );
+
+  const indirect_buffer = createBuffer(
+    device, 
+    'indirect buffer', 
+    20, 
+    GPUBufferUsage.COPY_DST | GPUBufferUsage.INDIRECT,
+    new Uint32Array([6, 0, 0, 0, 0])
+  );
+
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
   // ===============================================
+
   const preprocess_pipeline = device.createComputePipeline({
-    label: 'preprocess',
+    label: 'preprocess pipeline',
     layout: 'auto',
     compute: {
       module: device.createShaderModule({ code: preprocessWGSL }),
       entryPoint: 'preprocess',
       constants: {
         workgroupSize: C.histogram_wg_size,
         sortKeyPerThread: c_histogram_block_rows,
-      },
-    },
+        shDegree: pc.sh_deg
+      }
+    }
+  });
+
+  const camera_bind_group = device.createBindGroup({
+    label: 'preprocess camera bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } }
+    ]
+  });
+
+  const gaussian_bind_group = device.createBindGroup({
+    label: 'preprocess gaussians bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(1),
+    entries: [
+      { binding: 0, resource: { buffer: pc.gaussian_3d_buffer } }
+    ]
   });
 
   const sort_bind_group = device.createBindGroup({
-    label: 'sort',
+    label: 'preprocess sort bind group',
     layout: preprocess_pipeline.getBindGroupLayout(2),
     entries: [
       { binding: 0, resource: { buffer: sorter.sort_info_buffer } },
       { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_depths_buffer } },
       { binding: 2, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } },
-      { binding: 3, resource: { buffer: sorter.sort_dispatch_indirect_buffer } },
-    ],
+      { binding: 3, resource: { buffer: sorter.sort_dispatch_indirect_buffer } }
+    ]
   });
 
+  const splat_preprocess_bind_group = device.createBindGroup({
+    label: 'preprocess splat bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(3),
+    entries: [
+      { binding: 0, resource: { buffer: splat_buffer } },
+      { binding: 1, resource: { buffer: scaling_buffer } },
+      { binding: 2, resource: { buffer: pc.sh_buffer } }
+    ]
+  });
 
   // ===============================================
   //    Create Render Pipeline and Bind Groups
   // ===============================================
-
+
+  const gaussian_render_pipeline = device.createRenderPipeline({
+    label: 'gaussian render pipeline',
+    layout: 'auto',
+    vertex: {
+      module: device.createShaderModule({ code: renderWGSL }),
+      entryPoint: 'vs_main'
+    },
+    fragment: {
+      module: device.createShaderModule({ code: renderWGSL }),
+      entryPoint: 'fs_main',
+      targets: [{ 
+        format: presentation_format,
+        blend: {
+          color: {
+              srcFactor: 'one',
+              dstFactor: 'one-minus-src-alpha',
+              operation: 'add'
+          },
+          alpha: {
+              srcFactor: 'one',
+              dstFactor: 'one-minus-src-alpha',
+              operation: 'add'
+          }
+        }
+      }]
+    }
+  });
+
+  const splat_render_bind_group = device.createBindGroup({
+    label: 'gaussian render splat bind group',
+    layout: gaussian_render_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: splat_buffer } },
+      { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } },
+      { binding: 2, resource: { buffer: camera_buffer } }
+    ]
+  });
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
+
+  const preprocess_pass = (encoder: GPUCommandEncoder) => {
+    const preprocess_pass = encoder.beginComputePass();
+
+    preprocess_pass.setPipeline(preprocess_pipeline);
+
+    preprocess_pass.setBindGroup(0, camera_bind_group);
+    preprocess_pass.setBindGroup(1, gaussian_bind_group);
+    preprocess_pass.setBindGroup(2, sort_bind_group);
+    preprocess_pass.setBindGroup(3, splat_preprocess_bind_group);
+
+    preprocess_pass.dispatchWorkgroups(Math.ceil(pc.num_points / C.histogram_wg_size));
+
+    preprocess_pass.end();
+  };
 
+  const gaussian_render_pass = (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+    const gaussian_pass = encoder.beginRenderPass({
+      label: 'gaussian render',
+      colorAttachments: [
+        {
+          view: texture_view,
+          loadOp: 'clear',
+          storeOp: 'store',
+          clearValue: [0, 0, 0, 1]
+        }
+      ],
+    });
+
+    gaussian_pass.setPipeline(gaussian_render_pipeline);
+    gaussian_pass.setBindGroup(0, splat_render_bind_group);
+
+    gaussian_pass.drawIndirect(indirect_buffer, 0);
+    gaussian_pass.end();
+  };
 
   // ===============================================
   //    Return Render Object
   // ===============================================
+
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+
+      encoder.copyBufferToBuffer(
+        nulling_buffer, 
+        0,
+        sorter.sort_info_buffer, 
+        0,
+        4
+      );
+
+      encoder.copyBufferToBuffer(
+        nulling_buffer, 
+        0,
+        sorter.sort_dispatch_indirect_buffer, 
+        0,
+        4
+      );
+
+      preprocess_pass(encoder);
+
       sorter.sort(encoder);
+
+      encoder.copyBufferToBuffer(
+        sorter.sort_info_buffer, 
+        0,
+        indirect_buffer, 
+        4,
+        4
+      );
+
+      gaussian_render_pass(encoder, texture_view);
+
     },
+
     camera_buffer,
+    scaling_buffer
   };
 }

src/renderers/renderer.ts

@@ -121,7 +121,9 @@ export default async function init(
       'gaussian_multiplier',
       {min: 0, max: 1.5}
     ).on('change', (e) => {
-      //TODO: Bind constants to the gaussian renderer.
+      if (gaussian_renderer) {
+        device.queue.writeBuffer(gaussian_renderer.scaling_buffer, 0, new Float32Array([params.gaussian_multiplier]));
+      }
     });
   }