Project 5: Aaron Tian

.github/workflows/static.yml

@@ -33,7 +33,7 @@ jobs:
       - name: Set up Node
         uses: actions/setup-node@v4
         with:
-          node-version: 20
+          node-version: 22
           cache: 'npm'
       - name: Install dependencies
         run: npm ci

.gitignore

@@ -24,4 +24,4 @@ dist-ssr
 *.sw?
 /.vite
 
-*/scenes
+scenes/

README.md

@@ -2,30 +2,65 @@
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Aaron Tian
+  * [LinkedIn](https://www.linkedin.com/in/aaron-c-tian/), [personal website](https://aarontian-stack.github.io/)
+* Tested on: Windows 22H2 (26100.6584), Intel Core Ultra 7 265k @ 3.90GHz, 32GB RAM, RTX 5070 12GB (release driver 581.15)
 
-### Live Demo
+## Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+[Link](https://aarontian-stack.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/)
 
-### Demo Video/GIF
+[![live demo](images/demo.gif)](https://aarontian-stack.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/)
 
-[![](img/video.mp4)](TODO)
+## Summary
 
-### (TODO: Your README)
+A WebGPU app for rendering Gaussian splats as described in [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). This allows for real time rendering of high quality radiance fields.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+## Implementation
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+The app renders pretrained Gaussian splat models in the `.ply` format. A preprocessing compute shader is used to perform tasks such as transforming the Gaussians' positions, calculating the covariance matrix, and calculating size and color (this is the "splatting"). The Gaussians are then sorted back-to-front using radix sort. Finally the Gaussians (now splats) are rendered as quads using a single indirect draw call that had its instance count set in the preprocessing step.
 
-### Credits
+### Preprocessing Compute Shader
+
+Gaussian positions are transformed into NDC space using the camera's view-projection matrix. The covariance matrix is calculated from the Gaussian's rotation and scale. Each Gaussian stores spherical harmonic coefficients for calculating color based off the view direction. This information is appended to a buffer of 2D splats to be rendered along with a separate buffer for sorting those splats. This also keeps track of the number of splats that survive view-frustum culling, allowing the number of splats to be sorted to be copied to the indirect draw buffer later (the workgroup size for the sorting pass is also calculated here).
+
+### Rendering
+
+Each splat is rendered as a quad. The size simply comes from the result in the preprocessing step. For each fragment of the quad we determine if it is inside the splat (ellipse) using the centered matrix equation. If the test passes, we output the color of the splat. The opacity of the color decays exponentially based off the distance of the center of the splat.
+
+## Performance Analysis
+
+With either the Bicycle or Bonsai splat scenes, the average frame rate is the same and never drops below my max refresh rate of 100 FPS, regardless of settings, so I discuss theoretical performance implications of different settings below.
+
+### Point Cloud vs Gaussian Splatting
+
+The point cloud renders each Gaussian using point primitives, so no triangles are involved compared to splatting which uses quads (2 triangles per Gaussian). There is also no shading work in the fragment shader for point clouds (outputs constant color). Thus the Gaussian splatting incurs the additional costs of:
+* Triangle rasterization
+* Shading/Blending
+
+### Workgroup Size
+
+A larger workgroup size in the preprocessing step may offer benefits such as better memory coalescing, especially considering our work is a 1D dispatch. We do not use shared memory in the shader, so that is not relevant here. If the preprocessing step uses too many registers on a CU it might be better to reduce the workgroup size to mitigate spilling to shared memory. However to my knowledge it is not possible to determine this kind of information in WebGPU.
+
+
+### View-frustum Culling
+
+By culling splats that are outside the view-frustum, we can reduce the number of splats that need to be sorted and rendered. This should reduce compute pressure (from sorting step) and graphics pipeline pressure (less rasterization, shading, blending). The check for culling in the preprocessing shader is very simple, but it causes an early return. Depending on the order of the Gaussians in memory, this could cause some divergence in the preprocessing step.
+
+### Number of Gaussians
+
+A larger amount of Gaussians should increase the workload in all parts of the application. Loading from disk will take longer, preprocessing will take longer, sorting will take longer, and rendering will take longer due to more primitives.
+
+## Bloopers
+
+My Gaussians got flattened into a line...
+
+![wtf](images/wtf.png)
+
+## Credits
 
 - [Vite](https://vitejs.dev/)
 - [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/)
 - [stats.js](https://github.com/mrdoob/stats.js)
 - [wgpu-matrix](https://github.com/greggman/wgpu-matrix)
-- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Guassian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization)
+- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Gaussian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization)

src/renderers/gaussian-renderer.ts

@@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
 export interface GaussianRenderer extends Renderer {
-
+  update_scaling: (scaling: number) => void;
 }
 
 // Utility to create GPU buffers
@@ -35,13 +35,120 @@ export default function get_renderer(
   // ===============================================
 
   const nulling_data = new Uint32Array([0]);
+  // To clear number of points to 0 before preprocess
+  const null_buffer = createBuffer(
+    device,
+    'null buffer',
+    4,
+    GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST,
+    nulling_data
+  );
+  device.queue.writeBuffer(null_buffer, 0, nulling_data);
+
+  // vertexCount: u32, instanceCount: u32, firstVertex: u32, firstInstance: u32
+  const indirect_draw_buffer = createBuffer(
+    device,
+    'indirect draw',
+    4 * 4,
+    GPUBufferUsage.INDIRECT | GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST,
+    new Uint32Array([6, pc.num_points, 0, 0])
+  );
+
+  const splat_buffer = createBuffer(
+    device,
+    'splat buffer',
+    pc.num_points * 5 * 4,
+    GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST
+  );
+
+  const render_settings_buffer = createBuffer(
+    device,
+    'render settings',
+    8, // 2 f32
+    GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
+    new Float32Array([1.0, pc.sh_deg]) // default scaling 1.0
+  );
+
+  // 'auto' seemingly removes unused bindings!!! So make them manually
+
+  const camera_bind_group_layout = device.createBindGroupLayout({
+    label: 'camera bind group layout',
+    entries: [
+      {
+        binding: 0,
+        visibility: GPUShaderStage.COMPUTE | GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT,
+        buffer: { type: 'uniform' }
+      },
+      {
+        binding: 1,
+        visibility: GPUShaderStage.COMPUTE | GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT,
+        buffer: { type: 'uniform' }
+      }
+    ]
+  });
+
+  const gaussian_bind_group_layout = device.createBindGroupLayout({
+    label: 'gaussian bind group layout',
+    entries: [
+      {
+        binding: 0,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'read-only-storage' }
+      },
+      {
+        binding: 1,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'storage' }
+      },
+      {
+        binding: 2,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'read-only-storage' }
+      }
+    ]
+  });
+
+  const sort_bind_group_layout = device.createBindGroupLayout({
+    label: 'sort bind group layout',
+    entries: [
+      {
+        binding: 0,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'storage' }
+      },
+      {
+        binding: 1,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'storage' }
+      },
+      {
+        binding: 2,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'storage' }
+      },
+      {
+        binding: 3,
+        visibility: GPUShaderStage.COMPUTE,
+        buffer: { type: 'storage' }
+      }
+    ]
+  });
+
+  const preprocess_pipeline_layout = device.createPipelineLayout({
+    label: 'preprocess pipeline layout',
+    bindGroupLayouts: [
+      camera_bind_group_layout,
+      gaussian_bind_group_layout,
+      sort_bind_group_layout
+    ]
+  });
 
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
   // ===============================================
   const preprocess_pipeline = device.createComputePipeline({
     label: 'preprocess',
-    layout: 'auto',
+    layout: preprocess_pipeline_layout,
     compute: {
       module: device.createShaderModule({ code: preprocessWGSL }),
       entryPoint: 'preprocess',
@@ -54,7 +161,7 @@ export default function get_renderer(
 
   const sort_bind_group = device.createBindGroup({
     label: 'sort',
-    layout: preprocess_pipeline.getBindGroupLayout(2),
+    layout: sort_bind_group_layout,
     entries: [
       { binding: 0, resource: { buffer: sorter.sort_info_buffer } },
       { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_depths_buffer } },
@@ -68,19 +175,147 @@ export default function get_renderer(
   //    Create Render Pipeline and Bind Groups
   // ===============================================
 
+  const render_splat_bind_group_layout = device.createBindGroupLayout({
+    label: 'render splat bind group layout',
+    entries: [
+      {
+        binding: 0,
+        visibility: GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT,
+        buffer: { type: 'read-only-storage' }
+      },
+      {
+        binding: 1,
+        visibility: GPUShaderStage.VERTEX,
+        buffer: { type: 'read-only-storage' }
+      }
+    ]
+  });
+
+  const render_pipeline_layout = device.createPipelineLayout({
+    label: 'render pipeline layout',
+    bindGroupLayouts: [
+      camera_bind_group_layout,
+      render_splat_bind_group_layout
+    ]
+  });
+
+  const render_shader = device.createShaderModule({code: renderWGSL});
+  const render_pipeline = device.createRenderPipeline({
+    label: 'gaussian render',
+    layout: render_pipeline_layout,
+    vertex: {
+      module: render_shader,
+      entryPoint: 'vs_main',
+    },
+    fragment: {
+      module: render_shader,
+      entryPoint: 'fs_main',
+      targets: [{
+        format: presentation_format,
+        blend: {
+          color: {
+            srcFactor: 'src-alpha',
+            dstFactor: 'one-minus-src-alpha',
+            operation: 'add',
+          },
+          alpha: {
+            srcFactor: 'one',
+            dstFactor: 'one-minus-src-alpha',
+            operation: 'add',
+          },
+        },
+      }],
+    },
+    primitive: {
+      topology: 'triangle-list',
+      cullMode: 'none',
+      frontFace: 'ccw',
+    }
+  });
+
+  const camera_settings_bind_group = device.createBindGroup({
+    label: 'gaussian camera',
+    layout: camera_bind_group_layout,
+    entries: [
+      {binding: 0, resource: { buffer: camera_buffer }},
+      {binding: 1, resource: { buffer: render_settings_buffer }},
+    ],
+  });
+
+  const gaussian_splat_bind_group = device.createBindGroup({
+    label: 'gaussian splats',
+    layout: gaussian_bind_group_layout,
+    entries: [
+      { binding: 0, resource: { buffer: pc.gaussian_3d_buffer } },
+      { binding: 1, resource: { buffer: splat_buffer } },
+      { binding: 2, resource: { buffer: pc.sh_buffer } },
+    ],
+  });
+
+  const splat_bind_group = device.createBindGroup({
+    label: 'splat bind group',
+    layout: render_splat_bind_group_layout,
+    entries: [
+      { binding: 0, resource: { buffer: splat_buffer } },
+      { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } },
+    ],
+  });
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
-
+  let preprocess_func = (encoder: GPUCommandEncoder) => {
+    let preprocess_pass = encoder.beginComputePass({
+        label: 'preprocess',
+    });
+    preprocess_pass.setPipeline(preprocess_pipeline);
+    preprocess_pass.setBindGroup(0, camera_settings_bind_group);
+    preprocess_pass.setBindGroup(1, gaussian_splat_bind_group);
+    preprocess_pass.setBindGroup(2, sort_bind_group);
+    const workgroups_needed = Math.ceil(pc.num_points / C.histogram_wg_size);
+    preprocess_pass.dispatchWorkgroups(workgroups_needed);
+    preprocess_pass.end();
+  };
+
+  let render_func = (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+    const pass = encoder.beginRenderPass({
+      label: 'gaussian render',
+      colorAttachments: [
+        {
+          view: texture_view,
+          loadOp: 'clear',
+          storeOp: 'store',
+        }
+      ],
+    });
+    pass.setPipeline(render_pipeline);
+    pass.setBindGroup(0, camera_settings_bind_group);
+    pass.setBindGroup(1, splat_bind_group);
+
+    pass.drawIndirect(indirect_draw_buffer, 0);
+    pass.end();
+  };
 
   // ===============================================
   //    Return Render Object
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+
+      encoder.copyBufferToBuffer(null_buffer, 0, sorter.sort_info_buffer, 0, 4); // Clear keys_size to 0
+      encoder.copyBufferToBuffer(null_buffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4); // Clear dispatch_x to 0
+
+      preprocess_func(encoder);
+
       sorter.sort(encoder);
+
+      encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, indirect_draw_buffer, 4, 4); // Copy key size to indirect draw buffer
+
+      render_func(encoder, texture_view);
     },
     camera_buffer,
+    update_scaling: (scaling: number) => {
+      device.queue.writeBuffer(render_settings_buffer, 0, new Float32Array([scaling, pc.sh_deg]));
+    },
   };
 }