Project 5: Marcus Hedlund

README.md

@@ -2,25 +2,44 @@
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Marcus Hedlund
+  * [LinkedIn](https://www.linkedin.com/in/marcushedlund/)
+* Tested on: Windows 11, Intel Core Ultra 9 185H @ 2.5 GHz 16GB, NVIDIA GeForce RTX 4070 Laptop GPU 8GB (Personal Computer)
+
 
 ### Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+[![](images/splat1.png)](http://mhedlund7.github.io/Project5-WebGPU-Gaussian-Splat-Viewer)
 
 ### Demo Video/GIF
 
-[![](img/video.mp4)](TODO)
+[![](images/bikegif.gif)](TODO)
+
+### Overview
+
+In this project I implement a real-time WebGPU Gaussian Splat Viewer that renders scenes using neural point clouds based on the paper [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). Scenes are loaded from .ply files and can be displayed as either point clouds or Gaussian splats. The Gaussian splat method uses 3D Gaussian distributions in order to represent geometry. It maps these 3D Gaussians to 2D screen-space conics using the Jacobian of the projection matrix, and performs View-Frustrum culling to only evaluate Gaussians that the camera can see. Additionally, during preprocessing it leverages spherical harmonics to evaluate color and sorts the splats by depth. Then in the rendering pass, we render a quad per splat and use the conic to apply exponential falloff for opacity and composite the Gaussians through alpha-blending to produce the final image.
+
+|![](images/pointcloud2.png)|![](images/splat2.png)|
+|:--:|:--:|
+| *Point Cloud Renderer* | *Gaussian Splat Renderer* |
+
+#### Comparing Point Cloud and Gaussian Renderer
+
+Visually, the point cloud method renders the scene as just that, a collection of points each of the same color. Surprisinngly this method is actually able to do a fairly good job of encoding what is represented in the scene as can be seen in the above image, but it still leaves much to be desired. Gaussian splatting on the other hand is able to leverage the process to fully recreate a scene. It has accurate color, texture, and overall has an extrextremelymely realistic appearance. Performance-wise though as expected the point cloud method performs much better. In this case for the above scene I got 4.11 ms/frame for the point cloud and 6.58 ms/frame for the Gaussian Splasplattingtting. This makes a lot of sense because of how much more involved the Gaussian splatting is, having to perform many precomputations like 3D covariance, 2D conics, color, sorting, and opacity falloff instead of just creating points. 
+
+#### Workgroup Size Effect on Performance
+
+Decreasing the workgroup size will generally lower the performance of Gaussian splatting. This is because it makes it so not as many splats can be processed in parallel, slowing down the computation. For me, I achieved maximum performance at a workgroup of size 256. This is likely because at higher workgroup sizes there is too high of an overhead in both dispatch time and memory pressure, and the GPU will already be saturated so the additional workgroups don't fully leverage additional parallelism, resulting in slower computation.
+
+#### View-Frustrum Culling Effect on Performance
+
+I additionally performed view-frustrum culling during preprocessing by creating a bounding box around the camera's frustrum (extended by 1.2x in x and y to account for gaussians slightly outside the frustrum that could still influence the image) and not processing gaussians that fall outside that range. This was able to greatly improve performance when the camera was positioned closer to the scene so that not all of the gaussians were in sight, however there was close to no effect when the camera was positioned farther away so that all gaussians were visible. This makes sense because when we are actually able to cull Gaussians and not process them through View-Frustrum Culling it just leads to us having to process fewer items in each of the subsequent steps, leading to quicker computation time.
+
+#### Number of Gaussians Effect on Performance
 
-### (TODO: Your README)
+We are able to compare the effect the number of Gaussians has on performance by looking at different scenes with a higher or lower amount of points in them. As can be expected increasing the number of Gaussians in a scene decreases the performance of our viewer. This is because we will have a larger number of Gaussians to compute information for and process, leading to higher compute time and lower performance.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
 
 ### Credits
 

src/renderers/gaussian-renderer.ts

@@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
 export interface GaussianRenderer extends Renderer {
-
+  setScalingMultiplier: (multiplier: number) => void;
 }
 
 // Utility to create GPU buffers
@@ -36,6 +36,47 @@ export default function get_renderer(
 
   const nulling_data = new Uint32Array([0]);
 
+  // settings for gaussian rendering (scaling, sh_deg)
+  const gaussian_rendering_settings_array = new Float32Array(4);
+  gaussian_rendering_settings_array[0] = 1.0; // scaling multiplier
+  gaussian_rendering_settings_array[1] = pc.sh_deg; // padding
+
+  const gaussian_rendering_settings_buffer = createBuffer(
+    device,
+    'gaussian uniforms',
+    gaussian_rendering_settings_array.byteLength,
+    GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
+    gaussian_rendering_settings_array
+  );
+
+  // splat buffer
+  const splatSize = 56;
+  const splatBuffer = createBuffer(
+    device,
+    'splat buffer',
+    pc.num_points * splatSize,
+    GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST,
+  );
+
+  // indirect draw
+  const drawArgsArray = new Uint32Array([6, 0, 0, 0]); // vertexCount, instanceCount, firstVertex, firstInstance
+  const indirectDrawBuffer = createBuffer(
+    device,
+    'indirect draw buffer',
+    drawArgsArray.byteLength,
+    GPUBufferUsage.INDIRECT | GPUBufferUsage.COPY_DST,
+    drawArgsArray,
+  );
+
+  // nulling buffer
+  const nulling_buffer = createBuffer(
+    device,
+    'nulling buffer',
+    nulling_data.byteLength,
+    GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST,
+    nulling_data,
+  );
+
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
   // ===============================================
@@ -52,6 +93,26 @@ export default function get_renderer(
     },
   });
 
+
+  const preprocess_camera_bind_group = device.createBindGroup({
+    label: 'camera bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } },
+    ],
+  });
+
+  const preprocess_gaussian_bind_group = device.createBindGroup({
+    label: 'gaussian bind group',
+    layout: preprocess_pipeline.getBindGroupLayout(1),
+    entries: [
+      { binding: 0, resource: { buffer: pc.gaussian_3d_buffer } },
+      { binding: 1, resource: { buffer: splatBuffer } },
+      { binding: 2, resource: { buffer: gaussian_rendering_settings_buffer } },
+      { binding: 3, resource: { buffer: pc.sh_buffer } }
+    ],
+  });
+
   const sort_bind_group = device.createBindGroup({
     label: 'sort',
     layout: preprocess_pipeline.getBindGroupLayout(2),
@@ -67,20 +128,132 @@ export default function get_renderer(
   // ===============================================
   //    Create Render Pipeline and Bind Groups
   // ===============================================
-
+  const render_pipeline = device.createRenderPipeline({
+    label: 'gaussian render',
+    layout: 'auto',
+    vertex: {
+      module: device.createShaderModule(
+        { 
+          code: renderWGSL 
+        }
+      ),
+      entryPoint: 'vs_main',
+    },
+    fragment: {
+      module: device.createShaderModule(
+        {
+          code: renderWGSL
+        }
+      ),
+      entryPoint: 'fs_main',
+      targets: [
+        { 
+          format: presentation_format,
+          blend: {
+            color: {
+              srcFactor: 'one',
+              dstFactor: 'one-minus-src-alpha',
+              operation: 'add'
+            },
+            alpha: {
+              srcFactor: 'one',
+              dstFactor: 'one-minus-src-alpha',
+              operation: 'add'
+            }
+          }
+        }
+      ],
+    }
+  });
+
+  const render_gaussian_bind_group = device.createBindGroup({
+    label: 'render gaussian bind group',
+    layout: render_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: splatBuffer } },
+      { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } },
+      { binding: 2, resource: { buffer: camera_buffer } }
+    ],
+  });
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
 
+  const preprocess = (encoder: GPUCommandEncoder) => {
+    const pass = encoder.beginComputePass({
+      label: 'preprocess pass',
+    });
+    pass.setPipeline(preprocess_pipeline);
+    pass.setBindGroup(0, preprocess_camera_bind_group);
+    pass.setBindGroup(1, preprocess_gaussian_bind_group);
+    pass.setBindGroup(2, sort_bind_group);
+    const numWorkgroups = Math.ceil(pc.num_points / C.histogram_wg_size);
+
+    pass.dispatchWorkgroups(numWorkgroups);
+    pass.end();
+  };
+
+
+  const render = (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+    const pass = encoder.beginRenderPass({
+      label: 'gaussian render pass',
+      colorAttachments: [
+        {
+          view: texture_view,
+          loadOp: 'clear',
+          storeOp: 'store',
+          clearValue: { r: 0, g: 0, b: 0, a: 1 },
+        }
+      ],
+    });
+    pass.setPipeline(render_pipeline);
+    pass.setBindGroup(0, render_gaussian_bind_group);
+    pass.drawIndirect(indirectDrawBuffer, 0);
+    pass.end();
+  };
+
+  // ===============================================
+  //    Scaling Multiplier Function
+  // ===============================================
+
+  // update scaling multiplier
+  function setScalingMultiplier(multiplier: number) {
+    gaussian_rendering_settings_array[0] = multiplier;
+    device.queue.writeBuffer(
+      gaussian_rendering_settings_buffer, 0,
+      gaussian_rendering_settings_array
+    );
+  }
 
   // ===============================================
   //    Return Render Object
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+
+      encoder.copyBufferToBuffer(
+        nulling_buffer, 0,
+        sorter.sort_info_buffer, 0, 4
+      );
+
+      encoder.copyBufferToBuffer(
+        nulling_buffer, 0,
+        sorter.sort_dispatch_indirect_buffer, 0, 4
+      );
+
+      preprocess(encoder);
+
       sorter.sort(encoder);
+
+      encoder.copyBufferToBuffer(
+        sorter.sort_info_buffer, 0,
+        indirectDrawBuffer, 4, 4
+      );
+
+      render(encoder, texture_view);
     },
     camera_buffer,
+    setScalingMultiplier,
   };
 }

src/renderers/renderer.ts

@@ -122,6 +122,9 @@ export default async function init(
       {min: 0, max: 1.5}
     ).on('change', (e) => {
       //TODO: Bind constants to the gaussian renderer.
+      if (gaussian_renderer) {
+        gaussian_renderer.setScalingMultiplier(e.value);
+      }
     });
   }
 

src/shaders/gaussian.wgsl

@@ -1,22 +1,86 @@
 struct VertexOutput {
     @builtin(position) position: vec4<f32>,
     //TODO: information passed from vertex shader to fragment shader
+    @location(0) color : vec4<f32>,
+    @location(1) conic : vec3<f32>,
+    @location(2) opacity : f32,
+    @location(3) center : vec2<f32>
 };
 
 struct Splat {
-    //TODO: information defined in preprocess compute shader
+    //TODO: store information for 2D splat rendering
+    ndc_center: vec2<f32>,
+    radius: vec2<f32>,
+    ndc_depth: f32,
+    pad: f32,
+    color: vec4<f32>,
+    conic_opacity: vec4<f32>,
 };
 
+struct CameraUniforms {
+    view: mat4x4<f32>,
+    view_inv: mat4x4<f32>,
+    proj: mat4x4<f32>,
+    proj_inv: mat4x4<f32>,
+    viewport: vec2<f32>,
+    focal: vec2<f32>
+};
+
+@group(0) @binding(0)
+var<storage, read> splats: array<Splat>;
+@group(0) @binding(1)
+var<storage, read> sorted_indices : array<u32>;
+@group(0) @binding(2)
+var<uniform> camera: CameraUniforms;
+
+fn quad_vertex_offset(vertex_index: u32) -> vec2<f32> {
+    var offsets = array<vec2<f32>, 6>(
+        vec2<f32>(-1.0, -1.0),
+        vec2<f32>(1.0, -1.0),
+        vec2<f32>(-1.0, 1.0),
+        vec2<f32>(-1.0, 1.0),
+        vec2<f32>(1.0, -1.0),
+        vec2<f32>(1.0, 1.0)
+    );
+    return offsets[vertex_index];
+}
+
 @vertex
-fn vs_main(
+fn vs_main(@builtin(vertex_index) vertex_index: u32, @builtin(instance_index) instance_index: u32
 ) -> VertexOutput {
     //TODO: reconstruct 2D quad based on information from splat, pass 
     var out: VertexOutput;
-    out.position = vec4<f32>(1. ,1. , 0., 1.);
+
+    let splat_idx = sorted_indices[instance_index];
+
+    // splat info
+    let splat = splats[splat_idx];
+    let offset = quad_vertex_offset(vertex_index % 6u);
+    let ndc_offset = vec2<f32>(
+    offset.x * splat.radius.x,
+    offset.y * splat.radius.y
+    );
+    let ndc_out = splat.ndc_center + ndc_offset;
+
+    // out info
+    out.position = vec4<f32>(ndc_out.x, ndc_out.y, splat.ndc_depth, 1.0);
+    out.color = splat.color;
+    out.conic = splat.conic_opacity.xyz;
+    out.opacity = splat.conic_opacity.w;
+    // convert center into pixel space
+    out.center = (0.5 + splat.ndc_center * vec2<f32>(0.5, -0.5)) * camera.viewport;
     return out;
 }
 
 @fragment
 fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
-    return vec4<f32>(1.);
+    var dist = in.position.xy - in.center.xy;
+    dist.x = -dist.x;
+    let power = -0.5 * (in.conic.x * dist.x * dist.x + in.conic.z * dist.y * dist.y) - in.conic.y * dist.x * dist.y;
+    if (power > 0.0) {
+        return vec4<f32>(0.0, 0.0, 0.0, 0.0);
+    }
+
+    let alpha = min(0.99, in.opacity * exp(power));
+    return in.color * alpha;
 }

src/shaders/point_cloud.wgsl

@@ -35,7 +35,9 @@ fn vs_main(
     let pos = vec4<f32>(a.x, a.y, b.x, 1.);
 
     // TODO: MVP calculations
-    out.position = pos;
+    let view_pos = camera.view * pos;
+    let clip_pos = camera.proj * view_pos;
+    out.position = clip_pos;
 
     return out;
 }