Project 5: Shreyas Singh

README.md

@@ -1,26 +1,51 @@
 # Project5-WebGPU-Gaussian-Splat-Viewer
 
-**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4**
+**University of Pennsylvania, CIS 5650: GPU Programming and Architecture, Project 3**
+
+* Shreyas Singh
+    * [LinkedIn](https://linkedin.com/in/shreyassinghiitr)
+* Tested on: Apple MacBook Pro, Apple M2 Pro @ 3.49 GHz, 19-core GPU
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
 
 ### Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+[Live Demo](https://shreyas3156.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/)
+
+Open it with any Gaussian splat input dataset.
+
 
 ### Demo Video/GIF
 
-[![](img/video.mp4)](TODO)
+[![](img/gaussian.gif)]()
+
+### Project Overview
+
+#### Gaussian Splatting
+
+Gaussian splatting is a way to draw 3D scenes by treating each point as a little “splat” instead of a triangle. Each splat is a 3D Gaussian defined by its center, shape, and color (often encoded with spherical harmonics).
+
+This WebGPU project builds a real-time viewer for a pre-trained Gaussian scene. First, a compute pass culls off-screen Gaussians, computes each splat’s 2D position, size, and color, and builds an indirect draw command. Then a single indirect draw call runs a vertex shader to place each quad and a fragment shader to shade it using the Gaussian equation.
+
+### Features
+#### Point Cloud Rendering
+![](img/pointcloud.png)
+This image shows a 30,000-point scan, rendered as small yellow points which are centers of as many number of Gaussians.
+
+#### Gaussian Rendering
+The image at the top of this page is rendered by transforming the aforementioned Gaussian splats into screen space, dropping the ones that lie outside the view, sorting the remaining Gaussians back-to-front, and drawing them as small quads whose opacity falls off smoothly from the center.
+
+### Performance Analysis
+#### Point-Cloud Renderer vs Gaussian Renderer Comparison
+The **point-cloud renderer** simply displays individual points in 3D space, which often look sparse and disconnected, especially in regions where data is less dense. They lack fine detail and do not capture subtle lighting effects, thus appearing far less realistic. On the other hand, the **Gaussian renderer** produces a much smoother and more continuous image by placing volumetric ellipsoids at each point. These splats overlap and blend, filling in gaps and creating the appearance of solid, soft-edged surfaces. With the addition of spherical harmonics to encode color, the Gaussian renderer is able to reproduce rich and realistic shading. However, this added visual fidelity comes at the expense of performance: the Gaussian renderer requires compute such as preprocessing, frustum culling, and sorting, which slows down rendering.
 
-### (TODO: Your README)
+#### Effect of Workgroup Size on Gaussian Renderer Performance 
+Using a larger workgroup size can improve performance by better utilizing the GPU capabilities, allowing more simultaneous computation. However, if the workgroup size becomes too large, it can lead to problems like buffer overflows or inefficiency. On the other hand, using a smaller workgroup size can result in more overhead for the GPU to manage many small groups.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+#### Effect of View-Frustum Culling 
+View-frustum culling does provide a performance improvement, especially in larger or denser scenes. By discarding Gaussians that are outside the camera’s view early in the compute shader, the renderer avoids unnecessary processing for splats that would never be visible on the screen. The effect is most noticeable when the camera is inside a large scene or when many points fall outside the view. However, the benefit can be less noticeable in simple or small scenes where most Gaussians are visible anyway, and the overhead of the culling itself could slightly offset the gains.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+#### Effect of Number of Gaussians
+As the number of Gaussians increases, the performance generally decreases because the renderer must process, sort, and potentially draw more data thus leading to more computation and higher memory usage. While GPUs handle small numbers efficiently, very large numbers of Gaussians can slow down the frame rate, especially if many are visible at once.
 
 ### Credits
 

src/renderers/gaussian-renderer.ts

@@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
 export interface GaussianRenderer extends Renderer {
-
+  rendering_buffer: GPUBuffer;
 }
 
 // Utility to create GPU buffers
@@ -36,6 +36,38 @@ export default function get_renderer(
 
   const nulling_data = new Uint32Array([0]);
 
+  const nulling_buffer = createBuffer(
+    device,
+    'nulling buffer',
+    Uint32Array.BYTES_PER_ELEMENT,
+    GPUBufferUsage.COPY_DST | GPUBufferUsage.COPY_SRC,
+    nulling_data
+  );
+
+  const indirect_buffer = createBuffer(
+    device,
+    'indirect buffer',
+    20,
+    GPUBufferUsage.COPY_DST | GPUBufferUsage.INDIRECT,
+    new Uint32Array([6, 0, 0, 0, 0])
+  );
+
+  const rendering_buffer = createBuffer(
+    device, 
+    'render settings buffer', 
+    Float32Array.BYTES_PER_ELEMENT,
+    GPUBufferUsage.COPY_DST | GPUBufferUsage.UNIFORM, 
+    new Float32Array([1.0])
+  );
+
+  const floatsPerSplat = 24;
+  const splat_buffer = createBuffer(
+    device,
+    'gauss splat buffer',
+    floatsPerSplat * pc.num_points,
+    GPUBufferUsage.STORAGE
+  );
+
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
   // ===============================================
@@ -48,13 +80,21 @@ export default function get_renderer(
       constants: {
         workgroupSize: C.histogram_wg_size,
         sortKeyPerThread: c_histogram_block_rows,
+        shDegree: pc.sh_deg,
       },
     },
   });
 
+  const [cameraLayout, gaussianLayout, sortingLayout, splatLayout] = [
+  preprocess_pipeline.getBindGroupLayout(0),
+  preprocess_pipeline.getBindGroupLayout(1),
+  preprocess_pipeline.getBindGroupLayout(2),
+  preprocess_pipeline.getBindGroupLayout(3),
+];
+
   const sort_bind_group = device.createBindGroup({
     label: 'sort',
-    layout: preprocess_pipeline.getBindGroupLayout(2),
+    layout: sortingLayout,
     entries: [
       { binding: 0, resource: { buffer: sorter.sort_info_buffer } },
       { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_depths_buffer } },
@@ -63,24 +103,132 @@ export default function get_renderer(
     ],
   });
 
+  const camera_bind_group = device.createBindGroup({
+    label: 'preprocess: camera bind group',
+    layout: cameraLayout,
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } }
+    ]
+  });
+
+  const gaussian_bind_group = device.createBindGroup({
+    label: 'preprocess: gaussian bind group',
+    layout: gaussianLayout,
+    entries: [
+      { binding: 0, resource: { buffer: pc.gaussian_3d_buffer }},
+    ],
+  });
+
+  const splat_bind_group = device.createBindGroup({
+    label: 'preprocess splat bind group',
+    layout: splatLayout,
+    entries: [
+      {  binding: 0, resource: { buffer: splat_buffer } }, 
+      { binding: 1, resource: { buffer: rendering_buffer } },
+      { binding: 2, resource: { buffer: pc.sh_buffer } }
+    ]
+  });
 
   // ===============================================
   //    Create Render Pipeline and Bind Groups
   // ===============================================
-
+
+  const render_pipeline = device.createRenderPipeline({
+    label: 'gauss:render-pipeline',
+    layout: 'auto',
+    vertex: {
+      module: device.createShaderModule({ code: renderWGSL }),
+      entryPoint: 'vs_main',
+    },
+    fragment: {
+      module: device.createShaderModule({ code: renderWGSL }),
+      entryPoint: 'fs_main',
+      targets: [{ 
+        format: presentation_format,
+        blend: {
+          color: {
+              srcFactor: 'one',
+              dstFactor: 'one-minus-src-alpha',
+              operation: 'add'
+          },
+          alpha: {
+              srcFactor: 'one',
+              dstFactor: 'one-minus-src-alpha',
+              operation: 'add'
+          }
+        }
+      }],
+    },
+  });
+
+  const render_bind_group = device.createBindGroup({
+  label: 'gauss: render bind group',
+  layout: render_pipeline.getBindGroupLayout(0),
+  entries: [
+    {
+      binding: 0,
+      resource: { buffer: splat_buffer },
+    },
+    { 
+      binding: 1,
+      resource: { buffer: sorter.ping_pong[0].sort_indices_buffer },
+    },
+    {
+      binding: 2,
+      resource: {buffer: camera_buffer}
+    }
+  ],
+});
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
-
+
+  const compute_pass = (encoder: GPUCommandEncoder) => {
+    const preprocess_compute_pass = encoder.beginComputePass()
+    preprocess_compute_pass.setPipeline(preprocess_pipeline);
+    preprocess_compute_pass.setBindGroup(0, camera_bind_group);
+    preprocess_compute_pass.setBindGroup(1, gaussian_bind_group);
+    preprocess_compute_pass.setBindGroup(2, sort_bind_group);
+    preprocess_compute_pass.setBindGroup(3, splat_bind_group);
+    preprocess_compute_pass.dispatchWorkgroups(Math.ceil(pc.num_points / C.histogram_wg_size));
+    preprocess_compute_pass.end();
+  };
+
+  const render_pass = (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+    const gaussian_render_pass = encoder.beginRenderPass({
+      label: 'gaussian render pass',
+      colorAttachments: [{
+          view: texture_view,
+          loadOp: 'clear',
+          storeOp: 'store',
+          clearValue: [0, 0, 0, 1],
+        }
+      ],
+    });
+    gaussian_render_pass.setPipeline(render_pipeline);
+    gaussian_render_pass.setBindGroup(0, render_bind_group);
+    gaussian_render_pass.drawIndirect(indirect_buffer, 0);
+    gaussian_render_pass.end();
+  }
+
 
   // ===============================================
   //    Return Render Object
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+
+      encoder.copyBufferToBuffer(nulling_buffer, 0, sorter.sort_info_buffer, 0, 4);
+
+      encoder.copyBufferToBuffer(nulling_buffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4);
+      compute_pass(encoder);
       sorter.sort(encoder);
+      encoder.copyBufferToBuffer(
+        sorter.sort_info_buffer, 0, indirect_buffer, 4, 4);
+      render_pass(encoder, texture_view);
     },
     camera_buffer,
+    rendering_buffer
   };
 }

src/renderers/renderer.ts

@@ -121,7 +121,14 @@ export default async function init(
       'gaussian_multiplier',
       {min: 0, max: 1.5}
     ).on('change', (e) => {
-      //TODO: Bind constants to the gaussian renderer.
+
+      if (gaussian_renderer) {
+        device.queue.writeBuffer(
+          gaussian_renderer.rendering_buffer,
+          0,
+          new Float32Array([params.gaussian_multiplier])
+  );
+}
     });
   }
 

src/shaders/gaussian.wgsl

@@ -1,22 +1,99 @@
 struct VertexOutput {
     @builtin(position) position: vec4<f32>,
-    //TODO: information passed from vertex shader to fragment shader
+    @location(0) size: vec2<f32>,
+    @location(1) color: vec4<f32>,
+    @location(2) conic_opacity: vec4<f32>,
+    @location(3) center: vec2<f32>
 };
 
 struct Splat {
-    //TODO: information defined in preprocess compute shader
+    xy: u32,
+    widthHeight: u32,
+    packed_color: array<u32, 2>,
+    co: u32,
+    cp: u32
 };
 
+struct CameraUniforms {
+    view: mat4x4<f32>,
+    view_inv: mat4x4<f32>,
+    proj: mat4x4<f32>,
+    proj_inv: mat4x4<f32>,
+    viewport: vec2<f32>,
+    focal: vec2<f32>
+};
+
+@group(0) @binding(0)
+var<storage, read> splats: array<Splat>;
+@group(0) @binding(1)
+var<storage, read> sort_indices : array<u32>;
+@group(0) @binding(2)
+var<uniform> camera: CameraUniforms;
+
 @vertex
 fn vs_main(
+    @builtin(instance_index) instanceIndex: u32,
+    @builtin(vertex_index) vertexIndex: u32
 ) -> VertexOutput {
-    //TODO: reconstruct 2D quad based on information from splat, pass 
     var out: VertexOutput;
-    out.position = vec4<f32>(1. ,1. , 0., 1.);
+
+    // unpack center XY from 2×16-bit floats
+    let splatIdx = sort_indices[instanceIndex];
+    let splat = splats[splatIdx];
+    let xy = unpack2x16float(splat.xy);
+    let size: vec2f = unpack2x16float(splat.widthHeight);
+
+    // apply corner offset
+    let corners = array<vec2f, 6>(
+        vec2f(xy.x - size.x, xy.y + size.y),
+        vec2f(xy.x - size.x, xy.y - size.y),
+        vec2f(xy.x + size.x, xy.y - size.y),
+        vec2f(xy.x + size.x, xy.y - size.y),
+        vec2f(xy.x + size.x, xy.y + size.y),
+        vec2f(xy.x - size.x, xy.y + size.y)
+    );
+    let corner = corners[vertexIndex];
+    out.position = vec4f(corner.x, corner.y, 0.0, 1.0);
+    out.size = size;
+
+    let conicA = unpack2x16float(splat.co);
+    let conicB = unpack2x16float(splat.cp);
+
+    out.conic_opacity = vec4f(
+        conicA.x,
+        conicA.y,
+        conicB.x,
+        conicB.y
+    );
+    out.center = xy;
+
+    let color = unpack2x16float(splat.packed_color[0]);
+    out.color = vec4<f32>(unpack2x16float(splat.packed_color[0]),
+                      unpack2x16float(splat.packed_color[1]));
+
     return out;
 }
 
 @fragment
 fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
-    return vec4<f32>(1.);
+    let windowPos = in.position.xy;
+    let ndc = ((windowPos / camera.viewport) * 2.0 - vec2f(1.0)) * vec2f(1.0, -1.0);
+
+    let d = ndc - in.center;
+    var d_flipped = vec2f(-d.x, d.y);
+    d_flipped *= camera.viewport * 0.5f;
+
+    let conic = in.conic_opacity;
+    let exponent = -0.5 * (
+        conic.x * d_flipped.x * d_flipped.x +
+        conic.z * d_flipped.y * d_flipped.y +
+        2.0 * conic.y * d_flipped.x * d_flipped.y
+    );
+
+    if (exponent > 0.0f) {
+        return vec4f(0.0f, 0.0f, 0.0f, 0.0f);
+    }
+
+    let alpha = min(0.99f, conic.w * exp(exponent));
+    return in.color * alpha;
 }