Project 5 - Adithya Rajeev

README.md

@@ -2,30 +2,88 @@
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* ADITHYA RAJEEV
+  * [LinkedIn](https://www.linkedin.com/in/adithyar262/)
+* Tested on: Windows 11, i7 13th Gen @ 2.40GHz 16GB, GeForce RTX 4050 8GB (Personal)
+
+# 3D Gaussian Splatting Renderer
+
+This project implements a 3D Gaussian Splatting renderer based on the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" [[1]](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). It utilizes WebGL and leverages GPU parallelism for efficient rendering of complex scenes represented by Gaussian splats.
+
+**Features:**
+
+  - **Gaussian Splatting:** Implements the core algorithm for rendering 3D scenes using Gaussian splats.
+  - **View Frustum Culling:** Optimizes rendering performance by discarding Gaussians outside the view frustum.
+  - **Depth Sorting:** Sorts Gaussians by depth to ensure correct occlusion and transparency.
+  - **Spherical Harmonics Lighting:** Uses spherical harmonics coefficients to evaluate color based on view direction.
+  - **Interactive Controls:** Allows users to explore the scene with camera controls.
 
 ### Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+[Live Demo](https://adithyar262.github.io/Project5-WebGPU-Gaussian-Splat-Viewer)
 
 ### Demo Video/GIF
 
-[![](img/video.mp4)](TODO)
+![](images/bonzai.gif)
+![](images/cycle.gif)
+
+## What is Gaussian Splatting?
+
+Gaussian splatting is a technique for rendering 3D scenes by representing objects as a collection of Gaussian functions. Each Gaussian function, or "splat," is defined by its position, orientation, scale, and color. By blending these splats together, complex shapes and appearances can be recreated.
+
+![](images/bonzai.png)
+![](images/cycle.png)
+
+
+**Advantages of Gaussian Splatting:**
+
+  - **Efficient Representation:** Gaussian splats can compactly represent complex shapes and appearances.
+  - **Real-time Rendering:** The rendering process can be highly parallelized and optimized for real-time performance.
+  - **View-dependent Effects:** Gaussian splatting can accurately capture view-dependent effects like lighting and reflections.
+
+**How Gaussian Splatting Works:**
+
+1.  **Scene Representation:** The 3D scene is represented as a set of Gaussian splats.
+2.  **Splatting:** Each splat is projected onto the screen as a 2D ellipse.
+3.  **Blending:** The splats are blended together based on their depth and opacity to create the final image.
+
+![Diagram of Gaussian splatting process](images/Workflow.png)
+
+**Performance Analysis:**
+
+  - **Point Cloud vs. Gaussian Renderer:** 
+
+    * **Visual Quality:** The Gaussian renderer produces significantly smoother and more visually appealing results than the point cloud renderer. Point clouds simply display individual points, leading to a sparse and disconnected representation.  Gaussian splats, on the other hand, effectively interpolate the surface between points, resulting in a continuous and more realistic appearance. They also inherently capture view-dependent effects like lighting and reflections, which are absent in a basic point cloud rendering.
+
+    * **Data Representation:** Gaussian splats offer a more efficient representation of the scene.  They can encode surface orientation and appearance information within each splat, allowing for a more compact representation compared to storing individual points and their attributes.
+
+  - **Workgroup Size:** 
+
+    * **GPU Occupancy:** Increasing the workgroup size generally improves performance up to an optimal point. This is because larger workgroups allow for better utilization of GPU compute units, increasing occupancy and reducing idle time. Each workgroup is scheduled on a compute unit, and having more threads per workgroup maximizes the parallel processing capacity.
+
+    * **Memory Contention:** However, excessively large workgroups can lead to decreased performance. This is due to increased memory access contention, where multiple threads within the same workgroup try to access the same memory locations simultaneously. This contention can cause delays and stall the execution, negating the benefits of increased parallelism. The optimal workgroup size depends on the specific GPU architecture and the complexity of the rendering task.
+
+  - **View Frustum Culling:** 
+
+    * **Reduced Computation:** View frustum culling provides a significant performance improvement. By discarding Gaussians that lie outside the camera's view frustum, we drastically reduce the number of splats that need to be processed in subsequent stages of the rendering pipeline. This leads to fewer computations in transformations, depth sorting, and rasterization, resulting in a noticeable speedup.
+
+    * **Early Rejection:** Culling is an early rejection step that prevents unnecessary work. Without it, all Gaussians would go through the entire pipeline even if they don't contribute to the final image. This optimization is particularly crucial in scenes with a large number of splats, where the computational cost savings can be substantial.
+
+  - **Number of Gaussians:** 
+
+    * **Increased Workload:** As the number of Gaussians increases, the rendering performance decreases. This is simply due to the increased workload on the GPU. More splats require more computations for transformation, culling, sorting, and rasterization. The memory requirements also increase, potentially leading to more data transfers and cache misses.
 
-### (TODO: Your README)
+    * **Bottlenecks:**  A higher number of Gaussians can exacerbate bottlenecks in the rendering pipeline. For example, depth sorting becomes more computationally expensive with more splats, potentially limiting the overall frame rate. The specific performance impact depends on the scene complexity, GPU capabilities, and the efficiency of the rendering algorithm.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+## Bloopers! 
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+Everyone loves a good blooper reel! Here's a glimpse of some of the hilarious (and sometimes frustrating) glitches encountered during the development of this project. From Gaussians going rogue to unexpected visual effects, these bloopers showcase the challenges and triumphs of bringing this renderer to life.
 
-### Credits
+![](images/blooper1.gif)
+![](images/blooper3.gif)
+![](images/blooper2.gif)
 
-- [Vite](https://vitejs.dev/)
-- [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/)
-- [stats.js](https://github.com/mrdoob/stats.js)
-- [wgpu-matrix](https://github.com/greggman/wgpu-matrix)
-- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Guassian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization)
+**Resources:**
+  - **3D Gaussian Splatting Paper:** [https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/)
+  - **Gaussian Splatting Notes:** [https://github.com/kwea123/gaussian\_splatting\_notes](https://github.com/kwea123/gaussian_splatting_notes)
+  - **Spherical Harmonics Post:** [https://beatthezombie.github.io/sh\_post\_1/](https://beatthezombie.github.io/sh_post_1/)

src/camera/camera.ts

@@ -187,4 +187,4 @@ export class Camera {
     this.update_buffer();
   }
 
-};
+};

src/renderers/gaussian-renderer.ts

@@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
 export interface GaussianRenderer extends Renderer {
-
+  settingsBuffer : GPUBuffer;
 }
 
 // Utility to create GPU buffers
@@ -35,6 +35,28 @@ export default function get_renderer(
   // ===============================================
 
   const nulling_data = new Uint32Array([0]);
+  const nullBuffer  = createBuffer(
+    device,
+    'Clear Buffer ',
+    4,
+    GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST,
+    nulling_data);
+
+  const settingsBuffer  = createBuffer(
+      device,
+      'Render Settings Buffer ',
+      8, 
+      GPUBufferUsage.COPY_DST | GPUBufferUsage.UNIFORM,
+      new Float32Array([1.0, pc.sh_deg])
+  );
+
+  const splatBuffer = createBuffer(
+    device,
+    'splatBuffer',
+    (8 + 8 + 8) * pc.num_points, // Position and size, color, conic and opacity
+    GPUBufferUsage.STORAGE,
+    null
+  );
 
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
@@ -54,7 +76,7 @@ export default function get_renderer(
 
   const sort_bind_group = device.createBindGroup({
     label: 'sort',
-    layout: preprocess_pipeline.getBindGroupLayout(2),
+    layout: preprocess_pipeline.getBindGroupLayout(1),
     entries: [
       { binding: 0, resource: { buffer: sorter.sort_info_buffer } },
       { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_depths_buffer } },
@@ -63,24 +85,105 @@ export default function get_renderer(
     ],
   });
 
-
   // ===============================================
   //    Create Render Pipeline and Bind Groups
   // ===============================================
-
+  const buffer = createBuffer(
+    device, //device
+    "Buffer", //label
+    16, //size
+    GPUBufferUsage.COPY_DST | GPUBufferUsage.INDIRECT, // Usage
+    new Uint32Array([6, pc.num_points, 0, 0, ]) // Data
+  )
+  const render_shader = device.createShaderModule({code: renderWGSL});
+  const render_pipeline = device.createRenderPipeline({
+    label: 'render',
+    layout: 'auto',
+    vertex: {
+      module: render_shader,
+      entryPoint: 'vs_main',
+    },
+    fragment: {
+      module: render_shader,
+      entryPoint: 'fs_main',
+      targets: [{ format: presentation_format,
+            blend: {
+              color: {
+                  srcFactor: 'one',
+                  dstFactor: 'one-minus-src-alpha',
+                  operation: 'add',
+              },
+              alpha: {
+                  srcFactor: 'one',
+                  dstFactor: 'one-minus-src-alpha',
+                  operation: 'add',
+              },
+          },
+       }],
+    },
+  });
+
+  const preprocessor_bind_group = device.createBindGroup({
+    label: 'preprocessor_bind_group',
+    layout: preprocess_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer }},
+      { binding: 1, resource: { buffer: pc.gaussian_3d_buffer }},
+      { binding: 2, resource: { buffer: splatBuffer } },
+      { binding: 3, resource: { buffer: settingsBuffer } },
+      { binding: 4, resource: { buffer: pc.sh_buffer } },
+    ],
+  });
+
+  const gaussian_render_bind_group = device.createBindGroup({
+      label: 'gaussian_render_bind_group',
+      layout: render_pipeline.getBindGroupLayout(0),
+      entries: [
+          { binding: 0, resource: { buffer: splatBuffer } },
+          { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } },
+          { binding: 2, resource: { buffer: camera_buffer } }, 
+      ],
+  });
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
-
+  const compute = (encoder: GPUCommandEncoder) => {
+      const compute_pass = encoder.beginComputePass();
+      compute_pass.setPipeline(preprocess_pipeline);
+      compute_pass.setBindGroup(0, preprocessor_bind_group);
+      compute_pass.setBindGroup(1, sort_bind_group);
+      compute_pass.dispatchWorkgroups(Math.ceil(pc.num_points / C.histogram_wg_size));
+      compute_pass.end();
+  };
 
   // ===============================================
   //    Return Render Object
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+      encoder.copyBufferToBuffer(nullBuffer , 0, sorter.sort_info_buffer, 0, 4);
+      encoder.copyBufferToBuffer(nullBuffer , 0, sorter.sort_dispatch_indirect_buffer, 0, 4);
+      compute(encoder);
       sorter.sort(encoder);
+      encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, buffer, 4, 4);
+      const pass = encoder.beginRenderPass({
+          label: 'Render  pass',
+          colorAttachments: [
+            {
+              view: texture_view,
+              loadOp: 'clear',
+              storeOp: 'store',
+              clearValue: [0.0, 0.0, 0.0, 1.0],
+          }
+        ],
+      });
+      pass.setPipeline(render_pipeline);
+      pass.setBindGroup(0, gaussian_render_bind_group);
+      pass.drawIndirect(buffer, 0); 
+      pass.end();
     },
     camera_buffer,
+    settingsBuffer ,
   };
 }

src/renderers/renderer.ts

@@ -157,4 +157,4 @@ export default async function init(
   }
 
   requestAnimationFrame(frame);
-}
+}

src/shaders/gaussian.wgsl

@@ -1,22 +1,85 @@
+@group(0) @binding(0)
+var<storage, read> splats: array<Splat>;
+@group(0) @binding(1)
+var<storage, read> sort_indices : array<u32>;
+@group(0) @binding(2)
+var<uniform> camera: CameraUniforms;
+
 struct VertexOutput {
     @builtin(position) position: vec4<f32>,
     //TODO: information passed from vertex shader to fragment shader
+
+    @location(0) color: vec4f,
+    @location(1) conic_opacity: vec4f,
+    @location(2) center: vec2f,
 };
 
 struct Splat {
     //TODO: information defined in preprocess compute shader
+    // Data is packed, need to unpack
+    position_and_size: array<u32,2>,
+    color_data: array<u32,2>,
+    conic_and_opacity: array<u32,2>,
+};
+
+struct CameraUniforms {
+    view: mat4x4f,
+    view_inv: mat4x4f,
+    proj: mat4x4f,
+    proj_inv: mat4x4f,
+    viewport: vec2f,
+    focal: vec2f,
 };
 
 @vertex
 fn vs_main(
+    @builtin(vertex_index) in_vertex_index: u32, @builtin(instance_index) in_instance_index: u32
 ) -> VertexOutput {
     //TODO: reconstruct 2D quad based on information from splat, pass 
     var out: VertexOutput;
-    out.position = vec4<f32>(1. ,1. , 0., 1.);
+
+    let splat = splats[sort_indices[in_instance_index]];
+    let xy = unpack2x16float(splat.position_and_size[0]);
+    let wh = unpack2x16float(splat.position_and_size[1]);
+    let rg = unpack2x16float(splat.color_data[0]);
+    let ba = unpack2x16float(splat.color_data[1]);
+    let conic = unpack2x16float(splat.conic_and_opacity[0]);
+    let opacity = unpack2x16float(splat.conic_and_opacity[1]);
+    let pos = array<vec2f, 6>(
+        vec2f(xy.x - wh.x * 2.0f, xy.y + wh.y * 2.0f),
+        vec2f(xy.x - wh.x * 2.0f, xy.y - wh.y * 2.0f),
+        vec2f(xy.x + wh.x * 2.0f, xy.y - wh.y * 2.0f),
+        vec2f(xy.x + wh.x * 2.0f, xy.y - wh.y * 2.0f),
+        vec2f(xy.x + wh.x * 2.0f, xy.y + wh.y * 2.0f),
+        vec2f(xy.x - wh.x * 2.0f, xy.y + wh.y * 2.0f),
+    );
+
+    out.position = vec4(pos[in_vertex_index].x, pos[in_vertex_index].y, 0.0f, 1.0f);
+    out.color = vec4(rg.x, rg.y, ba.x, ba.y);;
+    out.conic_opacity = vec4f(conic.x, conic.y, opacity.x, opacity.y);
+    out.center = vec2f(xy.x, xy.y);
     return out;
 }
 
 @fragment
 fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
-    return vec4<f32>(1.);
+
+    var position = (in.position.xy / camera.viewport) * 2.0f - 1.0f;
+    position.y *= -1.0f;
+
+    var offset = (position.xy - in.center.xy) * camera.viewport * 0.5f;
+    offset.x *= -1.0f;
+
+    var power = -0.5f * (
+        in.conic_opacity.x * pow(offset.x, 2.0f) + 
+        in.conic_opacity.z * pow(offset.y, 2.0f)
+    );
+    power -= in.conic_opacity.y * offset.x * offset.y;
+
+    if (power > 0.0f) {
+        return vec4f(0.0f, 0.0f, 0.0f, 0.0f);
+    }
+
+    let alpha = min(0.99f, in.conic_opacity.w * exp(power));
+    return in.color * alpha;
 }

src/shaders/point_cloud.wgsl

@@ -35,7 +35,7 @@ fn vs_main(
     let pos = vec4<f32>(a.x, a.y, b.x, 1.);
 
     // TODO: MVP calculations
-    out.position = pos;
+    out.position = camera.proj * camera.view * pos;
 
     return out;
 }