Project 5: MANVI AGARWAL

README.md

@@ -2,25 +2,69 @@
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* MANVI AGARWAL
+  * [LinkedIn](https://www.linkedin.com/in/manviagarwal27/)
+* Tested on: Windows 11, AMD Ryzen 5 7640HS @ 4.30GHz 16GB, GeForce RTX 4060 8GB(personal)
 
 ### Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+[![](images/GaussianSplatBicycleRendere.png)](https://manvi27.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/)
 
 ### Demo Video/GIF
 
-[![](img/video.mp4)](TODO)
+![](images/BonsaiDemo.gif)
 
-### (TODO: Your README)
+### Overview
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+  This project covers implementation [*3D gaussian splatting for real time radiance field rendering*](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_gaussian_splatting_high.pdf) on WebGPU. There are two renders implemented in the project :- Point cloud and Gaussian render. 
+  One of the main difference in the two renders is that in gaussian renderer, a preprocess compute shader is used to to make some calculations that facilitate rendering 3D gaussian as a splat on screen space in the graphics pipeline. 
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+  The way I understood gaussians is that it is a cloud in 3D space whose center position, rotation, scaling and opacity is given to us. The following diagram is a good way to get intuition of a 3D gaussian.
+
+  ![](images/kdGaussian.png)
+
+
+  Based on the provided information of this point cloud, we can identify the region in 2D space which it will eventually impact to compute color contribution of the gaussian. Normally projecting a gaussian cloud on 2D screen would result in an ellipse but for the sake of ease, we approximate projecting it as a circle with radius. This circle is nothing but 2D splat and to render this we further approximate it as a quad circumscribing the circle. 
+
+### Advantages of Gaussian Splatting:
+
+- Fast real-time rendering method '
+- Efficient representation of 3D space where complex shapes can be represented from group of gaussians.
+- High visual quality which looks almost natural.
+
+
+### Performance Analysis
+
+**Point-cloud vs gaussian renderer**
+
+- One main difference is in point cloud, we render point which is mean of each gaussian cloud while in gaussian we render a quad approximating a 3D gaussian projection on 2D screen.
+- Indirect draw call in used in gaussian rendered because for each gaussian point, 6 vertices are drawn to make a quad.
+- **Visually**, Gaussian renderer also produce more natural and continous appearances while point cloud rendered simply looks like a collection of points.
+
+**Workgroup size impact on performance**
+
+- Increase in workgroup size definitely allows for more speedup given num points in render pipeline are usually high enough to benefit sufficiently from parallelization.
+- **Resourced contention** issues can happen as multiple threads can access shared resource like uniform data buffers at the same time which can cause higher latency 
+
+**View-frustum culling**
+
+- Ideally reduced computation due to early removing the gaussians which do not contribute to the scene should show some evident speedup bu since there is limitation to workgroup size in webgpu, there is not much noticeable speedup even after culling gaussian.
+
+**Impact of number of gaussians**
+
+- The performance tends to remain same while increasing gaussians suggesting scope of more paralleization as number of gaussians increase.
+- After certain threshold value of gaussian count, the fps would typically reduce showing the bottleneck of parallelization scale in gpu.
+
+### Bloopers
+
+ ![](images/blooper_1.png)
+
+ I remember taking screenshot of this image as I was trying to debug gaussian splatting. I believe I would have hardcoded position to get so many splats with no blending.
+
+
+ ![](images/blooper_2.png)
+
+This screenshot came when the depth passed to sorter was incorrect and blending factor wasn't computed correctly.
 
 ### Credits
 

src/renderers/gaussian-renderer.ts

@@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
 
 export interface GaussianRenderer extends Renderer {
-
+  settings_buffer:GPUBuffer
 }
 
 // Utility to create GPU buffers
@@ -35,7 +35,40 @@ export default function get_renderer(
   // ===============================================
 
   const nulling_data = new Uint32Array([0]);
+  const nulling_buffer = createBuffer(device,
+                                      'null buffer',
+                                      4,
+                                      GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST,
+                                      nulling_data);
+
+  const splat_buffer = createBuffer(device,
+                                    'splat buffer',
+                                    pc.num_points*24,
+                                    GPUBufferUsage.COPY_SRC|GPUBufferUsage.COPY_DST|GPUBufferUsage.STORAGE,
+                                    null);
+
+  const settings_buffer = createBuffer(device,
+                                      'settings buffer',
+                                      8,
+                                      GPUBufferUsage.COPY_DST | GPUBufferUsage.UNIFORM,
+                                      new Float32Array([1.0, pc.sh_deg]));                       
 
+  const drawval = new Uint32Array(4);
+
+  drawval[0] = 6; // The vertexCount value
+  drawval[1] = pc.num_points; // The instanceCount value
+  drawval[2] = 0; // The firstVertex value
+  drawval[3] = 0; // The firstInstance value
+  const drawbuffer = createBuffer(device,
+                                  'draw buffer',
+                                  16,
+                                  GPUBufferUsage.COPY_DST | GPUBufferUsage.INDIRECT,
+                                  drawval
+
+  )
+
+
+
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
   // ===============================================
@@ -68,19 +101,109 @@ export default function get_renderer(
   //    Create Render Pipeline and Bind Groups
   // ===============================================
 
+  const preprocess_bind_group = device.createBindGroup({
+    label: 'preprocess',
+    layout: preprocess_pipeline.getBindGroupLayout(1),
+    entries: [
+      { binding: 0, resource: { buffer: pc.gaussian_3d_buffer } },
+      { binding: 1, resource: { buffer:  settings_buffer} },
+      { binding: 2, resource: { buffer:  pc.sh_buffer} },
+      { binding: 3, resource: { buffer:  splat_buffer} }
+    ],
+  });
+
+  const camera_bind_group = device.createBindGroup({
+    label: 'camera',
+    layout: preprocess_pipeline.getBindGroupLayout(0),
+    entries:[{
+      binding: 0, resource: {buffer: camera_buffer}
+    }]
+  });
 
   // ===============================================
   //    Command Encoder Functions
   // ===============================================
 
+  const compute_shader = (encoder: GPUCommandEncoder) => {
+    encoder.copyBufferToBuffer(nulling_buffer,0,sorter.sort_info_buffer,0,4);
+    encoder.copyBufferToBuffer(nulling_buffer,0,sorter.sort_dispatch_indirect_buffer,0,4);
 
+    const pass = encoder.beginComputePass({ label: 'compute preprocess pass' });   
+    pass.setPipeline(preprocess_pipeline);
+    pass.setBindGroup(0, camera_bind_group);
+    pass.setBindGroup(1, preprocess_bind_group);
+    pass.setBindGroup(2, sort_bind_group);
+    pass.dispatchWorkgroups(pc.num_points/C.histogram_wg_size);
+    pass.end();
+
+  }; 
+
+  const render_shader = device.createShaderModule({code: renderWGSL});
+  const render_pipeline = device.createRenderPipeline({
+    label: 'render',
+    layout: 'auto',
+    vertex: {
+      module: render_shader,
+      entryPoint: 'vs_main',
+    },
+    fragment: {
+      module: render_shader,
+      entryPoint: 'fs_main',
+      targets: [{ format: presentation_format,
+            blend: {
+              color: {
+                  srcFactor: 'one',
+                  dstFactor: 'one-minus-src-alpha',
+                  operation: 'add',
+              },
+              alpha: {
+                  srcFactor: 'one',
+                  dstFactor: 'one-minus-src-alpha',
+                  operation: 'add',
+              },
+          },
+       }],
+    },
+  });
+
+  const render_bind_group = device.createBindGroup({
+    label: 'preprocess',
+    layout: render_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } },
+      { binding: 1, resource: { buffer:  splat_buffer} },
+      { binding: 2, resource: { buffer:  sorter.ping_pong[0].sort_indices_buffer} }
+    ],
+  });
+  const render = (encoder: GPUCommandEncoder,texture_view: GPUTextureView) =>{
+    const pass = encoder.beginRenderPass({
+      label: 'point cloud render',
+      colorAttachments: [
+        {
+          view: texture_view,
+          loadOp: 'clear',
+          storeOp: 'store',
+          clearValue: [0.0, 0.0, 0.0, 1.0],
+        }
+      ],
+    });
+    pass.setPipeline(render_pipeline);
+    pass.setBindGroup(0, render_bind_group);
+
+    pass.drawIndirect(drawbuffer,0);
+    pass.end();
+  };
   // ===============================================
   //    Return Render Object
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+      compute_shader(encoder);
       sorter.sort(encoder);
+      encoder.copyBufferToBuffer(sorter.sort_info_buffer,0,drawbuffer,4,4);
+      render(encoder,texture_view);
     },
     camera_buffer,
+    settings_buffer,
   };
 }

src/shaders/gaussian.wgsl

@@ -1,22 +1,84 @@
 struct VertexOutput {
     @builtin(position) position: vec4<f32>,
     //TODO: information passed from vertex shader to fragment shader
+
+    @location(0) conic_opacity:vec4f,
+    @location(1) color:vec4f,
+    @location(2) center:vec2f
 };
 
 struct Splat {
     //TODO: information defined in preprocess compute shader
+    radii_depths_pos:array<u32,2>,
+    conic_opacity:array<u32,2>,
+    color_tiles_touched:array<u32,2>
+};
+
+struct CameraUniforms {
+    view: mat4x4<f32>,
+    view_inv: mat4x4<f32>,
+    proj: mat4x4<f32>,
+    proj_inv: mat4x4<f32>,
+    viewport: vec2<f32>,
+    focal: vec2<f32>
 };
 
+@group(0) @binding(0)
+var<uniform> camera: CameraUniforms;
+@group(0) @binding(1)
+var<storage> splats:array<Splat>;
+@group(0) @binding(2)
+var<storage> sort_indices : array<u32>;
+
 @vertex
 fn vs_main(
+    @builtin(vertex_index) in_vertexIndex: u32,
+    @builtin(instance_index) in_instanceIndex: u32,
 ) -> VertexOutput {
     //TODO: reconstruct 2D quad based on information from splat, pass 
     var out: VertexOutput;
-    out.position = vec4<f32>(1. ,1. , 0., 1.);
+
+    let radius = unpack2x16float(splats[sort_indices[in_instanceIndex]].radii_depths_pos[0]).x;
+    let ndcPos = unpack2x16float(splats[sort_indices[in_instanceIndex]].radii_depths_pos[1]);
+
+    let pos = array<vec2f, 6>(
+        vec2f((ndcPos.x - radius * 2.0f/camera.viewport.x), (ndcPos.y + radius * 2.0f/camera.viewport.y)),
+        vec2f((ndcPos.x - radius * 2.0f/camera.viewport.x), (ndcPos.y - radius * 2.0f/camera.viewport.y)),
+        vec2f((ndcPos.x + radius * 2.0f/camera.viewport.x), (ndcPos.y - radius * 2.0f/camera.viewport.y)),
+        vec2f((ndcPos.x + radius * 2.0f/camera.viewport.x), (ndcPos.y - radius * 2.0f/camera.viewport.y)),
+        vec2f((ndcPos.x + radius * 2.0f/camera.viewport.x), (ndcPos.y + radius * 2.0f/camera.viewport.y)),
+        vec2f((ndcPos.x - radius * 2.0f/camera.viewport.x), (ndcPos.y + radius * 2.0f/camera.viewport.y)),
+    );
+    out.position = vec4f(pos[in_vertexIndex].xy,0.,1.);
+    let color_xy = unpack2x16float(splats[sort_indices[in_instanceIndex]].color_tiles_touched[0]);
+    let color_zw = unpack2x16float(splats[sort_indices[in_instanceIndex]].color_tiles_touched[1]);
+    let conic_xy = unpack2x16float(splats[sort_indices[in_instanceIndex]].conic_opacity[0]);
+    let conic_zw = unpack2x16float(splats[sort_indices[in_instanceIndex]].conic_opacity[1]);
+    out.color = vec4f(color_xy,color_zw.x,1.);
+    out.conic_opacity = vec4f(conic_xy,conic_zw);
+    out.center = ndcPos.xy;
     return out;
 }
 
+
 @fragment
 fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
-    return vec4<f32>(1.);
-}
+
+    var center = vec2f((in.center.x + 1.)*0.5*camera.viewport.x,(1. - in.center.y)*0.5*camera.viewport.y );
+    let pixelPos = in.position.xy;
+    var offset = (pixelPos - center);
+    offset.x *= -1;
+    var power = -0.5f * (
+        in.conic_opacity.x * pow(offset.x, 2.0f) + 
+        in.conic_opacity.z * pow(offset.y, 2.0f)
+    );
+    power -= in.conic_opacity.y * offset.x * offset.y;
+
+    if (power > 0.0f) {
+        return vec4f(0.0f, 0.0f, 0.0f, 0.0f);
+    }
+
+    let alpha = min(0.99f, in.conic_opacity.w * exp(power));
+    return in.color * alpha;
+
+}

src/shaders/point_cloud.wgsl

@@ -35,8 +35,11 @@ fn vs_main(
     let pos = vec4<f32>(a.x, a.y, b.x, 1.);
 
     // TODO: MVP calculations
+    let viewprojmat = camera.proj*camera.view;
     out.position = pos;
 
+    out.position = viewprojmat*out.position;
+
     return out;
 }