Project5: Zixiao Wang

README.md

@@ -1,31 +1,54 @@
-# Project5-WebGPU-Gaussian-Splat-Viewer
+# WebGPU-Gaussian-Splat-Viewer
 
-**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4**
+![](images/ga_bonsai.png)
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* [Zixiao Wang](https://www.linkedin.com/in/zixiao-wang-826a5a255/)
+* Tested on: Google Chrome Version 130.0. on
+  Windows 11, i7-14700K @ 3.40 GHz 32GB, GTX 4070TI 12GB  4K Monitor (Personal PC)
 
 ### Live Demo
 
-[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred)
+[Live Demo](https://lanbiubiu1.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/)
 
 ### Demo Video/GIF
+![](images/ga_bike.png)
+![](images/ga_bike_move.gif)
+![](images/ga_playroom_1.gif)
 
-[![](img/video.mp4)](TODO)
 
-### (TODO: Your README)
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+### Introduction
+| Point-Cloud           | Gaussian Splat           |
+|-----------------------|--------------------------|
+| ![](images/pc_bike.png) | ![](images/ga_bike.png)  |
+
+In this Project, I implemented the 3D Gaussian Splat Viewer. The entire project is based on [3D Gaussian Splatting
+for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/).
+To start off, we need to create a point_cloud shader to cluster the points based on the scene input .ply file. After that, the preprocess shader will read the point cloud data as splats and transfer their position into the screen space. Then, radix sort is applied to sort the splats by their screen space position and depth. In the end, the processed data are fed back into Gaussian splat to render the points in the scene in real time.
+
+![](images/workflow.png)
+
+### Performance Analysis
+
+#### Difference between Point-Cloud and Gaussian Renderer
+The Point-Cloud image represents a point-cloud rendering, which shows the bicycle and bench represented by numerous distinct points. Each point in the cloud contributes to a sampled representation of the scene. The level of detail is sparse, especially at the edges, and some structure is lost due to the gaps between points, especially on finer details like spokes and the texture of the grass. The Gaussian renderer represents the scene with significantly fewer gaps, generating a much more filled and realistic depiction. This is due to representing each point as a Gaussian splat rather than a single pixel, allowing for smoother boundaries and a more cohesive structure. This representation results in better-defined shapes and surfaces, providing a higher fidelity to the original scene.
+
+#### Changing Workgroup Size Affects Performance
+Changing the workgroup size directly affects the efficiency of the compute shader execution on the GPU. Workgroup size determines how many threads are used to process a specific task. The default setting size on my machine has the best performance. Either higher or lower the workgroup size can lead to worse performance. I think it is because of the inefficiencies of many wasted computations. And smaller workgroup size  results in underutilization of GPU cores since fewer threads are running in parallel. 
+
+#### Utilization of View-Frustum Culling to Performance Improvement
+View-frustum culling improves performance by reducing the number of objects that need to be processed and rendered by the GPU. It essentially removes all points, quads, or objects that are outside the visible region (the frustum). However, as the current testing scene is relatively small, it does not really benefit from View-Frustum Culling. Instead, it might lead to some unnecessary computations. Ideally, for large scenes with large view space, the View-Frustum Culling will largely improve the performance.
+
+#### Number of Guassians to Performance
+Yes, the Number of Gaussians Affects Performance. The more Gaussians used, the more computations the GPU has to perform. Each Gaussian requires resources for rendering—whether it's calculating the size, position, or color—so increasing the number of Gaussians will linearly increase the GPU workload. More Gaussians mean more memory reads/writes and increased calculations per frame. This impacts both the GPU’s memory bandwidth and compute capacity. If the GPU is overloaded with too many Gaussians, it could lead to increased render times and drops in frame rate.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
 
 ### Credits
 
 - [Vite](https://vitejs.dev/)
 - [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/)
 - [stats.js](https://github.com/mrdoob/stats.js)
 - [wgpu-matrix](https://github.com/greggman/wgpu-matrix)
-- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Guassian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization)
+- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Gaussian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization).
+- [Blending code reference](https://webgpufundamentals.org/webgpu/lessons/webgpu-transparency.html)
+- [indirect_draw](https://developer.mozilla.org/en-US/docs/Web/API/GPURenderPassEncoder/drawIndirect)

src/renderers/gaussian-renderer.ts

@@ -3,9 +3,10 @@ import preprocessWGSL from '../shaders/preprocess.wgsl';
 import renderWGSL from '../shaders/gaussian.wgsl';
 import { get_sorter,c_histogram_block_rows,C } from '../sort/sort';
 import { Renderer } from './renderer';
+import { encode } from '@loaders.gl/core';
 
 export interface GaussianRenderer extends Renderer {
-
+    renderSettingBuffer:GPUBuffer;
 }
 
 // Utility to create GPU buffers
@@ -33,9 +34,54 @@ export default function get_renderer(
   // ===============================================
   //            Initialize GPU Buffers
   // ===============================================
+
+
+
+
+
+  const indirectdraw_buffersize = 4 * Uint32Array.BYTES_PER_ELEMENT;
+  const indirectdraw_buffer = device.createBuffer({
+    label:"indirect draw buffer",
+    size: indirectdraw_buffersize,
+    usage: GPUBufferUsage.INDIRECT | GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST,
+  })
+
+
+  const indirectdraw_host = new Uint32Array([6, pc.num_points, 0, 0]);
+  device.queue.writeBuffer(indirectdraw_buffer, 0, indirectdraw_host.buffer);
+
 
   const nulling_data = new Uint32Array([0]);
+  const nulling_buffer = device.createBuffer({
+    label: "indirect draw reset null buffer",
+    size: 1 * Uint32Array.BYTES_PER_ELEMENT,
+    usage: GPUBufferUsage.INDIRECT | GPUBufferUsage.COPY_DST | GPUBufferUsage.COPY_SRC,
+  })
+
+  device.queue.writeBuffer(nulling_buffer, 0, nulling_data.buffer);
+
+
+  const splatData = new Uint32Array(pc.num_points * 6);
 
+
+  const splatBuffer = createBuffer(
+    device,
+    'Splat Buffer',
+    splatData.byteLength,
+    GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST | GPUBufferUsage.VERTEX,
+  );
+
+  const renderSettingsData = new Float32Array([1.0, pc.sh_deg]);
+  const renderSettingBuffer = createBuffer(
+    device,
+    'Render setting Buffer',
+    renderSettingsData.byteLength,
+    GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
+    renderSettingsData
+  );
+
+
+
   // ===============================================
   //    Create Compute Pipeline and Bind Groups
   // ===============================================
@@ -52,6 +98,26 @@ export default function get_renderer(
     },
   });
 
+  const computeBindGroup = device.createBindGroup({
+    label: 'Compute Bind Group',
+    layout: preprocess_pipeline.getBindGroupLayout(0),
+    entries: [
+      { binding: 0, resource: { buffer: splatBuffer } }, 
+      { binding: 1, resource: { buffer: pc.sh_buffer } }, 
+      { binding: 2, resource: { buffer: pc.gaussian_3d_buffer } }, 
+    ],
+  });
+
+  const compute_setting_BindGroup = device.createBindGroup({
+    label: 'Compute setting Bind Group',
+    layout: preprocess_pipeline.getBindGroupLayout(1),
+    entries: [
+      { binding: 0, resource: { buffer: camera_buffer } }, 
+      { binding: 1, resource: { buffer: renderSettingBuffer } }, 
+
+    ],
+  });
+
   const sort_bind_group = device.createBindGroup({
     label: 'sort',
     layout: preprocess_pipeline.getBindGroupLayout(2),
@@ -67,7 +133,55 @@ export default function get_renderer(
   // ===============================================
   //    Create Render Pipeline and Bind Groups
   // ===============================================
-
+  const renderPipeline = device.createRenderPipeline({
+    label : "render pipeline",
+    layout: "auto",
+    vertex: {
+        module: device.createShaderModule({
+            label: "vert shader",
+            code: renderWGSL,
+        }),
+        entryPoint: "vs_main",
+        buffers: [],
+    },
+    fragment: {
+        module: device.createShaderModule({
+            label: "frag shader",
+            code: renderWGSL,
+        }),
+        entryPoint: "fs_main",
+        targets: [
+          {
+              format: presentation_format,
+              blend: {
+                  color: {
+                    operation: 'add',
+                    srcFactor: 'one',
+                    dstFactor: 'one-minus-src-alpha',
+                  },
+                  alpha: {
+                    operation: 'add',
+                    srcFactor: 'one',
+                    dstFactor: 'one-minus-src-alpha',
+                  },
+              },
+          },
+      ],
+    }
+  });
+
+  const splatBindGroup = device.createBindGroup({
+    label: 'Splat Bind Group',
+    layout: renderPipeline.getBindGroupLayout(0),
+    entries: [{ binding: 0, resource: { buffer: splatBuffer } },
+              { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } },
+              { binding: 2, resource: { buffer: camera_buffer } }
+      ]
+    ,
+
+  });
+
+
 
   // ===============================================
   //    Command Encoder Functions
@@ -79,8 +193,50 @@ export default function get_renderer(
   // ===============================================
   return {
     frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => {
+
+      encoder.copyBufferToBuffer(nulling_buffer, 0, sorter.sort_info_buffer, 0, 4)
+      encoder.copyBufferToBuffer(nulling_buffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4)
+
+
+      const computePass = encoder.beginComputePass({
+        label: "Compute Pass",
+      });
+      computePass.setPipeline(preprocess_pipeline);
+      computePass.setBindGroup(0, computeBindGroup);
+      computePass.setBindGroup(1, compute_setting_BindGroup);
+      computePass.setBindGroup(2, sort_bind_group);
+
+      const workgroupSize = C.histogram_wg_size;
+      const numWorkgroups = Math.ceil(pc.num_points / workgroupSize);
+
+
+      computePass.dispatchWorkgroups(numWorkgroups);
+
+      computePass.end();
+
+
+
       sorter.sort(encoder);
+
+      encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, indirectdraw_buffer, 4, 4)
+
+
+      const render_pass = encoder.beginRenderPass({
+        label:"render pass",
+        colorAttachments:[{
+          view: texture_view,
+          loadOp : "clear",
+          storeOp: "store",
+          clearValue: {r: 0, g:0, b:0, a:1},
+        }]
+      })
+      render_pass.setPipeline(renderPipeline);
+      render_pass.setBindGroup(0, splatBindGroup);
+      //render_pass.setVertexBuffer(0, splatBuffer);
+      render_pass.drawIndirect(indirectdraw_buffer, 0);
+      render_pass.end();
     },
     camera_buffer,
+    renderSettingBuffer,
   };
 }

src/renderers/renderer.ts

@@ -122,6 +122,10 @@ export default async function init(
       {min: 0, max: 1.5}
     ).on('change', (e) => {
       //TODO: Bind constants to the gaussian renderer.
+      if (!gaussian_renderer){
+        return;
+      }
+      device.queue.writeBuffer(gaussian_renderer.renderSettingBuffer, 0, new Float32Array([params.gaussian_multiplier]));
     });
   }
 

src/shaders/gaussian.wgsl

@@ -1,22 +1,87 @@
+struct CameraUniforms {
+    view: mat4x4<f32>,
+    view_inv: mat4x4<f32>,
+    proj: mat4x4<f32>,
+    proj_inv: mat4x4<f32>,
+    viewport: vec2<f32>,
+    focal: vec2<f32>
+};
+
+ const quad_verts = array<vec2<f32>, 6>(
+    vec2<f32>(-1.0f, 1.0f),  
+    vec2<f32>(-1.0f, -1.0f), 
+    vec2<f32>(1.0f,  -1.0f),  
+    vec2<f32>(1.0f, -1.0f),  
+    vec2<f32>(1.0f,  1.0f),  
+    vec2<f32>(-1.0f, 1.0f),  
+);
+
 struct VertexOutput {
     @builtin(position) position: vec4<f32>,
-    //TODO: information passed from vertex shader to fragment shader
+    @location(0) v_color: vec4<f32>,
+    @location(1) conic: vec3<f32>,
+    @location(2) opacity: f32,
+    @location(3) center: vec2<f32>,
 };
 
+
+
 struct Splat {
     //TODO: information defined in preprocess compute shader
+    packedposition: u32, 
+    packedsize: u32,          
+    packedcolor: array<u32,2>, 
+    packedconic_opacity: array<u32,2>
 };
 
+@group(0) @binding(0)var<storage, read> splats: array<Splat>;
+@group(0) @binding(1)var<storage, read> sorted_indices : array<u32>;
+@group(0) @binding(2) var<uniform> camera: CameraUniforms;
+
+
 @vertex
 fn vs_main(
-) -> VertexOutput {
-    //TODO: reconstruct 2D quad based on information from splat, pass 
+    @builtin(instance_index) instanceIndex: u32,
+    @builtin(vertex_index) vert_idx : u32,) -> VertexOutput {
     var out: VertexOutput;
-    out.position = vec4<f32>(1. ,1. , 0., 1.);
+
+    let index = sorted_indices[instanceIndex];
+    var splat = splats[index];
+    let posi = unpack2x16float(splat.packedposition);
+    let size = unpack2x16float(splat.packedsize);
+    let conicXY = unpack2x16float(splat.packedconic_opacity[0]);
+    let conicZO = unpack2x16float(splat.packedconic_opacity[1]);
+    let conic = vec3<f32>(conicXY, conicZO.x);
+    let op = conicZO.y;
+    let colorRG = unpack2x16float(splat.packedcolor[0]);
+    let colorBA = unpack2x16float(splat.packedcolor[1]);
+    let color = vec4<f32>(colorRG, colorBA);
+
+    let offset = quad_verts[vert_idx];
+    let position = vec2<f32>(posi.x + offset.x * size.x, posi.y + offset.y * size.y);
+
+    out.position = vec4<f32>(position, 0.0, 1.0);
+    out.v_color = color;
+    out.conic = conic;
+    out.opacity = op;
+    out.center = vec2<f32>(posi.xy);
     return out;
 }
 
 @fragment
 fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
-    return vec4<f32>(1.);
+    var NDCpos = (in.position.xy / camera.viewport) * 2.0f - 1.0f;
+    NDCpos.y = -NDCpos.y;
+
+    //back to pixel space
+    var offset = (NDCpos.xy - in.center.xy) * camera.viewport * 0.5f;
+    let power = -0.5f * (in.conic.x * (-offset.x) * (-offset.x) + in.conic.z * offset.y * offset.y) - in.conic.y * (-offset.x) * offset.y;
+
+    if (power > 0.0f){
+        return vec4<f32>(0.0f);
+    }
+    let alpha = min(0.99f, in.opacity * exp(power));
+
+    return in.v_color * alpha;
+    //return vec4<f32>(offset,0.0, 1.0);
 }

src/shaders/point_cloud.wgsl

@@ -35,7 +35,7 @@ fn vs_main(
     let pos = vec4<f32>(a.x, a.y, b.x, 1.);
 
     // TODO: MVP calculations
-    out.position = pos;
+    out.position = camera.proj * camera.view * pos;
 
     return out;
 }