diff --git a/README.md b/README.md index f99cdff..dfff3ae 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,47 @@ -# Project5-WebGPU-Gaussian-Splat-Viewer +# WebGPU-Gaussian-Splat-Viewer -**University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4** +![](images/ga_bonsai.png) -* (TODO) YOUR NAME HERE -* Tested on: (TODO) **Google Chrome 222.2** on - Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab) +* [Zixiao Wang](https://www.linkedin.com/in/zixiao-wang-826a5a255/) +* Tested on: Google Chrome Version 130.0. on + Windows 11, i7-14700K @ 3.40 GHz 32GB, GTX 4070TI 12GB 4K Monitor (Personal PC) ### Live Demo -[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred) +[Live Demo](https://lanbiubiu1.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/) ### Demo Video/GIF +![](images/ga_bike.png) +![](images/ga_bike_move.gif) +![](images/ga_playroom_1.gif) -[![](img/video.mp4)](TODO) -### (TODO: Your README) -*DO NOT* leave the README to the last minute! It is a crucial part of the -project, and we will not be able to grade you without a good README. +### Introduction +| Point-Cloud | Gaussian Splat | +|-----------------------|--------------------------| +| ![](images/pc_bike.png) | ![](images/ga_bike.png) | + +In this Project, I implemented the 3D Gaussian Splat Viewer. The entire project is based on [3D Gaussian Splatting +for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). +To start off, we need to create a point_cloud shader to cluster the points based on the scene input .ply file. After that, the preprocess shader will read the point cloud data as splats and transfer their position into the screen space. Then, radix sort is applied to sort the splats by their screen space position and depth. In the end, the processed data are fed back into Gaussian splat to render the points in the scene in real time. + +![](images/workflow.png) + +### Performance Analysis + +#### Difference between Point-Cloud and Gaussian Renderer +The Point-Cloud image represents a point-cloud rendering, which shows the bicycle and bench represented by numerous distinct points. Each point in the cloud contributes to a sampled representation of the scene. The level of detail is sparse, especially at the edges, and some structure is lost due to the gaps between points, especially on finer details like spokes and the texture of the grass. The Gaussian renderer represents the scene with significantly fewer gaps, generating a much more filled and realistic depiction. This is due to representing each point as a Gaussian splat rather than a single pixel, allowing for smoother boundaries and a more cohesive structure. This representation results in better-defined shapes and surfaces, providing a higher fidelity to the original scene. + +#### Changing Workgroup Size Affects Performance +Changing the workgroup size directly affects the efficiency of the compute shader execution on the GPU. Workgroup size determines how many threads are used to process a specific task. The default setting size on my machine has the best performance. Either higher or lower the workgroup size can lead to worse performance. I think it is because of the inefficiencies of many wasted computations. And smaller workgroup size results in underutilization of GPU cores since fewer threads are running in parallel. + +#### Utilization of View-Frustum Culling to Performance Improvement +View-frustum culling improves performance by reducing the number of objects that need to be processed and rendered by the GPU. It essentially removes all points, quads, or objects that are outside the visible region (the frustum). However, as the current testing scene is relatively small, it does not really benefit from View-Frustum Culling. Instead, it might lead to some unnecessary computations. Ideally, for large scenes with large view space, the View-Frustum Culling will largely improve the performance. + +#### Number of Guassians to Performance +Yes, the Number of Gaussians Affects Performance. The more Gaussians used, the more computations the GPU has to perform. Each Gaussian requires resources for rendering—whether it's calculating the size, position, or color—so increasing the number of Gaussians will linearly increase the GPU workload. More Gaussians mean more memory reads/writes and increased calculations per frame. This impacts both the GPU’s memory bandwidth and compute capacity. If the GPU is overloaded with too many Gaussians, it could lead to increased render times and drops in frame rate. -This assignment has a considerable amount of performance analysis compared -to implementation work. Complete the implementation early to leave time! ### Credits @@ -28,4 +49,6 @@ to implementation work. Complete the implementation early to leave time! - [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/) - [stats.js](https://github.com/mrdoob/stats.js) - [wgpu-matrix](https://github.com/greggman/wgpu-matrix) -- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Guassian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization) +- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Gaussian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization). +- [Blending code reference](https://webgpufundamentals.org/webgpu/lessons/webgpu-transparency.html) +- [indirect_draw](https://developer.mozilla.org/en-US/docs/Web/API/GPURenderPassEncoder/drawIndirect) diff --git a/images/ga_bike.png b/images/ga_bike.png new file mode 100644 index 0000000..132732a Binary files /dev/null and b/images/ga_bike.png differ diff --git a/images/ga_bike_move.gif b/images/ga_bike_move.gif new file mode 100644 index 0000000..6640e9b Binary files /dev/null and b/images/ga_bike_move.gif differ diff --git a/images/ga_bonsai.png b/images/ga_bonsai.png new file mode 100644 index 0000000..7d3b1d9 Binary files /dev/null and b/images/ga_bonsai.png differ diff --git a/images/ga_playroom_1.gif b/images/ga_playroom_1.gif new file mode 100644 index 0000000..c2882e5 Binary files /dev/null and b/images/ga_playroom_1.gif differ diff --git a/images/pc_bike.png b/images/pc_bike.png new file mode 100644 index 0000000..3f27a8c Binary files /dev/null and b/images/pc_bike.png differ diff --git a/images/workflow.png b/images/workflow.png new file mode 100644 index 0000000..910507d Binary files /dev/null and b/images/workflow.png differ diff --git a/src/renderers/gaussian-renderer.ts b/src/renderers/gaussian-renderer.ts index 1684523..140f465 100644 --- a/src/renderers/gaussian-renderer.ts +++ b/src/renderers/gaussian-renderer.ts @@ -3,9 +3,10 @@ import preprocessWGSL from '../shaders/preprocess.wgsl'; import renderWGSL from '../shaders/gaussian.wgsl'; import { get_sorter,c_histogram_block_rows,C } from '../sort/sort'; import { Renderer } from './renderer'; +import { encode } from '@loaders.gl/core'; export interface GaussianRenderer extends Renderer { - + renderSettingBuffer:GPUBuffer; } // Utility to create GPU buffers @@ -33,9 +34,54 @@ export default function get_renderer( // =============================================== // Initialize GPU Buffers // =============================================== + + + + + + const indirectdraw_buffersize = 4 * Uint32Array.BYTES_PER_ELEMENT; + const indirectdraw_buffer = device.createBuffer({ + label:"indirect draw buffer", + size: indirectdraw_buffersize, + usage: GPUBufferUsage.INDIRECT | GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST, + }) + + + const indirectdraw_host = new Uint32Array([6, pc.num_points, 0, 0]); + device.queue.writeBuffer(indirectdraw_buffer, 0, indirectdraw_host.buffer); + const nulling_data = new Uint32Array([0]); + const nulling_buffer = device.createBuffer({ + label: "indirect draw reset null buffer", + size: 1 * Uint32Array.BYTES_PER_ELEMENT, + usage: GPUBufferUsage.INDIRECT | GPUBufferUsage.COPY_DST | GPUBufferUsage.COPY_SRC, + }) + + device.queue.writeBuffer(nulling_buffer, 0, nulling_data.buffer); + + + const splatData = new Uint32Array(pc.num_points * 6); + + const splatBuffer = createBuffer( + device, + 'Splat Buffer', + splatData.byteLength, + GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST | GPUBufferUsage.VERTEX, + ); + + const renderSettingsData = new Float32Array([1.0, pc.sh_deg]); + const renderSettingBuffer = createBuffer( + device, + 'Render setting Buffer', + renderSettingsData.byteLength, + GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST, + renderSettingsData + ); + + + // =============================================== // Create Compute Pipeline and Bind Groups // =============================================== @@ -52,6 +98,26 @@ export default function get_renderer( }, }); + const computeBindGroup = device.createBindGroup({ + label: 'Compute Bind Group', + layout: preprocess_pipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: { buffer: splatBuffer } }, + { binding: 1, resource: { buffer: pc.sh_buffer } }, + { binding: 2, resource: { buffer: pc.gaussian_3d_buffer } }, + ], + }); + + const compute_setting_BindGroup = device.createBindGroup({ + label: 'Compute setting Bind Group', + layout: preprocess_pipeline.getBindGroupLayout(1), + entries: [ + { binding: 0, resource: { buffer: camera_buffer } }, + { binding: 1, resource: { buffer: renderSettingBuffer } }, + + ], + }); + const sort_bind_group = device.createBindGroup({ label: 'sort', layout: preprocess_pipeline.getBindGroupLayout(2), @@ -67,7 +133,55 @@ export default function get_renderer( // =============================================== // Create Render Pipeline and Bind Groups // =============================================== - + const renderPipeline = device.createRenderPipeline({ + label : "render pipeline", + layout: "auto", + vertex: { + module: device.createShaderModule({ + label: "vert shader", + code: renderWGSL, + }), + entryPoint: "vs_main", + buffers: [], + }, + fragment: { + module: device.createShaderModule({ + label: "frag shader", + code: renderWGSL, + }), + entryPoint: "fs_main", + targets: [ + { + format: presentation_format, + blend: { + color: { + operation: 'add', + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + }, + alpha: { + operation: 'add', + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + }, + }, + }, + ], + } + }); + + const splatBindGroup = device.createBindGroup({ + label: 'Splat Bind Group', + layout: renderPipeline.getBindGroupLayout(0), + entries: [{ binding: 0, resource: { buffer: splatBuffer } }, + { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } }, + { binding: 2, resource: { buffer: camera_buffer } } + ] + , + + }); + + // =============================================== // Command Encoder Functions @@ -79,8 +193,50 @@ export default function get_renderer( // =============================================== return { frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => { + + encoder.copyBufferToBuffer(nulling_buffer, 0, sorter.sort_info_buffer, 0, 4) + encoder.copyBufferToBuffer(nulling_buffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4) + + + const computePass = encoder.beginComputePass({ + label: "Compute Pass", + }); + computePass.setPipeline(preprocess_pipeline); + computePass.setBindGroup(0, computeBindGroup); + computePass.setBindGroup(1, compute_setting_BindGroup); + computePass.setBindGroup(2, sort_bind_group); + + const workgroupSize = C.histogram_wg_size; + const numWorkgroups = Math.ceil(pc.num_points / workgroupSize); + + + computePass.dispatchWorkgroups(numWorkgroups); + + computePass.end(); + + + sorter.sort(encoder); + + encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, indirectdraw_buffer, 4, 4) + + + const render_pass = encoder.beginRenderPass({ + label:"render pass", + colorAttachments:[{ + view: texture_view, + loadOp : "clear", + storeOp: "store", + clearValue: {r: 0, g:0, b:0, a:1}, + }] + }) + render_pass.setPipeline(renderPipeline); + render_pass.setBindGroup(0, splatBindGroup); + //render_pass.setVertexBuffer(0, splatBuffer); + render_pass.drawIndirect(indirectdraw_buffer, 0); + render_pass.end(); }, camera_buffer, + renderSettingBuffer, }; } diff --git a/src/renderers/renderer.ts b/src/renderers/renderer.ts index ffdf9ba..a8b5e65 100644 --- a/src/renderers/renderer.ts +++ b/src/renderers/renderer.ts @@ -122,6 +122,10 @@ export default async function init( {min: 0, max: 1.5} ).on('change', (e) => { //TODO: Bind constants to the gaussian renderer. + if (!gaussian_renderer){ + return; + } + device.queue.writeBuffer(gaussian_renderer.renderSettingBuffer, 0, new Float32Array([params.gaussian_multiplier])); }); } diff --git a/src/shaders/gaussian.wgsl b/src/shaders/gaussian.wgsl index 759226d..7f24f05 100644 --- a/src/shaders/gaussian.wgsl +++ b/src/shaders/gaussian.wgsl @@ -1,22 +1,87 @@ +struct CameraUniforms { + view: mat4x4, + view_inv: mat4x4, + proj: mat4x4, + proj_inv: mat4x4, + viewport: vec2, + focal: vec2 +}; + + const quad_verts = array, 6>( + vec2(-1.0f, 1.0f), + vec2(-1.0f, -1.0f), + vec2(1.0f, -1.0f), + vec2(1.0f, -1.0f), + vec2(1.0f, 1.0f), + vec2(-1.0f, 1.0f), +); + struct VertexOutput { @builtin(position) position: vec4, - //TODO: information passed from vertex shader to fragment shader + @location(0) v_color: vec4, + @location(1) conic: vec3, + @location(2) opacity: f32, + @location(3) center: vec2, }; + + struct Splat { //TODO: information defined in preprocess compute shader + packedposition: u32, + packedsize: u32, + packedcolor: array, + packedconic_opacity: array }; +@group(0) @binding(0)var splats: array; +@group(0) @binding(1)var sorted_indices : array; +@group(0) @binding(2) var camera: CameraUniforms; + + @vertex fn vs_main( -) -> VertexOutput { - //TODO: reconstruct 2D quad based on information from splat, pass + @builtin(instance_index) instanceIndex: u32, + @builtin(vertex_index) vert_idx : u32,) -> VertexOutput { var out: VertexOutput; - out.position = vec4(1. ,1. , 0., 1.); + + let index = sorted_indices[instanceIndex]; + var splat = splats[index]; + let posi = unpack2x16float(splat.packedposition); + let size = unpack2x16float(splat.packedsize); + let conicXY = unpack2x16float(splat.packedconic_opacity[0]); + let conicZO = unpack2x16float(splat.packedconic_opacity[1]); + let conic = vec3(conicXY, conicZO.x); + let op = conicZO.y; + let colorRG = unpack2x16float(splat.packedcolor[0]); + let colorBA = unpack2x16float(splat.packedcolor[1]); + let color = vec4(colorRG, colorBA); + + let offset = quad_verts[vert_idx]; + let position = vec2(posi.x + offset.x * size.x, posi.y + offset.y * size.y); + + out.position = vec4(position, 0.0, 1.0); + out.v_color = color; + out.conic = conic; + out.opacity = op; + out.center = vec2(posi.xy); return out; } @fragment fn fs_main(in: VertexOutput) -> @location(0) vec4 { - return vec4(1.); + var NDCpos = (in.position.xy / camera.viewport) * 2.0f - 1.0f; + NDCpos.y = -NDCpos.y; + + //back to pixel space + var offset = (NDCpos.xy - in.center.xy) * camera.viewport * 0.5f; + let power = -0.5f * (in.conic.x * (-offset.x) * (-offset.x) + in.conic.z * offset.y * offset.y) - in.conic.y * (-offset.x) * offset.y; + + if (power > 0.0f){ + return vec4(0.0f); + } + let alpha = min(0.99f, in.opacity * exp(power)); + + return in.v_color * alpha; + //return vec4(offset,0.0, 1.0); } \ No newline at end of file diff --git a/src/shaders/point_cloud.wgsl b/src/shaders/point_cloud.wgsl index 01dded1..617171e 100644 --- a/src/shaders/point_cloud.wgsl +++ b/src/shaders/point_cloud.wgsl @@ -35,7 +35,7 @@ fn vs_main( let pos = vec4(a.x, a.y, b.x, 1.); // TODO: MVP calculations - out.position = pos; + out.position = camera.proj * camera.view * pos; return out; } diff --git a/src/shaders/preprocess.wgsl b/src/shaders/preprocess.wgsl index bbc63f5..189a087 100644 --- a/src/shaders/preprocess.wgsl +++ b/src/shaders/preprocess.wgsl @@ -46,7 +46,7 @@ struct CameraUniforms { struct RenderSettings { gaussian_scaling: f32, - sh_deg: f32, + sh_deg: u32, } struct Gaussian { @@ -57,9 +57,31 @@ struct Gaussian { struct Splat { //TODO: store information for 2D splat rendering + packedposition: u32, + packedsize: u32, + packedcolor: array, + packedconic_opacity: array }; //TODO: bind your data here +@group(0) @binding(0) +var splatBuffer: array; + +@group(0) @binding(1) +var sh_buffer: array; + +@group(0) @binding(2) +var gaussian3d_buffer: array; + + +@group(1) @binding(0) +var camera: CameraUniforms; + +@group(1) @binding(1) +var renderSettings: RenderSettings; + + + @group(2) @binding(0) var sort_infos: SortInfos; @group(2) @binding(1) @@ -71,8 +93,14 @@ var sort_dispatch: DispatchIndirect; /// reads the ith sh coef from the storage buffer fn sh_coef(splat_idx: u32, c_idx: u32) -> vec3 { - //TODO: access your binded sh_coeff, see load.ts for how it is stored - return vec3(0.0); + let base_index = splat_idx * 24 + (c_idx / 2) * 3 + c_idx % 2; + let color01 = unpack2x16float(sh_buffer[base_index + 0]); + let color23 = unpack2x16float(sh_buffer[base_index + 1]); + + if (c_idx % 2 == 0) { + return vec3f(color01.x, color01.y, color23.x); + } + return vec3f(color01.y, color23.x, color23.y); } // spherical harmonics evaluation with Condon–Shortley phase @@ -113,6 +141,130 @@ fn preprocess(@builtin(global_invocation_id) gid: vec3, @builtin(num_workgr let idx = gid.x; //TODO: set up pipeline as described in instruction + if(idx >= arrayLength(&gaussian3d_buffer)){ + return; + } + + let gaussian = gaussian3d_buffer[idx]; + let xy = unpack2x16float(gaussian.pos_opacity[0]); + let z1 = unpack2x16float(gaussian.pos_opacity[1]); + let xyz = vec3(xy, z1.x); + let alpha = z1.y; + + let viewPos = (camera.view*vec4(xyz, 1.0f)).xyz; + var positionNDC = camera.proj * (camera.view * vec4(xyz, 1.0)); + + positionNDC/=positionNDC.w; + let boundary = 1.2f; + + if(positionNDC.x < -boundary || positionNDC.x> boundary ||positionNDC.y < -boundary || positionNDC.y > boundary + || positionNDC.z < 0.0 || positionNDC.z > 1.0f){ + return; + } + + //calculatin of 3D covariance + let quatWX = unpack2x16float(gaussian.rot[0]); + let quatYZ = unpack2x16float(gaussian.rot[1]); + let scaleXY = unpack2x16float(gaussian.scale[0]); + let scaleZ = unpack2x16float(gaussian.scale[1]); + let scale = exp(vec3( + scaleXY.x, + scaleXY.y, + scaleZ.x)); + + let r = quatWX.x; + let x = quatWX.y; + let y = quatYZ.x; + let z = quatYZ.y; + + let rotationMatrix = mat3x3( + 1.f - 2.f * (y * y + z * z), 2.f * (x * y - r * z), 2.f * (x * z + r * y), + 2.f * (x * y + r * z), 1.f - 2.f * (x * x + z * z), 2.f * (y * z - r * x), + 2.f * (x * z - r * y), 2.f * (y * z + r * x), 1.f - 2.f * (x * x + y * y) + ); + + let scaleMatrix = mat3x3( + scale.x * renderSettings.gaussian_scaling , 0.0, 0.0, + 0.0, scale.y * renderSettings.gaussian_scaling , 0.0, + 0.0, 0.0, scale.z * renderSettings.gaussian_scaling , + ); + + let covariance3D = transpose(scaleMatrix * rotationMatrix) * scaleMatrix * rotationMatrix; + let covarianceSymmetric = array(covariance3D[0][0], covariance3D[0][1], covariance3D[0][2], + covariance3D[1][1], covariance3D[1][2], covariance3D[2][2],); + + //2D covariance matrix + var t = (camera.view * vec4(xyz, 1.0)).xyz; + + + let J = mat3x3( + camera.focal.x / t.z, 0.0f, -(camera.focal.x * t.x)/(t.z * t.z), + 0.0f, camera.focal.y / t.z, -(camera.focal.y * t.y)/(t.z * t.z), + 0.0, 0.0, 0.0 + ); + + let W = mat3x3( + camera.view[0].x, camera.view[1].x, camera.view[2].x, + camera.view[0].y, camera.view[1].y, camera.view[2].y, + camera.view[0].z, camera.view[1].z, camera.view[2].z + ); + + let T = W * J; + + let Vrk = mat3x3( + covarianceSymmetric[0], covarianceSymmetric[1], covarianceSymmetric[2], + covarianceSymmetric[1], covarianceSymmetric[3], covarianceSymmetric[4], + covarianceSymmetric[2], covarianceSymmetric[4], covarianceSymmetric[5], + ); + + var cov = transpose(T) * transpose(Vrk) * T; + cov[0][0] += 0.3f; + cov[1][1] += 0.3f; + + let cov_2D = vec3(cov[0][0], cov[0][1], cov[1][1]); + //radius + let det = (cov_2D.x * cov_2D.z - cov_2D.y * cov_2D.y); + if (det == 0){ + return; + } + let mid = 0.5f * (cov_2D.x + cov_2D.z); + let lambda1 = mid + sqrt(max(0.1f, mid * mid - det)); + let lambda2 = mid - sqrt(max(0.1f, mid * mid - det)); + + let radius = ceil(3.0f * sqrt(max(lambda1, lambda2))); + + let maxNDCsize = vec2( 2.0f * radius, 2.0f * radius)/camera.viewport; + + //testing + let test12345 = sort_infos.padded_size; + let testing2 = sh_buffer[0]; + let render = renderSettings.gaussian_scaling; + let testing3 = sort_depths[0]; + let testing4 = sort_indices[0]; + let testing5 = sort_dispatch.dispatch_y; + + let index = atomicAdd(&sort_infos.keys_size, 1u); + sort_indices[index] = index; + sort_depths[index]= bitcast(100.0 - viewPos.z); + + splatBuffer[index].packedposition = pack2x16float(positionNDC.xy); + splatBuffer[index].packedsize = pack2x16float(maxNDCsize); + + let dir =normalize(xyz - camera.view_inv[2].xyz); + let color = computeColorFromSH(dir, idx, renderSettings.sh_deg); + splatBuffer[index].packedcolor = array(pack2x16float(color.rg), pack2x16float(vec2(color.b, 1.0f))); + + let conic = vec3( + cov_2D.z * (1.f / det), -cov_2D.y * (1.f / det), cov_2D.x * (1.f / det) + ); + let op = 1.0f / (1.0f + exp(-alpha)); + splatBuffer[index].packedconic_opacity = array(pack2x16float(conic.xy), pack2x16float(vec2(conic.z, op))); + + + let keys_per_dispatch = workgroupSize * sortKeyPerThread; + if (index % keys_per_dispatch == 0){ + atomicAdd(&sort_dispatch.dispatch_x, 1); + } // increment DispatchIndirect.dispatchx each time you reach limit for one dispatch of keys -} \ No newline at end of file +}