diff --git a/README.md b/README.md index edffdaf..9177f88 100644 --- a/README.md +++ b/README.md @@ -2,27 +2,83 @@ **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5** -* (TODO) YOUR NAME HERE -* Tested on: (TODO) **Google Chrome 222.2** on - Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab) +* Griffin Evans +* Tested on: Windows 11 Education, i9-12900F @ 2.40GHz 64.0GB, NVIDIA GeForce RTX 3090 (Levine 057 #1) on Google Chrome -### Live Demo +## Live Demo -[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred) +[![](https://github.com/grievans/Project5-WebGPU-Gaussian-Splat-Viewer/blob/main/images/Screenshot%202025-10-28%20194238%20packed%20f16s.png)](https://grievans.github.io/Project5-WebGPU-Gaussian-Splat-Viewer/) -### Demo Video/GIF +In addition to the page linked in the image above, use of the demo requires a .ply file of point cloud data and a .json file of camera positional data. -[![](img/video.mp4)](TODO) +## Demo Video/GIF -### (TODO: Your README) +https://github.com/user-attachments/assets/0f0d214b-f67a-47e7-85a1-ef576ebfa30a -*DO NOT* leave the README to the last minute! It is a crucial part of the -project, and we will not be able to grade you without a good README. +## Overview -This assignment has a considerable amount of performance analysis compared -to implementation work. Complete the implementation early to leave time! +![](images/Screenshot%202025-10-28%20200732.png) -### Credits +This is a WebGPU-based renderer for rendering point clouds, displaying them either as single pixels or as gaussian splats—ellipsoids projected onto the screen with colors varying with viewing angle in order to reconstruct views of the scene. The user can control the scale of the splats used, making the image more sparse or solid as may be necessary for the particular input to appear realistic. + +![](images/Screenshot%202025-10-28%20200822.png) + +![](images/Screenshot%202025-10-28%20200850.png) + +## Analysis + +- Compare your results from point-cloud and gaussian renderer, what are the differences? + +Screen Shot 2025-10-28 at 10 28 29 PM + +On the desktop used to test this project as mentioned above, both of the renderers produced frames in 16.67 ms—having their framerate capped by Chrome based on the monitor's refresh rate such that they were apparently identical in performance despite the additional work performed in the gaussian renderer. On a weaker machine, a MacBook Pro 15-inch 2019 running macOS 10.14.6, performance differences were more noticeable, with the bicycle scene shown in the above video varying between about 13.3ms and 18.2 ms to render as a point cloud but slowing to around 110.5 ms per frame to render it as gaussian splats. However, this number is inaccurate as it leaves out the depth sorting step of the gaussian renderer, which when enabled completely freezes the browser and produces a blank screen as output on that machine. This may be a matter of some compatibility issue with the sorting algorithm code, as this test was limited to an older version of Chrome (116.0.5845.187, the last version available for macOS below 10.15), though as it did not explicitly cause an error to occur it may also be a matter of the sorting step being such a bottleneck that the weaker hardware is unable to complete it successfully without the browser locking up. + +Screen Shot 2025-10-28 at 10 28 16 PM + +Without the sorting step, the size of the quads rendered (in comparison to the single pixels used to render the point-cloud mode) seems to be a major performance detriment, as reducing the gaussian multiplier (thus reducing the scale of each splat) significantly increased the speed, going from 110.5 ms per frame as mentioned above to 21.1 ms per frame on the MacBook. However, note that these larger sized quads are key to the ability of gaussian splatting to produce solid-appearing images from the input data, in contrast to the single-pixel view of the point cloud renderer which produces significant sparseness and gaps, and hence too low a gaussian multiplier produces unconvincing results as in the image below. + +Screen Shot 2025-10-28 at 11 02 12 PM + +Additional performance impacts potentially come from the greater number of calculations needed to compute the splat data (needing covariance, conics, etc.) as compared to simply needing to multiply the position by the view and projection matrices to render a point cloud, as well as from overhead that may come with the greater amount of reading and writing from memory in constructing and passing the splat data in the compute shader and vertex/fragment shaders. + +- For gaussian renderer, how does changing the workgroup-size affect performance? Why do you think this is? + +Attempts at testing different workgroup sizes with the full gaussian renderer did not reveal a noticeable performance impact, however testing was limited by the output appearing incomplete when using lower workgroup sizes. For sizes below 256, it appears that the sorting skips some number of the splats, with which are left out varying from frame-to-frame such that the image has parts flicker and disappear as in the image below: + +![](images/Screenshot%202025-10-28%20203758%20128.png) + +Changing only the size passed in to the splat preprocess pipeline without altering the sorting pipelines caused errors, so testing the full renderer with different workgroup sizes appears infeasible with the current codebase [^1]. Testing without sorting enabled, the workgroup size appeared to have no particular effect on the rest of the process, staying in the same range hovering around 9.10 fps (10.99 ms/frame) on the MacBook for workgroup sizes of 32, 64, 128, and 256 with the default gaussian multiplier of 1. I suspect the relative lack of factors like branching may be why this appeared insignificant—with the preprocess compute shader only branching to return when outside the input array length, to return when outside the view frustum, to return when the determinant is 0 (which does not appear to occur regularly, especially as we add an offset to the covariance matrix to attempt to prevent it), and at the very end to increment the dispatch count for the sorting step when at a size threshold. I suspect the sorting portion may be more heavily affected by workgroup count though cannot currently test it. + +- Does view-frustum culling give performance improvement? Why do you think this is? + +Screen Shot 2025-10-28 at 11 34 20 PM + +The impact of culling depends on the particular scene and viewing position, but it appears to give some degree of performance improvement in any situation where we are not viewing an entire scene at once. For example, with the camera rotated to only see about half of a scene as in the above image, our run time per frame is about 51.6 ms with frustum culling and 56.2 ms without frustum culling. When parts of the scene are off-screen, we still see some performance improvement without view-frustum culling, as in our fragment shader we still automatically skip many fragments that fall outside the screen bounds, however we see further benefit in performing our own culling in the preprocess compute step as we can skip entire splats in both the sort and vertex shader steps rather than waiting until the fragment shader to see if they are relevant. + +Culling based on the near and far clip planes also provides benefit in removing bugs that can occur when moving the camera close in to the scene, as in the below image which has the bike's handlebar behind the camera position yet appearing in the top left corner of the screen. + +Screen Shot 2025-10-28 at 11 43 37 PM + +- Does number of gaussians affect performance? Why do you think this is? + +Screen Shot 2025-10-28 at 11 52 19 PM + +Number of gaussians has a significant performance impact. For example, on the aforementioned MacBook, the bonsai scene as in the image above runs significantly faster than the bike scene shown earlier, which it has about a fourth of the points of (1063091 vs. 272956), taking about 27.39 ms per frame at an angle to see the entire scene (even with the gaussian multiplier raised to 1.5). Fewer gaussians means fewer workgroups needed for the preprocessing shader and in turn fewer elements to sort (and fewer workgroups used to sort them) in the sorting step and finally fewer splats to shade in the vertex and fragment shaders. + +## Bloopers + +![](images/Screenshot%202025-10-28%20192735.png) + +![](images/Screenshot%202025-10-28%20193414.png) + +Both of the above images show visual distortions caused by using the wrong values when changing the input variables in the vertex shader to use 16-bit floats packed into u32s; the first shows the max radius of the splat being used instead of the first term of the conic matrix, and the second shows the opacity value being the negation of what was intended. + + + +[^1]: Addendum: I've realized in writing this how to change this (via line 335 in preprocess.wgsl) but I don't have access to the other machine to test at the moment + + +## Credits - [Vite](https://vitejs.dev/) - [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/) diff --git a/images/Recording 2025-10-28 200500.mp4 b/images/Recording 2025-10-28 200500.mp4 new file mode 100644 index 0000000..6288e77 Binary files /dev/null and b/images/Recording 2025-10-28 200500.mp4 differ diff --git a/images/Screenshot 2025-10-28 192735.png b/images/Screenshot 2025-10-28 192735.png new file mode 100644 index 0000000..7cf0b8a Binary files /dev/null and b/images/Screenshot 2025-10-28 192735.png differ diff --git a/images/Screenshot 2025-10-28 193414.png b/images/Screenshot 2025-10-28 193414.png new file mode 100644 index 0000000..b19f434 Binary files /dev/null and b/images/Screenshot 2025-10-28 193414.png differ diff --git a/images/Screenshot 2025-10-28 193950 f32s.png b/images/Screenshot 2025-10-28 193950 f32s.png new file mode 100644 index 0000000..17b1d78 Binary files /dev/null and b/images/Screenshot 2025-10-28 193950 f32s.png differ diff --git a/images/Screenshot 2025-10-28 194238 packed f16s.png b/images/Screenshot 2025-10-28 194238 packed f16s.png new file mode 100644 index 0000000..79677e7 Binary files /dev/null and b/images/Screenshot 2025-10-28 194238 packed f16s.png differ diff --git a/images/Screenshot 2025-10-28 200732.png b/images/Screenshot 2025-10-28 200732.png new file mode 100644 index 0000000..036e51c Binary files /dev/null and b/images/Screenshot 2025-10-28 200732.png differ diff --git a/images/Screenshot 2025-10-28 200822.png b/images/Screenshot 2025-10-28 200822.png new file mode 100644 index 0000000..bff018c Binary files /dev/null and b/images/Screenshot 2025-10-28 200822.png differ diff --git a/images/Screenshot 2025-10-28 200850.png b/images/Screenshot 2025-10-28 200850.png new file mode 100644 index 0000000..3a48b18 Binary files /dev/null and b/images/Screenshot 2025-10-28 200850.png differ diff --git a/images/Screenshot 2025-10-28 203758 128.png b/images/Screenshot 2025-10-28 203758 128.png new file mode 100644 index 0000000..18bdfb2 Binary files /dev/null and b/images/Screenshot 2025-10-28 203758 128.png differ diff --git a/package-lock.json b/package-lock.json index 04843bd..694c409 100644 --- a/package-lock.json +++ b/package-lock.json @@ -12,6 +12,7 @@ "@loaders.gl/ply": "^4.2.2", "@petamoriken/float16": "^3.8.7", "tweakpane": "^3.1.8", + "tweakpane-plugin-file-import": "^0.2.0", "wgpu-matrix": "^3.2.0" }, "devDependencies": { diff --git a/src/renderers/gaussian-renderer.ts b/src/renderers/gaussian-renderer.ts index 1684523..0d3aff4 100644 --- a/src/renderers/gaussian-renderer.ts +++ b/src/renderers/gaussian-renderer.ts @@ -20,12 +20,15 @@ const createBuffer = ( if (data) device.queue.writeBuffer(buffer, 0, data); return buffer; }; +// ^TODO maybe should swap to using that; forgot it was here. same functionality though + export default function get_renderer( pc: PointCloud, device: GPUDevice, presentation_format: GPUTextureFormat, camera_buffer: GPUBuffer, + renderSettingsBuffer: GPUBuffer, ): GaussianRenderer { const sorter = get_sorter(pc.num_points, device); @@ -36,6 +39,37 @@ export default function get_renderer( const nulling_data = new Uint32Array([0]); + const nullBuffer = createBuffer( + device, + "null buffer", + 4, + GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST, + nulling_data + ); + + const splatBuffer = device.createBuffer({ + label: 'splat data buffer', + // TODO size for 16bit + size: pc.num_points * 4*5, // buffer size multiple of 4? + usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_SRC, + // mappedAtCreation: false, + }); + + // TODO + const indirectData = new Uint32Array(4); + indirectData[0] = 6; + indirectData[1] = pc.num_points; + indirectData[2] = 0; + indirectData[3] = 0; + // console.log(pc.num_points); + + // TODO should do mappedAtCreation thing I think? or wait we have to write later anyway so maybe fine + const indirectBuffer = device.createBuffer({ + label: "indirect draw buffer", + size: 16, // TODO + usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.INDIRECT + }); + device.queue.writeBuffer(indirectBuffer, 0, indirectData, 0, indirectData.length); // =============================================== // Create Compute Pipeline and Bind Groups // =============================================== @@ -51,9 +85,11 @@ export default function get_renderer( }, }, }); + // let test1 = preprocess_pipeline.getBindGroupLayout(2); + // console.log(test1); const sort_bind_group = device.createBindGroup({ - label: 'sort', + label: 'sort gaussian preprocess bind group', layout: preprocess_pipeline.getBindGroupLayout(2), entries: [ { binding: 0, resource: { buffer: sorter.sort_info_buffer } }, @@ -63,23 +99,171 @@ export default function get_renderer( ], }); + // const gaussian_bind_group = device.createBindGroup({ + // label: 'point cloud gaussians', + // layout: preprocess_pipeline.getBindGroupLayout(1), + // entries: [ + // {binding: 0, resource: { buffer: pc.gaussian_3d_buffer }}, + // ], + // }); + // TODO need to invoke preprocess pipeline I think still? // =============================================== // Create Render Pipeline and Bind Groups // =============================================== + + // TODO figure out what if any of this I need to change + // ^ different buffer I think? splats rather than Gaussian struct directly? + // or is it meant to be both? I don't totally understand the distinction + const render_shader = device.createShaderModule({code: renderWGSL}); + const render_pipeline = device.createRenderPipeline({ + label: 'gaussian render', + layout: 'auto', + vertex: { + module: render_shader, + entryPoint: 'vs_main', + }, + fragment: { + module: render_shader, + entryPoint: 'fs_main', + targets: [{ format: presentation_format, + blend: { + color: { + operation: "add", + srcFactor: "one", + dstFactor: "one-minus-src-alpha" + }, + alpha: { + operation: "add", + srcFactor: "one", + dstFactor: "one-minus-src-alpha" + } + } + }], + }, + // primitive: { + // topology: 'point-list', + // }, + }); + + const camera_bind_group = device.createBindGroup({ + label: 'gaussian splat camera', + layout: preprocess_pipeline.getBindGroupLayout(0), + entries: [ + {binding: 0, resource: { buffer: camera_buffer }} + ], + }); + const gaussian_bind_group = device.createBindGroup({ + label: 'gaussian splat compute bind group', + layout: preprocess_pipeline.getBindGroupLayout(1), + entries: [ + {binding: 0, resource: { buffer: pc.gaussian_3d_buffer }}, + {binding: 1, resource: { buffer: splatBuffer }}, + {binding: 2, resource: { buffer: pc.sh_buffer}} + ], + }); + const renderSettings_bind_group = device.createBindGroup({ + label: 'render settings bind group', + layout: preprocess_pipeline.getBindGroupLayout(3), + entries: [ + {binding: 0, resource: { buffer: renderSettingsBuffer }}, + ], + }); + + // const splatBuffer // TODO + const splat_bind_group = device.createBindGroup({ + label: 'splat bind group', + layout: render_pipeline.getBindGroupLayout(0), + entries: [ + {binding: 0, resource: {buffer: splatBuffer}}, + { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } }, + {binding: 2, resource: { buffer: camera_buffer }} + + ], + }); // =============================================== // Command Encoder Functions // =============================================== - + // TODO not sure where this should be done exactly/if this is what's meant to be in this part + const preprocess = (encoder: GPUCommandEncoder) => { + const pass = encoder.beginComputePass({ + label: 'gaussian splat preprocess', + // colorAttachments: [ + // { + // view: texture_view, + // loadOp: 'clear', + // storeOp: 'store', + // } + // ], + }); + pass.setPipeline(preprocess_pipeline); + pass.setBindGroup(0, camera_bind_group); + pass.setBindGroup(1, gaussian_bind_group); + pass.setBindGroup(2, sort_bind_group); + pass.setBindGroup(3, renderSettings_bind_group); + + + // pass.draw(pc.num_points); + pass.dispatchWorkgroups(Math.ceil(pc.num_points / C.histogram_wg_size)); + // TODO make sure dispatch is done right + pass.end(); + }; + const render = (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => { + const pass = encoder.beginRenderPass({ + label: 'splat render', + colorAttachments: [ + { + view: texture_view, + loadOp: 'clear', + storeOp: 'store', + } + ], + }); + pass.setPipeline(render_pipeline); + // pass.setBindGroup(0, camera_bind_group); + pass.setBindGroup(0, splat_bind_group); + pass.drawIndirect(indirectBuffer, 0); + // pass.draw(pc.num_points); + + pass.end(); + // console.log(splatBuffer); + }; // =============================================== // Return Render Object // =============================================== + return { frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => { - sorter.sort(encoder); + // reset incrementing values + // apparently copying better than calling write from CPU side (https://webgpufundamentals.org/webgpu/lessons/webgpu-optimization.html) + encoder.copyBufferToBuffer(nullBuffer, 0, sorter.sort_info_buffer, 0, 4); + encoder.copyBufferToBuffer(nullBuffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4); + // encoder.copyBufferToBuffer(indirectBuffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4); + + // TODO anything else to reset + preprocess(encoder); // TODO is this the right order? + // encoder.copyBufferToBuffer(nullBuffer, 0, sorter.sort_dispatch_indirect_buffer, 0, 4); + // const arr = new Uint32Array([30]); + // device.queue.writeBuffer(sorter.sort_info_buffer, 0, arr, 0, arr.length); + // encoder.copyBufferToBuffer(nullBuffer2, 0, sorter.sort_info_buffer, 0, 4); + + // console.log(sorter.sort_dispatch_indirect_buffer); + + + + + + // OH Might just be this out of date computer/webgpu version has issues running it? + // but weird it just freezes/slows to nothing rather than giving an error + sorter.sort(encoder); // TODO reenable + // TODO sort causes whole thing to freeze atm dunno why + encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, indirectBuffer, 4, 4); + // device.queue. + + render(encoder, texture_view); }, camera_buffer, }; diff --git a/src/renderers/renderer.ts b/src/renderers/renderer.ts index ffdf9ba..8164de1 100644 --- a/src/renderers/renderer.ts +++ b/src/renderers/renderer.ts @@ -53,6 +53,17 @@ export default async function init( cam_file: '', }; + const renderSettingsData = new Float32Array(2); + renderSettingsData[0] = params.gaussian_multiplier; + renderSettingsData[1] = 0; // TODO what is sh_deg meant to be set to + const renderSettingsBuffer = device.createBuffer({ + label: "render settings buffer", + size: 8, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST, + }); + device.queue.writeBuffer(renderSettingsBuffer, 0, renderSettingsData, 0, renderSettingsData.length); + + const pane = new Pane({ title: 'Config', expanded: true, @@ -74,6 +85,7 @@ export default async function init( }); } { + pane.addInput(params, 'ply_file', { view: 'file-input', lineCount: 3, @@ -85,7 +97,7 @@ export default async function init( if (uploadedFile) { const pc = await load(uploadedFile, device); pointcloud_renderer = get_renderer_pointcloud(pc, device, presentation_format, camera.uniform_buffer); - gaussian_renderer = get_renderer_gaussian(pc, device, presentation_format, camera.uniform_buffer); + gaussian_renderer = get_renderer_gaussian(pc, device, presentation_format, camera.uniform_buffer, renderSettingsBuffer); renderers = { pointcloud: pointcloud_renderer, gaussian: gaussian_renderer, @@ -122,6 +134,8 @@ export default async function init( {min: 0, max: 1.5} ).on('change', (e) => { //TODO: Bind constants to the gaussian renderer. + renderSettingsData[0] = params.gaussian_multiplier; + device.queue.writeBuffer(renderSettingsBuffer, 0, renderSettingsData, 0, renderSettingsData.length); }); } diff --git a/src/shaders/gaussian.wgsl b/src/shaders/gaussian.wgsl index 759226d..93e0166 100644 --- a/src/shaders/gaussian.wgsl +++ b/src/shaders/gaussian.wgsl @@ -1,22 +1,121 @@ struct VertexOutput { @builtin(position) position: vec4, + @location(0) color: vec4f, + // @location(1) center: vec2f, + @location(1) offset: vec2f, + @location(2) conic: vec3f, //TODO: information passed from vertex shader to fragment shader }; struct Splat { //TODO: information defined in preprocess compute shader + // screenPos: vec2f, // TODO 16bit instead? in NDC + // maxRadius: f32, + // // TODO what do I need here + // // ^ I think quad size comes from maxRadius (treat like a square in NDC) + // // also need conic and color info + // conic: vec3f, // might make sense to just do mat2x2 instead? or again should I do some 16bit version + // // TODO think will just use standard 32 bit for now then do the optimization later since extra credit? + // color: vec4f, + + // I am uncertain in regards to the XC if they want packed f16s just for the passing or if the calculation should all be in f16 (assume the latter), think I could do the latter w/o too much work but will not bother for now so can work on writeup + screenPos: u32, + maxRadiusConic: array, + colorOpacity: array +}; + +struct CameraUniforms { + view: mat4x4, + view_inv: mat4x4, + proj: mat4x4, + proj_inv: mat4x4, + viewport: vec2, + focal: vec2 }; +@group(0) @binding(0) +var splats : array; +@group(0) @binding(1) +var sort_indices : array; +@group(0) @binding(2) +var camera: CameraUniforms; + @vertex fn vs_main( + @builtin(vertex_index) vIdx : u32, + @builtin(instance_index) iIdx : u32 ) -> VertexOutput { //TODO: reconstruct 2D quad based on information from splat, pass var out: VertexOutput; - out.position = vec4(1. ,1. , 0., 1.); + let sIdx = sort_indices[iIdx]; + // out.position = vec4(1. ,1. , 0., 1.); + const quadVerts = array( + vec2f(-1.f,-1.f), + vec2f(1.f,-1.f), + vec2f(-1.f,1.f), + vec2f(-1.f,1.f), + vec2f(1.f,-1.f), + vec2f(1.f,1.f) + ); + // out.position = vec4f(splats[sIdx].screenPos + splats[sIdx].maxRadius * quadVerts[vIdx], 0.f, 1.f); + + let screenPos = vec2(unpack2x16float(splats[sIdx].screenPos)); + + let radiusConicX = vec2(unpack2x16float(splats[sIdx].maxRadiusConic[0])); + let conicYZ = vec2(unpack2x16float(splats[sIdx].maxRadiusConic[1])); + let cXY = vec2(unpack2x16float(splats[sIdx].colorOpacity[0])); + let cZW = vec2(unpack2x16float(splats[sIdx].colorOpacity[1])); + + out.position = vec4f(screenPos + radiusConicX.x * quadVerts[vIdx], 0.f, 1.f); + + + // out.color = vec4f( + // select(0.f,1.f,iIdx < 272950), + // select(0.f,1.f,iIdx == 272955), + // select(0.f,1.f,iIdx >= 272956), 1.f); + // out.color = vec4f(1.f,1.f,0.f, 1.f); + // if (all(splats[sIdx].color == vec3f(0.f,0.f,0.f))) { + + // out.color = vec4f(vec3f(1.f,0.f,0.f), 1.f); + // } else { + + // let d = (out.position.xy - splats[sIdx].screenPos) * vec2f(-camera.viewport.x, camera.viewport.y); + + // out.color = vec4f(d.x, d.y, 0.f, 1.f); + // out.color = vec4f(splats[sIdx].color.xyz, 1.f / (1.f + exp(-splats[sIdx].color.w))); + // out.center = splats[sIdx].screenPos; + // out.offset = d; + // out.conic = splats[sIdx].conic; + + + let d = (out.position.xy - screenPos) * vec2f(-camera.viewport.x, camera.viewport.y); + + out.color = vec4f(cXY.x, cXY.y, cZW.x, 1.f / (1.f + exp(-cZW.y))); + out.offset = d; + out.conic = vec3f(radiusConicX.y, conicYZ.x, conicYZ.y); + // } return out; } @fragment fn fs_main(in: VertexOutput) -> @location(0) vec4 { - return vec4(1.); + + // let sPos = (in.position.xy / camera.viewport * 2.f - 1.f) * vec2f(1.f, -1.f); + // let center = (in.center.xy / camera.viewport * 2.f - 1.f) * vec2f(1.f, -1.f); + // // let center = + // let d = (sPos - center) * vec2f(-camera.viewport.x, camera.viewport.y); + + let d = in.offset; + + let power = -0.5f * (in.conic.x * d.x * d.x + + in.conic.z * d.y * d.y) - in.conic.y * d.x * d.y; + if (power > 0.f) { + discard; + } + + let alpha = clamp(0.f, 1.f, in.color.w * exp(power)); + // return vec4f(d.x, d.y, 0.f, 1.f); + return vec4f(in.color.xyz * alpha, alpha); + // return in.color; + // return vec4(1.); } \ No newline at end of file diff --git a/src/shaders/point_cloud.wgsl b/src/shaders/point_cloud.wgsl index 01dded1..bc4a5b0 100644 --- a/src/shaders/point_cloud.wgsl +++ b/src/shaders/point_cloud.wgsl @@ -35,8 +35,9 @@ fn vs_main( let pos = vec4(a.x, a.y, b.x, 1.); // TODO: MVP calculations - out.position = pos; - + // TODO model matrix? doesn't seem to be one used here; wouldn't expect one necessary but they do call it "MVP" + out.position = camera.proj * camera.view * pos; + return out; } diff --git a/src/shaders/preprocess.wgsl b/src/shaders/preprocess.wgsl index bbc63f5..074e37a 100644 --- a/src/shaders/preprocess.wgsl +++ b/src/shaders/preprocess.wgsl @@ -57,9 +57,35 @@ struct Gaussian { struct Splat { //TODO: store information for 2D splat rendering + + // TODO optimize to 16 bit XC + // screenPos: vec2f, // in NDC + // maxRadius: f32, + // conic: vec3f, + // color: vec4f, + + screenPos: u32, + maxRadiusConic: array, + colorOpacity: array }; +// struct SHBufferData { +// data: array +// } + //TODO: bind your data here +@group(0) @binding(0) +var camera: CameraUniforms; + +@group(1) @binding(0) +var gaussians : array; +@group(1) @binding(1) +var splats : array; + +@group(1) @binding(2) +var sh_buffer: array; + + @group(2) @binding(0) var sort_infos: SortInfos; @group(2) @binding(1) @@ -69,10 +95,28 @@ var sort_indices : array; @group(2) @binding(3) var sort_dispatch: DispatchIndirect; +@group(3) @binding(0) +var renderSettings: RenderSettings; + /// reads the ith sh coef from the storage buffer fn sh_coef(splat_idx: u32, c_idx: u32) -> vec3 { //TODO: access your binded sh_coeff, see load.ts for how it is stored - return vec3(0.0); + // 3 * 16 * (2byte/4byte) = 24 indices per splat? + // want to get set of 3 f16s from that + // c_idx : [0,16] -> [0,23] + // c_idx * 3 / 2 + // offset by 1/2 when c_idx odd + let arrIdx = splat_idx * 24 + c_idx * 3 / 2; + let c1 = unpack2x16float(sh_buffer[arrIdx]); + let c2 = unpack2x16float(sh_buffer[arrIdx + 1]); + if (c_idx % 2 == 1) { + // odd + return vec3(c1.y, c2.x, c2.y); + } else { + // even + return vec3(c1.x, c1.y, c2.x); + } + // return vec3(0.0); } // spherical harmonics evaluation with Condon–Shortley phase @@ -112,7 +156,191 @@ fn computeColorFromSH(dir: vec3, v_idx: u32, sh_deg: u32) -> vec3 { fn preprocess(@builtin(global_invocation_id) gid: vec3, @builtin(num_workgroups) wgs: vec3) { let idx = gid.x; //TODO: set up pipeline as described in instruction + if (idx >= arrayLength(&gaussians)) { + return; + } + // TODO culling, conic etc. + let vertex = gaussians[idx]; + let a = unpack2x16float(vertex.pos_opacity[0]); + let b = unpack2x16float(vertex.pos_opacity[1]); + // let viewPos = camera.view * vec4(a.x, a.y, b.x, 1.); + // if (viewPos.z < ) + let worldPos = vec4(a.x, a.y, b.x, 1.); + let viewPos = camera.view * worldPos; + + var pos = camera.proj * viewPos; + pos /= pos.w; + // NDC -> 1.2x screen size = +/- 1.2 + // want [-1.2,1.2] unculled; and in front of the camera + // TODO is that last one needed (far clip)? probably not relevant but not wrong? + if (abs(pos.x) > 1.2 || abs(pos.y) > 1.2 || pos.z < 0.f || pos.z > 1.f) { + return; + } + + let rXY = unpack2x16float(vertex.rot[0]); + let rZW = unpack2x16float(vertex.rot[1]); + let sXY = unpack2x16float(vertex.scale[0]); + let sZW = unpack2x16float(vertex.scale[1]); + + let r = rXY.x; + let x = rXY.y; + let y = rZW.x; + let z = rZW.y; + let R = mat3x3f( + 1.f - 2.f * (y * y + z * z), 2.f * (x * y - r * z), 2.f * (x * z + r * y), + 2.f * (x * y + r * z), 1.f - 2.f * (x * x + z * z), 2.f * (y * z - r * x), + 2.f * (x * z - r * y), 2.f * (y * z + r * x), 1.f - 2.f * (x * x + y * y) + // 1.f - 2.f * (rXY.y * rXY.y + rZW.x * rZW.x), 2.f * (rXY.x *rXY.y - rZW.y *rZW.x), 2.f * (rXY.x *rZW.x + rZW.y *rXY.y), + // 2.f * (rXY.x *rXY.y + rZW.y *rZW.x), 1.f - 2.f * (rXY.x *rXY.x +rZW.x *rZW.x), 2.f * (rXY.y *rZW.x - rZW.y *rXY.x), + // 2.f * (rXY.x *rZW.x - rZW.y *rXY.y), 2.f * (rXY.y *rZW.x + rZW.y *rXY.x), 1.f - 2.f * (rXY.x *rXY.x +rXY.y *rXY.y) + ); + + // took way too long to find in the instructions that this is in log space gahhh + let S = mat3x3f( + exp(sXY.x) * renderSettings.gaussian_scaling, 0.f, 0.f, + 0.f, exp(sXY.y) * renderSettings.gaussian_scaling, 0.f, + 0.f, 0.f, exp(sZW.x) * renderSettings.gaussian_scaling, + ); + + // I'm not sure why all the matrices seem to be transpose of what math in paper does??? + let M = S * R; + // let cov3d = R * S * transpose(S) * transpose(R); + let cov3d = transpose(M) * M; + // let cov3d = R * S * transpose(S) * transpose(R); + // let cov3d = R * S * transpose(S) * transpose(R); + + // W = "viewing transformation" + // = 3x3 version of camera.view? + // J = "Jacobian of the affine approximation of the projective transformation" + // T = W * J + + // limx = 1.3 * tan_fovx + // focal2fov(focal: number, pixels: number): number { + // return 2 * Math.atan(pixels / (2 * focal)); + // tan_fovx = tan(1/2 [I think?] * 2 Math.atan(pixels / (2 * focal))) I think? + // = pixels / (2 * focal) + // Not sure if this is necessary? + let limx = 0.65f * camera.viewport.x / camera.focal.x; + let limy = 0.65f * camera.viewport.y / camera.focal.y; + let txtz = viewPos.x / viewPos.z; + let tytz = viewPos.y / viewPos.z; + let t = vec3f( + min(limx, max(-limx, txtz)) * viewPos.z, + min(limy, max(-limy, tytz)) * viewPos.z, + viewPos.z + ); + let zSquared = t.z * t.z; + let W = transpose(mat3x3f( + camera.view[0].xyz, + camera.view[1].xyz, + camera.view[2].xyz + )); + let J = mat3x3f( + camera.focal.x / t.z, 0.f, -(camera.focal.x * t.x) / zSquared, + 0.f, camera.focal.y / t.z, -(camera.focal.y * t.y) / zSquared, + 0.f, 0.f, 0.f + ); + // let J = mat3x3f( + // camera.focal.x / viewPos.z, 0.f, 0.f, + // 0.f, camera.focal.y / viewPos.z, 0.f, + // -(camera.focal.x * viewPos.x) / zSquared, -(camera.focal.y * viewPos.y) / zSquared, 0.f + // ); + + let Vrk = mat3x3f( + cov3d[0][0], cov3d[0][1], cov3d[0][2], + cov3d[0][1], cov3d[1][1], cov3d[1][2], + cov3d[0][2], cov3d[1][2], cov3d[2][2] + ); + let T = W * J; + // let cov2d = T * (Vrk) * transpose(T); + let cov2d = transpose(T) * transpose(Vrk) * T; + // let cov2d = transpose(T) * transpose(cov3d) * T; + // TODO just following formula but don't get why they transpose the whole thing + + // addition ensuring numerical stability of inverse: + // cov[0][0] += 0.3f; + // cov[1][1] += 0.3f; + let cov = vec3f( + cov2d[0][0] + 0.3f, + cov2d[0][1], + cov2d[1][1] + 0.3f + ); + + let det = cov.x * cov.z - cov.y * cov.y; + if (det == 0.f) { + // if (det < 0.00001f) { + return; + } + let detInv = 1.f / det; + let conic = vec3f( + cov.z * detInv, + -cov.y * detInv, + cov.x * detInv + ); + + let mid = 0.5f * (cov.x + cov.z); + let root = sqrt(max(0.01, mid * mid - det)); + let lambda1 = mid + root; + let lambda2 = mid - root; // isn't there no reason to do this? sqrt >= 0 in this right? unless the below is also supposed to be abs of these or something + + let radius = ceil(3.f * sqrt(max(lambda1, lambda2))); + + + // can I use sort_infos.keys_size in the above? I don't totally get how we want to set up the data in splats + // oh wait looking at the specification it returns the original value + let splatIdx = atomicAdd(&sort_infos.keys_size, 1u); + // splats[splatIdx] = Splat(pos.xy, 0.05, vec3f(0.f,0.f,0.f), vec3f(1.f,0.f,0.f)); + // let shorterDir : f32 = min(camera.viewport.x, camera.viewport.y); + let quadDims = vec2f(radius, radius) / camera.viewport; + // TODO pass in both dims? (or pass camera into vs and divide there). I think square pixel space might make more sense but I think might've been implied meant to be same NDC width height? + // ^For now keeping it square in NDC (= rectangle on screen), passing in both (or passing in pixel space then doing viewport division in vertex shader) might be ideal but I'm content to leave it as is + // splats[splatIdx] = Splat(pos.xy, 0.01, vec3f(0.f,0.f,0.f), vec3f((radius / shorterDir),1.f,0.f)); + // splats[splatIdx] = Splat(pos.xy, max(quadDims.x, quadDims.y), vec3f(0.f,0.f,0.f), vec3f(1.f,1.f,0.f)); + + let cameraPos = camera.view_inv[3].xyz; + // surely sh_deg should just be passed in from renderSettings as a u32 already but it's not set up that way so think can just cast + let color = computeColorFromSH(normalize(worldPos.xyz - cameraPos), idx, u32(renderSettings.sh_deg)); + // splats[splatIdx] = Splat(pos.xy, max(quadDims.x, quadDims.y), conic, vec4f(color, b.y)); + + + splats[splatIdx] = Splat( + pack2x16float(pos.xy), + array( + pack2x16float(vec2f(max(quadDims.x, quadDims.y), conic.x)), + pack2x16float(conic.yz) + ), + array( + pack2x16float(color.xy), + pack2x16float(vec2f(color.z, b.y)) + )); + // splats[splatIdx] = Splat(pos.xy, max(quadDims.x, quadDims.y), vec3f(0.f,0.f,0.f), vec3f(quadDims.x,quadDims.y,0.f)); + // splats[splatIdx] = Splat(pos.xy, max(quadDims.x, quadDims.y), vec3f(0.f,0.f,0.f), vec3f(1.f,1.f,1.f)); + // splats[splatIdx] = Splat(pos.xy, max(quadDims.x, quadDims.y), vec3f(0.f,0.f,0.f), vec3f(quadDims.x,quadDims.y,0.f)); + // splats[splatIdx] = Splat(pos.xy, 10.f / camera.viewport.x, vec3f(0.f,0.f,0.f), vec3f(quadDims.x,1.f,0.f)); + // splats[splatIdx] = Splat(pos.xy, (radius / shorterDir), vec3f(0.f,0.f,0.f), vec3f(radius / 10.f,1.f,0.f)); + // atomicAdd(&sort_dispatch.dispatch_x, 1u); + + + // depths in u32, I don't see a specified way we particularly need to do mapping + // further first so higher z + // I think I could do a different range but the one I'm using right now shows fine so not touching at the moment + // sort_depths[splatIdx] = bitcast(1.f - pos.z); + sort_depths[splatIdx] = bitcast(100.f - viewPos.z); + // sort_depths[splatIdx] = bitcast(100.f * (1.f - pos.z)); + // sort_depths[splatIdx] = u32(100.f * (1.f - pos.z)); // TODO what range? + // sort_depths[splatIdx] = 0; + sort_indices[splatIdx] = splatIdx; + // } let keys_per_dispatch = workgroupSize * sortKeyPerThread; // increment DispatchIndirect.dispatchx each time you reach limit for one dispatch of keys + if (splatIdx % keys_per_dispatch == 0) { + // ^TODO is that the right comparison? + // I guess it's splatIdx / keys_per_dispatch > dispatch_x? + // can't directly check dispatch_x for that comparison though w/ being atomic + // so modulo to check when at a multiple? + atomicAdd(&sort_dispatch.dispatch_x, 1u); + + + } } \ No newline at end of file diff --git a/src/utils/load.ts b/src/utils/load.ts index 030b857..b5fe516 100644 --- a/src/utils/load.ts +++ b/src/utils/load.ts @@ -71,7 +71,7 @@ export async function load(file: string, device: GPUDevice) { // Spherical harmonic function coeffs const sh_buffer = device.createBuffer({ - label: 'ply input 3d gaussians data buffer', + label: 'ply input spherical harmonics 3d gaussians data buffer', size: num_points * c_size_sh_coef, // buffer size multiple of 4? usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE, mappedAtCreation: true,