diff --git a/README.md b/README.md index f99cdff..e0d5c49 100644 --- a/README.md +++ b/README.md @@ -2,30 +2,88 @@ **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 4** -* (TODO) YOUR NAME HERE -* Tested on: (TODO) **Google Chrome 222.2** on - Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab) +* ADITHYA RAJEEV + * [LinkedIn](https://www.linkedin.com/in/adithyar262/) +* Tested on: Windows 11, i7 13th Gen @ 2.40GHz 16GB, GeForce RTX 4050 8GB (Personal) + +# 3D Gaussian Splatting Renderer + +This project implements a 3D Gaussian Splatting renderer based on the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" [[1]](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). It utilizes WebGL and leverages GPU parallelism for efficient rendering of complex scenes represented by Gaussian splats. + +**Features:** + + - **Gaussian Splatting:** Implements the core algorithm for rendering 3D scenes using Gaussian splats. + - **View Frustum Culling:** Optimizes rendering performance by discarding Gaussians outside the view frustum. + - **Depth Sorting:** Sorts Gaussians by depth to ensure correct occlusion and transparency. + - **Spherical Harmonics Lighting:** Uses spherical harmonics coefficients to evaluate color based on view direction. + - **Interactive Controls:** Allows users to explore the scene with camera controls. ### Live Demo -[![](img/thumb.png)](http://TODO.github.io/Project4-WebGPU-Forward-Plus-and-Clustered-Deferred) +[Live Demo](https://adithyar262.github.io/Project5-WebGPU-Gaussian-Splat-Viewer) ### Demo Video/GIF -[![](img/video.mp4)](TODO) +![](images/bonzai.gif) +![](images/cycle.gif) + +## What is Gaussian Splatting? + +Gaussian splatting is a technique for rendering 3D scenes by representing objects as a collection of Gaussian functions. Each Gaussian function, or "splat," is defined by its position, orientation, scale, and color. By blending these splats together, complex shapes and appearances can be recreated. + +![](images/bonzai.png) +![](images/cycle.png) + + +**Advantages of Gaussian Splatting:** + + - **Efficient Representation:** Gaussian splats can compactly represent complex shapes and appearances. + - **Real-time Rendering:** The rendering process can be highly parallelized and optimized for real-time performance. + - **View-dependent Effects:** Gaussian splatting can accurately capture view-dependent effects like lighting and reflections. + +**How Gaussian Splatting Works:** + +1. **Scene Representation:** The 3D scene is represented as a set of Gaussian splats. +2. **Splatting:** Each splat is projected onto the screen as a 2D ellipse. +3. **Blending:** The splats are blended together based on their depth and opacity to create the final image. + +![Diagram of Gaussian splatting process](images/Workflow.png) + +**Performance Analysis:** + + - **Point Cloud vs. Gaussian Renderer:** + + * **Visual Quality:** The Gaussian renderer produces significantly smoother and more visually appealing results than the point cloud renderer. Point clouds simply display individual points, leading to a sparse and disconnected representation. Gaussian splats, on the other hand, effectively interpolate the surface between points, resulting in a continuous and more realistic appearance. They also inherently capture view-dependent effects like lighting and reflections, which are absent in a basic point cloud rendering. + + * **Data Representation:** Gaussian splats offer a more efficient representation of the scene. They can encode surface orientation and appearance information within each splat, allowing for a more compact representation compared to storing individual points and their attributes. + + - **Workgroup Size:** + + * **GPU Occupancy:** Increasing the workgroup size generally improves performance up to an optimal point. This is because larger workgroups allow for better utilization of GPU compute units, increasing occupancy and reducing idle time. Each workgroup is scheduled on a compute unit, and having more threads per workgroup maximizes the parallel processing capacity. + + * **Memory Contention:** However, excessively large workgroups can lead to decreased performance. This is due to increased memory access contention, where multiple threads within the same workgroup try to access the same memory locations simultaneously. This contention can cause delays and stall the execution, negating the benefits of increased parallelism. The optimal workgroup size depends on the specific GPU architecture and the complexity of the rendering task. + + - **View Frustum Culling:** + + * **Reduced Computation:** View frustum culling provides a significant performance improvement. By discarding Gaussians that lie outside the camera's view frustum, we drastically reduce the number of splats that need to be processed in subsequent stages of the rendering pipeline. This leads to fewer computations in transformations, depth sorting, and rasterization, resulting in a noticeable speedup. + + * **Early Rejection:** Culling is an early rejection step that prevents unnecessary work. Without it, all Gaussians would go through the entire pipeline even if they don't contribute to the final image. This optimization is particularly crucial in scenes with a large number of splats, where the computational cost savings can be substantial. + + - **Number of Gaussians:** + + * **Increased Workload:** As the number of Gaussians increases, the rendering performance decreases. This is simply due to the increased workload on the GPU. More splats require more computations for transformation, culling, sorting, and rasterization. The memory requirements also increase, potentially leading to more data transfers and cache misses. -### (TODO: Your README) + * **Bottlenecks:** A higher number of Gaussians can exacerbate bottlenecks in the rendering pipeline. For example, depth sorting becomes more computationally expensive with more splats, potentially limiting the overall frame rate. The specific performance impact depends on the scene complexity, GPU capabilities, and the efficiency of the rendering algorithm. -*DO NOT* leave the README to the last minute! It is a crucial part of the -project, and we will not be able to grade you without a good README. +## Bloopers! -This assignment has a considerable amount of performance analysis compared -to implementation work. Complete the implementation early to leave time! +Everyone loves a good blooper reel! Here's a glimpse of some of the hilarious (and sometimes frustrating) glitches encountered during the development of this project. From Gaussians going rogue to unexpected visual effects, these bloopers showcase the challenges and triumphs of bringing this renderer to life. -### Credits +![](images/blooper1.gif) +![](images/blooper3.gif) +![](images/blooper2.gif) -- [Vite](https://vitejs.dev/) -- [tweakpane](https://tweakpane.github.io/docs//v3/monitor-bindings/) -- [stats.js](https://github.com/mrdoob/stats.js) -- [wgpu-matrix](https://github.com/greggman/wgpu-matrix) -- Special Thanks to: Shrek Shao (Google WebGPU team) & [Differential Guassian Renderer](https://github.com/graphdeco-inria/diff-gaussian-rasterization) +**Resources:** + - **3D Gaussian Splatting Paper:** [https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) + - **Gaussian Splatting Notes:** [https://github.com/kwea123/gaussian\_splatting\_notes](https://github.com/kwea123/gaussian_splatting_notes) + - **Spherical Harmonics Post:** [https://beatthezombie.github.io/sh\_post\_1/](https://beatthezombie.github.io/sh_post_1/) diff --git a/images/Workflow.png b/images/Workflow.png new file mode 100644 index 0000000..4ff0e52 Binary files /dev/null and b/images/Workflow.png differ diff --git a/images/blooper1.gif b/images/blooper1.gif new file mode 100644 index 0000000..266882f Binary files /dev/null and b/images/blooper1.gif differ diff --git a/images/blooper2.gif b/images/blooper2.gif new file mode 100644 index 0000000..04c084d Binary files /dev/null and b/images/blooper2.gif differ diff --git a/images/blooper3.gif b/images/blooper3.gif new file mode 100644 index 0000000..d7a778b Binary files /dev/null and b/images/blooper3.gif differ diff --git a/images/bonzai.gif b/images/bonzai.gif new file mode 100644 index 0000000..4b2da18 Binary files /dev/null and b/images/bonzai.gif differ diff --git a/images/bonzai.mp4 b/images/bonzai.mp4 new file mode 100644 index 0000000..ab8ea34 Binary files /dev/null and b/images/bonzai.mp4 differ diff --git a/images/bonzai.png b/images/bonzai.png new file mode 100644 index 0000000..6bf625e Binary files /dev/null and b/images/bonzai.png differ diff --git a/images/bonzai_dark.png b/images/bonzai_dark.png new file mode 100644 index 0000000..faefd74 Binary files /dev/null and b/images/bonzai_dark.png differ diff --git a/images/bonzai_green.png b/images/bonzai_green.png new file mode 100644 index 0000000..02a060c Binary files /dev/null and b/images/bonzai_green.png differ diff --git a/images/bonzai_point_cloud.png b/images/bonzai_point_cloud.png new file mode 100644 index 0000000..b84c0f4 Binary files /dev/null and b/images/bonzai_point_cloud.png differ diff --git a/images/bonzai_weird.mp4 b/images/bonzai_weird.mp4 new file mode 100644 index 0000000..e00566e Binary files /dev/null and b/images/bonzai_weird.mp4 differ diff --git a/images/bonzai_weird2.mp4 b/images/bonzai_weird2.mp4 new file mode 100644 index 0000000..0ba4ea3 Binary files /dev/null and b/images/bonzai_weird2.mp4 differ diff --git a/images/bonzai_weird3.mp4 b/images/bonzai_weird3.mp4 new file mode 100644 index 0000000..733f2bc Binary files /dev/null and b/images/bonzai_weird3.mp4 differ diff --git a/images/cycle.gif b/images/cycle.gif new file mode 100644 index 0000000..d78e8a6 Binary files /dev/null and b/images/cycle.gif differ diff --git a/images/cycle.mp4 b/images/cycle.mp4 new file mode 100644 index 0000000..26ca27d Binary files /dev/null and b/images/cycle.mp4 differ diff --git a/images/cycle.png b/images/cycle.png new file mode 100644 index 0000000..80fa30c Binary files /dev/null and b/images/cycle.png differ diff --git a/images/cycle_green.png b/images/cycle_green.png new file mode 100644 index 0000000..adf653c Binary files /dev/null and b/images/cycle_green.png differ diff --git a/src/camera/camera.ts b/src/camera/camera.ts index 47ea1dc..884d0f6 100644 --- a/src/camera/camera.ts +++ b/src/camera/camera.ts @@ -187,4 +187,4 @@ export class Camera { this.update_buffer(); } -}; +}; \ No newline at end of file diff --git a/src/renderers/gaussian-renderer.ts b/src/renderers/gaussian-renderer.ts index 1684523..96c7ff0 100644 --- a/src/renderers/gaussian-renderer.ts +++ b/src/renderers/gaussian-renderer.ts @@ -5,7 +5,7 @@ import { get_sorter,c_histogram_block_rows,C } from '../sort/sort'; import { Renderer } from './renderer'; export interface GaussianRenderer extends Renderer { - + settingsBuffer : GPUBuffer; } // Utility to create GPU buffers @@ -35,6 +35,28 @@ export default function get_renderer( // =============================================== const nulling_data = new Uint32Array([0]); + const nullBuffer = createBuffer( + device, + 'Clear Buffer ', + 4, + GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST, + nulling_data); + + const settingsBuffer = createBuffer( + device, + 'Render Settings Buffer ', + 8, + GPUBufferUsage.COPY_DST | GPUBufferUsage.UNIFORM, + new Float32Array([1.0, pc.sh_deg]) + ); + + const splatBuffer = createBuffer( + device, + 'splatBuffer', + (8 + 8 + 8) * pc.num_points, // Position and size, color, conic and opacity + GPUBufferUsage.STORAGE, + null + ); // =============================================== // Create Compute Pipeline and Bind Groups @@ -54,7 +76,7 @@ export default function get_renderer( const sort_bind_group = device.createBindGroup({ label: 'sort', - layout: preprocess_pipeline.getBindGroupLayout(2), + layout: preprocess_pipeline.getBindGroupLayout(1), entries: [ { binding: 0, resource: { buffer: sorter.sort_info_buffer } }, { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_depths_buffer } }, @@ -63,24 +85,105 @@ export default function get_renderer( ], }); - // =============================================== // Create Render Pipeline and Bind Groups // =============================================== - + const buffer = createBuffer( + device, //device + "Buffer", //label + 16, //size + GPUBufferUsage.COPY_DST | GPUBufferUsage.INDIRECT, // Usage + new Uint32Array([6, pc.num_points, 0, 0, ]) // Data + ) + const render_shader = device.createShaderModule({code: renderWGSL}); + const render_pipeline = device.createRenderPipeline({ + label: 'render', + layout: 'auto', + vertex: { + module: render_shader, + entryPoint: 'vs_main', + }, + fragment: { + module: render_shader, + entryPoint: 'fs_main', + targets: [{ format: presentation_format, + blend: { + color: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add', + }, + alpha: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add', + }, + }, + }], + }, + }); + + const preprocessor_bind_group = device.createBindGroup({ + label: 'preprocessor_bind_group', + layout: preprocess_pipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: { buffer: camera_buffer }}, + { binding: 1, resource: { buffer: pc.gaussian_3d_buffer }}, + { binding: 2, resource: { buffer: splatBuffer } }, + { binding: 3, resource: { buffer: settingsBuffer } }, + { binding: 4, resource: { buffer: pc.sh_buffer } }, + ], + }); + + const gaussian_render_bind_group = device.createBindGroup({ + label: 'gaussian_render_bind_group', + layout: render_pipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: { buffer: splatBuffer } }, + { binding: 1, resource: { buffer: sorter.ping_pong[0].sort_indices_buffer } }, + { binding: 2, resource: { buffer: camera_buffer } }, + ], + }); // =============================================== // Command Encoder Functions // =============================================== - + const compute = (encoder: GPUCommandEncoder) => { + const compute_pass = encoder.beginComputePass(); + compute_pass.setPipeline(preprocess_pipeline); + compute_pass.setBindGroup(0, preprocessor_bind_group); + compute_pass.setBindGroup(1, sort_bind_group); + compute_pass.dispatchWorkgroups(Math.ceil(pc.num_points / C.histogram_wg_size)); + compute_pass.end(); + }; // =============================================== // Return Render Object // =============================================== return { frame: (encoder: GPUCommandEncoder, texture_view: GPUTextureView) => { + encoder.copyBufferToBuffer(nullBuffer , 0, sorter.sort_info_buffer, 0, 4); + encoder.copyBufferToBuffer(nullBuffer , 0, sorter.sort_dispatch_indirect_buffer, 0, 4); + compute(encoder); sorter.sort(encoder); + encoder.copyBufferToBuffer(sorter.sort_info_buffer, 0, buffer, 4, 4); + const pass = encoder.beginRenderPass({ + label: 'Render pass', + colorAttachments: [ + { + view: texture_view, + loadOp: 'clear', + storeOp: 'store', + clearValue: [0.0, 0.0, 0.0, 1.0], + } + ], + }); + pass.setPipeline(render_pipeline); + pass.setBindGroup(0, gaussian_render_bind_group); + pass.drawIndirect(buffer, 0); + pass.end(); }, camera_buffer, + settingsBuffer , }; } diff --git a/src/renderers/renderer.ts b/src/renderers/renderer.ts index ffdf9ba..25bf94c 100644 --- a/src/renderers/renderer.ts +++ b/src/renderers/renderer.ts @@ -157,4 +157,4 @@ export default async function init( } requestAnimationFrame(frame); -} +} \ No newline at end of file diff --git a/src/shaders/gaussian.wgsl b/src/shaders/gaussian.wgsl index 759226d..44e37f9 100644 --- a/src/shaders/gaussian.wgsl +++ b/src/shaders/gaussian.wgsl @@ -1,22 +1,85 @@ +@group(0) @binding(0) +var splats: array; +@group(0) @binding(1) +var sort_indices : array; +@group(0) @binding(2) +var camera: CameraUniforms; + struct VertexOutput { @builtin(position) position: vec4, //TODO: information passed from vertex shader to fragment shader + + @location(0) color: vec4f, + @location(1) conic_opacity: vec4f, + @location(2) center: vec2f, }; struct Splat { //TODO: information defined in preprocess compute shader + // Data is packed, need to unpack + position_and_size: array, + color_data: array, + conic_and_opacity: array, +}; + +struct CameraUniforms { + view: mat4x4f, + view_inv: mat4x4f, + proj: mat4x4f, + proj_inv: mat4x4f, + viewport: vec2f, + focal: vec2f, }; @vertex fn vs_main( + @builtin(vertex_index) in_vertex_index: u32, @builtin(instance_index) in_instance_index: u32 ) -> VertexOutput { //TODO: reconstruct 2D quad based on information from splat, pass var out: VertexOutput; - out.position = vec4(1. ,1. , 0., 1.); + + let splat = splats[sort_indices[in_instance_index]]; + let xy = unpack2x16float(splat.position_and_size[0]); + let wh = unpack2x16float(splat.position_and_size[1]); + let rg = unpack2x16float(splat.color_data[0]); + let ba = unpack2x16float(splat.color_data[1]); + let conic = unpack2x16float(splat.conic_and_opacity[0]); + let opacity = unpack2x16float(splat.conic_and_opacity[1]); + let pos = array( + vec2f(xy.x - wh.x * 2.0f, xy.y + wh.y * 2.0f), + vec2f(xy.x - wh.x * 2.0f, xy.y - wh.y * 2.0f), + vec2f(xy.x + wh.x * 2.0f, xy.y - wh.y * 2.0f), + vec2f(xy.x + wh.x * 2.0f, xy.y - wh.y * 2.0f), + vec2f(xy.x + wh.x * 2.0f, xy.y + wh.y * 2.0f), + vec2f(xy.x - wh.x * 2.0f, xy.y + wh.y * 2.0f), + ); + + out.position = vec4(pos[in_vertex_index].x, pos[in_vertex_index].y, 0.0f, 1.0f); + out.color = vec4(rg.x, rg.y, ba.x, ba.y);; + out.conic_opacity = vec4f(conic.x, conic.y, opacity.x, opacity.y); + out.center = vec2f(xy.x, xy.y); return out; } @fragment fn fs_main(in: VertexOutput) -> @location(0) vec4 { - return vec4(1.); + + var position = (in.position.xy / camera.viewport) * 2.0f - 1.0f; + position.y *= -1.0f; + + var offset = (position.xy - in.center.xy) * camera.viewport * 0.5f; + offset.x *= -1.0f; + + var power = -0.5f * ( + in.conic_opacity.x * pow(offset.x, 2.0f) + + in.conic_opacity.z * pow(offset.y, 2.0f) + ); + power -= in.conic_opacity.y * offset.x * offset.y; + + if (power > 0.0f) { + return vec4f(0.0f, 0.0f, 0.0f, 0.0f); + } + + let alpha = min(0.99f, in.conic_opacity.w * exp(power)); + return in.color * alpha; } \ No newline at end of file diff --git a/src/shaders/point_cloud.wgsl b/src/shaders/point_cloud.wgsl index 01dded1..617171e 100644 --- a/src/shaders/point_cloud.wgsl +++ b/src/shaders/point_cloud.wgsl @@ -35,7 +35,7 @@ fn vs_main( let pos = vec4(a.x, a.y, b.x, 1.); // TODO: MVP calculations - out.position = pos; + out.position = camera.proj * camera.view * pos; return out; } diff --git a/src/shaders/preprocess.wgsl b/src/shaders/preprocess.wgsl index bbc63f5..c6fc0e0 100644 --- a/src/shaders/preprocess.wgsl +++ b/src/shaders/preprocess.wgsl @@ -57,22 +57,44 @@ struct Gaussian { struct Splat { //TODO: store information for 2D splat rendering + // Data is packed, need to unpack + position_and_size: array, + color_data: array, + conic_and_opacity: array, }; //TODO: bind your data here -@group(2) @binding(0) +@group(0) @binding(0) +var camera: CameraUniforms; +@group(0) @binding(1) +var gaussians: array; +@group(0) @binding(2) +var splat_elements: array; +@group(0) @binding(3) +var settings: RenderSettings; +@group(0) @binding(4) +var color_data: array; + +@group(1) @binding(0) var sort_infos: SortInfos; -@group(2) @binding(1) +@group(1) @binding(1) var sort_depths : array; -@group(2) @binding(2) +@group(1) @binding(2) var sort_indices : array; -@group(2) @binding(3) +@group(1) @binding(3) var sort_dispatch: DispatchIndirect; /// reads the ith sh coef from the storage buffer fn sh_coef(splat_idx: u32, c_idx: u32) -> vec3 { //TODO: access your binded sh_coeff, see load.ts for how it is stored - return vec3(0.0); + let base = splat_idx * 24 + (c_idx / 2) * 3 + c_idx % 2; + let rg = unpack2x16float(color_data[base]); + let ba = unpack2x16float(color_data[base + 1]); + if (c_idx % 2 == 0) { + return vec3f(rg.x, rg.y, ba.x); + } else { + return vec3f(rg.y, ba.x, ba.y); + } } // spherical harmonics evaluation with Condon–Shortley phase @@ -112,7 +134,135 @@ fn computeColorFromSH(dir: vec3, v_idx: u32, sh_deg: u32) -> vec3 { fn preprocess(@builtin(global_invocation_id) gid: vec3, @builtin(num_workgroups) wgs: vec3) { let idx = gid.x; //TODO: set up pipeline as described in instruction - + // Reference - https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/forward.cu + if (idx >= arrayLength(&gaussians)) { + return; + } + + let current_element = gaussians[idx]; + + let xy = unpack2x16float(current_element.pos_opacity[0]); + let opacity = unpack2x16float(current_element.pos_opacity[1]); + let position = vec4f(xy.x, xy.y, opacity.x, 1.0f); + + let clip_pos = camera.proj * camera.view * position; + let ndc = clip_pos.xy / clip_pos.w; + let view_depth = (camera.view * position).z; + + if (abs(ndc.x) > 1.0f || + abs(ndc.y) > 1.0f || + view_depth < 0.0f) { + return; + } + // 3D Covariance + let S1 = unpack2x16float(current_element.scale[0]); + let S2 = unpack2x16float(current_element.scale[1]); + let R1 = unpack2x16float(current_element.rot[0]); + let R2 = unpack2x16float(current_element.rot[1]); + let S = mat3x3f( + // 1 + exp(S1.x) * settings.gaussian_scaling, 0.0f, 0.0f, + // 2 + 0.0f, exp(S1.y) * settings.gaussian_scaling, 0.0f, + // 3 + 0.0f, 0.0f, exp(S2.x) * settings.gaussian_scaling + ); + let R = mat3x3f( + // 1 + 1.0f - 2.0f * (R2.x * R2.x + R2.y * R2.y), + 2.0f * (R1.y * R2.x - R1.x * R2.y), + 2.0f * (R1.y * R2.y + R1.x * R2.x), + // 2 + 2.0f * (R1.y * R2.x + R1.x * R2.y), + 1.0f - 2.0f * (R1.y * R1.y + R2.y * R2.y), + 2.0f * (R2.x * R2.y - R1.x * R1.y), + // 3 + 2.0f * (R1.y * R2.y - R1.x * R2.x), + 2.0f * (R2.x * R2.y + R1.x * R1.y), + 1.0f - 2.0f * (R1.y * R1.y + R2.x * R2.x) + ); + + let M = S * R; + let sigma = transpose(M) * M; + let cov3D = array( + sigma [0][0], + sigma [0][1], + sigma [0][2], + sigma [1][1], + sigma [1][2], + sigma [2][2], + ); + + // 2D + var t = (camera.view * position).xyz; + let limx = 0.65f * camera.viewport.x / camera.focal.x; + let limy = 0.65f * camera.viewport.y / camera.focal.y; + let txtz = t.x / t.z; + let tytz = t.y / t.z; + t.x = min(limx, max(-limx, txtz)) * t.z; + t.y = min(limy, max(-limy, tytz)) * t.z; + + let J = mat3x3f( + camera.focal.x / t.z, 0.0f, -(camera.focal.x * t.x) / (t.z * t.z), + 0.0f, camera.focal.y / t.z, -(camera.focal.y * t.y) / (t.z * t.z), + 0.0f, 0.0f, 0.0f + ); + let W = transpose(mat3x3f( + camera.view[0].xyz, camera.view[1].xyz, camera.view[2].xyz + )); + + let T = W * J; + + let Vrk = mat3x3f( + // 1 + cov3D[0], cov3D[1], cov3D[2], + // 2 + cov3D[1], cov3D[3], cov3D[4], + // 3 + cov3D[2], cov3D[4], cov3D[5] + ); + + var cov = transpose(T) * transpose(Vrk) * T; + cov[0][0] += 0.3f; + cov[1][1] += 0.3f; + + let cov2d = vec3f( + cov[0][0], + cov[0][1], + cov[1][1] + ); + + var det = (cov2d.x * cov2d.z) - (cov2d.y * cov2d.y); + if (det == 0.0f) { + return; + } + var det_inv = 1.f / det; + let conic = vec3f( + cov2d.z * det_inv, -cov2d.y * det_inv, cov2d.x * det_inv, + ); + + let mid = 0.5f * (cov2d.x + cov2d.z); + let lambda1 = mid + sqrt(max(0.1f, mid * mid - det )); + let lambda2 = mid - sqrt(max(0.1f, mid * mid - det )); + let my_radius = ceil(3.0f * sqrt(max(lambda1, lambda2))); + let size = vec2f(my_radius , my_radius ) / camera.viewport; + let keys_per_dispatch = workgroupSize * sortKeyPerThread; // increment DispatchIndirect.dispatchx each time you reach limit for one dispatch of keys -} \ No newline at end of file + let splat_index = atomicAdd(&sort_infos.keys_size, 1u); + let direction = normalize(position.xyz - camera.view_inv[3].xyz); + let color = computeColorFromSH(direction, idx, u32(settings.sh_deg)); + + splat_elements[splat_index].position_and_size[0] = pack2x16float(ndc); + splat_elements[splat_index].position_and_size[1] = pack2x16float(size); + splat_elements[splat_index].color_data[0] = pack2x16float(color.rg); + splat_elements[splat_index].color_data[1] = pack2x16float(vec2f(color.b, 1.0f)); + splat_elements[splat_index].conic_and_opacity[0] = pack2x16float(conic.xy); + splat_elements[splat_index].conic_and_opacity[1] = pack2x16float(vec2f(conic.z, 1.0f / (1.0f + exp(-opacity.y)))); + + sort_depths[splat_index] = bitcast(100.0f - view_depth); + sort_indices[splat_index] = splat_index; + if (splat_index % keys_per_dispatch == 0) { + atomicAdd(&sort_dispatch.dispatch_x, 1u); + } +} diff --git a/src/utils/load.ts b/src/utils/load.ts index 030b857..5bd9f84 100644 --- a/src/utils/load.ts +++ b/src/utils/load.ts @@ -118,4 +118,4 @@ export async function load(file: string, device: GPUDevice) { gaussian_3d_buffer, sh_buffer, }; -} +} \ No newline at end of file