Implementation 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes (3DCS)

[AlexMorgand]

This is the reimplementation of 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes

Here are the main contributions:

• Adding convex points, delta, sigma
• Dedicated projection, rasterization and spherical harmonics computation
• Dedicated densification (only split and prune) strategy
• Saving as pt instead of ply. Output from official implementation can be loaded as well (WIP for the splat export of the new viewer)
• Adding light and outdoor mode and adapt the hyperparameters depending on the situation

How to test:

Here is an example for MipNerf360/garden

CUDA_VISIBLE_DEVICES=0   python examples/simple_trainer_3dcs.py 3dcs  --data-dir  path/to/MipNerf/360_v2/kitchen --result-dir /tmp/garden --max-steps 30000 --data_factor 4

[WIP] Metrics are quite similar to the paper but since I had to rebase on top of the new version, I need to recompute all of them.

Garden: PSNR: 26.611, SSIM: 0.8459, LPIPS: 0.097 Time: 0.063s/image Number of GS: 4473125

This work was approved and supervised by two authors of the paper @rvandeghen and @heldJan

[Copilot]

Pull Request Overview

This PR reimplements the 3D Convex Splatting radiance field rendering pipeline with a dedicated implementation for 3D smooth convexes (3DCS). Key changes include new CUDA kernels and helper functions for spherical harmonics, rasterization and projection in 3DCS, new splatting strategy functions in Python, and extended viewer backend support for the new "3dcs" mode.

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
gsplat/utils.py	Introduces new utilities; adds a PLY saving function for convex splats.
gsplat/strategy/*	Implements new splitting, removal and resetting functions for 3DCS.
gsplat/cuda/*	Adds/updates CUDA kernels for spherical harmonics, rasterization, and projection tailored to 3DCS.
gsplat/_helper.py	Adds a stub for load_ply_data.
examples/simple_viewer.py	Extends the viewer to support the new 3DCS backend.

Comments suppressed due to low confidence (2)

examples/simple_viewer.py:223

• It appears 'viewmat' is used here while the test data loader returns 'viewmats'. Please verify that the correct variable is used and update the naming accordingly.

viewmat[None],  # [1, 4, 4]

gsplat/utils.py:100

• The variable 'scales' is referenced here but is not defined in the function. Please ensure that either 'scales' is provided as part of the function's inputs or is correctly defined before use.

for i in range(scales.shape[1]):

[liruilong940607]

Hi @AlexMorgand thanks for the effort on implementing it in gsplat!

I'm not familiar with this paper -- by skimming through it I don't get what benefit we got from this convex formulation. It is certainly a more flexible representation than GS but it seems to train & render much slower with only marginal quality improvement.

Would you mind first elaborate the motivation of bringing it into gsplat?

[imlixinyang]

@liruilong940607 Hi Ruilong, from the perspective of my personal research, I would like to explain the necessity of this method which has more parameters for primitives, degraded rendering speed, but a smaller number of primitives. I use gsplat in my research for feed-forward formed 3DGS. Due to the large number of viewpoints and pixels, the number of gs is also large. However, if we can use a smaller number but slightly more complex forms of primitives, the method will be more efficient. Because the time consumption of rendering during training is only a small part compared to the time consumption of network feed-forwarding. I believe that the introduction of this type of method will significantly benefit gsplat in more related fields beyond 3D reconstruction based on optimization.

@AlexMorgand, thank you very much for your contribution. I'm really looking forward to the day when this pull request can be merged.

[AlexMorgand]

Hey @liruilong940607 !

thanks for the effort on implementing it in gsplat!

My pleasure, happy to give back to gsplat :)

I'm not familiar with this paper -- by skimming through it I don't get what benefit we got from this convex formulation. It is certainly a more flexible representation than GS but it seems to train & render much slower with only marginal quality improvement.
Would you mind first elaborate the motivation of bringing it into gsplat?

While it's true that the standard metrics show only marginal improvements, the visual quality is noticeably better in many cases, especially in fine details and background regions. Let me add some comparison videos in this post.

As noted by @imlixinyang, 3DCS is able to solve finer details while using much less primitives. The introduction of 3D convexes as the primtive for splatting is expanding exciting research possibilities for compression and meshing (some minor spoilers for a future work from the authors? :) ) Speed is currently a trade-off but from what I've tested, convergence is quite fast and has a lot of potential for great optimisation. Currently convex hull computation is a bit suboptimal and takes a lot of the time. The current paper focused mostly on introducing a new primitive and how it allowed for finer details.

Additionally, there's been strong interest from the community in a real-time viewer for 3D Convex Splatting. Integrating it into gsplat not only makes the method more accessible but also opens the door to further exploration and real-time applications.
The new viser updates allows us to really highlight the fine details of the method.

Also this method was selected as a highlight for CVPR 2025 which is always interesting to get extra eyes on the gsplat project!

Here are two good sequences highlighting the benefits of the method

drone_2.mp4

3DCS: PSNR: 33.672, SSIM: 0.9276, LPIPS: 0.122 Time: 0.455s/image Number of GS: 552333
3DGS: PSNR: 24.956, SSIM: 0.8922, LPIPS: 0.225 Time: 0.021s/image Number of GS: 394897

drone.mp4

3DCS: PSNR: 37.847, SSIM: 0.9716, LPIPS: 0.037 Time: 0.017s/image Number of GS: 447636
3DGS: PSNR: 30.804, SSIM: 0.9569, LPIPS: 0.062 Time: 0.003s/image Number of GS: 455068

The potential also for depth supervision is also quite interesting as you can see in the depth map from 3DCS.

[liruilong940607]

Hey @AlexMorgand Thanks for the explanation and the videos are super cool!

But honestly It doesn't feel like it's the convex formulation that leads to the huge quality difference in the two videos.

One of the reason is exactly as you explained, the benefit of this convex formulation is better modeling finer details. However the biggest difference in the provided videos are the close-to-white background, which is not related to finer details at all.

Another reason is that I also noticed the densification strategy is modified. I guess it makes sense to work with the convex formulation, but it seems like many hyper-parameters of the densification strategy are different from the original one. Additionally I realized an extra loss term is used for 3DCS. All of these could actually have a huge impact on the performance, especially case like this -- pushing background away. In fact if tweak the hyper parameter of the original 3DGS strategy, or even simply switch to MCMC strategy, most of the time you could also cleanup a lot of floaters.

Plus, considering 3DCS trains/render much slower, I think a fair comparison should be under same training time / same render fps, what would be the difference. Because we know 3DGS can always get better quality with longer training or introducing more GSs.

For the record I do like this formulation and I do see its potential on flexibility. But I guess at this point I'm not confident enough that this formula -- in its current form -- would bring enough benefits to gsplat which worth the maintainness efforts going forward.

Happy to be convinced other way around!

[AlexMorgand]

Hey @liruilong940607 thank you for your answer.

For the finer details, I would say the strength of 3DCS is to get finer details really quickly compare to 3DGS. Quickly here means more in a few iterations while each iterations takes longer. Getting the rasterization down to something similar to 3DGS would be tremendous!
Here is a quick comparison for 3k/2k/2k iterations for 3DGS (top) and 3DCS (bottom) on stump/bicycle/garden sequences

Discussing with the authors @rvandeghen and @heldJan, it feels like most of the latest iterations (above 9k) could be redundant and probably halted for similar results.

I can see that the major issue with the current implementation is the speed tradeoff compare to how fast 3DGS in gsplat is! (I'm still amazed at how fast you guys managed to do it).

I tried to make the method as isolated as possible while following most of the gsplat pipeline so it doesn't disturb the current 3DGS implementation with the extra options and hyperparameters but I can understand if it presents an issue and extra work for maintaining the current code base.