glTF I/O path produces incorrect opacity values

[warrenm]

The KHR_gaussian_splatting extension spec states that when an ellipse kernel shape (the default) is used:

The opacity of a Gaussian splat is defined by the KHR_gaussian_splatting:OPACITY attribute for this kernel. It stores a normalized value between 0.0 (transparent) and 1.0 (opaque). [...] Out-of-range values are invalid.

In the current implementation, opacity values stored in memory are treated as if they are linear and normalized when reading and writing glTF assets, while they are treated as in logit space when read from a PLY file, leading to a situation where transforming a PLY into a glTF results in invalid opacity values in the glTF.

This is readily verified by inspecting an asset produced by this pipeline: if an in-memory asset stores the value 20 for the opacity of a splat (signifying near-opacity), the glTF writer will output a splat with an opacity of 20, rather than the preferred $\frac{1}{1 + e^{-20}} \approx 1$.

[javagl]

Thanks. A PR that will fix that is in #9

There had been a bit of back and forth during the development of the extension, with some relevant comments around KhronosGroup/glTF#2490 (comment) or KhronosGroup/glTF#2490 (comment) . I think that at some point, I suggested to make it clear(er) with the proper name: The opacity of splats seems to be in [-Inf,+Inf]. An alpha value is usually in [0,1]. So one could make a strong case to call it KHR_gaussian_splatting:OPACITY only when it stores [-Inf,+Inf] values, and call it KHR_gaussian_splatting:ALPHA when it stores actual alpha values in [0,1] - but apparently, that wasn't picked up.

[javagl]

@warrenm I'm curious: Are you implementing a renderer for this extension?

To my knowledge, there currently is no renderer that can actually render the KHR_gaussian_splatting extension (except for JSplat). The fact that this error remained unnoticed in JSplat is because the error was cancelled out by the reader and the writer both making the same (wrong) assumption. (Yeah, some of that could be avoided with unit tests, but even these would hardly check actual accessor values, and might then still hide the error by making a wrong assumption...).

It would be nice to have a bit more options for verifying things here (also in view of KhronosGroup/glTF#2562 (comment) ...)

[warrenm]

I'm not working on a renderer myself; my main interest is in ensuring that cgltf and GLTFKit2 are able to faithfully read and write the GS extension, so I just adapted the open-source MetalSplatter renderer to develop and test my implementations. You're one of the few people producing test assets for this extension, so I came across your sample assets and projects in the course of my explorations. I've also succeeded in using Norbert Nopper's 3DGS ply2gltf project to convert larger GS scenes.

[javagl]

The ecosystem is just starting to build.

Quickly skimming over the cgltf PR and the changes, I don't see a point where the accessors are read/written? (But maybe that's already covered elsewhere in the code, implicitly).

I've been testing most of the other file formats using https://sandbox.babylonjs.com/ which already supports several different splat formats (PLY(binLE)/SPZ/SOG), but some aspects (like #6 ) make it hard to find the "ground truth" for some cases.

The test data sets in the linked issue are generated with the state from #7 , which is only a draft for now. The current master here only contains these "grid" and "roundtrip" examples (with the "roundtrip" always bearing the risk of having a wrong reader/writer cancelling out an error). I hope that it will be possible to consolidate all that quickly.

[warrenm]

cgltf is quite low-level, as glTF libraries go. It doesn't pay much attention to semantics, so it's up to the library user to ensure that vertex attributes are laid out correctly and that attribute names are appropriate to the content (when writing), or inspect the contents of primitives and accessors to adapt them to whatever renderer they might be using (when reading). Hence, the only GS-specific code in that PR pertains to the primitive extension object, rather than the splat attributes.