Model generates "thin shell" double parallel surfaces with small hole leaks

[samuele-bortolato]

First of all, thank you for the amazing work on this project!

I have been testing the model via the Hugging Face space and noticed that it frequently generates a thin-shell mesh composed of two parallel surfaces sandwiched tightly against one another (likely the inner and outer approximations of the surface boundary).

While this wouldn't normally be an issue for visualization, the mesh occasionally contains tiny holes that bridge the gap between the inner and outer space. This makes it practically impossible to separate the two sides of the mesh or modify it using standard sculpting tools. Furthermore, because the model behaves like an empty shell, it cannot easily be used even for 3D printing. These holes are usually microscopic compared to the overall mesh size and are likely caused by a single mispredicted voxel.

Here are a few screenshots of the mesh inspected in Blender with face orientation (normals) coloring enabled:

It looks like the surface extraction step wraps a polygon skin around both sides of a thin voxel boundary layer. When a voxel is missing or incorrectly predicted, it creates a bridge between the inner and outer walls, creating a single mesh that also wraps the inside.
This behavior might have been introduced during the architectural transition from the original Direct3D-S2 backbone (which used a Signed Distance Field format) to the TRELLIS-2 backend (which uses a different algorithm to reconstruct the mesh from the O-Voxels).

[antonzub99]

afaik both TRELLIS-2 and Pixal use o-voxels to get the final mesh; however Pixal by default doesn't enforce inner and outer shells to be close to each other (remesh_project=0 in https://github.com/TencentARC/Pixal3D/blob/master/inference.py#L268). Even if you change that value to 0.95 or 0.99, the gap will become tighter but the mesh itself won't become manifold = you won't be able to 3d print it. This is actually a feature of o-voxels, your best bet would be to find some algorithm that makes the mesh manifold, though I'm not sure if the visual surface quality will be the same.

I personally tried to render RGBD images of the Pixal outputs, fuse them into a TSDF and then extract the surface with marching cubes - this way only the outer visible surface remains, but it comes at a cost of the surface quality; highly detailed examples from the repo with a lot of tiny fine components will be definitely lost in the process.

[samuele-bortolato]

Not what I meant, I'll try to be clearer. I don't want the inner and outer shell to be closer, I want the inner shell to be gone.

TRELLIS-2 is trained not to output a manifold, on the other hand Pixal seems to be using something like marching cubes (or similar algorithm) to convert an occupancy volume to a mesh, which does output a manifold (once you apply some vertex and edge deduplication). If I wanted I could try to 3d print the output if Pixal on a 3d printer, the only problem is that the mesh is already an extremely thin shell and would need a lot of work to manually add internal support.

Quoting from the Trellis2 paper, from section 3.1.1

O-Voxel → Mesh.
From an O-Voxel, we recover mesh surfaces by connecting dual vertices across intersected edges, forming quadrilateral faces among neighboring active voxels. Each quadrilateral can be adaptively subdivided into two triangles guided by the splitting weights, allowing the surface to better conform to local geometric features.
The Flexible Dual Grid offers several key advantages:

1. Instant bidirectional conversion — it enables rapid mapping with meshes and supports high-resolution conversion with minimal computation overhead. The costly processes in prior works such as SDF evaluation, floodfill procedure, and iterative optimization are not needed.
2. Arability-topology modeling — it is free from the watertight and manifold constraints, enabling robust handling of arbitrary geometry including self-intersecting surfaces and fully-enclosed interior structures.
3. High precision and sharp feature preservation — the dual vertices are aligned with local geometric features by algorithm design [25], allowing for the preservation of sharp features. Additionally, the dual vertex positions and splitting weights can receive learnable adjustments by neural network using other supervisions such as rendering loss (see Section 3.2.2), further enhancing the flexibility and precision of the geometry

My point is, instead of using marching cubes, the output mesh would be much more usefull if it wasn't a manifold, and instead aligned with how Trellis 2 was trained, wich would ouptut a mesh (eventually with holes) that doesn't try to wrap the inside of the object. This because fixing the holes it's much simpler than trying to remove the inner shell (since the inner and outer shells are connected and there is no easy way to just select the outer part), and if we wanted to add thickness to the non manifold mesh it's quite easily done with most 3d softwares.

[Premik]

Doesn't Trellis actually suffer from the very same issue? microsoft/TRELLIS.2#140

[antonzub99]

@samuele-bortolato If you don't want the inner shell, then use the generated mesh directly (https://github.com/TencentARC/Pixal3D/blob/master/inference.py#L259) and do not use the postprocessing with o-voxels. o-voxels will always create multiple shells by design, this happens in both TRELLIS-2 and Pixal.

[samuele-bortolato]

@antonzub99 Thank you!
That's it, calling postprocess with remesh=True is triggering the dual conturing https://github.com/microsoft/TRELLIS.2/blob/main/o-voxel/o_voxel/postprocess.py#L170, which is the culprit of creating the two shells, passing it as False (as the default) seems to be the solution.

I had missed that part, since I have yet to manage to get the model to work on a low VRAM gpu, and I was just testing the huggingface space and inspecting the code.
Still, I think it should probably not use remesh by default.