A continuously-updated, data-centric map of 3D vision — bridging representations, learning paradigms, and applications in one coherent framework.
Last updated: 2026-06-04
 RGB-D |
 Multi-view Images |
 Point Cloud |
 Voxels |
 Mesh |
 CAD |
 Implicit Field |
 3D Gaussians |
Eight 3D representations of the Stanford bunny — the structural spectrum that this cookbook unifies.
🎉 News (June 2026) — Our paper “A Cookbook of 3D Vision: Data, Learning Paradigms, and Application” is now available at the CVPR 2026 OpenSUN3D Workshop (CVF Open Access) and on arXiv:2606.04291. This repository is the living companion to the paper — pull requests, issues, and suggestions are very welcome.
- A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
3D vision has rapidly evolved, driven by increasingly diverse data representations, learning paradigms, and modeling strategies. Yet the field remains fragmented across representations and benchmarks, making it difficult to develop unified perspectives on efficiency, fidelity, and scalability. This work provides a data-centric taxonomy of 3D vision that connects geometric representations, datasets, learning frameworks, and applications within a single conceptual map.
We begin by analysing the principal structural representations of 3D data — point clouds, meshes, voxels, and 3D Gaussians — along with their acquisition pipelines. We then examine how dataset design, benchmark construction, and supervision regimes shape recent advances, spanning 2D-supervised 3D learning, implicit neural representations, and 4D world modeling. Through this integrative lens, we clarify the relationships among representations, learning paradigms, and downstream tasks in reconstruction, generation, and video modeling, offering a consolidated view of emerging trends toward balancing efficiency and fidelity, and toward multimodal geometric grounding.
Paper: Official (CVF Open Access) · arXiv:2606.04291
If you find this repository useful for your research, please consider citing:
@InProceedings{Du_2026_CVPR,
author = {Du, Hongyang and Li, Zongxia and Liu, Dawei and Li, Runhao and Song, Haoyuan and Zhang, Qingyu and Wang, Yubo and Ni, Jingcheng and Gui, Shihang and Dong, Congchao and Hu, Tao},
title = {A Cookbook of 3D Vision: Data, Learning Paradigms, and Application},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2026},
pages = {8376-8390}
}3D vision relies on diverse data representations, each with distinct trade-offs in structure, efficiency, and fidelity. The table below summarizes the major formats we cover in the cookbook.
| Representation | Structure | Efficiency | Fidelity | Typical Applications |
|---|---|---|---|---|
| RGB-D | 2.5D grid (RGB + depth) | High | Medium | SLAM, indoor mapping, pose estimation |
| Multi-view Images | 2D views + camera poses | High | High* | SfM, MVS, NeRF/3DGS input |
| Point Cloud | Unstructured 3D points | High | Low–Medium | Detection, mapping, robotics |
| Mesh | Vertex–edge–face graph | Medium | High | Modeling, animation, simulation |
| Voxel Grid | Dense 3D lattice | Low | Medium | Volumetric CNN, segmentation |
| Implicit Field | Neural function |
Low | Very High | View synthesis, scene modeling |
| 3D Gaussians | Sparse 3D Gaussian distributions | Very High | High | Real-time NeRF-style rendering |
| CAD Model | Parametric surfaces (NURBS / B-Rep) | Very High | Very High | CAD design, reverse engineering |
* Fidelity for multi-view images refers to visual fidelity (appearance); geometric structure must be inferred. See §3 (A Taxonomy of 3D Representations) of the paper for the formal definitions and acquisition pipelines of each format.
Modern 3D vision has shifted from explicit geometry pipelines toward learned systems that couple representation design, supervision, and practical utility. We highlight three pillars below.
By back-propagating through the image-formation process, differentiable renderers replace explicit 3D supervision with image-plane losses on color, depth, or silhouettes:
where
| Title | Year | Paper | Website | Code |
|---|---|---|---|---|
| NeRF: Representing Scenes as Neural Radiance Fields | 2020 | 📄 Paper | 🌍 Website | 💾 Code |
| 3D Gaussian Splatting for Real-Time Radiance Field Rendering | 2023 | 📄 Paper | 🌍 Website | 💾 Code |
| Neural 3D Mesh Renderer | 2018 | 📄 Paper | 🌍 Website | 💾 Code |
| Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning | 2019 | 📄 Paper | - | 💾 Code |
A wave of image-aligned geometric foundation models predicts dense per-pixel 3D structure directly from uncalibrated imagery, replacing the brittle SfM/MVS bottleneck. Representative formulations:
- DUSt3R — confidence-weighted regression on image-aligned point maps without explicit multi-view optimization at training time.
- VGGT — scales the image-aligned paradigm with a multi-task camera + depth + point-map + tracking objective for reusable backbones.
- RayZer — factorizes input into camera and scene representations, training entirely through 2D self-supervised reconstruction without explicit 3D geometry.
-
$\pi^3$ — enforces permutation-equivariant supervision over unordered image sets via local point maps and relative poses. -
Depth Anything 3 — collapses multiple geometric heads into a unified depth-plus-ray representation
$R \in \mathbb{R}^{H \times W \times 6}$ .
See the 3D Reconstruction table for direct links.
When explicit 3D data is scarce, learning paradigms shift toward distilling priors from large-scale 2D models or training native 3D foundation models that learn structured latent spaces:
- Score Distillation Sampling (SDS) — DreamFusion, Magic3D, ProlificDreamer optimize neural fields through 2D diffusion priors.
- Native 3D Latents — TRELLIS, Hunyuan3D 2.5, MeshFormer learn structured 3D latents decodable into radiance fields, Gaussians, or meshes.
- Rectified Conditional Flow Matching (RCFM) — SAM 3D introduces a Model-in-the-Loop (MITL) data engine where generative outputs are human-vetted to create recursive supervision, breaking the 3D-data barrier.
| Title | Year | Paper | Website | Code |
|---|---|---|---|---|
| Score Distillation Sampling (DreamFusion) | 2022 | 📄 Paper | 🌍 Website | - |
| ProlificDreamer: Variational Score Distillation | 2023 | 📄 Paper | 🌍 Website | 💾 Code |
| TRELLIS: Structured 3D Latents for Scalable and Versatile 3D Generation | 2025 | 📄 Paper | 🌍 Website | 💾 Code |
| Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation | 2025 | 📄 Paper | 🌍 Website | 💾 Code |
| SAM 3D: 3Dfy Anything in Images | 2025 | 📄 Paper | 🌍 Website | 💾 Code |
3D reconstruction recovers object or scene geometry from visual inputs. Modern feed-forward image-aligned backbones largely replace classical SfM/MVS bottlenecks.
| Title | Year | Paper | Website | Code | HuggingFace |
|---|---|---|---|---|---|
| DUSt3R: Geometric 3D Vision Made Easy | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| MASt3R: Grounding Image Matching in 3D | 2024 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views in 2 Seconds | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Mickey: Matching 2D Images in 3D — Metric Relative Pose from Metric Correspondences | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| VGGT: Visual Geometry Grounded Transformer | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
|
|
2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| StreamVGGT: Streaming 4D Visual Geometry Transformer | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second | 2025 | 📄 Paper | 🌍 Website | 💾 Code | - |
| RayZer: A Self-Supervised Large View Synthesis Model | 2025 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Depth Anything 3: Recovering the Visual Space from Any Views | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer | 2025 | 📄 Paper | 🌍 Website | 💾 Code | - |
| TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction | 2025 | 📄 Paper | - | 💾 Code | - |
| SAM 3D Body: Promptable Full-Body Human Mesh Recovery | 2025 | 📄 Paper | 🌍 Website | 💾 Code | - |
From per-prompt SDS optimization to feed-forward Large Reconstruction Models (LRMs) and native 3D latent diffusion, 3D asset and scene generation has matured into a fast, controllable pipeline.
| Title | Year | Paper | Website | Code | HuggingFace |
|---|---|---|---|---|---|
| DreamFusion: Text-to-3D using 2D Diffusion | 2022 | 📄 Paper | 🌍 Website | - | - |
| Magic3D: High-Resolution Text-to-3D Content Creation | 2023 | 📄 Paper | 🌍 Website | - | - |
| Zero-1-to-3: Zero-Shot One Image to 3D Object | 2023 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| LRM: Large Reconstruction Model for Single Image to 3D | 2023 | 📄 Paper | 🌍 Website | - | - |
| MVDream: Multi-View Diffusion for 3D Generation | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation | 2024 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| InstantMesh: Efficient 3D Mesh Generation from a Single Image | 2024 | 📄 Paper | - | 💾 Code | 😊 HuggingFace |
| AnyHome: Open-Vocabulary Generation of Structured and Embodied 3D Homes | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation | 2024 | 📄 Paper | - | - | - |
| TRELLIS: Structured 3D Latents for Scalable and Versatile 3D Generation | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Details | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation | 2025 | 📄 Paper | 🌍 Website | - | - |
| Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| AssetFormer: Modular 3D Assets Generation with Autoregressive Transformer | 2026 | 📄 Paper | - | 💾 Code | - |
Large Video Diffusion Models (VDMs) generate visually stunning content but struggle to preserve stable geometry across time. Recent work injects 3D paradigms — geometric preference alignment, feature-level forcing, and 3D-aware control — to regulate generation.
| Title | Year | Paper | Website | Code | HuggingFace |
|---|---|---|---|---|---|
| 3D-Aware Video Generation | 2023 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets | 2023 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| Lumiere: A Space-Time Diffusion Model for Video Generation | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning | 2024 | 📄 Paper | 🌍 Website | - | - |
| CogVideoX: Text-to-Video Diffusion Models with an Expert Transformer | 2024 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| World-Consistent Video Diffusion with Explicit 3D Modeling | 2024 | 📄 Paper | 🌍 Website | - | - |
| PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| CamI2V: Camera-Controlled Image-to-Video Diffusion Model | 2024 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| Diffusion as Shader: 3D-Aware Video Diffusion for Versatile Video Generation Control | 2025 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Tora: Trajectory-Oriented Diffusion Transformer for Video Generation | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| Force Prompting: Video Generation Models Can Learn and Generalize Physics-Based Control Signals | 2025 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Wan: Open and Advanced Large-Scale Video Generative Models | 2025 | 📄 Paper | 🌍 Website | 💾 Code | 😊 HuggingFace |
| IM-Portrait: Learning 3D-Aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos | 2025 | 📄 Paper | 🌍 Website | - | - |
| Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling | 2025 | 📄 Paper | 🌍 Website | - | - |
| Epipolar Geometry Improves Video Generation Models | 2025 | 📄 Paper | - | - | - |
| SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer | 2026 | 📄 Paper | 🌍 Website | 💾 Code | - |
| VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation | 2026 | 📄 Paper | - | - | - |
| daVinci-MagiHuman: A Single-Stream Architecture for Fast Audio-Video Foundation Model | 2026 | 📄 Paper | - | 💾 Code | 😊 HuggingFace |
Static 3D evolves into temporally persistent simulation: 4D Gaussian fields, dynamic SLAM, particle-based world models, and benchmarks that evaluate models as usable simulators under long-horizon, physically grounded tasks.
| Title | Year | Paper | Website | Code | HuggingFace |
|---|---|---|---|---|---|
| Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids | 2019 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Learning to Simulate Complex Physics with Graph Networks | 2020 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Learning Mesh-Based Simulation with Graph Networks | 2021 | 📄 Paper | 🌍 Website | 💾 Code | - |
| SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation | 2021 | 📄 Paper | 🌍 Website | 💾 Code | - |
| 3D Gaussian Splatting for Real-Time Radiance Field Rendering | 2023 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis | 2023 | 📄 Paper | 🌍 Website | 💾 Code | - |
| 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Gaussian Splatting SLAM | 2024 | 📄 Paper | 🌍 Website | 💾 Code | - |
| Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussians | 2024 | 📄 Paper | - | 💾 Code | - |
| CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models | 2024 | 📄 Paper | 🌍 Website | - | - |
| WorldSimBench: Towards Video Generation Models as World Simulators | 2024 | 📄 Paper | 🌍 Website | - | - |
| ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation | 2025 | 📄 Paper | 🌍 Website | - | - |
| RoboScape: Physics-Informed Embodied World Model | 2025 | 📄 Paper | - | 💾 Code | - |
| LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics | 2025 | 📄 Paper | - | 💾 Code | - |
| World-in-World: World Models in a Closed-Loop World | 2026 | 📄 Paper | - | 💾 Code | - |
| FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction | 2026 | 📄 Paper | - | 💾 Code | 😊 HuggingFace |
| Latent Particle World Models: Self-Supervised Object-Centric Stochastic Dynamics Modeling | 2026 | 📄 Paper | 🌍 Website | 💾 Code | - |
| PointWorld: Scaling 3D World Models for In-the-Wild Robotic Manipulation | 2026 | 📄 Paper | - | - | - |
Modern 3D-VLA systems ground perception, language, and robotic control in shared 3D representations, replacing direct 2D-token-to-motor mappings. Representing intent as 3D point flows or spatial trajectories improves viewpoint robustness, enables cross-embodiment generalization, and unlocks complex spatial reasoning.
| Title | Year | Paper | Website | Code |
|---|---|---|---|---|
| 3D-LLM: Injecting the 3D World into Large Language Models | 2023 | 📄 Paper | 🌍 Website | 💾 Code |
| PointLLM: Empowering Large Language Models to Understand Point Clouds | 2023 | 📄 Paper | 🌍 Website | 💾 Code |
| 3D-VLA: A 3D Vision-Language-Action Generative World Model | 2024 | 📄 Paper | 🌍 Website | 💾 Code |
| PointWorld: Scaling 3D World Models for In-the-Wild Robotic Manipulation | 2026 | 📄 Paper | - | - |
Representative 3D datasets across object, scene (indoor & outdoor), and human (face, hand, body) granularities — including Objaverse, Fusion 360 Gallery, ScanNet++, 3D-FRONT, RELLIS-3D, Virtual KITTI 2, FaceScape, GigaHands, and H3WB.
Drawing from the 50 representative datasets reviewed in the paper, we observe two clear trends.
Releases per year (selected datasets):
'15 ▏ 1
'16 █████ 5
'17 ██ 2
'18 ████ 4
'19 ████ 4
'20 ████████ 8
'21 ███████ 7
'22 ██ 2
'23 ████████ 8
'24 ████ 4
'25 █████ 5
Coverage by modality (multi-label, 50 datasets):
| Modality | Count | Coverage |
|---|---|---|
| Mesh | 28 | ██████████████████████████ 56% |
| Multi-view | 25 | ████████████████████████ 50% |
| RGB-D | 16 | ████████████████ 32% |
| Point Cloud | 13 | █████████████ 26% |
| Voxel | 3 | ███ 6% |
| Implicit | 1 | █ 2% |
Coverage by spatial granularity (multi-label, 50 datasets):
| Granularity | Count | Coverage |
|---|---|---|
| Object | 18 | ██████████████████ 36% |
| Indoor Scene | 13 | █████████████ 26% |
| Human-centric | 7 | ███████ 14% |
| Scene | 6 | ██████ 12% |
| Outdoor Scene | 5 | █████ 10% |
| Mixed | 1 | █ 2% |
Modality and granularity bars are not mutually exclusive — a single dataset can fall under multiple categories. Object-centric and indoor-scene benchmarks dominate the landscape, while voxel and implicit-field corpora remain rare.
| Dataset | Year | Granularity | Tasks | Size | Site | Description |
|---|---|---|---|---|---|---|
| SAM 3D Body | 2025 | Human Body | Full-Body Human Mesh Recovery | 5M | Github | A Meta-released dataset for single-image full-body reconstruction. Adapts the Segment Anything philosophy to 3D human recovery with the Momentum Human Rig (MHR) representation. |
| InterAct | 2025 | Human Body | 3D Motion Generation | 1.1K | Github | A comprehensive Human-Human and Human-Object interaction dataset focused on socially accurate 3D contacts (e.g., handshakes, sitting on chairs). |
| GigaHands | 2025 | Human Hand | 3D bimanual hand | 14K | Github | A large dataset of bimanual hand–object interactions with 183M frames, dense annotations (per hand & object), and 51 camera views for motion clips. |
| EgoHumans | 2023 | Human Body | 3D Multi-Human Tracking, 3D Pose Estimation | 125K | Github | A multi-human egocentric 3D benchmark addressing the limitations of single-subject indoor egocentric datasets. |
| EgoExo4D | 2023 | Human Body | Ego-Exo Relation, Ego-Exo Translation, 3D Pose Estimation | 1.4K | Github | Massive-scale, multimodal, multi-view dataset bridging egocentric and exocentric perspectives. |
| H3WB | 2022 | Human Body | 3D Pose Estimation | 100K | Github | Augments Human3.6M with 133 whole-body 3D keypoint annotations (body, hands, face, feet) for 100k images via a multi-view annotation pipeline. |
| FaceScape | 2020 | Human Face | Classification, Segmentation, Reconstruction, Completion, Recognition | 18K | Github | 18,760 high-quality textured 3D face meshes from 938 people with pore-level geometry and 20 expressions per subject. |
| Dataset | Year | Granularity | Tasks | Size | Site | Description |
|---|---|---|---|---|---|---|
| uCO3D | 2025 | Object | Few-shot 3D Reconstruction | 1K | Github | The largest publicly-available collection of high-resolution videos of objects with 3D annotations and full 360° coverage. |
| HPSketch | 2025 | Object | - | 151.9K | Link | A history-based parametric CAD sketch dataset with advanced engineering commands; 151,984 sketches, 377,623 loops, and 29 command types. |
| CBF | 2025 | Object | - | 20K | Link | 20,000 CAD B-rep models with per-face JSON labels; released with BRepFormer for benchmarking complex feature recognition on B-reps. |
| Parametric 20000 | 2024 | Object | - | 20K | Link | Multi-modal CAD shapes: each instance includes a point cloud, a triangle mesh, and a B-Rep file. |
| WildRGB-D | 2024 | Object | View Synthesis, Pose Estimation, 6D Object Tracking, 3D Reconstruction | 8.5K | Github | A large-scale real-world RGB-D object video collection (~20K videos, 8.5K objects) with 360° views, masks, real-scale poses, and aggregated point clouds. |
| BRep2Seq | 2024 | Object | - | 1M | - | A synthetic CAD dataset (~1M models) of B-rep solids paired with feature-based construction sequences. |
| Objaverse | 2023 | Object | 3D Asset Collection, Annotation, Multimodal Learning | 800K | Github | 800K+ free 3D object models with rich metadata (captions, tags, categories) and some animations. |
| DIVA-360 | 2023 | Object | Novel View Synthesis, NeRF Pretraining | 50 | Github | High-resolution synchronized multi-view video of dynamic table-scale scenes (53 RGB cameras), with masks, audio, and text descriptions. |
| StrobeNet | 2021 | Object | 3D Reconstruction | 120K | Link | Articulated-object categories with rendered RGB views, joint + part segmentation, and ground-truth implicit / point cloud geometry. |
| Amazon Berkeley Objects | 2021 | Object | 3D Reconstruction, Multi-view Retrieval, Material Estimation | 8K | Website | Real household objects with high-resolution CAD models, PBR materials, and product images & metadata. |
| Fusion 360 Gallery Dataset | 2021 | Object | 3D reconstruction, segmentation, assembly prediction, sequential modeling | 8K | Github | A parametric CAD dataset (~20K designs) with sketch & extrude sequences, face segmentation, and multi-part assemblies. |
| CO3Dv2 | 2021 | Object | Novel View Synthesis, Category-level 3D Reconstruction | 19K | Github | Large-scale real-object dataset with object-centric multi-view images, annotated camera poses, and ground-truth 3D point clouds across 50 categories. |
| 3D-FUTURE | 2020 | Object | Navigation, Exploration, Interaction | 10K | Github | A furniture CAD + texture dataset with ~10K detailed instances used in realistic room scenes. |
| SketchGraphs | 2020 | Object | - | 15M | Github | ~15M 2D parametric CAD sketches represented as geometric-constraint graphs. |
| ABC | 2019 | Object | Shape Analysis, Segmentation, Surface Fitting | 1M | Link | A huge collection of CAD models with analytic parametric curves & surfaces, sharp feature annotations, and patch decompositions. |
| ScanObjectNN | 2019 | Object | Classification, Segmentation | 700 | Link | Real-world indoor object point clouds (with background clutter, occlusion, partial scans) over 15 categories. |
| Thingi10K | 2016 | Object | Scene Understanding, Semantic Segmentation, Layout Prediction | 300 | Github | 10,000 real-world 3D printing meshes from Thingiverse across 72 categories, with non-manifold and self-intersection cases included. |
| A Large Dataset of Object Scans | 2016 | Object | Object Scanning, 3D Reconstruction, Object Categorization | 10K | Github | 10,000+ consumer-grade real-object 3D scans, diverse in category and size. |
| ShapeNet | 2015 | Object | Single-view Reconstruction, Multi-view Reconstruction | 300M | Link | 3D CAD models (~3M shapes), with ~220K classified models, part annotations, symmetry planes, alignments, and physical sizes. |
| Dataset | Year | Granularity | Tasks | Size | Site | Description |
|---|---|---|---|---|---|---|
| SceneSplat++ | 2025 | Mixed | 3D Scene Captioning, Open-Vocabulary 3D Segmentation, 3D Visual Grounding | 48K | Hugging Face | Currently the world's largest open-source 3D scene dataset, bridging vision and language by enriching reconstructed scenes with semantic features. |
| InteriorGS | 2025 | Scene Indoor |
3D Scene Understanding, Controllable Scene Generation, Embodied Agent Navigation | 100K | Hugging Face | 100K procedurally generated indoor scenes provided directly in the 3D Gaussian Splatting regime with 6DoF trajectories, 3D floor-plans, and segmentation/depth labels. |
| MegaSynth | 2025 | Scene Indoor |
Scene Reconstruction Pretraining | 700K | Website | Synthetic non-semantic scenes scaling 3D scene reconstruction pretraining; CVPR 2025 work demonstrating large generic synthesis can match real-data pipelines. |
| Aria Synthetic Environments | 2023 | Scene Indoor |
3D Question Answering, Spatial Reasoning, Scene Understanding | 100K | Link | 100K procedurally generated indoor scenes with realistic object placement, simulated Aria-glass camera, full 6DoF trajectories, 3D floor-plans, segmentation, and depth. |
| DL3DV-10K | 2023 | Scene | Novel View Synthesis, NeRF Pretraining | 10K | Website | A real-world multi-view video dataset capturing 10,510 4K videos across 65 POI scene types. |
| Aria Digital Twin | 2023 | Scene Indoor |
3D Question Answering, Spatial Reasoning, Scene Understanding | 400 | Link | An egocentric dataset captured with wearable glasses, with synchronized RGB+monochrome cameras, IMU, depth, 6-DoF poses, human pose & eye gaze. |
| PointOdyssey | 2023 | Scene | 3D Generation, Multimodal Learning, Simulation | 104 | Link | Synthetic dataset with natural motion, deformable characters, diverse scenes & materials, and long videos for fine-grained point-tracking evaluation. |
| ScanNet++ | 2023 | Scene Indoor |
Reconstruction, Segmentation, Novel View Synthesis | 1K | Website | High-fidelity indoor scenes with sub-mm laser scans, high-res DSLR + iPhone RGB-D, dense mesh & semantic + instance annotations. |
| Kubric | 2022 | Mixed | Semantic Mapping, 2.5D Reconstruction, View-consistent Semantics | N/A | Github | A Python framework for generating photo-realistic synthetic scenes (Blender + PyBullet) with rich annotations. |
| HM3D | 2021 | Scene Indoor |
Embodied Navigation | 1K | Website | 1,000 high-fidelity real-world indoor 3D meshes with extensive navigable space and clean reconstructions. |
| HyperSim | 2021 | Scene Indoor |
Multi-task Scene Understanding | 461 | Github | Photorealistic synthetic indoor dataset with full scene geometry, materials, lighting, dense semantic + instance segmentation. |
| Habitat 2.0 | 2021 | Scene Indoor |
Pick, Place, Navigate, Open, Close, Rearrange | 111 | Link | Reconfigurable, artist-authored apartments with articulated objects, semantic class & surface annotations, and collision proxies. |
| Virtual KITTI 2 | 2020 | Scene Outdoor |
6D Pose Estimation, Object Detection, Benchmarking | 5 | Link | Synthetic driving-scene dataset with fully annotated RGB, depth, optical flow, semantic & instance segmentation, and weather/camera variants. |
| RELLIS-3D | 2020 | Scene Outdoor |
3D Semantic Segmentation, Sensor Fusion, Autonomous Navigation | 13K | Website | Multimodal off-road robotics dataset with 13,556 LiDAR scans, 6,235 RGB images, point-wise & pixel-wise semantic labels. |
| 3D-FRONT | 2020 | Scene Indoor |
Scene Understanding, Layout Analysis, Object Arrangement | 18K | HuggingFace | Synthetic indoor scenes with professional layouts, textured furniture, consistent style curation, and semantic annotations. |
| Structured3D | 2020 | Scene Indoor |
Reconstruction, Segmentation, Object Detection | 3.5K | Link | Photo-realistic indoor scenes with “primitive + relationship” structure annotations (planes, lines, junctions, layouts, floorplans). |
| Mapillary | 2020 | Scene Outdoor |
Reconstruction, Semantics, Viewpoint Estimation | 1.6M | Link | A street-level image sequence dataset with 1.6M geo-tagged images covering diverse cities, seasons, and weather. |
| BlendedMVS | 2019 | Scene | Reconstruction, Alignment, Evaluation | 113 | Github | Multi-view stereo with 113 textured mesh scenes, rendered + blended image inputs, and ground-truth depth maps. |
| Replica | 2019 | Scene Indoor |
Scene Graph Generation, Object Detection, Relationship Modeling | 18 | Github | 18 photo-realistic indoor scenes with dense meshes, HDR textures, semantic & instance annotations, plus mirror and glass reflectors. |
| RealEstate10K | 2018 | Scene | Camera-Pose Learning, View Synthesis | 10K | Link | Camera trajectories from ~10K YouTube real-estate videos, with pose + intrinsics for 10M frames over ~80K clips. |
| MegaDepth | 2018 | Scene | Depth Estimation, Representation Learning | 200 | Link | Diverse depth dataset built from Internet multi-view photo collections, with ~130K images and dense / ordinal depth labels. |
| DeepMVS | 2018 | Scene | CAD Alignment, 3D Matching, Pose Estimation | 120 | Link | Photorealistic synthetic multi-view dataset (MVS-SYNTH: 120 urban sequences) plus real indoor/outdoor sets for disparity prediction. |
| ScanNet | 2017 | Scene Indoor |
Feature Matching, Registration, 3D Reconstruction | 1.5K | Link | Indoor RGB-D dataset with 1,513 scans across 707 spaces, ~2.5M frames, dense surfaces, semantic + instance labels, and aligned CAD models. |
| Matterport3D | 2017 | Scene Indoor |
Scene Understanding | 90 | Website | Indoor RGB-D dataset with 90 scenes, 194,400 RGB-D images, textured meshes and semantic/instance annotations. |
| Semantic3D | 2016 | Scene Outdoor |
Point Cloud Classification, Semantic Segmentation | 30 | Link | Outdoor laser-scanned benchmark of ~30 high-density static scans (~4B points), with manual semantic labels across 8 classes. |
| SceneNN / ObjectNN | 2016 | Scene Indoor |
Multi-view Fusion, 3D Reconstruction, Semantic Segmentation | 100 | Link | ~100 RGB-D reconstructed indoor scenes with per-vertex / per-pixel semantic & instance annotations and bounding boxes. |
| Virtual KITTI | 2016 | Scene Outdoor |
Detection, Tracking, Scene Segmentation | 35 | Link | Photo-realistic synthetic video dataset cloned from KITTI, with automatic ground truth and weather & camera-angle variants. |
✅ indicates supported modality, * indicates CAD mesh.
📝 Modality includes available signals like RGB, Depth, Pose, Segmentation, Flow, Mesh, Action…
| Dataset | RGB-D | Point Cloud | Mesh | Multi-view | Voxel | Implicit Field |
|---|---|---|---|---|---|---|
| SAM 3D Body | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| GigaHands | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ |
| InterAct | ❌ | ❌ | ✅* | ❌ | ❌ | ❌ |
| EgoHumans | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| EgoExo4D | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
| H3WB | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| FaceScape | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ |
| InteriorGS | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
| MegaSynth | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ |
| HPSketch | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| CBF | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Parametric 20000 | ❌ | ✅ | ✅* | ❌ | ❌ | ❌ |
| WildRGB-D | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| BRep2seq | ❌ | ❌ | ✅* | ❌ | ❌ | ❌ |
| Aria Synthetic Environments | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| DL3DV-10K | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| PointOdyssey | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ |
| Aria Digital Twin | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
| ScanNet++ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Objaverse | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| DIVA-360 | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Kubric | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Amazon Berkeley Objects | ❌ | ❌ | ✅* | ✅ | ❌ | ❌ |
| HM3D | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Fusion 360 Gallery Dataset | ❌ | ❌ | ✅* | ❌ | ❌ | ❌ |
| CO3Dv2 | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ |
| HyperSim | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Habitat 2.0 | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| StrobeNet | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ |
| Virtual KITTI 2 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| RELLIS-3D | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| 3D-FRONT | ❌ | ❌ | ✅* | ❌ | ❌ | ❌ |
| 3D-FUTURE | ❌ | ❌ | ✅* | ❌ | ❌ | ❌ |
| SketchGraphs | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Structured3D | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
| Mapillary | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| ScanObjectNN | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| ABC | ❌ | ❌ | ✅* | ❌ | ✅ | ❌ |
| BlendedMVS | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
| Replica | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| RealEstate10K | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| MegaDepth | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
| DeepMVS | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| ScanNet | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Matterport3D | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Thingi10K | ❌ | ❌ | ✅* | ❌ | ✅ | ❌ |
| Semantic3D | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| SceneNN / ObjectNN | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| A Large Dataset of Object Scans | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Virtual KITTI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| ShapeNet | ❌ | ❌ | ✅* | ❌ | ✅ | ❌ |
| uCO3D | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ |
| SceneSplat++ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
- Hongyang Du* — Brown University
- Zongxia Li* — University of Maryland, College Park
- Dawei Liu* — University of Pennsylvania
- Runhao Li* — University of Southern California
- Haoyuan Song — New York University
- Qingyu Zhang — New York University
- Yubo Wang — New York University
- Jingcheng Ni — Brown University
- Shihang Gui — Brown University
- Congchao Dong — The University of Sydney
- Tao Hu — Stability AI
* Equal contribution.
- Hongyang Du
- Dawei Liu
- Haoyuan Song
- Qingyu Zhang
- Yubo Wang
- Shihang Gui
We welcome and encourage contributions! If you have a paper, dataset, or method that should appear here, please:
- Open a pull request with the new entry, keeping the alphabetical / chronological order of the relevant table.
- Or open an issue to suggest a new section, fix a bug, or request a topic.
If this repository is useful to your work, consider giving it a ⭐ — it helps others discover the cookbook.
Maintained with ❤ for the 3D vision community.
If you'd like to be added as a contributor, just open a PR!