Python Only 3D Computer Vision

Supported Features

• Camera Model
  • Focal Length Estimation (Metadata, DepthPro, MoGe, GeoCalib)
• Camera Calibration
  • Single-Camera Calibration
  • Multi-Camera Calibration
  • PTZ-Camera Calibration
  • LiDAR-Camera Calibration
• Depth Estimation
  • Monocular Depth Estimation (DepthPro, MoGe)
  • Stereo Depth Estimation (FoundationStereo)
• Pose Estimation
  • Relative Pose Estimation (Essential, Fundamental, Homography)
  • Absolute Pose Estimation (PnP)
  • Visual Localization (PnP)
  • Joint Optimization of Pose and NeRF/GS (iNeRF, iCoMa)
• Point-set Registration
  • Rigid Registration (Procrustes, ICP)
  • Non-Rigid Registration (CPD)
• 3D Reconstruction
  • Global Image Retrieval (NetVLAD, SALAD)
  • Local Feature Detection and Matching (LoFTR, RoMa)
  • Triangulation
  • Structure-from-Motion (Incremental, Global)
• Global Alignment
  • Bundle Adjustment
  • Differentiable Bundle Adjustment
  • Video Depth Alignment
• Post Processing
  • Mesh Reconstruction (SDF, Marching Cubes, PSR)
  • Voxelization
  • Point Cloud Completion

WARNING: Some features or methods may be missing for now. They will be implemented slowly. The codebase may have significant changes.

A lot of 3D computer vision tasks will be supported with a simple inference script with SOTA models. However, the following tasks will not be supported:

• Object-Level 3D Tasks (Classification, Detection, Segmentation)
• Scene Flow & Motion (Optical Flow)
• Learning-based SfM (DUSt3R, MASt3R, VGGSfM, etc.) (may be later)
• Visual SLAM (may be later)
• Absolute Pose Regression
• Scene Coordinate Regression

Installation

bash install.sh

Monocular Depth Estimation

The goal is to get the pixel-wise depth values (and bonus: focal length).

Supported Metric-Depth Models:

• DepthPro (ArXiv 2024)

Supported Relative-Depth Models:

• MoGe (CVPR 2025)

Run this command to get the estimated depth and focal length (if not exist in metadata):

python scripts/estimate_depth.py

Notes: For outdoor images, mask the sky.

Stereo Depth Estimation

The goal is to get the pixel-wise depth values given stereo images. Usually the depth values are in metric-scale and accurate compared to monocular depth methods.

Supported Models:

• FoundationStereo (CVPR 2025 Oral)

Notes: For FoundationStereo, download the pre-trained model from official repo and put them in checkpoints/ folder.

Run this command to get the estimated depth for the left image:

python scripts/estimate_stereo_depth.py

Ground-truth	Estimated

Image from MiddleBury 2005 Dataset.

Global Image Retrieval

The goal is to search for the most similar image from a database with respect to the given image.

Supported Models:

Local Feature Detection and Matching

Given two images, to find the matches (pixel locations) based on features correspondences.

Supported Feature Detectors/Matching Models:

• LoFTR (CVPR 2021)
• RoMa (CVPR 2024)

Run this command to match the two images:

python scripts/match_views.py

Matched Keypoints	Matched 3D Effect

Notes: You can also specify the number of matches for dense methods.

Pose Estimation (Relative/Absolute)

Given 2D correspondences, to find the relative or absolute pose between two images.

Relative Pose Estimation

• Up to an unknown scale
• 2D-2D Correspondences
• Pose relative to an another image

Absolute Pose Estimation or Camera Relocalization

• Up to a real scale
• 2D-3D Correspondences
• Pose relative to a world/map

With relative pose from 2D-2D correspondences, you can only recover the direction of motion (translation vector is only correct up to scale), but not how far you moved. So the resulting pose is in an arbitary scale.

Pose Solvers:

• 2D-2D Correspondences
  • Essential Matrix (if you know the camera intrinsics)
  • Fundamental Matrix (if you don't know the camera intrinsics)
  • Homography
• 2D-3D Correspondences
  • PnP (if you know the camera intrinsics and depth of the first image)
• 3D-3D Corrspondences
  • Procrustes (if you know the camera intrinsics and depth of both images)

No matter which algorithm you choose, the initial correspondences will be pixel correspondences.

Run this command to match the two images:

python scripts/find_pose.py

Relative Pose Estimation	Absolute Pose Estimation

Point Cloud Registration

Rigid Registration

Given two point clouds, find the global transformation and align them.

Global Registration:

• Procrustes or Kabsch

Local Registration:

• ICP

Feature-based Registration:

• Deep Gloabl Registration (CVPR 2020 Oral)

Run this command to align the point clouds:

python scripts/align_pcd.py

Gloabl Registration	Local Registration

Before registration/alignment, the two point clouds are in their own coordinate system.

Non-Rigid or Deformable Registration

Given two point clouds, find the transformation of each point and align them.

Supported Methods:

• CPD (TPAMI 2010)

Run this command to align the point clouds:

python scripts/align_pcd_non_rigid.py

Gloabl Registration	Non-Rigid Registration

References

• https://github.com/colmap/colmap
• https://github.com/naver/dust3r
• https://github.com/apple/ml-depth-pro
• https://github.com/microsoft/moge