Skip to content

feat: benchmark suite comparing performance against PCL and Open3D #150

Description

@rajgandhi1

Overview

Create a published benchmark suite comparing 3DCrate's performance against PCL (C++) and Open3D (Python/C++) on common robotics operations. Publish results in the README and as a GitHub Pages report.

Motivation

The robotics community is conservative about switching tools. Performance numbers are the strongest adoption argument. Community research shows this is how teams evaluate new libraries — they won't migrate without data.

Operations to Benchmark

Operation Input size Metric
Voxel grid downsampling 300K, 1M, 5M pts ms/cloud
Statistical outlier removal 300K, 1M pts ms/cloud
Normal estimation (KNN k=10) 300K, 1M pts ms/cloud
ICP (point-to-point) 50K → 50K pts ms/iter, final MSE
RANSAC plane segmentation 300K pts ms/cloud
Euclidean clustering 300K pts ms/cloud
PLY read (binary) 1M pts ms, MB/s
PLY write (binary) 1M pts ms, MB/s

Methodology

  • Same input data for all libraries (shared .ply/.pcd files checked into benches/data/)
  • 100 warm-up runs, report median ± p95
  • Hardware: document CPU/RAM/GPU used
  • Single-thread and multi-thread results where applicable
  • For Python (Open3D): use timeit with process isolation
  • For Rust (3DCrate, PCL via C FFI): use criterion

Output

  • benches/README.md with tables and charts
  • GitHub Actions job that regenerates results on main pushes
  • Badge in top-level README: ![ICP speed](https://...)

Notes

  • PCL benchmarks should use their standard CMake build (Release mode, -O3)
  • Open3D benchmarks should use their official PyPI wheels
  • The threecrate-bench crate already exists — extend it with these cases

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationenhancementNew feature or requestperformancePerformance improvements

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions