diff --git a/docker/development/Dockerfile b/docker/development/Dockerfile index 17838d69..ba585728 100644 --- a/docker/development/Dockerfile +++ b/docker/development/Dockerfile @@ -42,6 +42,7 @@ RUN apt-get update && apt-get install -y \ gnupg \ iputils-ping \ jq \ + libturbojpeg0-dev \ nano \ net-tools \ ninja-build \ diff --git a/docs/configuration.md b/docs/configuration.md index 3b8def50..4c2b2d64 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -22,6 +22,7 @@ a YAML configuration file for runtime parameters. - [Capping the outbound forward rate](#capping-the-outbound-forward-rate) - [Latched topics (`latched: true`)](#latched-topics-latched-true) - [Data track encoding (`encoding`)](#data-track-encoding-encoding) + - [Outbound video](#outbound-video) - [Video Options](#video-options) ## Prerequisites @@ -423,6 +424,28 @@ Notes: - Like `max_rate_hz` and `latched`, `encoding` is matched by **literal topic name**, not regex. +#### Outbound video + +ROS Portal sends an outbound topic of type `sensor_msgs/Image` or +`sensor_msgs/CompressedImage` as a LiveKit video track. Other types use a data +track. + +- `sensor_msgs/Image` supports `rgba8`, `rgb8`, `bgr8`, `bgra8`, and `mono8`. + The track has the same name as the topic. +- `sensor_msgs/CompressedImage` supports JPEG only. The `format` field must + contain `jpeg` or `jpg` (case-insensitive). + ROS Portal decodes each frame before LiveKit encodes it as video. +- Video tracks follow the same naming convention as data tracks. + +```yaml +topics: + # Sends the LiveKit video track /insta/cam0/image_raw/compressed. + - topic: "/insta/cam0/image_raw/compressed" + direction: "out" +``` + +The track size comes from the first frame. + ### Video Options | Field | Type | Required | Description | diff --git a/docs/diagnostics.md b/docs/diagnostics.md index d02c9a31..0dc65ba6 100644 --- a/docs/diagnostics.md +++ b/docs/diagnostics.md @@ -199,9 +199,9 @@ inbound reader thread has stopped. | `inbound.data_tracks` | Number of inbound LiveKit data tracks being republished. | | `inbound.failures` | Cumulative inbound failures, aggregating tracks rejected because no ROS type could be resolved, because they matched no topic pattern, or because a ROS topic name could not be produced, plus failures publishing inbound frames on ROS, invalid inbound JSON frames, empty inbound CDR payloads, streams that ended with a terminal error, and inbound schema validation rejections. | | `outbound.data_tracks` | Number of discovered outbound data topics. | -| `outbound.failures` | Cumulative outbound failures, aggregating LiveKit data-frame push failures, ROS-to-JSON conversion failures, ROS subscription creation failures, and outbound schema define, render, and encoding-mismatch failures. | +| `outbound.failures` | Cumulative outbound failures, aggregating LiveKit data-frame push failures, ROS-to-JSON conversion failures, ROS subscription creation failures, and outbound schema define, render, and encoding-mismatch failures. Also counts `CompressedImage` frames that are not JPEG or fail to decode, and outbound video topics skipped because another topic already uses their track name. | | `outbound.subscriptions` | Number of active outbound ROS subscriptions. | -| `outbound.video_tracks` | Number of discovered outbound image topics. | +| `outbound.video_tracks` | Number of discovered outbound image topics, `sensor_msgs/Image` and `sensor_msgs/CompressedImage`. | `outbound.failures` and `inbound.failures` are deliberately coarse. Every increment is logged individually with its specific cause, including the track, diff --git a/docs/limitations.md b/docs/limitations.md index 1df57e6b..bf7c8cba 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -6,6 +6,9 @@ recreated. - Only `rgba8`, `rgb8`, `bgr8`, `bgra8`, and `mono8` encodings are handled. Other encodings are dropped with a throttled warning. +- Outbound `sensor_msgs/CompressedImage` topics support JPEG only. Each frame + is decoded on the CPU before LiveKit encodes it as video. A 1472x1440 frame + takes about 6.5 ms to decode on an aarch64 devcontainer. - Packed `rgba8`, `bgra8`, and `rgb8` images go to LiveKit without a pixel copy. `bgr8`, `mono8`, and images with row padding need one CPU pass inside the subscription callback. The LiveKit SDK then converts each frame to I420. diff --git a/src/livekit_scarf_slam/CMakeLists.txt b/src/livekit_scarf_slam/CMakeLists.txt new file mode 100644 index 00000000..e9882095 --- /dev/null +++ b/src/livekit_scarf_slam/CMakeLists.txt @@ -0,0 +1,43 @@ +# +# Copyright 2026 LiveKit +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cmake_minimum_required(VERSION 3.8) +project(livekit_scarf_slam) + +find_package(ament_cmake_auto REQUIRED) +find_package(ament_cmake_python REQUIRED) + +ament_python_install_package(${PROJECT_NAME}) + +install(PROGRAMS + scripts/convert_stereo_calibration.py + scripts/slam_bag_recorder_node + scripts/stereo_rectify_node + DESTINATION lib/${PROJECT_NAME}) + +install(FILES README.md calibration.md + DESTINATION share/${PROJECT_NAME}) + +if(BUILD_TESTING) + find_package(ament_lint_auto REQUIRED) + ament_lint_auto_find_test_dependencies() + + find_package(ament_cmake_pytest REQUIRED) + ament_add_pytest_test(livekit_scarf_slam_unit_tests test + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + TIMEOUT 120) +endif() + +ament_auto_package(INSTALL_TO_SHARE config launch) diff --git a/src/livekit_scarf_slam/README.md b/src/livekit_scarf_slam/README.md new file mode 100644 index 00000000..5d3fae40 --- /dev/null +++ b/src/livekit_scarf_slam/README.md @@ -0,0 +1,402 @@ +# livekit_scarf_slam + +This package runs a +[ScaRF-SLAM](https://github.com/ori-drs/ScaRF-SLAM) demo through `ros_portal`. +The robot (a Jetson) sends one stereo camera stream through LiveKit. A SLAM +host (a MacBook Pro for now) does the localization and mapping. + +The robot has no IMU, so the demo does not use OpenVINS, the default +ScaRF-SLAM pose source. RTAB-Map stereo SLAM (stereo odometry, loop closure, +and pose graph optimization) gives the poses instead. + +The data flows in this order: + +1. The camera driver on the robot publishes one side-by-side (SBS) JPEG frame + on `/stereo/image_raw/compressed`. An SBS frame holds the left and right + images next to each other, so both eyes have one stamp. +2. `ros_portal` sends the topic as the LiveKit video track `/stereo/image_raw`. +3. `ros_portal` on the SLAM host publishes the track on + `/stereo/image_raw/compressed`. +4. `stereo_rectify_node` splits each frame and rectifies both fisheye images + for RTAB-Map. Rectified images have matching points on the same row. +5. RTAB-Map calculates the poses. +6. `slam_bag_recorder_node` writes the ScaRF-SLAM input bag. For each RTAB-Map + keyframe, the bag has the left fisheye image, the odometry pose, and the + loop-corrected trajectory. +7. After the run, ScaRF-SLAM builds the dense map from the bag. + +ScaRF-SLAM always reads a recorded bag, also in its upstream online mode. +RTAB-Map runs live on the SLAM host. The dense map comes after the run. + +## Python Environment + +The upstream page uses a conda environment with Python 3.11. Do not use conda +in the devcontainer. ROS 2 Jazzy, `rclpy`, and colcon use the system Python +3.12 at `/usr/bin/python3`, and a different Python version breaks them. + +Instead, you create a venv (an isolated Python package folder) from the system +Python with `--system-site-packages`. With this flag, the venv sees the ROS +Python packages and colcon. The ScaRF-SLAM packages go into the venv only. + +Do not use `/livekit_ws/venv`. That folder is a macOS venv from the host +mount, and it does not run in the container. + +The devcontainer ships `numpy` 1.26.4. Depth Anything 3 requires `numpy<2`, and +the Jazzy apt packages, for example `cv_bridge`, use numpy 1.x. Do not upgrade +numpy to 2.x in the venv. + +## Prerequisites + +You run all commands in this page inside the devcontainer, as `root`. + +1. Start the devcontainer from the repository root on the host: + + ```bash + devcontainer up --workspace-folder . + ``` + +2. Open a shell in the container: + + ```bash + devcontainer exec --workspace-folder . bash + ``` + +3. Install the apt packages: + + ```bash + apt update + apt install -y python3-venv ros-$ROS_DISTRO-rtabmap-ros + ``` + + The development image installs `ros-$ROS_DISTRO-rtabmap-ros`. If you use an + older container, install it here. + +The venv in `/opt` and the clones in `~/git` are outside the host mount. If +you recreate the container, do all the steps again. + +## Create The Venv + +1. Create the venv from the system Python: + + ```bash + python3 -m venv --system-site-packages /opt/scarf_venv + ``` + +2. Add a `COLCON_IGNORE` file, so that colcon does not scan the venv: + + ```bash + touch /opt/scarf_venv/COLCON_IGNORE + ``` + +3. Source ROS 2, then activate the venv: + + ```bash + source /opt/ros/jazzy/setup.bash + source /opt/scarf_venv/bin/activate + ``` + +4. Make sure that the venv uses Python 3.12 and sees `rclpy`: + + ```bash + python -c "import sys, rclpy, numpy; print(sys.version, numpy.__version__)" + ``` + +Source ROS 2 before you activate the venv in every new shell. If you do it in +the opposite order, the ROS setup script can put the system Python first. + +## Install The Python Dependencies + +Do these steps in a shell with the venv active. + +1. Clone ScaRF-SLAM and Depth Anything 3: + + ```bash + mkdir -p ~/git && cd ~/git + git clone https://github.com/ori-drs/ScaRF-SLAM.git + git clone https://github.com/ByteDance-Seed/Depth-Anything-3.git + ``` + +2. Create a pip constraints file (a list of version limits for every + `pip install`): + + ```bash + printf 'numpy<2\nsetuptools>=77.0.3,<80\n' > /opt/scarf_venv/constraints.txt + ``` + + Depth Anything 3 and the Jazzy apt packages need `numpy<2`. torch needs + `setuptools>=77.0.3`, and `colcon-core` needs `setuptools<80`. If + setuptools is 80 or newer, `colcon build --symlink-install` can fail for + Python packages. Pass `-c /opt/scarf_venv/constraints.txt` to every + `pip install` in this section. + +3. Install PyTorch and xformers: + + ```bash + pip install -c /opt/scarf_venv/constraints.txt "torch>=2" torchvision xformers + ``` + +4. On an `x86_64` host, install Depth Anything 3 in editable mode: + + ```bash + cd ~/git/Depth-Anything-3 + pip install -c /opt/scarf_venv/constraints.txt -e . addict + ``` + + Depth Anything 3 imports `addict` but does not list it as a dependency, so + you install it in the same command. + +5. On an `aarch64` host, do the steps in + [Depth Anything 3 On aarch64](#depth-anything-3-on-aarch64) instead of + step 4. + +6. Install the other ScaRF-SLAM dependencies: + + ```bash + pip install -c /opt/scarf_venv/constraints.txt rosbags open3d gtsam vismatch + ``` + +7. Make sure that the packages import, that numpy is still 1.x, and that + setuptools is still below 80: + + ```bash + python -c "import torch, depth_anything_3.api, open3d, gtsam, rosbags, vismatch, numpy, setuptools; print(torch.__version__, numpy.__version__, setuptools.__version__)" + ``` + + The warning about `gsplat` is normal. ScaRF-SLAM does not need 3D Gaussian + splatting rendering. + +The devcontainer on an Apple silicon host is `aarch64` and has no NVIDIA GPU. +On that host, PyTorch installs the CUDA build, but it runs on the CPU only. +xformers has no `aarch64` wheel, so pip compiles it without the CUDA kernels. +For the real-time demo, use an `x86_64` host with an NVIDIA GPU. + +Optional: to add sky masks to the Depth Anything 3 output, edit `da3.py`. In +the `forward` function of `NestedDepthAnything3Net`, add this line just before +the `return` statement: + +```python +output.sky = metric_output.sky +``` + +The upstream page also points to +[pull request #7](https://github.com/ori-drs/ScaRF-SLAM/pull/7). It is a +faster implementation for large environments, but nobody reviewed it +independently. + +### Depth Anything 3 On aarch64 + +`pycolmap` has no Linux `aarch64` wheel on PyPI, so `pip install -e .` fails +with this error: + +```text +ERROR: No matching distribution found for pycolmap +``` + +Depth Anything 3 uses `pycolmap` only for COLMAP export, but +`depth_anything_3.api` imports it at load time. You install the other +dependencies, and then you make that import optional. + +1. Install the Depth Anything 3 dependencies without `pycolmap`: + + ```bash + cd ~/git/Depth-Anything-3 + pip install -c /opt/scarf_venv/constraints.txt pre-commit trimesh einops \ + huggingface_hub imageio opencv-python open3d fastapi uvicorn requests \ + typer pillow omegaconf evo e3nn "moviepy==1.0.3" plyfile pillow_heif \ + safetensors addict + ``` + +2. Install Depth Anything 3 without its dependency list: + + ```bash + pip install -c /opt/scarf_venv/constraints.txt --no-deps -e . + ``` + +3. In `src/depth_anything_3/utils/export/colmap.py`, replace + `import pycolmap` with this block: + + ```python + try: + import pycolmap + except ImportError: # no aarch64 wheel; only export_to_colmap needs it + pycolmap = None + ``` + +4. In the same file, add this check in `export_to_colmap`, just before the + `# 2. Set Reconstruction` comment: + + ```python + if pycolmap is None: + raise ImportError("pycolmap is required for COLMAP export") + ``` + +After these steps, `pip check` reports +`depth-anything-3 0.0.0 requires pycolmap, which is not installed`. This +message is expected on `aarch64`. COLMAP export does not work on this host. + +## Build The Workspace + +ScaRF-SLAM and Depth Anything 3 are plain Python projects, so colcon does not +build them. colcon builds this package and `ros_portal`. + +1. Build the demo packages: + + ```bash + cd /livekit_ws + colcon build --symlink-install --packages-up-to livekit_scarf_slam + ``` + + `--packages-up-to` builds only this package and its dependencies. Do + not build the full workspace. The full build also builds the + `client-sdk-cpp` checkout as the `livekit` package, and that build needs + Rust. `ros_portal` downloads a prebuilt LiveKit SDK, so it does not need + that package. Without Rust, the full build fails with this error: + + ```text + Could not find CARGO_EXECUTABLE using the following names: cargo + ``` + + If the build fails with + `c++: fatal error: Killed signal terminated program cc1plus`, the + container ran out of memory. Build one package at a time with two compile + jobs: + + ```bash + MAKEFLAGS="-j2" colcon build --symlink-install \ + --executor sequential \ + --packages-up-to livekit_scarf_slam + ``` + +2. Source the workspace: + + ```bash + source install/setup.bash + ``` + +3. Run the tests: + + ```bash + colcon test --packages-select livekit_scarf_slam + colcon test-result --verbose + ``` + +If you built the workspace before without `--symlink-install`, the build can +fail with this error: + +```text +failed to create symbolic link '/livekit_ws/build/ros_portal_msgs/ament_cmake_python/ros_portal_msgs/ros_portal_msgs' because existing path cannot be removed: Is a directory +``` + +The old build copied the Python package as a folder, and colcon cannot +replace that folder with a symlink. Delete the build and install output of +the package in the error, then build again: + +```bash +rm -rf build/ros_portal_msgs install/ros_portal_msgs +colcon build --symlink-install --packages-up-to livekit_scarf_slam +``` + +## Configuration + +Before the first run, calibrate the stereo camera. See +[calibration.md](calibration.md). The launch file reads the calibration from +`/livekit_ws/stereo-calibration/stereo_calibration.json` by default. + +| File | Purpose | +| --- | --- | +| `config/ros_portal/edge_robot.yaml` | ROS Portal on the robot. Sends `/stereo/image_raw/compressed` as a video track at 8000 kbps. | +| `config/ros_portal/slam_host.yaml` | ROS Portal on the SLAM host. Receives the `/stereo/image_raw` track. | +| `config/rtabmap/stereo.yaml` | RTAB-Map parameters. `frame_id` is `stereo_link`. RTAB-Map adds at most 3 keyframes per second. | +| `config/scarf_slam/cobra_stereo.yaml` | ScaRF-SLAM configuration. The camera blocks come from the calibration. | + +The camera driver on the robot must use the calibrated mode, 3840x1080. A +frame rate of 15 fps is enough for RTAB-Map. If LiveKit lowers the stream +resolution, `stereo_rectify_node` scales the intrinsics, and the rectified +images keep the same size. A frame with a different aspect ratio is dropped. + +`stereo_slam.launch.py` has these arguments: + +| Argument | Default | Description | +| --- | --- | --- | +| `calibration_path` | `/livekit_ws/stereo-calibration/stereo_calibration.json` | The stereo calibration. | +| `sbs_topic` | `/stereo/image_raw/compressed` | The SBS JPEG topic. | +| `output_dir` | `~/scarf_slam_runs/_