Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ permissions:
env:
NGC_API_KEY: ${{ secrets.NGC_API_KEY }}
ISAACSIM_BASE_IMAGE: ${{ vars.ISAACSIM_BASE_IMAGE || 'nvcr.io/nvidia/isaac-sim' }}
ISAACSIM_BASE_VERSION: ${{ vars.ISAACSIM_BASE_VERSION || '5.0.0' }}
ISAACSIM_BASE_VERSION: ${{ vars.ISAACSIM_BASE_VERSION || '5.1.0' }}
DOCKER_IMAGE_TAG: isaac-lab-dev:${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.ref_name }}-${{ github.sha }}

jobs:
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ Guidelines for modifications:
* Ritvik Singh
* Rosario Scalise
* Ryley McCarroll
* Sergey Grizan
* Shafeef Omar
* Shaoshu Su
* Shaurya Dewan
Expand Down
20 changes: 13 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

# Isaac Lab

[![IsaacSim](https://img.shields.io/badge/IsaacSim-5.0.0-silver.svg)](https://docs.isaacsim.omniverse.nvidia.com/latest/index.html)
[![IsaacSim](https://img.shields.io/badge/IsaacSim-5.1.0-silver.svg)](https://docs.isaacsim.omniverse.nvidia.com/latest/index.html)
[![Python](https://img.shields.io/badge/python-3.11-blue.svg)](https://docs.python.org/3/whatsnew/3.11.html)
[![Linux platform](https://img.shields.io/badge/platform-linux--64-orange.svg)](https://releases.ubuntu.com/22.04/)
[![Windows platform](https://img.shields.io/badge/platform-windows--64-orange.svg)](https://www.microsoft.com/en-us/)
Expand All @@ -14,6 +14,11 @@
[![License](https://img.shields.io/badge/license-Apache--2.0-yellow.svg)](https://opensource.org/license/apache-2-0)


Please note that this branch is currently under active development in preparation for the Isaac Lab 2.3.0 release.
Some features may not be fully functional while we continue development.
Additional features may depend on unreleased Isaac Sim features that will be available in Isaac Sim 5.1.0.


**Isaac Lab** is a GPU-accelerated, open-source framework designed to unify and simplify robotics research workflows,
such as reinforcement learning, imitation learning, and motion planning. Built on [NVIDIA Isaac Sim](https://docs.isaacsim.omniverse.nvidia.com/latest/index.html),
it combines fast and accurate physics and sensor simulation, making it an ideal choice for sim-to-real
Expand Down Expand Up @@ -140,12 +145,13 @@ Isaac Lab is built on top of Isaac Sim and requires specific versions of Isaac S
release of Isaac Lab. Below, we outline the recent Isaac Lab releases and GitHub branches and their corresponding
dependency versions for Isaac Sim.

| Isaac Lab Version | Isaac Sim Version |
| ----------------------------- | ------------------- |
| `main` branch | Isaac Sim 4.5 / 5.0 |
| `v2.2.X` | Isaac Sim 4.5 / 5.0 |
| `v2.1.X` | Isaac Sim 4.5 |
| `v2.0.X` | Isaac Sim 4.5 |
| Isaac Lab Version | Isaac Sim Version |
| ----------------------------- | ------------------------- |
| `main` branch | Isaac Sim 4.5 / 5.0 |
| `v2.3.X` | Isaac Sim 4.5 / 5.0 / 5.1 |
| `v2.2.X` | Isaac Sim 4.5 / 5.0 |
| `v2.1.X` | Isaac Sim 4.5 |
| `v2.0.X` | Isaac Sim 4.5 |


## Contributing to Isaac Lab
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.2.1
2.3.0
12 changes: 6 additions & 6 deletions apps/isaaclab.python.headless.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python Headless"
description = "An app for running Isaac Lab headlessly"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "isaaclab", "python", "headless"]
Expand All @@ -15,7 +15,7 @@ keywords = ["experience", "app", "isaaclab", "python", "headless"]
app.versionFile = "${exe-path}/VERSION"
app.folder = "${exe-path}/"
app.name = "Isaac-Sim"
app.version = "5.0.0"
app.version = "5.1.0"

##################################
# Omniverse related dependencies #
Expand Down Expand Up @@ -108,7 +108,7 @@ metricsAssembler.changeListenerEnabled = false
###############################
[settings.exts."omni.kit.registry.nucleus"]
registries = [
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/106/shared" },
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/107/shared" },
{ name = "kit/sdk", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/sdk/${kit_version_short}/${kit_git_hash}" },
{ name = "kit/community", url = "https://dw290v42wisod.cloudfront.net/exts/kit/community" },
]
Expand Down Expand Up @@ -215,6 +215,6 @@ enabled=true # Enable this for DLSS
# set the S3 directory manually to the latest published S3
# note: this is done to ensure prior versions of Isaac Sim still use the latest assets
[settings]
persistent.isaac.asset_root.default = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.cloud = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.default = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.cloud = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
12 changes: 6 additions & 6 deletions apps/isaaclab.python.headless.rendering.kit
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
[package]
title = "Isaac Lab Python Headless Camera"
description = "An app for running Isaac Lab headlessly with rendering enabled"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "isaaclab", "python", "camera", "minimal"]
Expand All @@ -32,7 +32,7 @@ cameras_enabled = true
app.versionFile = "${exe-path}/VERSION"
app.folder = "${exe-path}/"
app.name = "Isaac-Sim"
app.version = "5.0.0"
app.version = "5.1.0"

### FSD
app.useFabricSceneDelegate = true
Expand Down Expand Up @@ -105,7 +105,7 @@ metricsAssembler.changeListenerEnabled = false

[settings.exts."omni.kit.registry.nucleus"]
registries = [
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/106/shared" },
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/107/shared" },
{ name = "kit/sdk", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/sdk/${kit_version_short}/${kit_git_hash}" },
{ name = "kit/community", url = "https://dw290v42wisod.cloudfront.net/exts/kit/community" },
]
Expand Down Expand Up @@ -156,6 +156,6 @@ folders = [
# set the S3 directory manually to the latest published S3
# note: this is done to ensure prior versions of Isaac Sim still use the latest assets
[settings]
persistent.isaac.asset_root.default = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.cloud = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.default = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.cloud = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
12 changes: 6 additions & 6 deletions apps/isaaclab.python.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python"
description = "An app for running Isaac Lab"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "usd"]
Expand Down Expand Up @@ -161,7 +161,7 @@ show_menu_titles = true

[settings.app]
name = "Isaac-Sim"
version = "5.0.0"
version = "5.1.0"
versionFile = "${exe-path}/VERSION"
content.emptyStageOnStart = true
fastShutdown = true
Expand Down Expand Up @@ -255,7 +255,7 @@ outDirectory = "${data}"
###############################
[settings.exts."omni.kit.registry.nucleus"]
registries = [
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/106/shared" },
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/107/shared" },
{ name = "kit/sdk", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/sdk/${kit_version_short}/${kit_git_hash}" },
{ name = "kit/community", url = "https://dw290v42wisod.cloudfront.net/exts/kit/community" },
]
Expand Down Expand Up @@ -302,6 +302,6 @@ fabricUseGPUInterop = true
# set the S3 directory manually to the latest published S3
# note: this is done to ensure prior versions of Isaac Sim still use the latest assets
[settings]
persistent.isaac.asset_root.default = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.cloud = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.default = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.cloud = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
12 changes: 6 additions & 6 deletions apps/isaaclab.python.rendering.kit
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
[package]
title = "Isaac Lab Python Camera"
description = "An app for running Isaac Lab with rendering enabled"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "isaaclab", "python", "camera", "minimal"]
Expand All @@ -33,7 +33,7 @@ cameras_enabled = true
app.versionFile = "${exe-path}/VERSION"
app.folder = "${exe-path}/"
app.name = "Isaac-Sim"
app.version = "5.0.0"
app.version = "5.1.0"

### FSD
app.useFabricSceneDelegate = true
Expand Down Expand Up @@ -109,7 +109,7 @@ fabricUseGPUInterop = true

[settings.exts."omni.kit.registry.nucleus"]
registries = [
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/106/shared" },
{ name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/107/shared" },
{ name = "kit/sdk", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/sdk/${kit_version_short}/${kit_git_hash}" },
{ name = "kit/community", url = "https://dw290v42wisod.cloudfront.net/exts/kit/community" },
]
Expand Down Expand Up @@ -145,6 +145,6 @@ folders = [
# set the S3 directory manually to the latest published S3
# note: this is done to ensure prior versions of Isaac Sim still use the latest assets
[settings]
persistent.isaac.asset_root.default = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.cloud = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.default = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.cloud = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
10 changes: 5 additions & 5 deletions apps/isaaclab.python.xr.openxr.headless.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python OpenXR Headless"
description = "An app for running Isaac Lab with OpenXR in headless mode"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "usd", "headless"]
Expand All @@ -15,7 +15,7 @@ keywords = ["experience", "app", "usd", "headless"]
app.versionFile = "${exe-path}/VERSION"
app.folder = "${exe-path}/"
app.name = "Isaac-Sim"
app.version = "5.0.0"
app.version = "5.1.0"

### FSD
app.useFabricSceneDelegate = true
Expand Down Expand Up @@ -59,6 +59,6 @@ folders = [
]

[settings]
persistent.isaac.asset_root.default = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.cloud = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.default = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.cloud = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
10 changes: 5 additions & 5 deletions apps/isaaclab.python.xr.openxr.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python OpenXR"
description = "An app for running Isaac Lab with OpenXR"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "usd"]
Expand All @@ -15,7 +15,7 @@ keywords = ["experience", "app", "usd"]
app.versionFile = "${exe-path}/VERSION"
app.folder = "${exe-path}/"
app.name = "Isaac-Sim"
app.version = "5.0.0"
app.version = "5.1.0"

### async rendering settings
# omni.replicator.asyncRendering needs to be false for external camera rendering
Expand Down Expand Up @@ -88,6 +88,6 @@ folders = [
# set the S3 directory manually to the latest published S3
# note: this is done to ensure prior versions of Isaac Sim still use the latest assets
[settings]
persistent.isaac.asset_root.default = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.cloud = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0"
persistent.isaac.asset_root.default = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.cloud = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
persistent.isaac.asset_root.nvidia = "https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1"
2 changes: 1 addition & 1 deletion apps/isaacsim_4_5/isaaclab.python.headless.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python Headless"
description = "An app for running Isaac Lab headlessly"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "isaaclab", "python", "headless"]
Expand Down
2 changes: 1 addition & 1 deletion apps/isaacsim_4_5/isaaclab.python.headless.rendering.kit
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
[package]
title = "Isaac Lab Python Headless Camera"
description = "An app for running Isaac Lab headlessly with rendering enabled"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "isaaclab", "python", "camera", "minimal"]
Expand Down
2 changes: 1 addition & 1 deletion apps/isaacsim_4_5/isaaclab.python.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python"
description = "An app for running Isaac Lab"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "usd"]
Expand Down
2 changes: 1 addition & 1 deletion apps/isaacsim_4_5/isaaclab.python.rendering.kit
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
[package]
title = "Isaac Lab Python Camera"
description = "An app for running Isaac Lab with rendering enabled"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "isaaclab", "python", "camera", "minimal"]
Expand Down
2 changes: 1 addition & 1 deletion apps/isaacsim_4_5/isaaclab.python.xr.openxr.headless.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python OpenXR Headless"
description = "An app for running Isaac Lab with OpenXR in headless mode"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "usd", "headless"]
Expand Down
2 changes: 1 addition & 1 deletion apps/isaacsim_4_5/isaaclab.python.xr.openxr.kit
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[package]
title = "Isaac Lab Python OpenXR"
description = "An app for running Isaac Lab with OpenXR"
version = "2.2.1"
version = "2.3.0"

# That makes it browsable in UI with "experience" filter
keywords = ["experience", "app", "usd"]
Expand Down
4 changes: 2 additions & 2 deletions docker/.env.base
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
ACCEPT_EULA=Y
# NVIDIA Isaac Sim base image
ISAACSIM_BASE_IMAGE=nvcr.io/nvidia/isaac-sim
# NVIDIA Isaac Sim version to use (e.g. 5.0.0)
ISAACSIM_VERSION=5.0.0
# NVIDIA Isaac Sim version to use (e.g. 5.1.0)
ISAACSIM_VERSION=5.1.0
# Derived from the default path in the NVIDIA provided Isaac Sim container
DOCKER_ISAACSIM_ROOT_PATH=/isaac-sim
# The Isaac Lab path in the container
Expand Down
2 changes: 1 addition & 1 deletion docker/.env.cloudxr-runtime
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
# NVIDIA CloudXR Runtime base image
CLOUDXR_RUNTIME_BASE_IMAGE_ARG=nvcr.io/nvidia/cloudxr-runtime
# NVIDIA CloudXR Runtime version to use
CLOUDXR_RUNTIME_VERSION_ARG=5.0.0
CLOUDXR_RUNTIME_VERSION_ARG=5.1.0
2 changes: 1 addition & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@
{
"name": "Isaac Sim",
"url": "https://developer.nvidia.com/isaac-sim",
"icon": "https://img.shields.io/badge/IsaacSim-5.0.0-silver.svg",
"icon": "https://img.shields.io/badge/IsaacSim-5.1.0-silver.svg",
"type": "url",
},
{
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/teleop/hand_asset.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/teleop/teleop_diagram.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Install Isaac Sim 5.0:

.. code-block:: bash

pip install "isaacsim[all,extscache]==5.0.0" --extra-index-url https://pypi.nvidia.com
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com

Install Isaac Lab extensions and dependencies:

Expand Down
25 changes: 24 additions & 1 deletion docs/source/how-to/cloudxr_teleoperation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ There are two options to run the CloudXR Runtime Docker container:
-p 48005:48005/udp \
-p 48008:48008/udp \
-p 48012:48012/udp \
nvcr.io/nvidia/cloudxr-runtime:5.0.0
nvcr.io/nvidia/cloudxr-runtime:5.1.0

.. note::
If you choose a particular GPU instead of ``all``, you need to make sure Isaac Lab also runs
Expand Down Expand Up @@ -593,6 +593,12 @@ Isaac Lab provides three main retargeters for hand tracking:
* Handles both left and right hands, converting hand poses to joint angles for the GR1T2 robot's hands
* Supports visualization of tracked hand joints

.. dropdown:: UnitreeG1Retargeter (:class:`isaaclab.devices.openxr.retargeters.UnitreeG1Retargeter`)

* Retargets OpenXR hand tracking data to Unitree G1 using Inspire 5-finger hand end-effector commands
* Handles both left and right hands, converting hand poses to joint angles for the G1 robot's hands
* Supports visualization of tracked hand joints

Retargeters can be combined to control different robot functions simultaneously.

Using Retargeters with Hand Tracking
Expand Down Expand Up @@ -634,6 +640,23 @@ Here's an example of setting up hand tracking:

.. _control-robot-with-xr-callbacks:

Here's a diagram for the dataflow and algorithm used in humanoid teleoperation. Using Apple Vision Pro, we collect 26 keypoints for each hand.
The wrist keypoint is used to control the hand end-effector, while the remaining hand keypoints are used for hand retargeting.

.. figure:: ../_static/teleop/teleop_diagram.jpg
:align: center
:figwidth: 80%
:alt: teleop_diagram

For dex-retargeting, we are currently using the Dexpilot optimizer, which relies on the five fingertips and the palm for retargeting. It is essential
that the links used for retargeting are defined exactly at the fingertips—not in the middle of the fingers—to ensure accurate optimization.Please refer
to the image below for hand asset selection, find a suitable hand asset, or add fingertip links in IsaacLab as needed.

.. figure:: ../_static/teleop/hand_asset.jpg
:align: center
:figwidth: 60%
:alt: hand_asset

Adding Callbacks for XR UI Events
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
Loading