Dev/hougantc/teleop controllers

[rwiltz]

Description

This MR does the following:

• Introduces Quest retargeters for G1 env loco-manipulation tasks. This enables lower body control via the quest controller joysticks, and upper body control via controller tracking.
• Refactors the retargeters to not depend on OpenXRDevice, instead move enums into DeviceBase and allow retargeters to be used across devices.
• Adds XrAnchor "pinning" to a specific robot prim so that the XR view follows the robot in the scene.

Fixes # (issue)

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (existing functionality will not work without user modification)
• Documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Overview

Greptile Summary

This PR introduces Quest controller support for G1 loco-manipulation tasks and refactors the retargeter architecture for device independence.

Key Changes:

• Architecture Refactoring: Moved tracking enums (TrackingTarget, MotionControllerDataRowIndex, MotionControllerInputIndex) from OpenXRDevice to DeviceBase, enabling retargeters to work across different device types
• Feature Discovery System: Added Requirement enum and GetRequirments() method to RetargeterBase, allowing devices to query which data features each retargeter needs (hand tracking, head tracking, motion controllers)
• XR Anchor Pinning: New XrAnchorSynchronizer class enables the XR anchor to dynamically follow a robot prim (e.g., pelvis) with configurable rotation modes (FIXED, FOLLOW_PRIM, FOLLOW_PRIM_SMOOTHED, CUSTOM)
• Quest Controller Integration: Two new retargeters for G1:
  • G1LowerBodyStandingMotionControllerRetargeter: Maps thumbstick inputs to locomotion commands (x/y movement, yaw rotation, hip height)
  • G1TriHandUpperBodyMotionControllerRetargeter: Maps trigger/squeeze/buttons to hand joint commands for manipulation
• Breaking Change: All existing retargeters updated to use DeviceBase.TrackingTarget instead of OpenXRDevice.TrackingTarget

Issues Found:

• Method name typo: GetRequirments() should be GetRequirements() (marked as intentional but is still a spelling error that should be corrected across all 10 files)

Confidence Score: 4/5

• Safe to merge with one systematic typo correction needed
• The refactoring is well-architected and properly decouples retargeters from OpenXR-specific dependencies. The XR anchor synchronizer implementation is solid with proper error handling. The only issue is a systematic method name typo (GetRequirments vs GetRequirements) that needs correction across all implementations. All existing retargeters have been updated consistently, tests have been updated, and the breaking changes are well-documented.
• All files using GetRequirments() need the typo corrected to GetRequirements() before merge (10 files total)

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/device_base.py	4/5	Added shared enums (TrackingTarget, MotionControllerDataRowIndex, MotionControllerInputIndex) to support cross-device retargeting and feature aggregation system
source/isaaclab/isaaclab/devices/retargeter_base.py	3/5	Added Requirement enum and GetRequirments() method for feature discovery - typo in method name should be corrected to GetRequirements()
source/isaaclab/isaaclab/devices/openxr/openxr_device.py	4/5	Refactored to use DeviceBase enums, added motion controller support, integrated XrAnchorSynchronizer for dynamic anchor tracking
source/isaaclab/isaaclab/devices/openxr/xr_anchor_utils.py	4/5	New XrAnchorSynchronizer class enables XR anchor to follow robot prim with configurable rotation modes and smoothing
source/isaaclab/isaaclab/devices/openxr/xr_cfg.py	5/5	Added XrAnchorRotationMode enum and new config fields for dynamic anchor positioning (anchor_prim_path, anchor_rotation_mode, smoothing parameters)
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/g1_motion_controller_locomotion.py	4/5	New retargeter maps Quest controller thumbsticks to G1 locomotion commands (movement and hip height adjustment)
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/trihand/g1_upper_body_motion_ctrl_retargeter.py	4/5	New retargeter maps Quest controller inputs (trigger, squeeze, buttons) to G1 hand joints for manipulation

Sequence Diagram

sequenceDiagram
    participant User
    participant Env as Environment
    participant Device as OpenXRDevice
    participant Retargeter1 as G1UpperBodyRetargeter
    participant Retargeter2 as G1LowerBodyRetargeter
    participant XRCore as XRCore/Hardware
    participant Anchor as XrAnchorSynchronizer
    participant Robot as G1 Robot

    User->>Env: Initialize environment
    Env->>Device: Create OpenXRDevice(cfg, retargeters)
    Device->>Retargeter1: Query GetRequirments()
    Retargeter1-->>Device: [HAND_TRACKING]
    Device->>Retargeter2: Query GetRequirments()
    Retargeter2-->>Device: [MOTION_CONTROLLER]
    Device->>Device: Aggregate required features
    Device->>Anchor: Initialize XrAnchorSynchronizer
    Anchor->>Robot: Track pelvis prim
    
    loop Every Simulation Step
        User->>Env: env.step()
        Env->>Device: advance()
        Device->>Device: _get_raw_data()
        Device->>XRCore: get_input_device("/user/hand/left")
        XRCore-->>Device: Hand tracking data
        Device->>XRCore: get_input_device("/user/hand/right")
        XRCore-->>Device: Controller + inputs data
        Device->>Device: Package data with TrackingTarget keys
        
        Device->>Retargeter1: retarget(raw_data)
        Retargeter1->>Retargeter1: Extract HAND_LEFT/RIGHT from data
        Retargeter1->>Retargeter1: Calculate wrist poses
        Retargeter1->>Retargeter1: Retarget hand joints
        Retargeter1-->>Device: tensor[wrist_poses + hand_joints]
        
        Device->>Retargeter2: retarget(raw_data)
        Retargeter2->>Retargeter2: Extract CONTROLLER_LEFT/RIGHT
        Retargeter2->>Retargeter2: Map thumbsticks to locomotion
        Retargeter2-->>Device: tensor[x, y, yaw, hip_height]
        
        Device->>Device: torch.cat([ret1_output, ret2_output])
        Device-->>Env: Combined command tensor
        
        Env->>Robot: Apply upper body IK
        Env->>Robot: Apply lower body joint positions
        
        Anchor->>Robot: Query pelvis world pose
        Robot-->>Anchor: Position + Rotation
        Anchor->>Anchor: Apply rotation mode (FOLLOW_PRIM_SMOOTHED)
        Anchor->>Anchor: Slerp smoothing
        Anchor->>XRCore: set_world_transform_matrix()
        XRCore->>User: Update VR view to follow robot
    end

Loading

_{40 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[rwiltz]

@greptile

[greptile-apps]

Greptile Overview

Greptile Summary

This PR introduces Quest controller support for G1 humanoid locomotion-manipulation tasks and decouples the retargeter architecture from OpenXR-specific dependencies.

Key Changes:

• Moved TrackingTarget and motion controller enums from OpenXRDevice to DeviceBase, enabling retargeters to work across different device types
• Added Requirement enum to RetargeterBase allowing retargeters to declare required device features (hand tracking, head tracking, motion controllers)
• Implemented feature requirement aggregation in DeviceBase.__init__() to collect requirements from all retargeters
• Created two new retargeters for G1:
  • G1LowerBodyStandingMotionControllerRetargeter: Maps joystick inputs to locomotion commands with dynamic hip height adjustment
  • G1TriHandUpperBodyMotionControllerRetargeter: Maps trigger/squeeze/button inputs to hand joint angles
• Refactored all existing retargeters to use DeviceBase.TrackingTarget instead of OpenXRDevice.TrackingTarget
• Added retargeter_type field to all RetargeterCfg dataclasses for factory instantiation
• Moved config dataclasses to end of files for better organization

Issues Found:

• Missing super().__init__(cfg) call in G1LowerBodyStandingMotionControllerRetargeter.__init__() (already noted in previous comments)

Confidence Score: 4/5

• This PR is safe to merge with one constructor fix needed
• The refactoring is well-structured and follows good design principles by decoupling device-specific code. The new locomotion retargeter has a missing super().init() call that should be fixed, but otherwise the changes are clean and all existing retargeters were updated consistently
• source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/g1_motion_controller_locomotion.py needs the super().init() fix

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/device_base.py	5/5	Added enums for tracking targets and motion controller indices, plus teleoperation config fields and feature requirement aggregation in init
source/isaaclab/isaaclab/devices/retargeter_base.py	5/5	Added Requirement enum and GetRequirements() method for declaring required device features, plus retargeter_type field to RetargeterCfg
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/g1_motion_controller_locomotion.py	4/5	New retargeter for G1 lower body locomotion using motion controller joysticks; missing super().init() call in constructor
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/trihand/g1_upper_body_motion_ctrl_retargeter.py	5/5	New retargeter mapping motion controller inputs (trigger/squeeze/buttons) to G1 hand joints with proper requirements declaration

Sequence Diagram

sequenceDiagram
    participant Device as DeviceBase
    participant Retargeter as RetargeterBase
    participant G1Locomotion as G1MotionControllerRetargeter
    participant Quest as Quest Controller

    Note over Device: Device initialization
    Device->>Retargeter: __init__(retargeters)
    loop For each retargeter
        Device->>Retargeter: GetRequirements()
        Retargeter-->>Device: [MOTION_CONTROLLER, HAND_TRACKING, ...]
        Device->>Device: Aggregate into _required_features
    end

    Note over Device: Teleoperation loop
    Quest->>Device: Raw input data (controllers, hands)
    Device->>Device: _get_raw_data()
    Device->>Device: advance()
    
    loop For each retargeter
        Device->>G1Locomotion: retarget(raw_data)
        G1Locomotion->>G1Locomotion: Extract controller inputs
        G1Locomotion->>G1Locomotion: Scale thumbstick values
        G1Locomotion->>G1Locomotion: Update hip_height with dt
        G1Locomotion-->>Device: torch.Tensor([x, y, yaw, height])
    end
    
    Device->>Device: torch.cat(all_outputs)
    Device-->>Quest: Concatenated robot commands

Loading

_{11 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: Missing super().__init__(cfg) call - the base class constructor should be called to properly initialize self._sim_device

Suggested change

	def __init__(self, cfg: G1LowerBodyStandingMotionControllerRetargeterCfg):
	"""Initialize the retargeter."""
	self.cfg = cfg
	self._hip_height = cfg.hip_height
	def __init__(self, cfg: G1LowerBodyStandingMotionControllerRetargeterCfg):
	"""Initialize the retargeter."""
	super().__init__(cfg)
	self.cfg = cfg
	self._hip_height = cfg.hip_height

[rwiltz]

@greptile

[greptile-apps]

Greptile Overview

Greptile Summary

This PR refactors the teleoperation retargeting system to decouple retargeters from specific device implementations, and adds Quest motion controller support for G1 robot loco-manipulation tasks.

Key Changes:

• Moved TrackingTarget, MotionControllerDataRowIndex, and MotionControllerInputIndex enums from OpenXRDevice to DeviceBase for device-agnostic retargeting
• Added Requirement enum to RetargeterBase allowing retargeters to declare which features they need (hand tracking, head tracking, motion controllers)
• Introduced GetRequirements() method that devices call to aggregate feature requirements across all retargeters
• Added device_type and retargeter_type fields to config classes, removing the need for factory registration maps
• Implemented new motion controller retargeters for G1 lower body locomotion (g1_motion_controller_locomotion.py) and TriHand upper body control (g1_upper_body_motion_ctrl_retargeter.py)
• Added XR anchor "pinning" functionality to track robot prims in the scene
• Updated all existing retargeters to use DeviceBase.TrackingTarget enums instead of device-specific enums

Architecture Improvements:
The refactoring improves modularity by making retargeters work across different device types (OpenXR, ManusVive, future devices) and optimizes performance by only querying the tracking data actually needed by retargeters.

Confidence Score: 5/5

• This PR is safe to merge with no critical issues found
• The refactoring is well-structured with proper abstraction, all retargeters correctly call super().init(), enum usage is consistent throughout, the GetRequirements() implementation is sound, and the changes maintain backward compatibility while improving architecture
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/fourier/gr1t2_retargeter.py	5/5	Updated to use DeviceBase.TrackingTarget enum instead of OpenXRDevice.TrackingTarget for device-agnostic retargeting
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/g1_motion_controller_locomotion.py	5/5	Motion controller locomotion retargeter using thumbsticks for movement and hip height control with proper enum usage
source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/trihand/g1_upper_body_motion_ctrl_retargeter.py	5/5	Motion controller retargeter mapping trigger/squeeze inputs to TriHand finger joints with proper pose transformations

Sequence Diagram

sequenceDiagram
    participant User
    participant Device as DeviceBase/OpenXRDevice
    participant Retargeter as RetargeterBase
    participant Robot as Robot Controller

    Note over Device,Retargeter: Initialization Phase
    User->>Device: Create device with retargeters config
    Device->>Retargeter: Call GetRequirements() for each retargeter
    Retargeter-->>Device: Returns required features (HAND_TRACKING, MOTION_CONTROLLER, etc)
    Device->>Device: Aggregate required features into _required_features set

    Note over Device,Robot: Teleoperation Loop
    User->>Device: Call advance()
    Device->>Device: Call _get_raw_data()
    alt Hand Tracking Required
        Device->>Device: Query hand poses from XRCore
        Device->>Device: Store in data[TrackingTarget.HAND_LEFT/RIGHT]
    end
    alt Motion Controller Required
        Device->>Device: Query controller poses & inputs
        Device->>Device: Store in data[TrackingTarget.CONTROLLER_LEFT/RIGHT]
    end
    alt Head Tracking Required
        Device->>Device: Query head pose
        Device->>Device: Store in data[TrackingTarget.HEAD]
    end
    
    Device->>Retargeter: Call retarget(data) for each retargeter
    Retargeter->>Retargeter: Extract required tracking data using enums
    Retargeter->>Retargeter: Transform to robot-specific commands
    Retargeter-->>Device: Return torch.Tensor
    
    Device->>Device: Concatenate all retargeter outputs
    Device-->>Robot: Return combined command tensor
    Robot->>Robot: Apply commands to robot joints

Loading

_{6 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}