【Question】Demo Augmentation Failing Due to Coordinate System Mismatch for Robot with Offset Arm Base

[1804sasami]

Environment

• Isaac Lab Version: 2.2.0
• Isaac Sim Version: 5.0.0
• OS: Windows
• Python Version: 3.11
• Robot Configuration: 6-DOF robot with arm base offset from robot root

Problem Description

I'm trying to use generate_dataset.py to augment demonstrations for a custom robot stacking task, but the generated trajectories have large position errors (~88cm) and the IK solver fails to converge properly.
The demo file was originally recorded using teleoperation, and I'm attempting to augment it using the mimic environment.

Observed Behavior

Robot Configuration Details

My robot has the following structure:

• Robot root position (world): [1.5, 0.0, -1.0]
• Arm base position (world): [0.239, -0.3, 0.898]

The robot has a mobile base, and the manipulator arm is mounted on top with a significant offset from the robot root.

Environment Configuration

Action Configuration

self.actions.arm_action = DifferentialInverseKinematicsActionCfg(
    asset_name="robot",
    joint_names=["arm_joint[1-6]"],
    body_name="end_effector_link",
    controller=DifferentialIKControllerCfg(
        command_type="pose",
        use_relative_mode=False,  # Absolute mode
        ik_method="dls",
        ik_params={"lambda_val": 0.05},
    ),
    body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(
        pos=[0.0, 0.0, 0.3]  # TCP offset
    ),
)

Observation Configuration

self.scene.ee_frame = FrameTransformerCfg(
    prim_path="{ENV_REGEX_NS}/Robot/footprint",
    target_frames=[
        FrameTransformerCfg.FrameCfg(
            prim_path="{ENV_REGEX_NS}/Robot/end_effector_link",
            name="end_effector",
            offset=OffsetCfg(pos=[0.0, 0.0, 0.3]),
        ),
    ],
)

# observations.py
def ee_frame_pos(env, ee_frame_cfg):
    ee_frame = env.scene[ee_frame_cfg.name]
    ee_frame_pos = ee_frame.data.target_pos_w[:, 0, :] - env.scene.env_origins[:, 0:3]
    return ee_frame_pos

Note: env.scene.env_origins is [0, 0, 0]

Root Cause Analysis

The coordinate system mismatch appears to be caused by:

1. Observation data: Uses FrameTransformer from footprint to end-effector

   • Results in coordinates relative to footprint frame: [0.906, 0.299, 0.014]

2. Action data: World coordinates from teleoperation device

   • Results in world coordinates: [1.110, 0.344, 0.008]

3. Coordinate offset: The ~20.4cm difference appears to correspond to the offset between footprint and the robot root or arm base

Questions

1. For robots where the arm base is offset from the robot root, what should be used as the source prim_path for FrameTransformer?

   • Robot root (e.g., {ENV_REGEX_NS}/Robot/base)?
   • Arm base link (e.g., {ENV_REGEX_NS}/Robot/arm_base_link)?

2. How should coordinate systems be aligned between teleoperation recording and mimic augmentation for robots with this structure?

3. What is the correct workflow for ensuring observation and action coordinates are in the same frame during:

   • Teleoperation demo recording
   • Demo augmentation with generate_dataset.py

4. Should env.scene.env_origins be set differently for robots where the arm base is significantly offset from the robot root?

Expected Behavior

The augmented demonstrations should closely follow the original trajectory with position errors <5cm, and IK should converge without singularity warnings.

---

Additional Verification with Franka Panda Robot

Test Setup

• Environment: Custom environment (same as above) with Franka Panda replacing the custom robot
• Configuration: All task-related objects (robot, table, etc.) were offset by the same amount from the origin

Observed Behavior

• Result: Task success rate decreased when all objects were uniformly offset from the origin
• Implication: This suggests that coordinate frame handling may have underlying issues even with standard robots like Franka, when positioned away from the world origin

Code Modification

• Location: task_space_actions.py in the process_actions method
• Change: Modified the coordinate transformation logic to compensate for the offset of objects from the origin
• Purpose: To ensure that the object offset does not affect the action processing and IK computation

This verification indicates that the coordinate system issue may not be specific to custom robots with offset arm bases, but could be a more general problem with how the framework handles environments where task components are positioned away from the world origin.

---

Any guidance on the correct coordinate frame setup for robots with offset arm bases would be greatly appreciated!