[Question] Issue with Franka Teleoperation (Hand Tracking) using a Publicly Available URDF/USD Model

[tou-tatsuro]

I'm working on IsaacLab to collect imitation learning data by controlling a robot arm via hand tracking using GR00T-Teleop + MetaQuest3 + SteamVR.
I confirmed that the default Franka sample task works correctly with the following command:

isaaclab.bat -p D:\isaac\IsaacLab\scripts\environments\teleoperation\teleop_se3_agent.py --task Isaac-Stack-Cube-Franka-IK-Abs-v0 --teleop_device handtracking

My next goal is to use a publicly available Franka URDF model and get it to work with teleoperation (hand tracking), just like the sample.
However, the arm's movement doesn't follow my hand (the red dot) as expected. It moves slightly, but not correctly.

IsaacLab_teleop.mp4

I suspect the problem is not in the code but in the imported USD file (robot model) configuration.
I am unsure if additional settings are required after importing a public URDF/USD model to use it for teleoperation in IsaacLab, or if my URDF/USD import process is wrong.

Could you please provide a solution for correctly setting up a public URDF/USD model to enable teleoperation functionality similar to the sample task?
Thank you!

[tou-tatsuro]

We have confirmed two additional key findings regarding the instability of the IK control:

Model Isolation (HandTracking):
When using other standard USD models provided by Isaac Lab (e.g., Denso/CobottaPro900) and ensuring proper Joint configuration, teleoperation worked without any issues using the HandTracking device. This suggests the root cause is not in the HandTracking interface or the basic IK controller logic, but rather in the specific physical configuration of the USD model created from the URDF conversion.

Input Isolation (Keyboard):
We rebuilt the task using the more stable input source, --teleop_device keyboard, instead of --teleop_device handtracking. In this setup, the standard Isaac Lab USD models worked correctly, but the USD model created from the URDF conversion failed to operate properly under the same conditions.

[tou-tatsuro]

We have successfully identified the root cause of the unstable IK calculation after URDF import, and the issue is now resolved.

Root Cause:
The problem was caused by spurious (unnecessary) colliders being added to the model during the URDF-to-USD conversion process. These extra colliders likely introduced unintentional internal contacts (self-collisions) or high constraint forces that interfered with the PhysX Jacobian solver, leading to the erratic behavior.

Resolution:
The issue was resolved by manually deleting the redundant colliders from the converted USD model. The robot now performs stable IK tracking with the HandTracking device.

Thank you for your valuable input during the debugging process.