LearningHumanoidArmMotion-RAL2025-Code

This repository is an open-sourced code for the following paper presented at the 2025 IEEE Robotics and Automation Letters (RA-L).

Title: Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning

Paper Link: https://ieeexplore.ieee.org/abstract/document/11177014
Video Link: https://youtu.be/BNYML7QZyWQ
Project Website: https://hojae-io.github.io/LearningHumanoidArmMotion-RAL2025-Website

Table of Contents

1. Getting Started
   • Installation
   • Code Structure
2. User Manual
• Generate CusADi Function (Optional)
• Train the policy in IsaacLab
• Play the policy in IsaacLab
3. System Info
4. Troubleshooting
5. Acknowledgement

Getting Started

Installation

1. Git clone the repository.

2. Set up the git submodules.

   git submodule init
   git submodule update

   This should set up IsaacLab, cusadi, rsl_rl submodules (branch: dev_RAL2025).

3. Go to the IsaacLab directory.
   Follow the instructions to install IsaacSim and IsaacLab Anaconda virtual environment.
   

   cd IsaacLab
   conda create -n ral2025 python=3.10
   conda activate ral2025
   pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
   pip install "isaacsim[all,extscache]==4.5.0" --extra-index-url https://pypi.nvidia.com
   ./isaaclab.sh --install none

   💡 Note
   You must use the customized IsaacLab, not the official version.
   IsaacSim version: 4.5.0
   Python version: 3.10

4. Go to the rsl_rl directory.
   Check if rsl_rl is already installed during IsaacLab installation by running pip show rsl_rl; if so, uninstall it by running pip uninstall rsl_rl.
   Run the following command to install customized rsl_rl.

   cd rsl_rl
   pip install -e .

5. Go to the cusadi directory.
   

   cd cusadi
   pip install -e .

6. Install required libraries by running pip install -r requirements.txt at root directory.

   pip install -r requirements.txt

Important

By now, you should be able to run

pip show isaacsim
pip show IsaacLab
pip show rsl_rl
pip show cusadi

And all these packages should be located in $HOME/anaconda3/envs/ral2025/lib/python3.10/site-packages and their Editable project located in LearningHumanoidArmMotion-RAL2025-Code/

Code Structure

Directory	Description
extensions/	Core algorithms (MIT Humanoid env, CAM computation, etc.)
IsaacLab/	RL wrappers (observation, action, reward, termination managers etc.)
cusadi/	GPU parallelization of CasADi functions (pinocchio etc.)
rsl_rl/	RL algorithms (PPO etc.)
scripts/	Entry points (play_modular.py, train_modular.py etc.)
logs/	Saved trained models
resources/	MIT Humanoid URDF

User Manual

Activate Anaconda virtual environment:

conda activate ral2025
cd LearningHumanoidArmMotion-RAL2025-Code

0. Generate CusADi Function (Optional)

Necessary CusADi functions for running this code are already generated. So you don't need to run this section. But if you want to generate custom CusADi functions, follow the instructions below.

1. Install the CasADi extension of Pinocchio by following the instructions here or by running conda install pinocchio -c conda-forge.
   You should be able to run from pinocchio import casadi as cpin from the script.

2. Run

   python ./extensions/humanoid/dynamics/forward_kinematics.py

   It will generate CasADi functions in extensions/humanoid/dynamics/casadi_fns directory.

3. Copy manually these CasADi functions to cusadi/src/casadi_functions

4. Run

   cd cusadi
   python run_codegen.py
   

   It will generate *.cu files in cusadi/codegen. Now we are ready to use CusADi functions!

1. Train the policy in IsaacLab

python scripts/train_modular.py --task=humanoid_full_modular --headless --max_iterations=2000

The trained models will be saved in logs/rsl_rl/ directory.

For the single-agent controller, run

python scripts/train.py --task=humanoid_vanilla --headless --max_iterations=2000
python scripts/train.py --task=humanoid_full_vanilla --headless --max_iterations=2000

2. Play the policy in IsaacLab

python scripts/play_modular.py --task=humanoid_full_modular_play --num_envs=3 --device cpu

You can control velocity command by L / H / K / J / U / I keys.
By setting RECORD=True in play_modular.py and pressing W key, you can save recorded videos and plots in /logs/rsl_rl/[YOUR MODEL]/analysis/ directory.
By pressing F key, you can push the robot (i.e., apply external perturbation).

For the single-agent controller, run

python scripts/play.py --task=humanoid_vanilla_play --headless --max_iterations=2000
python scripts/play.py --task=humanoid_full_vanilla_play --headless --max_iterations=2000

By training with 2000 iterations, you should be able to see the motions as below:

Walking Forward / Backward

Push Recovery

3. Deploy the policy to the robot hardware

This repository does not include a code stack for deploying a policy to MIT Humanoid hardware. Please check the Cheetah-Software for our lab's hardware code stack.

To deploy the trained policy, you would need to set EXPORT_POLICY=TRUE in the scripts/play_modular.py script. Then you would get a *.onnx file to run on C++ code.

System Info

Operating System: Ubuntu 22.04
GPU: Geforce 3090 / Geforce 4070 Ti

Troubleshooting

1. If you have any issues with numpy version conflict, consider downgrading to version 1.23.1 by pip install numpy==1.23.1
2. If you have ModuleNotFoundError: No module named 'extensions' error, add ~/anaconda3/envs/ral2025/etc/conda/activate.d/env_vars.sh with the following contents:

   export PATH="/opt/openrobots/bin:$PATH"
   export PKG_CONFIG_PATH="/opt/openrobots/lib/pkgconfig:$PKG_CONFIG_PATH"
   export LD_LIBRARY_PATH="/opt/openrobots/lib:$LD_LIBRARY_PATH"
   export PYTHONPATH="$HOME/anaconda3/envs/ral2025/lib/python3.10/site-packages:$HOME/git/LearningHumanoidArmMotion-RAL2025-Code:/opt/openrobots/lib/python3.10/site-packages:$PYTHONPATH"
   export CMAKE_PREFIX_PATH="/opt/openrobots:$CMAKE_PREFIX_PATH"

Acknowledgement

We would appreciate it if you would cite it in academic publications:

@article{lee2025learning,
  title={Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning},
  author={Lee, Ho Jae and Jeon, Se Hwan and Kim, Sangbae},
  journal={IEEE Robotics and Automation Letters},
  year={2025},
  publisher={IEEE}
}