MT-MARL-SG

🔍 Overview

This repository supports our paper: Multi-Task Multi-Agent Reinforcement Learning via Skill Graphs. The proposed skill graph offers three key benefits:

• Effective handling of unrelated tasks in MT-MARL.
• Improved knowledge transfer over standard hierarchical methods.
• Decoupled training of skill graphs and low-level policies for better generalization.

📋 Requirements

• Operating System: Ubuntu 20.04 (recommended)
• Python: 3.8 or later
• GPU: CUDA-compatible GPU

📁 Project Structure

MTRL-SG/
├── mt_marl_sg/                          # Core framework implementation
│ ├── algorithm/                         # Reinforcement learning algorithms
│ ├── cfg/                               # Training configurations
│ ├── eval/                              # Evaluation scripts
│ ├── skill_graph/                       # Skill graph modules
│ ├── train/                             # Training scripts
│ └── requirements.txt                   # Module-level dependencies
├── cus_gym/                             # Custom Gym environments
│ ├── gym/                               # Gym environment implementation
│ └── ...
└── README.md                            # This document

⚙️ Installation

1. Create Virtual Environment

Create a new virtual environment using the following command:

   conda create -n xxx(your env name) python=3.8 # or > 3.8

2. Install Dependencies

Navigate to the mt_marl_sg folder and run the following command to install dependencies:

   pip install -r requirements.txt

If you encounter any other missing packages during the process, feel free to install them manually using pip install xxx

3. Install PyTorch

Visit the PyTorch official website and install the GPU version of PyTorch according to your system configuration, such as

   pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121

4. Install Custom Environment

Navigate to the cus_gym folder and run the following command to install the MT-MARL environment:

   pip install -e .

For more information about the environment, please refer to Gym-PPS

5. Set Environment Variables

To make the project accessible within your Python environment, add the following path to your ~/.bashrc:

echo 'export PYTHONPATH="$PYTHONPATH:/path/to/your/mt_marl_sg/"' >> ~/.bashrc
source ~/.bashrc

6. Compile the C++ Library

Several environment functions are implemented in C++ to improve performance. To build the C++ extension, execute the following:

cd cus_gym/gym/envs/customized_envs/envs_cplus
chmod +x build.sh
./build.sh

🚀 Usage

1. Train Low-Level Skills (Flocking Task)

To train low-level skills using the flocking task, navigate to the training script directory and run the following command:

cd mt_marl_sg/train
python ./train_flocking.py

The corresponding training configuration is located in cfg/flocking_cfg.py

2. Skill Evaluation

After training, you can evaluate the learned skills by modifying the experiment identifier in your evaluation script. Replace the curr_run variable with the directory name corresponding to your experiment:

curr_run = '2025-01-19-15-58-03'  # Replace with your experiment timestamp

then, run

python ./eval_flocking.py

3. Train High-Level Skill Graph

To train the high-level skill graph over the learned low-level skills, navigate to the skill graph training directory and run:

cd skill_graph
python ./train.py

4. Skill Inference

Once training is completed, you can perform inference by updating the experiment path in inference.py. Replace the your_experiment_name placeholder with your actual run directory:

log_dir = os.path.join(current_dir, 'runs', 'your_experiment_name')

then, run

python ./inference.py

📝 Getting Help

Please open an Issue if you have some trouble and advice.

📝 Citation

If you find this code useful in your research, please consider citing our paper:

@article{zhu2025mtmarlsg,
  author={Zhu, Guobin and Zhou, Rui and Ji, Wenkang and Zhang, Hongyin and Wang, Donglin and Zhao, Shiyu},
  journal={IEEE Robotics and Automation Letters}, 
  title={Multi-Task Multi-Agent Reinforcement Learning via Skill Graphs}, 
  volume={10},
  number={9},
  pages={8650-8657},
  year={2025},
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Note: This documentation is continuously being updated. For the latest information, please check the repository regularly.