[docker, doc] feat: add Ascend NPU Dockerfiles and install guide#225
[docker, doc] feat: add Ascend NPU Dockerfiles and install guide#225Sky-Trigger wants to merge 10 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for Ascend Atlas A2 and A3 NPU environments by adding corresponding Dockerfiles (Dockerfile.a2.npu and Dockerfile.a3.npu) and updating the installation documentation (docs/start/install.md). Feedback on the changes highlights potential cross-building issues in both Dockerfiles due to the evaluation of environment variables and shell commands inside double quotes during the build phase. Additionally, the reviewer suggests improving the documented Docker launch commands for both NPU environments to mount the local repository, cache directories, and persistent volumes, preventing runtime failures and data loss.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| DEVICES="" | ||
| for i in $(seq 0 15); do | ||
| DEVICES="$DEVICES --device=/dev/davinci$i" | ||
| done | ||
|
|
||
| docker run -it --rm \ | ||
| --name verl_omni_16npu \ | ||
| --network host \ | ||
| --ipc host \ | ||
| $DEVICES \ | ||
| --device=/dev/davinci_manager \ | ||
| --device=/dev/devmm_svm \ | ||
| --device=/dev/hisi_hdc \ | ||
| -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro \ | ||
| -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \ | ||
| -v /mnt/data:/mnt/data \ | ||
| verl-omni:npu-a3 \ | ||
| bash |
There was a problem hiding this comment.
The current launch command for the Ascend Atlas A3 container does not mount the local repository or persistent volumes (such as the dataset directory, checkpoints, and Hugging Face cache). This leads to several issues:
- The container's default working directory (
/workspace) does not contain theexamples/folder, causing the dataset preparation and training commands to fail immediately withNo such file or directory. - Any prepared datasets or training checkpoints will be lost when the container exits.
- Hugging Face models will be re-downloaded on every run.
Aligning the NPU launch command with the CUDA launch command by mounting the repository, setting the working directory, and mounting persistent directories resolves these issues.
export REPO=/path/to/verl-omni # this repository
export WORKSPACE=$HOME # data, checkpoints, HF cache root
DEVICES=""
for i in $(seq 0 15); do
DEVICES="$DEVICES --device=/dev/davinci$i"
done
docker run -it --rm \
--name verl_omni_16npu \
--network host \
--ipc host \
$DEVICES \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v "$REPO:/workspace/verl-omni" \
-v "$WORKSPACE/data:$WORKSPACE/data" \
-v "$WORKSPACE/checkpoints:$WORKSPACE/checkpoints" \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
-e WORKSPACE="$WORKSPACE" \
-e HF_HOME=/root/.cache/huggingface \
-e WANDB_API_KEY="${WANDB_API_KEY:-}" \
-w /workspace/verl-omni \
verl-omni:npu-a3 \
bashThere was a problem hiding this comment.
For the NPU image, the verl-omni source is already included in the image, so I prefer not to mount the host repository by default to avoid overwriting the in-image source tree.
For changes inside the container, users can either keep the container running or save it with docker commit if they want to preserve the environment. Datasets, checkpoints, Hugging Face models, and other persistent files are expected to be stored in host-mounted directories, for example /mnt/data:/mnt/data in our setup.
Signed-off-by: Trigger <129651635+Sky-Trigger@users.noreply.github.com>
What does this PR do?
This PR adds Ascend NPU Docker support and updates the installation guide for running VeRL-Omni on Ascend NPUs.
Main changes:
docker/Dockerfile.a2.npudocker/Dockerfile.a3.npudocs/start/install.mdwith Ascend NPU Docker build and runtime instructionsChecklist Before Starting
[{modules}] {type}: {description}Test
The following checks have passed:
The Dockerfiles and installation instructions were also checked manually for command consistency.
CI unit tests are not added because this PR only adds Dockerfiles and documentation.
API and Usage Example
No Python API changes.
Example usage is documented in
docs/start/install.md.Design & Code Changes
This PR introduces two Ascend NPU Dockerfiles for different NPU targets and documents how to build and run them.
The Dockerfiles provide reproducible Ascend NPU environments, while the installation guide explains the expected build and runtime commands.
Checklist Before Submitting
pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always