Skip to content

[docker, doc] feat: add Ascend NPU Dockerfiles and install guide#225

Open
Sky-Trigger wants to merge 10 commits into
verl-project:mainfrom
Sky-Trigger:add-npu-dockerfiles-install-guide
Open

[docker, doc] feat: add Ascend NPU Dockerfiles and install guide#225
Sky-Trigger wants to merge 10 commits into
verl-project:mainfrom
Sky-Trigger:add-npu-dockerfiles-install-guide

Conversation

@Sky-Trigger

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR adds Ascend NPU Docker support and updates the installation guide for running VeRL-Omni on Ascend NPUs.

Main changes:

  • Add docker/Dockerfile.a2.npu
  • Add docker/Dockerfile.a3.npu
  • Update docs/start/install.md with Ascend NPU Docker build and runtime instructions

Checklist Before Starting

Test

The following checks have passed:

pre-commit install
pre-commit run --all-files --show-diff-on-failure --color=always

The Dockerfiles and installation instructions were also checked manually for command consistency.

CI unit tests are not added because this PR only adds Dockerfiles and documentation.

API and Usage Example

No Python API changes.

Example usage is documented in docs/start/install.md.

Design & Code Changes

This PR introduces two Ascend NPU Dockerfiles for different NPU targets and documents how to build and run them.

The Dockerfiles provide reproducible Ascend NPU environments, while the installation guide explains the expected build and runtime commands.

Checklist Before Submitting

@Sky-Trigger Sky-Trigger requested a review from SamitHuang as a code owner July 2, 2026 06:11

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for Ascend Atlas A2 and A3 NPU environments by adding corresponding Dockerfiles (Dockerfile.a2.npu and Dockerfile.a3.npu) and updating the installation documentation (docs/start/install.md). Feedback on the changes highlights potential cross-building issues in both Dockerfiles due to the evaluation of environment variables and shell commands inside double quotes during the build phase. Additionally, the reviewer suggests improving the documented Docker launch commands for both NPU environments to mount the local repository, cache directories, and persistent volumes, preventing runtime failures and data loss.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread docker/Dockerfile.a2.npu Outdated
Comment thread docker/Dockerfile.a3.npu Outdated
Comment thread docs/start/install.md
Comment on lines +260 to +277
DEVICES=""
for i in $(seq 0 15); do
DEVICES="$DEVICES --device=/dev/davinci$i"
done

docker run -it --rm \
--name verl_omni_16npu \
--network host \
--ipc host \
$DEVICES \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
-v /mnt/data:/mnt/data \
verl-omni:npu-a3 \
bash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current launch command for the Ascend Atlas A3 container does not mount the local repository or persistent volumes (such as the dataset directory, checkpoints, and Hugging Face cache). This leads to several issues:

  1. The container's default working directory (/workspace) does not contain the examples/ folder, causing the dataset preparation and training commands to fail immediately with No such file or directory.
  2. Any prepared datasets or training checkpoints will be lost when the container exits.
  3. Hugging Face models will be re-downloaded on every run.

Aligning the NPU launch command with the CUDA launch command by mounting the repository, setting the working directory, and mounting persistent directories resolves these issues.

export REPO=/path/to/verl-omni          # this repository
export WORKSPACE=$HOME                  # data, checkpoints, HF cache root

DEVICES=""
for i in $(seq 0 15); do
  DEVICES="$DEVICES --device=/dev/davinci$i"
done

docker run -it --rm \
  --name verl_omni_16npu \
  --network host \
  --ipc host \
  $DEVICES \
  --device=/dev/davinci_manager \
  --device=/dev/devmm_svm \
  --device=/dev/hisi_hdc \
  -v "$REPO:/workspace/verl-omni" \
  -v "$WORKSPACE/data:$WORKSPACE/data" \
  -v "$WORKSPACE/checkpoints:$WORKSPACE/checkpoints" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
  -e WORKSPACE="$WORKSPACE" \
  -e HF_HOME=/root/.cache/huggingface \
  -e WANDB_API_KEY="${WANDB_API_KEY:-}" \
  -w /workspace/verl-omni \
  verl-omni:npu-a3 \
  bash

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the NPU image, the verl-omni source is already included in the image, so I prefer not to mount the host repository by default to avoid overwriting the in-image source tree.

For changes inside the container, users can either keep the container running or save it with docker commit if they want to preserve the environment. Datasets, checkpoints, Hugging Face models, and other persistent files are expected to be stored in host-mounted directories, for example /mnt/data:/mnt/data in our setup.

Comment thread docs/start/install.md
Signed-off-by: Trigger <129651635+Sky-Trigger@users.noreply.github.com>
Signed-off-by: Trigger <129651635+Sky-Trigger@users.noreply.github.com>
@SamitHuang SamitHuang requested review from panshaowu and wtomin July 2, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant