RCCL & Aorta documentation update

cvs/input/config_file/aorta/aorta_benchmark.yaml

@@ -68,7 +68,9 @@ analysis:
   gemm_script: scripts/gemm_analysis/run_tracelens_analysis.sh
   skip_if_exists: false
 
-# Expected results (tuned for host raw-trace parsing on MI300; use stricter values with TraceLens Excel reports)
+# Expected results: default thresholds for gfx942 (e.g. MI300). Change these as per your
+# testing config (GPU, node count, workload); see docs/reference/configuration-files/aorta.rst.
+# Tuned for host raw-trace parsing; use stricter values (e.g. min_compute_ratio 0.5+) with TraceLens Excel reports.
 expected_results:
   max_avg_iteration_ms: 12000
   min_compute_ratio: 0.01   # host parsing often 1–10%; use 0.5+ for TraceLens/Excel metrics

cvs/tests/benchmark/README.md

@@ -7,25 +7,40 @@ For details on arguments and their purpose, see the main README under the CVS pa
 1. **Config file:** Edit `cvs/input/config_file/aorta/aorta_benchmark.yaml` and set `aorta_path` to the absolute path of your Aorta repository. Do not leave the default `<changeme>`.
 2. **Cluster file:** Provide a valid cluster file (e.g. `input/cluster_file/cluster.json`) with node and user settings.
 
-Example from the ``cvs`` directory (repository root):
+Example from the CVS repo root (directory containing ``cvs`` and ``input``):
 
 ```bash
-(myenv) [user@host]~/cvs:(main)$ pwd
-/home/user/cvs/cvs
-(myenv) [user@host]~/cvs:(main)$ pytest -vvv --log-file=/tmp/test.log -s ./tests/benchmark/test_aorta.py --cluster_file input/cluster_file/cluster.json --config_file input/config_file/aorta/aorta_benchmark.yaml --html=/var/www/html/cvs/aorta.html --capture=tee-sys --self-contained-html
+cvs run test_aorta \
+    --cluster_file input/cluster_file/cluster.json \
+    --config_file input/config_file/aorta/aorta_benchmark.yaml \
+    -v --log-cli-level=INFO
 ```
 
-With verbose logging:
+With HTML report and full logging (see also `docs/reference/configuration-files/aorta.rst`):
 
 ```bash
-pytest ./tests/benchmark/test_aorta.py --cluster_file input/cluster_file/cluster.json --config_file input/config_file/aorta/aorta_benchmark.yaml -v --log-cli-level=INFO
+cvs run test_aorta \
+    --cluster_file input/cluster_file/cluster.json \
+    --config_file input/config_file/aorta/aorta_benchmark.yaml \
+    --html=/var/www/html/cvs/aorta.html --capture=tee-sys --self-contained-html \
+    --log-file=/tmp/test.log -vvv -s
 ```
 
 # Config and expected results
 
 Configuration options (paths, Docker image, RCCL build, environment, analysis, and expected-result thresholds) are documented in the reference docs under `docs/reference/configuration-files/aorta.rst`. Key settings:
 
 - **aorta_path** – Path to Aorta repo on the host (bind-mounted into the container).
-- **expected_results** – Optional thresholds for validation: `max_avg_iteration_ms`, `min_compute_ratio`, `min_overlap_ratio`, `max_time_variance_ratio`.
+- **expected_results** – Validation thresholds; the test fails if any is not met:
+  - **max_avg_iteration_ms** – Maximum acceptable average iteration time (ms).
+  - **min_compute_ratio** – Minimum compute ratio (compute time / total iteration time).
+  - **min_overlap_ratio** – Minimum compute–communication overlap ratio.
+  - **max_time_variance_ratio** – Maximum iteration time variance (e.g. std/mean) across ranks.
 
-The test parses results from host-side trace parsing (raw PyTorch profiler traces or TraceLens reports when present) and fails if any configured threshold is not met. Artifacts include training logs, profiler traces, and an optional TraceLens analysis directory when enabled.
+The values in `aorta_benchmark.yaml` are **default thresholds for gfx942** (e.g. MI300) and should be **changed as per your testing config** (GPU, node count, workload). The test parses results from host-side trace parsing (raw PyTorch profiler traces or TraceLens reports when present). Artifacts include training logs, profiler traces, and an optional TraceLens analysis directory when enabled.
+
+# Note for users: where to put Aorta (`aorta_path`)
+
+**Prefer a path on local or scratch storage** (e.g. `/scratch/...`) for `aorta_path` when running this benchmark.
+
+If `aorta_path` points to a directory on **NFS** (for example your home directory under `/home`), the container may fail with **Permission denied** when creating the `artifacts/` output directory. Many NFS exports use *root_squash*, so the process running as root inside the container is treated as a non-privileged user on the NFS server and cannot create directories in your tree. Using a path on local disk or on a non–root-squashed filesystem (e.g. `/scratch`) avoids this. No code changes are required—use a suitable path in `aorta_benchmark.yaml` for `aorta_path`.

cvs/tests/rccl/README.md

@@ -1,45 +1,158 @@
-RCCL (ROCm Communication Collectives Library) tests are comprehensive benchmarks that validate distributed GPU communication performance across AMD GPU clusters. These tests ensure optimal performance for AI training, HPC workloads, and distributed computing
+# RCCL Performance Tests
 
-# How to run the tests
+RCCL (ROCm Communication Collectives Library) tests validate distributed GPU communication performance across AMD GPU clusters. These tests run RCCL collectives through CVS, compare results against optional thresholds, and generate HTML artifacts for review.
 
-This Pytest script can be run in the following fashion (for the details on arguments and their purpose, please refer the main README under the CVS parent folder
+## Supported test suites
 
+CVS currently provides these RCCL suites:
+
+1. `rccl_multinode_cvs`  
+   Multinode sweep across collective, algorithm, protocol, queue-pair scaling (`qp_scale`), and PXN toggle (`nccl_pxn_disable`).
+
+2. `rccl_multinode_default_cvs`  
+   Multinode collectives using RCCL default algorithm/protocol behavior.
+
+3. `rccl_singlenode_cvs`  
+   Single-node collective performance and threshold checks.
+
+4. `rccl_heatmap_cvs`  
+   Multinode matrix sweep over collective, `gpu_count_list`, `data_type_list`, and `channel_config_list`, then builds a heatmap against a golden reference JSON.
+
+All suites also collect host/network info and validate that firewall services are not blocking the run.
+
+## Collective tests covered
+
+These RCCL binaries can be selected through `rccl_collective`:
+
+- `all_reduce_perf`: all ranks reduce and receive the reduced value.
+- `all_gather_perf`: each rank gathers data from all other ranks.
+- `scatter_perf`: one rank scatters data to all ranks.
+- `gather_perf`: all ranks gather data to one root rank.
+- `reduce_scatter_perf`: reduction followed by shard distribution.
+- `sendrecv_perf`: point-to-point send/receive performance.
+- `alltoall_perf`: all ranks exchange equal-sized payloads.
+- `alltoallv_perf`: all ranks exchange variable-sized payloads.
+- `broadcast_perf`: one rank broadcasts data to all ranks.
+
+## Prerequisites
+
+1. **Cluster file**: Provide a valid cluster file (for example `input/cluster_file/cluster.json`) with node definitions, username, key path, and reachable node IPs.
+2. **RCCL test binaries**: Install `rccl-tests` and ensure paths in config are correct (`rccl_dir`, `rccl_tests_dir`, `rccl_path_var`).
+3. **MPI/ROCm paths**: Ensure `mpi_dir`, `mpi_path_var`, and `rocm_path_var` in config match your environment.
+4. **Passwordless SSH**: Nodes should be able to run distributed MPI launch commands without interactive prompts.
+5. **Expected results**: Update `results` thresholds for your hardware and cluster size (defaults are sample values only).
+
+## How to run with CVS
+
+Run from the CVS repo root (directory containing `cvs` and `input`):
+
+### List available RCCL suites
+
+```bash
+cvs list rccl_multinode_cvs
+cvs list rccl_multinode_default_cvs
+cvs list rccl_singlenode_cvs
+cvs list rccl_heatmap_cvs
 ```
-(myenv) [user@host]~/cvs:(main)$
-(myenv) [user@host]~/cvs:(main)$pwd
-/home/user/cvs/cvs
-(myenv) [user@host]~/cvs:(main)$pytest -vvv --log-file=/tmp/test.log -s ./tests/rccl/rccl_multinode_cvs.py --cluster_file input/cluster_file/cluster.json  --config_file input/config_file/rccl/rccl_config.json --html=/var/www/html/cvs/rccl.html --capture=tee-sys --self-contained-html
 
+### Multinode parameter sweep
+
+```bash
+cvs run rccl_multinode_cvs \
+  --cluster_file input/cluster_file/cluster.json \
+  --config_file input/config_file/rccl/rccl_config.json \
+  --html=/var/www/html/cvs/rccl_multinode.html --capture=tee-sys --self-contained-html \
+  --log-file=/tmp/rccl_multinode.log -vvv -s
 ```
 
-# Additional RCCL env setting to use AINIC RCCL net plugin
+### Multinode (RCCL defaults)
 
-## Overview
+```bash
+cvs run rccl_multinode_default_cvs \
+  --cluster_file input/cluster_file/cluster.json \
+  --config_file input/config_file/rccl/rccl_config.json \
+  --html=/var/www/html/cvs/rccl_multinode_default.html --capture=tee-sys --self-contained-html \
+  --log-file=/tmp/rccl_multinode_default.log -vvv -s
+```
 
-The `ainic_env_script.sh` script (located at `input/config_file/rccl/ainic_env_script.sh`) sets recommended environment variables for optimal performance when using RCCL (ROCm Communication Collectives Library) with AMD AI Network Interface Cards (AINIC). These settings are designed to enhance communication efficiency in multinode GPU clusters.
+### Single-node run
 
-This script is part of the CVS (Cluster Validation Suite) and is used to configure the environment for RCCL-based collective operations, such as all-reduce, all-gather, etc.
+```bash
+cvs run rccl_singlenode_cvs \
+  --cluster_file input/cluster_file/cluster.json \
+  --config_file input/config_file/rccl/single_node_mi355_rccl.json \
+  --html=/var/www/html/cvs/rccl_singlenode.html --capture=tee-sys --self-contained-html \
+  --log-file=/tmp/rccl_singlenode.log -vvv -s
+```
 
-## Prerequisites
+### Heatmap run
+
+```bash
+cvs run rccl_heatmap_cvs \
+  --cluster_file input/cluster_file/cluster.json \
+  --config_file input/config_file/rccl/rccl_config.json \
+  --html=/var/www/html/cvs/rccl_heatmap.html --capture=tee-sys --self-contained-html \
+  --log-file=/tmp/rccl_heatmap.log -vvv -s
+```
+
+## Parameter quick reference
+
+Edit `input/config_file/rccl/rccl_config.json` (and `single_node_mi355_rccl.json` for single-node baseline) before running:
 
-- AMD ANP (AMD Network Plugin) built/installed
+- **Scale and topology**: `no_of_nodes`, `no_of_global_ranks`, `no_of_local_ranks`, `ranks_per_node`.
+- **Sweep controls**:
+  - `rccl_collective`, `rccl_algo`, `rccl_protocol`, `qp_scale`, `nccl_pxn_disable`
+  - `gpu_count_list`, `data_type_list`, `channel_config_list` (used by `rccl_heatmap_cvs`)
+- **Message sweep**: `start_msg_size`, `end_msg_size`, `step_function`, `warmup_iterations`, `no_of_iterations`, `no_of_cycles`.
+- **Network and transport**: `ib_hca_list`, `net_dev_list`, `oob_port`, `gid_index`, `nccl_socket_ifname`, `ucx_tls`, `mpi_pml`.
+- **Validation controls**: `verify_bus_bw`, `verify_bw_dip`, `verify_lat_dip`, `results`.
+- **Artifacts/reference**: `rccl_result_file`, `golden_reference_json_file`, `output_dir`, `heatmap_title`.
 
-## Usage
+Additional tuning parameters used in multinode runs:
+
+- **Runtime and verbosity**: `threads_per_gpu`, `debug_level`, `cluster_snapshot_debug`.
+- **IB/UCX tuning**: `nccl_ib_timeout`, `ib_rx_queue_len`, `ucx_tls`, `hcoll_enable_mcast_all`.
+- **RCCL/NCCL behavior**: `nccl_cumem_enable`, `nccl_ib_sl`, `nccl_ib_tc`, `nccl_ib_split_data_on_qps`, `nccl_net_plugin`.
+- **Heatmap-specific validation**: `nic_model` and `golden_reference_json_file`.
+
+Placeholder handling:
+
+- `{user-id}` and other supported placeholders in cluster/config files are resolved at runtime by CVS.
+- You can keep placeholders or replace them with explicit absolute paths.
+
+Expected-results format (`results`):
+
+- Thresholds are keyed by collective and message size in bytes.
+- Typical structure:
+
+```json
+"results": {
+  "all_reduce_perf": {
+    "bus_bw": {
+      "8589934592": "330.00",
+      "17179869184": "350.00"
+    }
+  }
+}
+```
 
-1. **Edit the Script**: Update the `ANP_HOME_DIR` variable in `ainic_env_script.sh` to point to your AMD ANP installation directory.
+## Optional AINIC/ANP net-plugin setup
 
-   ```bash
-   export ANP_HOME_DIR=/path/to/your/amd-anp/installation
-   ```
+If using AINIC + ANP plugin:
 
-   Replace `/path/to/your/amd-anp/installation` with the actual absolute path.
+1. Ensure AMD ANP is installed and accessible on all target nodes.
+2. Edit `input/config_file/rccl/ainic_env_script.sh` and set `ANP_HOME_DIR` to your absolute ANP install path.
+3. Set `env_source_script` in your RCCL JSON config to that script path.
+4. Run RCCL with `cvs run ...`; CVS sources the script automatically before test execution.
 
-2. **Configure rccl_config.json**: Ensure the `env_source_script` field in `rccl_config.json` points to the correct path of `ainic_env_script.sh`. For example:
+If you are not using AINIC/ANP, set `env_source_script` to your standard environment setup script (or `"None"` to skip script sourcing).
 
-   ```json
-   "env_source_script": "/path/to/cvs/input/config_file/rccl/ainic_env_script.sh"
-   ```
+## Expected output artifacts
 
-   CVS will automatically source this script before running the tests.
+- Pytest HTML summary from `--html`.
+- RCCL perf graph HTML under `/tmp/rccl_perf_report_*.html` or `/tmp/rccl_singlenode_perf_report_*.html`.
+- Heatmap output under `/tmp/rccl_heatmap_*.html` plus JSON result files.
+- Optional copied heatmap artifacts under `output_dir` (from config) for easier sharing.
+- Graph/heatmap files under `/tmp` can be copied to your web server path (for example `/var/www/html/cvs`) for browser viewing.
 
-3. **Run RCCL Tests**: Use the pytest command as described in the "How to run the tests" section. The environment variables will be set automatically from the script.
+The run passes when command execution succeeds and configured validations (`results`, dip checks, and optional bus bandwidth checks) are met.

docs/how-to/run-cvs-tests.rst

@@ -361,17 +361,58 @@ You can list all available RCCL multinode test cases using the CLI:
     - test_singlenode_perf
     - test_gen_graph
 
-You can run all RCCL multinode tests using the CVS CLI:
+.. code:: bash
+
+  cvs list rccl_multinode_default_cvs
+
+.. code:: text
+
+  Available tests in rccl_multinode_default_cvs:
+    - test_collect_hostinfo
+    - test_collect_networkinfo
+    - test_disable_firewall
+    - test_rccl_perf
+    - test_gen_graph
+
+.. code:: bash
+
+  cvs list rccl_heatmap_cvs
+
+.. code:: text
+
+  Available tests in rccl_heatmap_cvs:
+    - test_collect_hostinfo
+    - test_collect_networkinfo
+    - test_disable_firewall
+    - test_rccl_perf
+    - test_gen_graph
+    - test_gen_heatmap
+
+Use these scripts to start RCCL tests with CVS:
+
+1. Run RCCL multinode parameter sweep:
+
+.. code:: bash
+
+  cvs run rccl_multinode_cvs --cluster_file input/cluster_file/cluster.json --config_file input/config_file/rccl/rccl_config.json --html=/var/www/html/cvs/rccl_multinode.html --capture=tee-sys --self-contained-html --log-file=/tmp/rccl_multinode.log -vvv -s
+
+2. Run RCCL multinode with RCCL defaults:
 
 .. code:: bash
 
-  cvs run rccl_multinode_cvs --cluster_file input/cluster_file/cluster.json --config_file input/config_file/rccl/rccl_multinode_config.json --html=/var/www/html/cvs/rccl_multinode.html --capture=tee-sys --self-contained-html --log-file=/tmp/rccl_multinode.log -vvv -s
+  cvs run rccl_multinode_default_cvs --cluster_file input/cluster_file/cluster.json --config_file input/config_file/rccl/rccl_config.json --html=/var/www/html/cvs/rccl_multinode_default.html --capture=tee-sys --self-contained-html --log-file=/tmp/rccl_multinode_default.log -vvv -s
 
-You can run all RCCL singlenode tests using the CVS CLI:
+3. Run RCCL single-node:
 
 .. code:: bash
 
-  cvs run rccl_singlenode_cvs --cluster_file input/cluster_file/cluster.json --config_file input/config_file/rccl/rccl_singlenode_config.json --html=/var/www/html/cvs/rccl_singlenode.html --capture=tee-sys --self-contained-html --log-file=/tmp/rccl_singlenode.log -vvv -s
+  cvs run rccl_singlenode_cvs --cluster_file input/cluster_file/cluster.json --config_file input/config_file/rccl/single_node_mi355_rccl.json --html=/var/www/html/cvs/rccl_singlenode.html --capture=tee-sys --self-contained-html --log-file=/tmp/rccl_singlenode.log -vvv -s
+
+4. Run RCCL heatmap:
+
+.. code:: bash
+
+  cvs run rccl_heatmap_cvs --cluster_file input/cluster_file/cluster.json --config_file input/config_file/rccl/rccl_config.json --html=/var/www/html/cvs/rccl_heatmap.html --capture=tee-sys --self-contained-html --log-file=/tmp/rccl_heatmap.log -vvv -s
 
 JAX training test scripts
 You can run all JAX llama training tests using the CVS CLI:
@@ -469,27 +510,6 @@ You can list all available Megatron training test cases using the CLI:
     - test_cleanup_stale_containers
     - test_launch_megatron_containers
     - test_llama_3_1_fp8_single_node
-.. code:: bash
-
-  cvs list rccl_multinode_default_cvs
-
-.. code:: text
-
-  Available tests in rccl_multinode_default_cvs:
-    - test_collect_hostinfo
-    - test_collect_networkinfo
-    - test_disable_firewall
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_rccl_perf
-    - test_gen_graph
-
 Use these scripts to run the Megatron tests.
 
 Single Node 8b MI3XX