Port install_rvs to the orch fixture#174
Merged
Merged
Conversation
Switch cvs/tests/health/install/install_rvs.py from the legacy phdl/shdl Pssh handles to the shared orch fixture in cvs/tests/conftest.py, mirroring rvs_cvs.py. 21 phdl.exec / hdl.exec callsites become orch.exec. Drop the head-only-on-NFS dispatch (config_dict['nfs_install'] is True). On the canonical mi300_health_config.json the JSON value is the string "True" while the test compared with Python `is True`, so the branch was dead in shipped configs and head-only behavior was never engaged. Custom configs that relied on JSON boolean true to install from a single head node will now install on all nodes in parallel; this matches what cvs run install_rvs already did via parallel SSH. Layer-3 doc updates so install_rvs is no longer omitted from the "which suites consume the orchestrator" claims: - docs/_includes/orchestrator-scope.rst: scope note now names rvs_cvs and install_rvs. - docs/reference/configuration-files/cluster-file.rst: L42 (container backend scope) and L107 (orchestrator key scope) updated. - cvs/tests/health/README.md: Container-mode example pins both suites. - docs/how-to/run-cvs-tests.rst: container-mode note pins both suites. Verified via cvs run install_rvs in baremetal Tier-2 on a single MI300X node (a06u01): pytest exit 0, "RVS installation and verification completed successfully" emitted, auto-bundle written. Container Tier-2 not exercised in this commit because the orch fixture's container backend has two pre-existing regressions independent of this port: MultiProcessPssh.exec drops the `detailed=` kwarg (PR #136 backward-compat regression) which blocks orch.setup_containers(), and cvs/core/runtimes/docker.py exec does not wrap commands in `bash -c` so shell chains like `cd X; cmake ...` get split between host and container. Both will be fixed in follow-up PRs.
atnair-amd
added a commit
that referenced
this pull request
May 19, 2026
…ild [AIMVT-168] (#175) * fix(install_rvs): single-command strings for container-mode source build [AIMVT-168] In container orchestrator mode, the source-build path passes multi-command shell strings like 'cd X; cmd Y' to orch.exec. The docker runtime expands that to 'sudo docker exec <name> cd X; cmd Y'; the host shell parses ';' first, so 'docker exec ... cd X' runs and fails with 'OCI runtime exec failed: "cd": executable file not found in $PATH' ('cd' is a shell builtin, not an executable), and 'cmd Y' then runs on the host instead of inside the container. Follow-up to AIMVT-168 / PR #174 (port-install-rvs-to-orch): the original port wired install_rvs onto the orch fixture; this fix unbreaks the container-mode source-build path that the port surfaced. Rewrite each affected orch.exec call as a single command: git clone <url> <dest> (was: cd X; git clone Y) cmake -S <src> -B <build> ... (was: cd X; cmake -B ./build) cmake --build <build> --parallel (was: cd X; make -j$(nproc)) cmake --build <build> --target package ... (was: cd X; make -j$(nproc) package) sudo cmake --install <build> (was: cd X; sudo make install; echo "RVS_INSTALL_STATUS:$?") Also add -DRVS_BUILD_TESTS=OFF: the bundled googletest in release/rocm-rel-7.2 declares cmake_minimum_required(VERSION 2.6.4) which newer cmake versions reject. RVS's own unit tests are not needed for install_rvs's verification. The RVS_INSTALL_STATUS:0 regex check is dropped: cmake --install exits non-zero on failure, and the existing 'which rvs' + gst_single.conf existence checks below already gate install success. The baremetal path was unaffected by the original bug (SSH delivers the string to a shell that interprets ';'); single commands work there too. * Suppress stderr in final RVS config verification ls When MI300X config subdir is absent, the first ls prints "No such file" to stderr even if the default-path fallback succeeds; the regex then false-fails. Match the earlier verification block by adding 2>/dev/null. * Apply ruff format to install_rvs.py
atnair-amd
added a commit
that referenced
this pull request
May 20, 2026
…181) * [AIMVT-170] Port install_agfhc.py to orch fixture and drop nfs_install Mechanical translation of phdl/shdl -> orch per orch fixture pattern (parent: AIMVT-2). Deletes the local phdl/shdl fixtures (orch lives in cvs/tests/conftest.py per AIMVT-172). Also removes the nfs_install knob from this suite. Precedent: install_rvs.py (merged via PR #174) already silently ignores the field. The head-only optimization saved only a cp+tar extraction step -- sudo ./install was already always fleet-wide (legacy L172). After this PR, install runs unconditionally on every node via orch.exec. Behavior change: package_tar_ball must now exist at the same local path on every node. Operators who previously relied on nfs_install=True with the tarball only on a shared mount must pre-stage the tarball per-node. Out of scope: the rvs.nfs_install and transferbench.nfs_install keys in mi300_health_config.json remain (covered by separate PRs). * [AIMVT-170] install_agfhc: one logical command per orch.exec; --rocm-tar Splits `cd X; cp/tar/install` shell-string chains into separate orch.exec calls, each invoking a single binary with absolute paths. This makes the test's command flow legible at the call site and works correctly under ContainerOrchestrator's docker-exec transport, which does not spawn a shell to interpret `cd`, `;`, `&&` and friends. The single AGFHC `./install` call still requires cwd = install_dir (installer reads sibling files via relative paths), so that one call is wrapped in an explicit `sudo bash -c 'cd X && ./install'` to make the cwd dependency visible at the call site rather than smuggling it in via a shell chain. Also passes `--rocm-tar` to AGFHC's installer: this bypasses apt-based dependency resolution (which fails on TheRock-rocm systems because the libs aren't tracked by dpkg) in favour of dpkg-deb direct extraction. Safe on apt-rocm hosts too: dpkg-deb extracts the bundled debs into /, which is correct either way. Validated on a06u43 (single MI300X node, ROCm 7.13.0 in cvs-harness/ rocm-dev:latest container, externally-managed via launch:false): 1 passed in 26.13s. Container /opt/amd/agfhc/agfhc now exists post-run.
atnair-amd
added a commit
that referenced
this pull request
May 21, 2026
…VT-171] (#182) * [AIMVT-171] Port install_transferbench.py to orch fixture and drop nfs_install Mechanical translation of phdl/shdl -> orch per orch fixture pattern (parent: AIMVT-2). Deletes the local phdl/shdl fixtures (orch lives in cvs/tests/conftest.py per AIMVT-172). After this edit, the suite-local detect_rocm_path / detect_hip_compiler helpers match install_rvs.py byte-for-byte. Also removes the nfs_install knob from this suite. Precedent: install_rvs.py (merged via PR #174) already silently ignores the field. After this PR, git clone and build run on every node via orch.exec; wall-clock equivalent since builds are independent. Side-effect fix: the legacy existence check at L256 was head- only while the conditional mkdir could run fleet-wide -- inconsistent. Both are now fleet-wide; mkdir -p is idempotent. Out of scope: the rvs.nfs_install and agfhc.nfs_install keys in mi300_health_config.json remain (covered by separate PRs). * [AIMVT-171] install_transferbench: one logical command per orch.exec Splits `cd X; cmd` / `test ... && echo Y` / `ls -d X/* | sort | tail` shell-string chains into separate orch.exec calls. Per-call now invokes a single binary or a single explicit `bash -c '...'`, which works under ContainerOrchestrator's docker-exec transport (no implicit shell). Specific changes: - detect_rocm_path: glob+pipe wrapped in explicit `bash -c`. Legacy fallback uses `test -d X` exit code via detailed=True instead of `test -d X && echo Y`. - detect_hip_compiler: same exit-code pattern for both hipcc and amdclang++ probes. - TransferBench clone: `git clone URL DEST` with explicit destination drops the `cd X; git clone URL` cwd dependency. - TransferBench build: Makefile genuinely needs cwd in source tree, so wrapped in explicit `bash -c 'cd X && ENV=v make'`; the wrap is now visible at the call site instead of buried in an implicit shell chain. Validated on a06u43 (single MI300X node, cvs-harness/rocm-dev:latest container with TheRock ROCm 7.13.0, externally-managed): 1 passed in 45.63s. TransferBench source tree extracted into container under /home/atnair/INSTALL/TransferBench/. * [AIMVT-171] ruff format: install_transferbench.py Mechanical ruff format pass to satisfy make fmt-check. Splits the `git clone` orch.exec call across two lines (trailing comma in the multi-arg call forces multi-line per ruff's formatting rules); no behavior change.
amd-droy
pushed a commit
that referenced
this pull request
Jun 25, 2026
…181) * [AIMVT-170] Port install_agfhc.py to orch fixture and drop nfs_install Mechanical translation of phdl/shdl -> orch per orch fixture pattern (parent: AIMVT-2). Deletes the local phdl/shdl fixtures (orch lives in cvs/tests/conftest.py per AIMVT-172). Also removes the nfs_install knob from this suite. Precedent: install_rvs.py (merged via PR #174) already silently ignores the field. The head-only optimization saved only a cp+tar extraction step -- sudo ./install was already always fleet-wide (legacy L172). After this PR, install runs unconditionally on every node via orch.exec. Behavior change: package_tar_ball must now exist at the same local path on every node. Operators who previously relied on nfs_install=True with the tarball only on a shared mount must pre-stage the tarball per-node. Out of scope: the rvs.nfs_install and transferbench.nfs_install keys in mi300_health_config.json remain (covered by separate PRs). * [AIMVT-170] install_agfhc: one logical command per orch.exec; --rocm-tar Splits `cd X; cp/tar/install` shell-string chains into separate orch.exec calls, each invoking a single binary with absolute paths. This makes the test's command flow legible at the call site and works correctly under ContainerOrchestrator's docker-exec transport, which does not spawn a shell to interpret `cd`, `;`, `&&` and friends. The single AGFHC `./install` call still requires cwd = install_dir (installer reads sibling files via relative paths), so that one call is wrapped in an explicit `sudo bash -c 'cd X && ./install'` to make the cwd dependency visible at the call site rather than smuggling it in via a shell chain. Also passes `--rocm-tar` to AGFHC's installer: this bypasses apt-based dependency resolution (which fails on TheRock-rocm systems because the libs aren't tracked by dpkg) in favour of dpkg-deb direct extraction. Safe on apt-rocm hosts too: dpkg-deb extracts the bundled debs into /, which is correct either way. Validated on a06u43 (single MI300X node, ROCm 7.13.0 in cvs-harness/ rocm-dev:latest container, externally-managed via launch:false): 1 passed in 26.13s. Container /opt/amd/agfhc/agfhc now exists post-run.
amd-droy
pushed a commit
that referenced
this pull request
Jun 25, 2026
…VT-171] (#182) * [AIMVT-171] Port install_transferbench.py to orch fixture and drop nfs_install Mechanical translation of phdl/shdl -> orch per orch fixture pattern (parent: AIMVT-2). Deletes the local phdl/shdl fixtures (orch lives in cvs/tests/conftest.py per AIMVT-172). After this edit, the suite-local detect_rocm_path / detect_hip_compiler helpers match install_rvs.py byte-for-byte. Also removes the nfs_install knob from this suite. Precedent: install_rvs.py (merged via PR #174) already silently ignores the field. After this PR, git clone and build run on every node via orch.exec; wall-clock equivalent since builds are independent. Side-effect fix: the legacy existence check at L256 was head- only while the conditional mkdir could run fleet-wide -- inconsistent. Both are now fleet-wide; mkdir -p is idempotent. Out of scope: the rvs.nfs_install and agfhc.nfs_install keys in mi300_health_config.json remain (covered by separate PRs). * [AIMVT-171] install_transferbench: one logical command per orch.exec Splits `cd X; cmd` / `test ... && echo Y` / `ls -d X/* | sort | tail` shell-string chains into separate orch.exec calls. Per-call now invokes a single binary or a single explicit `bash -c '...'`, which works under ContainerOrchestrator's docker-exec transport (no implicit shell). Specific changes: - detect_rocm_path: glob+pipe wrapped in explicit `bash -c`. Legacy fallback uses `test -d X` exit code via detailed=True instead of `test -d X && echo Y`. - detect_hip_compiler: same exit-code pattern for both hipcc and amdclang++ probes. - TransferBench clone: `git clone URL DEST` with explicit destination drops the `cd X; git clone URL` cwd dependency. - TransferBench build: Makefile genuinely needs cwd in source tree, so wrapped in explicit `bash -c 'cd X && ENV=v make'`; the wrap is now visible at the call site instead of buried in an implicit shell chain. Validated on a06u43 (single MI300X node, cvs-harness/rocm-dev:latest container with TheRock ROCm 7.13.0, externally-managed): 1 passed in 45.63s. TransferBench source tree extracted into container under /home/atnair/INSTALL/TransferBench/. * [AIMVT-171] ruff format: install_transferbench.py Mechanical ruff format pass to satisfy make fmt-check. Splits the `git clone` orch.exec call across two lines (trailing comma in the multi-arg call forces multi-line per ruff's formatting rules); no behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cvs/tests/health/install/install_rvs.pyfrom the legacyphdl/shdlPssh handles to the sharedorchfixture incvs/tests/conftest.py, mirroring the priorrvs_cvs.pymigration (PR-B2 / commit02ff2fe). 21 call sites becomeorch.exec.config_dict['nfs_install'] is Truehead-only dispatch. On the canonicalmi300_health_config.jsonthe JSON value is the string"True"(not a boolean), sois Truewas always false and head-only mode was never engaged in shipped configs.Tracking: AIMVT-168.
Commit
eb43701— Port install_rvs to the orch fixture; update orchestrator scope docs (5 files, +34 / -91).Test plan
Mirrors
rvs_cvsprecedent: source-only port, no suite-level dispatch unit tests in this PR. Coverage comes from Tier-2 cluster verification + the existing orchestrator unit tests incvs/core/orchestrators/unittests/.Tier-2 — real cluster (single MI300X node
10.245.128.47/a06u43, ROCm 7.2.0, RVS pre-installed)Baremetal orchestrator (1 passed in 20.30s):
Auto-bundle:
install_rvs_2026-05-14T183458.zip(attached to AIMVT-168).Container orchestrator (1 passed in 6.03s, image
cvs-harness/rocm-dev:rvs-installed):