Add IFoE L2 connectivity preflight check [AIMVT-180]

[speriaswamy-amd]

Summary

Adds an opt-in preflight check that validates IFoE L2 reachability across the cluster by running afmctl test ping on every reachable node and parsing the per-port pass/fail table plus the aggregate Summary: block from afmctl's output. Disabled by default, so existing preflight runs on clusters that don't use IFoE are unaffected.

Motivation

The existing preflight covers RDMA connectivity (GID consistency, RDMA interface presence, ibv_rc_pingpong) but doesn't exercise the IFoE-based fabric path. AIMVT-180 fills that gap with a fast L2-level gate before more expensive downstream tests.

Technical Details

New module — cvs/lib/preflight/ifoe_l2_connectivity.py

• IfoeL2ConnectivityCheck: orchestrates one afmctl test ping invocation per (bdf, dst_accelerator) pairing on every reachable node via parallel SSH. No pairwise client/server coordination needed — afmctl drives the request/response state machine in the device.
• AfmctlPingParser: tolerant parser for the per-port table (one row per (local_accel, port#), three traffic-type columns) and the aggregate Summary: block.
• parse_afmctl_show_device: parses afmctl show device for BDF auto-discovery.

New pytest entry — cvs/tests/preflight/preflight_checks.py

• test_ifoe_l2_connectivity is wired in between the interface-presence check and the existing RDMA connectivity check. Opt-in via connectivity_check.ifoe.connectivity_mode ("run" or "skip"; default "skip").

New config block — cvs/input/config_file/preflight/preflight_config.json

• connectivity_check.ifoe: afmctl_path, use_sudo, bdf_discovery (auto/config), bdfs, dst_accelerators, ports, pings_per_port, per_ping_timeout, traffic_types (ifoe_req/ifoe_resp/non_ifoe), loss_threshold_pct, ssh_timeout. All keys have inline _comment_* docs.

Reporting — cvs/lib/preflight/report.py

• Adds the IFoE L2 result to the executive summary and emits a dedicated HTML section with a per-node breakdown of failing invocations, failed ports, parsed Summary lines, raw afmctl output (expandable), and the exact command that was run.

PASS/FAIL logic

• A node fails if any enabled traffic type's loss exceeds loss_threshold_pct (default 0.0) or any per-port row reports FAIL.
• Skipped/unreachable nodes do not pollute the PASS/FAIL accounting.

Documentation

• cvs/tests/preflight/README.md and cvs/input/config_file/preflight/README_preflight_config.md updated with the new check, configuration reference, and example config block.

Test Plan

• Unit tests: python3 -m unittest cvs.lib.preflight.unittests.test_ifoe_l2_connectivity → 20/20 pass. Coverage:
  • Parser: passing / failing / partial-loss / empty / garbage afmctl output; multi-device afmctl show device.
  • Orchestrator: command rendering (defaults, sudo, port lists/strings, -t timeout, --traffic-type subsetting and aliases), passing/failing runs, loss-threshold leniency, auto-discovery fallback, multiple destination accelerators.
• Config validation: preflight_config.json parses cleanly with python -m json.tool.
• Backwards compatibility: default connectivity_mode = "skip" means no behavioral change for existing clusters; the new pytest entry records a SKIPPED result and returns immediately without contacting nodes.

Out of Scope

• Real-traffic / load-store validation (this PR only covers the L2 ping prerequisite; existing tools like TransferBench / RCCL handle data-path validation).
• Triggering or driving the IFoE driver / afmctl install — operators are expected to install these and apply fabric config before flipping connectivity_mode to "run".

Refs: AIMVT-180

Made with Cursor