Skip to content

Aorta benchmark runner and host-side parsing fallback#88

Merged
speriaswamy-amd merged 3 commits into
mainfrom
surya/aorta
Feb 23, 2026
Merged

Aorta benchmark runner and host-side parsing fallback#88
speriaswamy-amd merged 3 commits into
mainfrom
surya/aorta

Conversation

@speriaswamy-amd

Copy link
Copy Markdown
Contributor

Summary

This PR improves the existing Aorta CVS benchmark implementation by making benchmark execution and result parsing more resilient across different cluster environments.

Key changes

  • Add optional Aorta repo auto-clone support when aorta_path is missing:
    • aorta_auto_clone
    • aorta_clone_url
  • Improve runner robustness:
    • pass explicit base config to launch script
    • propagate training overrides through AORTA_OVERRIDE_ARGS
    • discover torch_profiler artifacts recursively (latest directory wins)
    • add optional GEMM analysis hook and TraceLens dependency pre-check
    • chown host Aorta path on teardown after root container execution
  • Improve parsing semantics:
    • add ParseStatus.NO_DATA
    • distinguish between missing Excel reports with raw traces present vs true parse failures
  • Update benchmark test flow:
    • prefer container-generated reports when available
    • fallback to host raw-trace parsing when reports are missing/unusable
    • validate thresholds with the same parser that produced benchmark metrics
  • Refresh benchmark config defaults and add documentation:
    • update aorta_benchmark.yaml to practical benchmark defaults
    • add benchmark README and full Aorta config reference docs
    • include docs in config index and Sphinx TOC

Why

The benchmark already exists on main, but had brittle assumptions around artifact paths, report availability, and environment setup. These changes make benchmark execution and validation reliable even when container-side TraceLens outputs are absent, while preserving optional richer analysis paths.

Test plan

  • Run benchmark test:
    • pytest cvs/tests/benchmark/test_aorta.py --cluster_file <cluster.json> --config_file <aorta_benchmark.yaml> -v --log-cli-level=INFO
  • Verify parse/validation pipeline completes end-to-end
  • Confirm threshold validation and report generation pass

Observed result (sample run)

  • 5 tests passed - gfx942 arch
  • Aggregated metrics:
    • avg iteration ~2161.6 ms
    • compute ratio ~4.87%
    • comm ratio ~3.09%
    • overlap ratio ~0.00%

@urtiwari urtiwari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In aorta.rst and READme can you update the command using cvs run instead of pytest

@urtiwari urtiwari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look fine now.

@speriaswamy-amd speriaswamy-amd merged commit 08ea066 into main Feb 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants