[hipDNN] ALMIOPEN-2075 Add real-workload benchmark graph collection (DVC)#8795
[hipDNN] ALMIOPEN-2075 Add real-workload benchmark graph collection (DVC)#8795SamuelReeder wants to merge 4 commits into
Conversation
… (DVC) Curate ~9,000 real-workload hipDNN graphs (SDPA, GEMM, Conv, Normalization, etc.) for the Perf & benchmarking app, tracked via DVC: only .tar.gz.dvc pointers are committed, blobs live on the DVC remote. - Split bundles into per-model (Workloads/models/) and per-library shape-sweep (Workloads/microbench/) tarballs; tarball internal paths kept at op-folder level. - Every graph sourced from a real workload (OSS model configs + open model-crawler op dump); quantized dtypes mapped to nearest supported; unsupported ops documented and skipped, never fabricated. No customer/MAD-internal data included. - Add MANIFEST.md/INDEX.md provenance, sample graph fixtures, and per-bucket .gitignore for tarball blobs.
SamuelReeder
force-pushed
the
users/sareeder/sdpa-benchmark-suites
branch
from
079fd4b to
633eaac
Compare
Standalone helper (tools/check_deserialize.py) that verifies benchmark graph JSON deserializes/validates at three levels (json / graph / opgraph) without building a plan or running a kernel. Document it under the Workloads section of the README.
…s no GPU build_operation_graph only assembles/finalizes the operation-graph descriptor; the sole device-properties query is in the engine-heuristics path the script never reaches. Drop the inaccurate GPU/ranked-engine claims from the docstring and README.
There was a problem hiding this comment.
I kept this script separate from dnn-benchmarking to narrow scope. Its a low-effort utility that may occasionally be useful. Could move elsewhere or drop if desired.
There was a problem hiding this comment.
I have some other scripts for generating JSON graphs for each op from their direct params, and another for scraping modelcrawler to generate graphs. I didn't commit them because the use-cases would be scarce, but I could throw them in another tools directory or mlse-tools-internal.
opgraph is a strict superset of graph with identical prerequisites and per-stage error messages, so graph added nothing. Keep json (no build) and opgraph (full); default to opgraph.
There was a problem hiding this comment.
modelcrawler has a convenient way to sort shapes by the number of models that use it and number of cumulative calls across models, so I only included the most frequent 256 shapes per-op out of ~173,500 total unique shapes.
This subset may possess more value than the complete set due to its feasibility to run and the incidence of the selected shapes.
Perhaps the collection script should be added to mlse-tools-internal at the very least to recollect the popular shapes on-demand.
Codecov Report✅ All modified and coverable lines are covered by tests. ❌ Your project status has failed because the head coverage (77.89%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #8795 +/- ##
========================================
Coverage 71.43% 71.44%
========================================
Files 2612 2612
Lines 406992 407023 +31
Branches 60771 60771
========================================
+ Hits 290730 290770 +40
+ Misses 94960 94953 -7
+ Partials 21302 21300 -2
*This pull request uses carry forward flags. Click here to find out more. 🚀 New features to boost your workflow:
|
2 participants
Summary
Adds a curated collection of ~9,000 real-workload hipDNN graphs (SDPA, GEMM, Conv, Normalization, and the other support hipDNN node types) for the Perf & benchmarking app, tracked via DVC so the JSON blobs live on the remote and only pointers are committed. Graphs come from real OSS model configs and an open model-crawler op dump; no customer data is included.
Risk Assessment
Low risk. In-repo footprint is data/metadata only — DVC pointers, per-bucket
.gitignores, and two sample graph fixtures. No product/library code, build configuration, or public APIs change. The collection blobs are served from the DVC remote, so blast radius is limited to consumers of the benchmark Workloads.Testing Summary
hipdnn_frontend.Testing Checklist
python check_deserialize.py --level opgraph "<collection>/**/*.json"- Status: Passeddvc status -c- Status: Passed (0 out-of-sync)Technical Changes
Workloads/, split into per-model bundles (Workloads/models/) and per-library shape-sweep bundles (Workloads/microbench/); only.tar.gz.dvcpointers are committed, blobs are gitignored and pushed to the DVC remote.{gemm,conv,sdpa}/...) so each bundle extracts cleanly without a redundant source/model prefix.MANIFEST.md/INDEX.mdprovenance bundled alongside the collection, two sample graph fixtures (sample_add,sample_relu), and per-bucket.gitignoreentries for the tarball blobs.