Skip to content

community-benchmarks.md

Latest commit

 

History

History
76 lines (60 loc) · 3.18 KB

community-benchmarks.md

File metadata and controls

76 lines (60 loc) · 3.18 KB
Raw

Community benchmarks

Use this checklist to propose an AMD GPU / ROCm result for the Hardware matrix. Open a GitHub issue with the information below and link an immutable result summary. Contributors only need to submit evidence; they do not need to declare a result verified or assign platform review status.

Trust is governed by the central ROCmDoc ROCMDOC_STANDARD.md and COMPATIBILITY.md. Under that model, producer assurance records submission completeness (submitted or evidence-complete) while platform_review is a separate, downstream action performed by maintainers or the central platform. When a result is imported into the central platform, its platform review starts as not-reviewed. Contributors do not assign this state or self-assign an accepted review or reproduction assurance.

If you use this repository's evaluation path, reuse its model cards and engine-produced bundle files (run_summary, provenance, metric_result, run_stats, prediction manifest, and dataset identity). Do not create a second result format just for the submission.

Required information

Platform and hardware

  • GPU or APU name and GPU architecture, such as gfx1100 or gfx1151
  • OS and kernel version
  • ROCm and HIP versions

Workload

  • MinerU version
  • model ID and exact model revision
  • backend and backend version or commit
  • dtype and any runtime parameters that can affect output

Evaluation

  • benchmark name/version, dataset revision, and scorer revision
  • pages attempted, succeeded, and failed
  • Overall and relevant submetrics
  • inference wall time

Reproducibility

  • MinerU-ROCm repository/adapter commit, if used
  • prediction manifest SHA-256 or an equivalent integrity record
  • whether MinerU was patched
  • whether ROCm or the runtime was patched
  • whether HSA_OVERRIDE_GFX_VERSION or a similar override was used
  • immutable evidence URL: a release/tag or commit-pinned permalink to the result summary and provenance

Raw predictions, downloadable result artifacts, performance logs, and memory/resource observations are welcome but optional. Raw predictions are not a requirement for submitting evidence.

Shortest submission flow

  1. Complete the benchmark and preserve the fields above.
  2. Publish a concise result summary and provenance at an immutable tag or commit permalink.
  3. Open an issue and link that evidence. Maintainers review the revisions, counts, integrity record, score semantics, and any patches or overrides.
  4. Maintainers may add a Hardware Matrix row with the exact platform/backend path and producer evidence state. Any later platform review remains a separate central-platform action.

Issue #21 is an external community result submission example. Issue or draft-PR discussion may document the review process, while Hardware Matrix inclusion must link finalized, immutable evidence. Inclusion records one tested configuration and is not an official hardware-support claim or a cross-platform ranking.