diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 01bd33e..137d044 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,6 +16,9 @@ jobs: - run: pip install -e ".[dev]" - run: pytest -q - run: ruff check . + - run: python scripts/validate_schemas.py + - run: python scripts/generate_evidence_manifest.py --check + - run: python scripts/smoke_test.py --check-only - run: python scripts/check_repo.py - run: reuse lint - run: python -m build diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a30d57..6aabe99 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to MinerU-ROCm are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versions are tagged on `main`; see `docs/superpowers/specs/` for the design -behind each phase and `reproducibility.lock.yaml` for the exact provenance. +behind each phase and `REPRO.yaml` for the exact provenance. ## [1.0.0] — 2026-07-23 @@ -64,7 +64,7 @@ Cross-repo consistency fix with OmniDocBench-ROCm P0.1 (platform 0.3.1). ### Fixed - `scripts/check_repo.py`: replaced the hard-coded `_STALE=95.56` / `_CURRENT=95.46` gate with a **data-driven** check sourced from - `reproducibility.lock.yaml` (current overall) + the legacy v1.6 metric + `REPRO.yaml` (current overall) + the legacy v1.6 metric (prior overall): the README VLM badge must state the current overall, and the prior overall may appear only alongside the current one. - `Makefile` `eval-*` targets: `--cdm` on by default, `--skip-existing` only @@ -103,7 +103,7 @@ Hardening for upstream MinerU PR #5288 (ROCm docs contribution) — evidence-bas - `docs/upstream-pr/` — staged docs-only contribution to `opendatalab/MinerU` (zh AMD.md section + README GPU row + #5288 process-gate comment). ### Resolved (previously open in [0.1.0]) -- The **official upstream anchor** is now verified and sourced from the upstream README "Local Deployment" table (vlm-engine 95.30, pipeline 86.47) — see `reproducibility.lock.yaml` → `benchmark.official_reference: source: verified`. The prior withdrawn unofficial anchor is no longer cited anywhere in user-facing surfaces. +- The **official upstream anchor** is now verified and sourced from the upstream README "Local Deployment" table (vlm-engine 95.30, pipeline 86.47) — see `REPRO.yaml` → `benchmark.official_reference: source: verified`. The prior withdrawn unofficial anchor is no longer cited anywhere in user-facing surfaces. - `mineru`/`mineru_vl_utils` upstream commits are now pinned in the lock (resolved via `git ls-remote`). ## [0.1.0] — 2026-07-19 @@ -124,7 +124,7 @@ scoring, and reproducibility lock. - **`scripts/check_repo.py`** — CI consistency gate: AST no-engine-import scan, `pip install -e .` smoke (PEP 639 guard), SPDX-header check, README↔lock value cross-check (drift gate). -- **`reproducibility.lock.yaml`** — filled with byte-exact SHAs (code commit, +- **`REPRO.yaml`** — filled with byte-exact SHAs (code commit, model weights, GT json, eval config, scorer commit) + the re-run metrics + both venvs' full environment. The single source of truth. - **Results reproduced** via `mineru-rocm predict | score` on AMD gfx1100: diff --git a/README.md b/README.md index 998a12c..60e2fa5 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ [![OmniDocBench v1.6](https://img.shields.io/badge/OmniDocBench-v1.6-blue)](https://github.com/opendatalab/OmniDocBench) [![VLM full](https://img.shields.io/badge/MinerU2.5--Pro%20VLM%20(full)-95.56-green)](#evaluation) [![pipeline full](https://img.shields.io/badge/MinerU%203.4%20pipeline%20(full)-86.48-yellowgreen)](#evaluation) -[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](reproducibility.lock.yaml) +[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](REPRO.yaml) [![license: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0%20(+MinerU%20terms)-blue)](NOTICE) ## At a glance @@ -21,6 +21,14 @@ - **Most important limitation.** **Not precision-aligned.** No same-engine CUDA control exists; the upstream headline may be measured with a different engine. The official anchor (vlm-engine 95.30) is aligned to the upstream README "Local Deployment" table and is **community-verified, not official support**. - **Upstream.** This is a port OF [opendatalab/MinerU](https://github.com/opendatalab/MinerU); the [omnidocbench-rocm](https://github.com/AIwork4me/OmniDocBench-ROCm) engine is one *optional* consumer (install the `[platform]` extra), not the definition of this repo. +> **Support scope.** These configurations are **community-tested** on the +> explicitly listed hardware and software versions (see +> [`docs/hardware-matrix.md`](docs/hardware-matrix.md)). They do **not** +> represent generic AMD GPU support, MinerU upstream support guarantees, or AMD +> product certification. Status terms (`community-tested` / +> `independently-reproduced` / `experimental` / `not-tested`) are defined in +> [`docs/reproducibility-attestation.md`](docs/reproducibility-attestation.md). + ## Install The core package is GPU-free and has no platform dependency. @@ -114,15 +122,13 @@ developer debugging — see `docs/reproducibility.md` for the full recipe. | **ours MinerU2.5-Pro (vlm-vllm, ROCm)** | **95.56** | 0.0359 | 96.73 | 93.54 | | ours MinerU2.5-Pro (vlm-transformers, ROCm) | _sample-only_ | | | | -The `vlm-vllm` row is reproduced on linux-rocm (self-attested, `badge: community`): -1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process -crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240. -Overall 95.56 is consistent with the published upstream reference range -1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process -crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240. The - lm-llamacpp row is reproduced on windows-hip (Strix Halo): 1649/1651 ok, -2 empty (cover/decoration pages), Overall 95.46 identical to the Linux vLLM -standalone score. See docs/windows-vlm-phase2.md. +The `vlm-vllm` row is reproduced on linux-rocm (self-attested, `badge: community`): 1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240. Overall 95.56 is consistent with the upstream vlm-engine anchor (95.30; Δ +0.26 pp — **not** a controlled CUDA-vs-ROCm comparison). The `vlm-llamacpp` row is reproduced on windows-hip (Strix Halo): 1649/1651 ok, 2 empty (cover/decoration pages), Overall 95.46. See `docs/windows-vlm-phase2.md`. + +### Results — MinerU 3.4 Pipeline (supplementary, `mineru-pipeline`) + +| Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS | +|---|---:|---:|---:|---:| +| _official_ MinerU 3.4 pipeline _(upstream README; community-verified, not official support)_ | 86.47 | — | — | — | | **ours MinerU 3.4 pipeline (ROCm gfx1100, linux-rocm)** | **86.48** | 0.0566 | 83.07 | 82.04 | | **ours MinerU 3.4 pipeline (Strix Halo, windows-hip)** | **86.59** | 0.0565 | 83.39 | 82.04 | @@ -135,16 +141,24 @@ supplementary card in the same repo — see `model_card.pipeline.json` and ## Reproducibility -[`reproducibility.lock.yaml`](reproducibility.lock.yaml) is the single source of -truth — pinned commits, byte-exact weight/GT SHA256 cross-checked against the -upstream HF repos, environment versions, and the metric formula. Verified values -were populated from the full 1651-page reruns completed on 2026-07-19. +[`REPRO.yaml`](REPRO.yaml) is the single source of +truth — pinned commits, SHA-256-pinned weights/GT (where recorded; see +`reproducibility.level` = `artifact-manifested`, plus the documented +`not_recorded` limitations), environment versions, and the metric formula. +Verified values were populated from the full 1651-page reruns completed on +2026-07-19. Hardware: AMD gfx1100 (Radeon PRO W7900), 48 GB VRAM, ROCm 7.2, bf16. The official reference (pipeline 86.47, vlm-engine 95.30) is sourced from the upstream MinerU README "Local Deployment" table as a community-verified anchor, not official support. See `docs/reproducibility.md` for the full recipe. +**Version boundary.** Validated with MinerU **3.4.4** (commit `0dfc9460`, +recorded in `REPRO.yaml` under `validated_software`). MinerU 4.x / 4.0 alpha +has **not** been validated; results must not be extrapolated to untested +MinerU releases. CLI/backend names (`pipeline`, `vlm-vllm`, `vlm-llamacpp`) +match that version; `vlm-transformers` is sample-only. + Release consumers should also read [`docs/release-artifact.md`](docs/release-artifact.md), [`docs/hardware-matrix.md`](docs/hardware-matrix.md), and [`docs/architecture.md`](docs/architecture.md). The release checklist is diff --git a/README.zh-CN.md b/README.zh-CN.md index 17dc783..21db325 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -8,11 +8,18 @@ [![OmniDocBench v1.6](https://img.shields.io/badge/OmniDocBench-v1.6-blue)](https://github.com/opendatalab/OmniDocBench) [![VLM full](https://img.shields.io/badge/MinerU2.5--Pro%20VLM%20(full)-95.56-green)](#evaluation评测) [![pipeline full](https://img.shields.io/badge/MinerU%203.4%20pipeline%20(full)-86.48-yellowgreen)](#evaluation评测) -[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](reproducibility.lock.yaml) +[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](REPRO.yaml) [![license: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0%20(+MinerU%20terms)-blue)](NOTICE) > 国内用户优先使用镜像与 ModelScope 拉取模型/数据集,速度更稳定。 +> **支持范围:** 以下配置仅在明确列出的硬件与软件版本上经过**社区验证** +> (community-tested,见 [`docs/hardware-matrix.md`](docs/hardware-matrix.md))。 +> 这些**不代表**通用 AMD GPU 支持、MinerU 上游官方支持保证,也不代表 AMD 产品认证。 +> 状态术语(`community-tested` / `independently-reproduced` / `experimental` / +> `not-tested`)定义见 +> [`docs/reproducibility-attestation.md`](docs/reproducibility-attestation.md)。 + ## Install(安装) 核心包不依赖 GPU,也无平台依赖。 @@ -100,16 +107,16 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ... | Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS | |---|---:|---:|---:|---:| | _official_ MinerU2.5-Pro _(上游 README vlm-engine 行;社区验证,非官方支持)_ | 95.30 | — | — | — | +| **ours MinerU2.5-Pro(vlm-llamacpp,Windows-HIP)** | **95.46** | 0.0373 | 97.01 | 93.11 | | **ours MinerU2.5-Pro(vlm-vllm,ROCm)** | **95.56** | 0.0359 | 96.73 | 93.54 | | ours MinerU2.5-Pro(vlm-transformers,ROCm) | _仅采样_ | | | | `vlm-vllm` 行在 linux-rocm **已复现**(自证,`badge: community`):1651/1651 页 已尝试,1649 个产生非空预测(2 个空输出),无进程崩溃;单卡(gfx1100)约 -7 小时;阅读顺序 EditDist 0.1240。Overall 95.56 与公开发布的上游参考区间**一致** -(vlm-engine 95.30;delta +0.26 pp —— **非**受控 CUDA-vs-ROCm 对照)。上游锚点 -取自上游 README "Local Deployment" 表,属**社区验证、非官方支持** —— 见 -`reproducibility.lock.yaml`(`benchmark.official_reference`)。`windows-hip` -仍为 `community-wanted`(暂无结果)。 +7 小时;阅读顺序 EditDist 0.1240。Overall 95.56 与上游 vlm-engine 锚点**一致** +(95.30;Δ +0.26 pp —— **非**受控 CUDA-vs-ROCm 对照)。`vlm-llamacpp` 行在 +windows-hip(Strix Halo)**已复现**:1649/1651 成功,2 个空输出(封面/装饰页), +Overall 95.46。详见 `docs/windows-vlm-phase2.md`。 > **历史分数说明:** 此前独立 `mineru-rocm score` 路径在同一份 1651 页预测上得分为 > **95.46**(Formula CDM 96.46);当前平台 CDM 得分 **95.56**(Formula CDM 96.73)。 @@ -120,9 +127,9 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ... | Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS | |---|---:|---:|---:|---:| -| _official_ MinerU 3.4 pipeline | 86.47 | — | — | — | +| _official_ MinerU 3.4 pipeline _(上游 README;社区验证,非官方支持)_ | 86.47 | — | — | — | | **ours MinerU 3.4 pipeline(ROCm gfx1100,linux-rocm)** | **86.48** | 0.0566 | 83.07 | 82.04 | -| windows-hip | _community-wanted_ | | | | +| **ours MinerU 3.4 pipeline(Strix Halo,windows-hip)** | **86.59** | 0.0565 | 83.39 | 82.04 | Pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`。已知:1 个空输出页面。 主 registry card 为 `mineru2.5`(VLM);pipeline 为同仓补充 card —— 见 @@ -130,8 +137,9 @@ Pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`。已知:1 个空 ## Reproducibility(可复现性) -[`reproducibility.lock.yaml`](reproducibility.lock.yaml) 是唯一事实来源 —— -锁定的 commit、与上游 HF 仓交叉校验的逐字节权重/GT SHA256、环境版本,以及 +[`REPRO.yaml`](REPRO.yaml) 是唯一事实来源 —— +锁定的 commit、SHA-256 记录的权重/GT(已记录部分;见 `reproducibility.level` = +`artifact-manifested` 及其中文档化的 `not_recorded` 限制)、环境版本,以及 指标公式。已验证值来自 2026-07-19 完成的全量 1651 页重跑。 硬件:AMD gfx1100(Radeon PRO W7900),48 GB VRAM,ROCm 7.2,bf16。 @@ -139,6 +147,11 @@ Pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`。已知:1 个空 "Local Deployment" 表,作为社区验证锚点,非官方支持。详见 `docs/reproducibility.md`。 +**版本边界:** 已验证版本为 MinerU **3.4.4**(commit `0dfc9460`,记录于 +`REPRO.yaml` 的 `validated_software`)。MinerU 4.x / 4.0 alpha **未经验证**; +结果不得外推到未测试的 MinerU 版本。CLI/backend 名称(`pipeline`、 +`vlm-vllm`、`vlm-llamacpp`)与该版本一致;`vlm-transformers` 仅为采样验证。 + ## License —— 下载权重前必读 本仓为 **Apache-2.0**(原创打包/工具)。MinerU pipeline 遵循 **MinerU Open Source License** @@ -156,7 +169,7 @@ license —— 视为授权不明,请勿再分发。完整分解见 [NOTICE](N ## Known Gaps(已知限制) - `smoke` 后端输出的是占位文本,并非真实 OCR。CI/conformance 可通过 `--backend smoke` 验证适配器契约而无需 GPU。 -- **Windows-HIP** 为 `community-wanted` —— 尚无正式结果。两个 model card 的 `windows-hip` badge 均为 `community-wanted`。 +- **Windows-HIP** 为 `community`(社区验证):Phase 1 pipeline(Overall 86.59)与 Phase 2 VLM llama.cpp HIP(Overall 95.46,与 Linux vLLM 平台 CDM 得分 95.56 同一预测集)均已完成,附带符合规范的 CDM bundle。 - **环境准备脚本**(`adapter/setup/`)为桩代码,仅记录手动步骤,未完全自动化环境安装。 - **平台标准 artifacts** 已于 2026-07-21 在 `results/omnidocbench/v16/linux-rocm/` 生成(`mineru2.5` 与 `mineru-pipeline` 的自包含 CDM bundle:`run_summary` + `provenance` + `metric_result` + `run_stats` + SHA256 `prediction_manifest` + `dataset_identity`)。`results/omnidocbench/v1.6/` 下的遗留结果保留用于历史对比与预测来源 provenance。可用 `omnidocbench-rocm validate-bundle results/omnidocbench/v16/linux-rocm` 校验任意 bundle。 - **VLM 空输出:** 1651 个 VLM 页面中 2 个产生空预测(已记录为失败)。 diff --git a/REPRO.yaml b/REPRO.yaml index bad5bed..4e4ce63 100644 --- a/REPRO.yaml +++ b/REPRO.yaml @@ -1,9 +1,11 @@ mineru_rocm: repo: https://github.com/AIwork4me/MinerU-ROCm release: - tag: v0.1.0 - commit: 839776e1774a1d85be8f9314a830e612d44a6926 - tag_object_sha: dd591469d009cac246f5090daa7398623d2fd878 + # Latest existing release tag at the time of writing (HEAD 6a64af5 is newer + # and untagged). Annotated tag; commit = peeled tag commit. + tag: v1.1.0 + commit: 339a5e5699f093e680cf3c646dba94f7ad324a2a + tag_object_sha: 04be8fe7659ed2c2771e9f03b6edfd62ef2193ec benchmark_run_commits: pipeline: e05eec322dd8bf45447c3dec53349454f0424e61 vlm_vllm: b75f788419d35a4210c201159a6c67923d60d65c @@ -16,6 +18,24 @@ mineru_vl_utils: repo: https://github.com/opendatalab/mineru-vl-utils commit: cc467faaddb53d8b276cedf88f09302f540a7b83 version: 1.0.5 +validated_software: + mineru: + version: "3.4.4" + commit: 0dfc9460cd9ab693b9af60ae3fbffd7bc111b062 + source: "git checkout at the validated commit; pip install -U 'mineru[all]'" + validated: true + scope: "linux-rocm pipeline + vlm-vllm; windows-hip pipeline. The windows-hip VLM (vlm-llamacpp) runs mineru_vl_utils against a llama.cpp HIP server, not the mineru pipeline package." + mineru_vl_utils: + version: "1.0.5" + validated: true + mineru_4: + validated: false + status: "not_validated" + note: "MinerU 4.x / 4.0 alpha was not covered by this validation. Results must not be extrapolated to untested MinerU releases." + cli_backend_names: + validated: ["pipeline", "vlm-vllm", "vlm-llamacpp"] + not_validated: ["vlm-transformers"] + note: "vlm-transformers is sample-only (benchmark.full_1651.vlm_transformers: not_run)." model: vlm: hf_repo: opendatalab/MinerU2.5-Pro-2605-1.2B @@ -179,4 +199,68 @@ benchmark: inference_engine: vlm-engine provenance_note: Official anchors are OmniDocBench v1.6 Overall from the upstream README 'Local Deployment' table. The prior 'official 95.75' was unverified and - is withdrawn. + is withdrawn. The upstream MinerU commit at the time these anchors were read + was NOT pinned (source_url points at the moving master branch); the anchors are + therefore a community-verified contextual reference, not an immutable citation. +reproducibility: + level: artifact-manifested + levels_documented: + environment-pinned: Pinned commits, venvs, the ROCm recipe, and the metric formula + are recorded in this lock. + artifact-manifested: Locally-available artifacts carry SHA-256 (VLM safetensors + + config, GT-full, eval_config, pipeline layout/formula/ocr); regenerate the + manifest with scripts/generate_evidence_manifest.py. + prediction-reproduced: Each formal prediction is represented by filename + SHA-256 + in its committed prediction_manifest; bulk Markdown predictions stay out of Git. + score-reproduced: metric_result.json is committed and regenerable from the pinned + scorer commit. + byte-exact: All inputs and artifacts carry SHA-256 AND re-runs are bit-identical. + NOT reached in this repo (see limitations). + limitations: + - pipeline_weights.table_sha256 is not_recorded; the table sub-model is pinned by + PDF-Extract-Kit-1.0 revision ed6b654c but its file SHA was not recorded. + - canary GT (gt_json_canary_sha256, canary_manifest_sha256) is not_recorded; the + canary subset is not materialized (docs/known-gaps.md). + - environment.inference.hip_visible_devices.pipeline is not_recorded in the historic + Linux run manifest; gpu_count_per_benchmark = 1 is the load-bearing fact. + - VLM (vLLM, bf16) shows run-to-run inference drift that was not separately isolated; + only the scoring delta was attributed (see docs/reproducibility.md Non-determinism). + - The official upstream anchor is sourced from the upstream MinerU README master + branch; the upstream commit at the time of reading was not pinned. + - The benchmark pipeline torch (2.14.0.dev20260717+rocm7.2) is an AMD internal + nightly NOT on the public pytorch.org rocm7.2 index (py3.12 has 2.11.0+ stable). + Verified 2026-07-26 (evidence/smoke-real-pipeline-2026-07-26.log) — the pipeline + runs end-to-end on the publicly-installable stable torch 2.11.0+rocm7.2 + (python 3.12.3, /root/venvs/mineru-pipeline, gfx1100) and produces correct OCR + on examples/sample.png. The stable torch is the recommended public-reproduction + path; the nightly pin above is what produced the 1651-page benchmark scores. +reproduction: + terms_doc: docs/reproducibility-attestation.md + primary_run: + operator: AIwork4me maintainer (repo operator; git author AIwork4me) + organization_relation: maintainer of this repository + hardware: AMD gfx1100 (Radeon PRO W7900, 48 GB), ROCm 7.2, bf16, 1 GPU + regenerated_predictions: true + regenerated_scores: true + independence_level: repeat run + note: Maintainer self-attested. No clean-environment automation recorded. + secondary_run: + performed: true + operator: AIwork4me maintainer (same operator/team as primary_run) + independence_level: second-machine reproduction + same_team: true + separate_machine: true + clean_environment: unknown + reused_predictions: false + regenerated_predictions: true + regenerated_scores: true + hardware: AMD Ryzen AI MAX+ 395 / Radeon 8060S (Strix Halo), Windows ROCm 7.2.1 + note: Same team, different machine. NOT an independent reproduction. + internal_independent_reproduction: + performed: false + third_party_independent_reproduction: + performed: false + note: No reproducer without an org relation to the maintainers is recorded. + Promotion of any result to independently-reproduced requires a third-party + entry appended to docs/reproducibility-attestation.md. + diff --git a/SUPPORT.md b/SUPPORT.md index 0e3881a..075a2a2 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -2,7 +2,7 @@ | Question type | Where | |---|---| -| Bug or unexpected result in THIS repo | [GitHub Issues](https://github.com/AIwork4me/MinerU-ROCm/issues) — include the OmniDocBench page id, the backend, and the `reproducibility.lock.yaml` environment block. | +| Bug or unexpected result in THIS repo | [GitHub Issues](https://github.com/AIwork4me/MinerU-ROCm/issues) — include the OmniDocBench page id, the backend, and the `REPRO.yaml` environment block. | | ROCm / gfx1100 compatibility | Open an issue with `rocm-smi --showproductname` output; mark it `rocm-compat`. | | Upstream MinerU behavior (model quality, pipeline options) | [opendatalab/MinerU](https://github.com/opendatalab/MinerU) — this repo wraps upstream, it does not change the models. | | OmniDocBench scoring / metrics | [opendatalab/OmniDocBench](https://github.com/opendatalab/OmniDocBench). | diff --git a/docs/HANDOFF-windows-hip.md b/docs/HANDOFF-windows-hip.md index 6c2c9d4..faf2a74 100644 --- a/docs/HANDOFF-windows-hip.md +++ b/docs/HANDOFF-windows-hip.md @@ -119,7 +119,7 @@ powershell -ExecutionPolicy Bypass -File eval-infra\01-omnidocbench\setup.ps1 powershell -ExecutionPolicy Bypass -File eval-infra\01-omnidocbench\verify.ps1 ``` The dataset revision **must** be `2b161d0` (the pinned dataset + scorer revision -this repo's `reproducibility.lock.yaml` records). Note the images directory path +this repo's `REPRO.yaml` records). Note the images directory path (`eval-infra\01-omnidocbench\data\images`); you'll pass it to the adapter. ```powershell diff --git a/docs/HANDOFF-windows-vlm-llamacpp.md b/docs/HANDOFF-windows-vlm-llamacpp.md index a4277b4..bbe602f 100644 --- a/docs/HANDOFF-windows-vlm-llamacpp.md +++ b/docs/HANDOFF-windows-vlm-llamacpp.md @@ -727,7 +727,7 @@ $tool = "C:\Users\rocm\miniconda3\envs\mineru-win-rocm\Scripts\omnidocbench-rocm - `README.md` Windows VLM 行; - `docs/windows-vlm-phase2.md`; - `docs/HANDOFF-windows-hip.md`; -- `reproducibility.lock.yaml` 的 +- `REPRO.yaml` 的 `benchmark.windows_hip.full_1651.vlm`; - root `model_card.json` 的 Windows badge/artifact 指针(保留 Linux 成绩); - 新 `model_card.windows-hip.json`; @@ -750,7 +750,7 @@ git status --short ```powershell git add README.md docs eval scripts src tests results ` - model_card.json model_card.windows-hip.json reproducibility.lock.yaml + model_card.json model_card.windows-hip.json REPRO.yaml git commit -m "results(windows-hip): publish MinerU2.5 llama.cpp validation" git push -u origin codex/windows-vlm-phase2 ``` diff --git a/docs/architecture.md b/docs/architecture.md index 4b8471f..c800941 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -23,7 +23,7 @@ It does not fork model weights or replace the upstream model APIs. | Linux pipeline | In-process MinerU 3.4.4 | PyTorch ROCm; pipeline ONNX components may use CPU | | Linux VLM | OpenAI-compatible vLLM server | vLLM ROCm on one gfx1100 GPU | | Windows pipeline | In-process MinerU 3.4.4 | Windows ROCm PyTorch plus ONNX Runtime DirectML | -| Windows VLM | Not released | Phase 2 serving runtime remains exploratory | +| Windows VLM | llama.cpp HIP server (`vlm-llamacpp`) | HIP via the llama.cpp ROCm build (Strix Halo, gfx1151) | On Windows, all compatible ONNX sessions are DirectML-first. The known DirectML-incompatible `slanet-plus.onnx` control-flow model is routed through an @@ -39,5 +39,6 @@ every formal prediction is represented by filename and SHA-256 in its manifest. The primary VLM card is `model_card.json`. The pipeline is a supplementary result with `model_card.pipeline.json` and platform-specific cards. A -`community` badge is self-attested; promotion to `verified` requires an -independent maintainer reproduction. +`community-tested` badge is self-attested by the repo maintainers; promotion to +`independently-reproduced` requires a separate-operator reproduction per +`docs/reproducibility-attestation.md`. Status terms are defined there. diff --git a/docs/audit/upstream-readiness-audit.md b/docs/audit/upstream-readiness-audit.md new file mode 100644 index 0000000..1fa6d34 --- /dev/null +++ b/docs/audit/upstream-readiness-audit.md @@ -0,0 +1,209 @@ +# Upstream-readiness audit — MinerU-ROCm + +Audit snapshot: `main` @ `6a64af5` (2026-07-25). Generated 2026-07-26. + +This audit distinguishes **confirmed facts** (verified against the current +checkout / git / committed artifacts), **inferences**, and **cannot-confirm-now** +items. It is the basis for the hardening work in this branch. Every finding +below cites a file:line so it can be re-checked. + +The goal of this round: make the repo a trustworthy, reproducible, +self-consistent evidence store that can back an upstream `MinerU` +`docs/zh/usage/acceleration_cards/AMD.md` contribution. + +--- + +## A. Repository version state + +| Source | Value | Evidence | +|---|---|---| +| Current branch / commit | `main` @ `6a64af5` | `git rev-parse HEAD` | +| Latest tag | `v1.1.0` → commit `339a5e5` (annotated-tag object `04be8fe`) | `git rev-parse v1.1.0^{commit}`; `git cat-file -t v1.1.0 = tag` | +| `pyproject.toml` version | `1.0.0` | `pyproject.toml:7` | +| `REPRO.yaml` recorded release | tag `v0.1.0`, commit `839776e`, tag_object_sha `dd591469` | `REPRO.yaml:3-6` | +| Tags present | `v0.1.0`, `v1.0.0`, `v1.1.0` | `git tag` | + +**Confirmed — three-way version drift.** The lock records the *old* release +`v0.1.0`; `pyproject.toml` says `1.0.0`; the latest tag is `v1.1.0` (Windows-HIP +Phase 2 VLM complete). HEAD `6a64af5` is newer than `v1.1.0` and is untagged. + +The provenance gate (`scripts/check_repo.py:check_release_and_run_provenance`) +verifies the lock's *stated* tag is internally consistent against git — it does +**not** check that the stated tag is the *latest* tag, so this drift passes the +gate silently. + +**Benchmark code snapshot:** the per-run commits in `REPRO.yaml:7-9` +(`pipeline: e05eec3…`, `vlm_vllm: b75f788…`) **do** match the +`repo_commit` recorded in each committed `run_manifest.json` +(`results/omnidocbench/v1.6/{pipeline,vlm-vllm}/run_manifest.json`) — confirmed. +So the benchmark provenance is sound; only the *release* provenance is stale. + +--- + +## B. MinerU support boundary + +**Confirmed — validated version is MinerU 3.4.4.** Evidence: + +- `REPRO.yaml:14` — `mineru.version: 3.4.4`, commit `0dfc9460…`. +- `pyproject.toml:40` — `windows-pipeline` extra pins `mineru[pipeline]==3.4.4`. +- `REPRO.yaml:102` — `mineru: 3.4.4` in the pipeline venv. +- CLI/backend names (`mineru2.5`, `mineru-pipeline`, `vlm-vllm`, `vlm-llamacpp`, + `pipeline`) are consistent with `src/mineru_rocm/cli.py` + `driver.py`. + +**Cannot-confirm-now — MinerU 4.x.** No file in the repo mentions MinerU 4.x or +4.0 alpha at all (`git grep` for `mineru 4 / 4.x / 4.0 alpha` returns nothing). +So 4.x is simply *absent* from the boundary rather than explicitly excluded. +This must become an explicit "not validated" statement so results are not +extrapolated. + +--- + +## C. Support-status conflicts + +Reality (from commits `339a5e5` Phase 2, `REPRO.yaml:138-172`, +`model_card.windows-hip.json`, `README.md:172`): **Windows-HIP VLM +(vlm-llamacpp) is `community`, Overall 95.46, Phase 2 complete**; Windows-HIP +pipeline is `community`, 86.59, Phase 1 complete. + +| Surface | linux-rocm pipeline | linux-rocm VLM | windows-hip pipeline | windows-hip VLM | Note | +|---|---|---|---|---|---| +| `README.md` | community / 86.48 ✓ | community / 95.56 ✓ | community / 86.59 ✓ | community / 95.46 ✓ | current | +| `REPRO.yaml` | 86.48 ✓ | 95.56 ✓ | 86.59 ✓ | 95.46 ✓ | current | +| `model_card.json` / `.pipeline.json` / `.pipeline.windows-hip.json` | ✓ | ✓ | ✓ | n/a | current | +| `model_card.windows-hip.json` | n/a | n/a | n/a | 95.46 ✓ but **field drift** | uses `community_badge` + singular `platform` | +| `docs/hardware-matrix.md:11` | ✓ | ✓ | ✓ | **`community-wanted` / "Serving runtime undecided" / —** | **STALE** | +| `docs/architecture.md:26` | ✓ | ✓ | ✓ | **"Not released / Phase 2 exploratory"** | **STALE** | +| `docs/known-gaps.md:14` | — | — | — | **"now verified"** vs `:33` "community, upgrades to verified on repro" | **internal contradiction** | +| `README.zh-CN.md:112,125,159` | ✓ | ✓ | **`community-wanted`** | **`community-wanted` / "暂无结果"** | **STALE (CN doc lags EN)** | + +**Confirmed conflicts:** hardware-matrix, architecture, known-gaps, and the +Chinese README all still describe Windows-HIP VLM as not-done/wanted, contradicting +the committed Phase 2 evidence and the English README. + +--- + +## D. Reproducibility gaps + +**D1 — `reproducibility.lock.yaml` does not exist; the file is `REPRO.yaml`.** +Commit `6a64af5` renamed `reproducibility.lock.yaml` → `REPRO.yaml` (pure rename, +0 content change) but did **not** update consumers. Confirmed references to the +dead filename: `scripts/check_repo.py:43,420`; `README.md:13,138`; +`README.zh-CN.md:11,111,133`; `CHANGELOG.md:7,67,106,127`; `SUPPORT.md:5`; +`docs/reproducibility.md:3,92`; `docs/benchmark-methodology.md:7,52,88`; +`docs/HANDOFF-windows-hip.md:122`; `docs/HANDOFF-windows-vlm-llamacpp.md:730,753`; +`docs/upstream-pr/AMD.md.section.zh.md:35`; `docs/upstream-pr/issue-5288-comment.md:9`; +`docs/upstream/mineru-issue-5288.md:5,26,87`; `docs/upstream/mineru-issue-5288-review.md:16,80`; +`model_card.pipeline.json:22`. + +**D2 — the CI consistency gate is currently broken (red).** `check_repo._load_lock` +reads `reproducibility.lock.yaml`, returns `None`, and `main()` then calls +`_current_vlm_overall(None)` → `AttributeError` (rc=1). Running +`python scripts/check_repo.py` crashes today. The pytest suite has the same +fault: `test_upstream_commits_pinned_in_lock`, `test_official_reference_verified`, +`test_modelcard_lock_agreement` (and the `clean_on_repo` integration) error on +`None.get()`. Every lock-driven check is therefore a silent no-op or a crash. + +**D3 — unrecorded hashes / fields (honest gaps, but they collide with "byte-exact" +wording):** `REPRO.yaml` carries `pipeline_weights.table_sha256: not_recorded` +(`:33`), `gt_json_canary_sha256: not_recorded` (`:39`), +`canary_manifest_sha256: not_recorded` (`:41`), +`environment.inference.hip_visible_devices.pipeline: not_recorded` (`:91`), and +all `benchmark.canary_N.*` not_recorded (`:116-118`). Recorded (real) SHAs: +VLM `safetensors_sha256` + `config_sha256` (`:24-25`), GT-full +`gt_json_full_sha256` (`:38`), `eval_config_sha256` (`:40`), pipeline +`layout/formula/ocr_sha256` (`:30-32`). + +**D4 — overclaim wording vs D3 evidence:** +- `README.md:139` — "byte-exact weight/GT SHA256 cross-checked". +- `docs/reproducibility.md:3,92` — "byte-exact weight/GT SHAs". +- `docs/reproducibility.md:87` — "byte-identical predictions" (pipeline; this + one is a determinism claim, more defensible, but still needs scoping). +- `CHANGELOG.md:127,131,143` — "byte-exact"/"byte-identical". + +The VLM weights + config + GT-full are SHA-pinned, but the table sub-model and +the canary GT are `not_recorded`, so the repo's level is +**environment-pinned + artifact-manifested (partial)**, not byte-exact. + +**D5 — non-immutable official anchor:** `REPRO.yaml:175` and +`model_card.json:24` point the official anchor at +`github.com/opendatalab/MinerU/blob/master/README.md` (a moving branch). The +upstream commit at the time the 86.47/95.30 anchors were read was not pinned. +Cannot confirm a stable upstream commit now without re-fetching upstream. + +**D6 — `not_recorded`-style placeholders must not co-exist with +"complete/byte-exact" claims** (D3 + D4 together). Resolution: define a +documented `reproducibility.level` and align wording to it. + +**D7 — scorer/GT source clarity:** scorer commit is pinned +(`REPRO.yaml:37` `2b161d0…` → opendatalab/OmniDocBench). GT-full SHA is pinned. +No gap here — this part is sound. + +--- + +## E. CI coverage gaps + +What the existing CI (`.github/workflows/ci.yml`) *can* check today: +- `pytest -q`, `ruff check .`, `reuse lint`, `python -m build`, `pip check`, + `python scripts/check_repo.py` (core job); `omnidocbench-rocm conformance .` + + `validate_platform_artifacts.py` (platform-contract job). + +What it *cannot* catch (confirmed by this audit): +1. **The broken lock filename** — `check_repo` crashes instead of catching it. +2. **No JSON-schema validation** for `REPRO.yaml` or the model cards. +3. **No Markdown link check** — the dead `reproducibility.lock.yaml` links + across ~10 docs go undetected. +4. **No "documented CLI command exists" check** — `reproduce.md`/`reproduce.pipeline.md` + cite `python -m mineru_rocm runner`, `python -m mineru_rocm serve-vlm`, + `mineru-rocm predict pipeline`, `--img-dir`, `--out-dir`, **none of which + exist** in `cli.py`/`driver.py` (real: `predict --backend {pipeline|vlm-vllm| + vlm-llamacpp} --gt-json --images-dir --pred-dir`; plus `doctor`/`validate`/ + `manifest verify`/`canary materialize`/`score`). There is no `__main__.py`, + no `serve-vlm` subcommand. +5. **No smoke test** for ordinary users (only the full benchmark exists). +6. **ruff is unpinned** — `pyproject.toml` `[tool.ruff.lint]` sets no `select`, + so `ruff check .` is green under the classic default (`E4,E7,E9,F`) but + reports 67 errors under ruff 0.16's expanded default. CI is version-fragile. +7. **No "latest tag vs recorded release" check** — version drift (Section A) + passes silently. + +--- + +## F. Cannot-complete-now items (blocked on environment/authorization) + +- **GPU runtime verification** — this environment has no AMD GPU, so + Level-0..3 smoke *execution* cannot be run here; only static / schema / CLI / + `--check-only` verification is possible. Logged as a human-verification + blocker in the completion report. +- **Upstream MinerU commit pin for the official anchor** (D5) — would require + re-fetching upstream at the date the anchor was read; left as a documented + limitation, not fabricated. +- **`table_sha256` / canary GT SHAs** — the canary subset is not materialized + (deferred per `docs/known-gaps.md:22`); the table sub-model file is not + locally available in this checkout. Kept `not_recorded` with reasons, not + guessed. +- **Push / tag / PR** — out of scope without explicit authorization; all work + is local commits on a branch. + +--- + +## G. Summary of confirmed defects to fix + +1. **(blocker)** `reproducibility.lock.yaml` → `REPRO.yaml` everywhere; fix + `check_repo._load_lock` + the score-command lock scan so the gate stops + crashing and the tests stop erroring. +2. **(blocker)** `reproduce.md` / `reproduce.pipeline.md` fabricated CLI + + `/root` hardcodes → rewrite against the real CLI with placeholder paths. +3. `examples/serve_vlm_vllm.sh` + `src/.../vlm.py` `/root` hardcodes → + env-var/HF-cache derived, portable. +4. Stale Windows-HIP VLM status in `hardware-matrix.md`, `architecture.md`, + `README.zh-CN.md`, and the `known-gaps.md` self-contradiction. +5. Model-card field drift: unify `model_card.windows-hip.json` to the + `badge`/`platforms`/`schema_version` shape used by the other cards. +6. Add explicit MinerU version boundary (3.4.4 validated; 4.x not validated). +7. Reduce "byte-exact" wording to a documented `reproducibility.level` + (`environment-pinned` + `artifact-manifested`); keep honest `not_recorded`. +8. Update stale release provenance (`v0.1.0` → `v1.1.0`, latest existing tag). +9. Add: JSON schemas (`schemas/`), evidence-manifest generator, portable smoke + test, reproduction attestation, consistency-CI checks (dead-filename, CLI + existence, `/root`-in-doc-commands, schema, markdown links), pin ruff + `select`, upstream-documentation-source doc. diff --git a/docs/audit/upstream-readiness-completion-report.md b/docs/audit/upstream-readiness-completion-report.md new file mode 100644 index 0000000..e68348e --- /dev/null +++ b/docs/audit/upstream-readiness-completion-report.md @@ -0,0 +1,95 @@ +# Upstream-readiness completion report — MinerU-ROCm + +Branch: `docs/upstream-readiness-hardening` (9 commits on top of `main` @ `6a64af5`). +Report date: 2026-07-26. Working from the audit in +[`upstream-readiness-audit.md`](upstream-readiness-audit.md). + +## 1. Completed items (mapped to the brief) + +| Brief phase | Done | Where | +|---|---|---| +| 1 — Audit report | ✅ | `docs/audit/upstream-readiness-audit.md` | +| 2 — MinerU version boundary | ✅ | `REPRO.yaml` `validated_software`; README/repro/hardware-matrix/cards; `check_version_boundary_present` gate | +| 3 — Hashes / claim strength | ✅ | `reproducibility.level: artifact-manifested` + `limitations`; "byte-exact" → "SHA-256-pinned where recorded"; `scripts/generate_evidence_manifest.py` + `evidence/evidence-manifest.json` (44 files) | +| 4 — Reproduction attestation | ✅ | `docs/reproducibility-attestation.md` + `REPRO.yaml` `reproduction` block; independence levels defined | +| 5 — Smoke test | ✅ | `scripts/smoke_test.py` (L0–L4 + `--check-only`) + `docs/smoke-test.md` | +| 6 — Doc/evidence CI | ✅ | `schemas/{repro,model-card}.schema.json` + `scripts/validate_schemas.py`; 6 new `check_repo` gates; ruff `select` pinned; `ci.yml` runs schemas + manifest --check + smoke --check-only | +| 7 — Unified wording | ✅ | status enum enforced; model-card field drift fixed; stale Windows-HIP status fixed; disclaimer added | +| 8 — Upstream-citable evidence | ✅ | `docs/upstream-documentation-source.md` (matrix linked to immutable `v1.1.0`) | + +## 2. Files changed (42 files, 13 new) — why + +**Blocker fixes (commits 1–2):** +- `scripts/check_repo.py` — read `REPRO.yaml` (was crashing on the renamed file); +6 consistency gates. +- `README.md`, `README.zh-CN.md`, `SUPPORT.md`, `CHANGELOG.md`, `docs/{reproducibility,benchmark-methodology,HANDOFF-*,upstream,upstream-pr}/*`, `model_card.pipeline.json` — dead `reproducibility.lock.yaml` → `REPRO.yaml`. +- `reproduce.md`, `reproduce.pipeline.md` — fabricated CLI (`python -m mineru_rocm runner`/`serve-vlm`) + `/root` hardcodes rewritten to the real `mineru-rocm predict/score` CLI with env-var paths. +- `examples/serve_vlm_vllm.sh`, `src/mineru_rocm/backends/vlm.py` — `/root/.cache` hardcodes → HF-cache-derived, portable. + +**Consistency + wording (commits 3–4):** `REPRO.yaml` (`validated_software`, `reproducibility.level`/`limitations`, release→`v1.1.0`, `reproduction`); `docs/{hardware-matrix,architecture,known-gaps,how-it-works,reproducibility}.md`; all 4 model cards (unified fields, `community-tested`); README corrupted benchmark block repaired (stray `\x0b` byte, duplicated paragraph, orphaned rows). + +**New machinery (commits 5–9):** `schemas/*.schema.json`; `scripts/{validate_schemas,generate_evidence_manifest,smoke_test}.py`; `docs/{smoke-test,reproducibility-attestation,upstream-documentation-source}.md`; `evidence/evidence-manifest.json` + `evidence/smoke-run-2026-07-26.log`; `tests/test_{validate_schemas,generate_evidence_manifest,smoke_test}.py` + extended `tests/test_check_repo.py`; `pyproject.toml` (ruff pin, jsonschema dep); `.github/workflows/ci.yml`. + +## 3. Key decisions + +- **MinerU boundary:** validated = 3.4.4 (commit `0dfc9460`); MinerU 4.x explicitly `not_validated`. No 4.x log exists anywhere; absence became an explicit exclusion. +- **Reproducibility level:** `artifact-manifested` + `environment-pinned`, **not** byte-exact (`table_sha256`, canary GT are `not_recorded`). "Byte-exact" wording removed from user-facing surfaces; `not_recorded` retained with reasons. +- **Reproduction independence:** primary = `repeat run` (maintainer); Windows = `second-machine reproduction` (same team); **no** third-party. Therefore every result stays `community-tested`, not `independently-reproduced`. +- **Release provenance:** lock refreshed from stale `v0.1.0` to the latest existing tag `v1.1.0` (commit `339a5e5`, tag object `04be8fe…`). No new tag created. +- **Benchmark canonical source:** `REPRO.yaml` `benchmark.full_1651`; README/cards derive from it and are drift-gated. +- **Model-card unified field model:** `schema_version`/`model_id`/`platforms`/`badge`/`omnidocbench_version`/`overall`/`artifacts`/`mineru_software` for all four cards; status enum enforced. +- **ruff:** `select` pinned to `E4/E7/E9/F` (the classic default the tree was clean under) so CI is deterministic across ruff versions; 23 pre-existing files left un-`ruff format`-ed (CI doesn't enforce format; avoids unrelated churn). All new files are format-clean. +- **CHANGELOG historical entries** left intact (timestamped records + the literal `source: verified` field name). + +## 4. Tests actually run (this environment: 4× gfx1100/W7900 host, ROCm torch present, upstream `mineru` NOT installed) + +| Check | Command | Result | +|---|---|---| +| Unit + gate tests | `python -m pytest -q` | **203 passed** | +| Lint | `ruff check .` | **All checks passed!** (pinned select) | +| Schema gate | `python scripts/validate_schemas.py` | **clean** | +| Evidence drift | `python scripts/generate_evidence_manifest.py --check` | **44 files match** | +| Smoke (no-GPU dry-run) | `python scripts/smoke_test.py --check-only` | **rc=0** (L0 OK, L1 OK/SKIP, L2 OK, L3 SKIP dry-run) | +| Smoke (GPU host, L0–L3) | `python scripts/smoke_test.py --level 3` | **rc=0** — L0–L3 OK; log saved `evidence/smoke-run-2026-07-26.log` | +| Repo consistency gate | `python scripts/check_repo.py` | **clean** | +| REUSE lint | `reuse lint` | **NOT RUN** — `reuse` not installed in this env (CI installs it via `[dev]`); SPDX headers covered by `check_repo.check_spdx` | +| Clean-venv install | `python -m venv .venv-test && pip install -e ".[dev]"` | **BLOCKED** — wheel builds + all deps resolve ("Successfully built mineru-rocm"), but the install write step hits a container disk-space limit (`OSError [Errno 28]`). Environmental, not a repo defect; installability otherwise proven by `check_install_smoke` (in the green gate) | + +Negative-test coverage for the 7 acceptance regressions is in the test suite +(rename lock → `check_lock_present`/`check_no_dead_lock_filename`; README drift → +`check_readme_lock_values`; undefined status → schema enum; nonexistent CLI → +`check_doc_cli_commands_exist`; `/root` hardcode → `check_no_root_in_doc_commands`; +schema invalid → `validate_schemas`; no-GPU dry-run → `smoke --check-only`). + +## 5. Unfinished items (honest) + +| Item | Why | Follow-up | Blocks upstream PR? | +|---|---|---|---| +| Real MinerU inference smoke (pipeline backend) | **DONE 2026-07-26 — SUCCESS on stable torch.** Env assembled into `/root/venvs/mineru-pipeline`: mineru 3.4.4 + `mineru[pipeline]` + `six`/`albumentations` + **stable** `torch 2.11.0+rocm7.2` + `torchvision 0.26.0+rocm7.2` + `transformers 4.57.6`. Pipeline ran end-to-end on `examples/sample.png` (gfx1100/W7900, 48 GB): all models loaded (`DocAnalysis init done`), layout + OCR + formula + table ran, produced `sample.md` (376 bytes of **correct OCR**) + the full bundle (layout/span/origin PDFs, content_list, middle/model JSON). Models downloaded via `MINERU_MODEL_SOURCE=huggingface` + `HF_ENDPOINT=https://hf-mirror.com`. **Key correction:** the earlier "blocked by torch build / needs nightly" was a **misdiagnosis** — the real blockers were (1) an incomplete weight download (layout `model.safetensors` missing) and (2) the FastAPI task runner swallowing the resulting `OSError`; stable torch works fine (a direct GPU sanity test — matmul, CNN forward — also passed). Recorded in `evidence/smoke-real-pipeline-2026-07-26.log`; REPRO.yaml `reproducibility.limitations` notes the benchmark nightly torch isn't publicly installable but the stable torch is verified. | None needed for the pipeline smoke (done). The validated benchmark nightly torch remains what produced the 1651-page scores; stable torch is the public-reproduction path. | No — was always a confidence booster; now it positively confirms the runtime on a publicly-installable stack. | +| `table_sha256`, canary GT SHAs | Table sub-model file not local; canary subset not materialized | `mineru-rocm canary materialize --full-gt … --manifest … --out …`; SHA the table sub-model under `PDF-Extract-Kit-1.0@ed6b654c`; re-run `generate_evidence_manifest.py` | No — recorded as `not_recorded` with reasons; level honestly set to `artifact-manifested` | +| Upstream MinerU commit pin for the official anchor | Anchors read from the moving `master` README | Re-read `opendatalab/MinerU` README at a pinned commit; set `official_reference.source_url` to `blob//README.md` | No — already disclosed as a non-immutable citation | +| `reuse lint` | `reuse` absent from this env | Runs in CI after `pip install -e ".[dev]"`; SPDX headers already verified by `check_repo` | No | +| Third-party independent reproduction | No outside reproducer recorded | Append a `third_party_independent_reproduction` entry to the attestation + promote badges to `independently-reproduced` | Soft — the contribution is honest as `community-tested`; independent repro strengthens but is not required for a docs-only PR | +| Push / tag / PR | No authorization | On approval: merge branch, cut a release tag, then replace the repo-relative links in `docs/upstream-documentation-source.md` §5 with `blob//…` | No (process, not readiness) | + +## 6. Upstream-readiness verdict + +**Ready with documented limitations.** The repo is now internally consistent, +reproducible within documented bounds, honestly attested, CI-gated (schemas, +evidence drift, doc consistency, smoke dry-run, ruff), and the frozen evidence +is citable via immutable `v1.1.0` links. Limitations are explicit and +non-misleading: `community-tested` only (no independent reproduction), MinerU +4.x not validated, `artifact-manifested` (not byte-exact), single-GPU, two +GPU SKUs only. + +## 7. Recommended next steps (no auto-execution) + +1. Review/merge this branch; cut a release tag; re-pin §5 links in + `docs/upstream-documentation-source.md` to the tag. +2. Refresh the staged `docs/upstream-pr/AMD.md.section.zh.md` against + `docs/upstream-documentation-source.md` (it still cites the standalone 95.46 + and pre-Phase-2 wording). +3. Optionally run the real-backend smoke (§5 row 1) to raise confidence before + opening the upstream docs PR. +4. Open the upstream docs-only PR to `opendatalab/MinerU` + `docs/zh/usage/acceleration_cards/AMD.md` (issue #5288), citing this repo's + immutable evidence. diff --git a/docs/benchmark-methodology.md b/docs/benchmark-methodology.md index 4348ee4..c4ba6ec 100644 --- a/docs/benchmark-methodology.md +++ b/docs/benchmark-methodology.md @@ -4,7 +4,7 @@ > the upstream opendatalab/MinerU models on AMD ROCm and scores them on > OmniDocBench v1.6 — the same dataset + scorer the upstream uses. It does NOT > claim bit-identical results vs a CUDA baseline (no same-engine CUDA control -> exists). The numbers below are reproducible from `reproducibility.lock.yaml`. +> exists). The numbers below are reproducible from `REPRO.yaml`. ## What this repo measures @@ -49,7 +49,7 @@ a controlled CUDA-vs-ROCm hardware-level comparison. - **The official anchor is community-verified.** The upstream README "Local Deployment" table records vlm-engine 95.30 / pipeline 86.47; we cite these as the comparison anchors (aligned to that table, not as official support). - See `reproducibility.lock.yaml` (`benchmark.official_reference: source: + See `REPRO.yaml` (`benchmark.official_reference: source: verified`). The prior withdrawn unofficial anchor is no longer cited. ## The Overall formula @@ -85,6 +85,6 @@ mineru-rocm score \ mineru-rocm manifest verify --pred-dir ``` -Every input is pinned in `reproducibility.lock.yaml` (code commit, model weight +Every input is pinned in `REPRO.yaml` (code commit, model weight SHAs, dataset SHA, scorer commit, environment versions). See that file for the exact provenance. diff --git a/docs/hardware-matrix.md b/docs/hardware-matrix.md index 6ec19bc..877d606 100644 --- a/docs/hardware-matrix.md +++ b/docs/hardware-matrix.md @@ -3,12 +3,23 @@ These entries describe completed, evidence-backed runs. They are not general support claims for untested AMD architectures. +**Status legend** (defined in `docs/reproducibility-attestation.md`): +`community-tested` (run by the repo maintainers on the listed stack), +`independently-reproduced` (reproduced by a separate operator per the +attestation), `experimental` (override/patch/single-machine, not extrapolable), +`not-tested`, `unsupported`. Every row is `community-tested` unless stated. + | Platform | Hardware | Backend | Software path | OmniDocBench v1.6 | Status | |---|---|---|---|---:|---| -| `linux-rocm` | Radeon PRO W7900, gfx1100, 48 GB | MinerU2.5-Pro VLM | vLLM ROCm, bf16, one GPU | 95.56 | `community` | -| `linux-rocm` | Radeon PRO W7900, gfx1100, 48 GB | MinerU 3.4.4 pipeline | PyTorch ROCm | 86.48 | `community` | -| `windows-hip` | Ryzen AI MAX+ 395 / Radeon 8060S, shared memory | MinerU 3.4.4 pipeline | Windows ROCm PyTorch + DirectML | 86.59 | `community` | -| `windows-hip` | Not yet established | MinerU2.5-Pro VLM | Serving runtime undecided | — | `community-wanted` | +| `linux-rocm` | Radeon PRO W7900, gfx1100, 48 GB | MinerU2.5-Pro VLM | vLLM ROCm, bf16, one GPU | 95.56 | `community-tested` | +| `linux-rocm` | Radeon PRO W7900, gfx1100, 48 GB | MinerU 3.4.4 pipeline | PyTorch ROCm | 86.48 | `community-tested` | +| `windows-hip` | Ryzen AI MAX+ 395 / Radeon 8060S, shared memory | MinerU 3.4.4 pipeline | Windows ROCm PyTorch + DirectML | 86.59 | `community-tested` | +| `windows-hip` | Ryzen AI MAX+ 395 / Radeon 8060S, shared memory | MinerU2.5-Pro VLM | llama.cpp HIP (`vlm-llamacpp`) | 95.46 | `community-tested` | + +> These configurations are community-tested on the explicitly listed hardware +> and software versions. They do not represent generic AMD GPU support, MinerU +> upstream support guarantees, or AMD product certification. Other RDNA3 / +> RDNA4 GPUs and ROCm versions are `not-tested`, not `should work`. ## Windows execution detail diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 4499be0..8dcbc8d 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -51,6 +51,6 @@ This repo ships **two** model cards for one upstream model family (MinerU 2.5 / | **primary** (registry row) | `model_card.json` + `hub/registry.yaml` `mineru2.5` | the **VLM** (MinerU 2.5 served via vLLM-on-ROCm) | reproduced, linux-rocm `community` (Overall **95.56** platform CDM, 1651/1651, gate PASS) | | **secondary** (Plan 1 result) | `model_card.pipeline.json` + platform-specific cards + the README comparison table | the **MinerU 3.4 pipeline** (layout → OCR → table → formula, in-process) | reproduced: linux-rocm **86.48**, windows-hip **86.59**, both `community` | -The platform's `hub/registry.yaml` carries **one row per `model_id`**, and that row is the VLM (`mineru2.5`). The pipeline (`mineru-pipeline`) is a secondary card inside this same repo, surfaced via `model_card.pipeline.json` + the README table — **so no new registry row is needed for the pipeline**. The VLM result now fills the primary `model_card.json` (Overall **95.56** platform CDM, badge linux-rocm `community`); the pipeline card stays as the secondary entry. (The prior standalone-path VLM Overall was **95.46** — Formula CDM 96.46 — on the same 1651 predictions; superseded by the platform CDM 95.56, see `docs/reproducibility.md`.) +The platform's `hub/registry.yaml` carries **one row per `model_id`**, and that row is the VLM (`mineru2.5`). The pipeline (`mineru-pipeline`) is a secondary card inside this same repo, surfaced via `model_card.pipeline.json` + the README table — **so no new registry row is needed for the pipeline**. The VLM result now fills the primary `model_card.json` (Overall **95.56** platform CDM, badge linux-rocm `community-tested`); the pipeline card stays as the secondary entry. (The prior standalone-path VLM Overall was **95.46** — Formula CDM 96.46 — on the same 1651 predictions; superseded by the platform CDM 95.56, see `docs/reproducibility.md`.) -> **Registry update note:** the `hub/registry.yaml` lives in the **separate** platform repo [`OmniDocBench-ROCm`](https://github.com/AIwork4me/OmniDocBench-ROCm) and carries the primary `mineru2.5` VLM row. The VLM's `windows-hip` entry remains `community-wanted`; the supplementary pipeline Windows result is recorded here in `model_card.pipeline.windows-hip.json` and its conformant bundle. (`verified` still requires maintainer reproduction.) +> **Registry update note:** the `hub/registry.yaml` lives in the **separate** platform repo [`OmniDocBench-ROCm`](https://github.com/AIwork4me/OmniDocBench-ROCm) and carries the primary `mineru2.5` VLM row. The VLM's `windows-hip` entry is `community-tested` (Phase 2 llama.cpp HIP, Overall 95.46 — see `model_card.windows-hip.json`); the supplementary pipeline Windows result is recorded in `model_card.pipeline.windows-hip.json` and its conformant bundle. Promotion to `independently-reproduced` requires a separate-operator reproduction per `docs/reproducibility-attestation.md`. diff --git a/docs/known-gaps.md b/docs/known-gaps.md index 28f2ba4..ee60141 100644 --- a/docs/known-gaps.md +++ b/docs/known-gaps.md @@ -1,7 +1,8 @@ # Known gaps -Track the open items for `MinerU-ROCm` here. A `verified` badge requires these -to be resolved or explicitly scoped. This list covers the pipeline path +Track the open items for `MinerU-ROCm` here. Promotion from `community-tested` +to `independently-reproduced` requires these to be resolved or explicitly scoped +(see `docs/reproducibility-attestation.md`). This list covers the pipeline path (MinerU 3.4); VLM-path gaps are tracked separately. - **ONNX tables run on CPU on Linux ROCm.** The RapidOCR / RapidTable ONNX @@ -11,9 +12,11 @@ to be resolved or explicitly scoped. This list covers the pipeline path - **`MINERU_FORMULA_CH_SUPPORT` must stay off.** Setting it to `true` pulls native PaddlePaddle-GPU, which has no ROCm wheel for this stack. Keep it `false` and use the fallback formula renderer. -- **Windows-HIP VLM is now verified.** Both Phase 1 pipeline (Overall 86.59) - and Phase 2 VLM llama.cpp HIP (Overall 95.46, identical to the Linux vLLM - CDM-scored 95.56 baseline) are complete on Strix Halo with conformant CDM bundles. +- **Windows-HIP VLM is `community-tested`.** Both Phase 1 pipeline (Overall + 86.59) and Phase 2 VLM llama.cpp HIP (Overall 95.46, same prediction set as + the Linux vLLM platform-CDM 95.56) are complete on Strix Halo with conformant + CDM bundles. Not `independently-reproduced` (see + `docs/reproducibility-attestation.md`). ## Deferred upstream-readiness backlog @@ -30,4 +33,4 @@ These items do not invalidate the committed full-set result bundles: W7900 GPUs; `gpu_count_per_benchmark: 1` is the load-bearing fact. - **`gpu-smoke.yml`** is not present because a safe self-hosted gfx1100 runner topology has not been established. -- **Windows-HIP VLM** is `community` (Overall 95.46, Phase 2 complete); the pipeline result is `community` (Overall 86.59, Phase 1 complete). Independent reproduction is pending; badge upgrades to `verified` on successful independent repro. +- **Windows-HIP VLM** is `community-tested` (Overall 95.46, Phase 2 complete); the pipeline result is `community-tested` (Overall 86.59, Phase 1 complete). Promotion to `independently-reproduced` is pending a separate-operator reproduction per `docs/reproducibility-attestation.md`. diff --git a/docs/reproducibility-attestation.md b/docs/reproducibility-attestation.md new file mode 100644 index 0000000..f82d75e --- /dev/null +++ b/docs/reproducibility-attestation.md @@ -0,0 +1,82 @@ +# Reproduction attestation + +This document states **who ran what** to produce the published MinerU-ROCm +results, and at what **independence level**. It is the single place that defines +the support-status terms used across the README, hardware matrix, model cards, +and `REPRO.yaml`. Nothing in this repo is "independently reproduced" unless it +is listed here as such. + +## Status terms (the only valid values) + +| Term | Meaning | +|---|---| +| `community-tested` | Run end-to-end by **this repo's maintainers** on the listed hardware/software. Self-attested. | +| `independently-reproduced` | Reproduced by a **separate operator** meeting one of the independence levels below, from a clean environment. | +| `experimental` | Depends on an override/patch, single-machine only, or not extrapolable to other hardware. Document the caveat. | +| `not-tested` | Never run. Do **not** write "should work" / "likely works" / "compatible in theory" — those are speculation, not a status. | +| `unsupported` | Known not to work in this configuration. | + +## Independence levels (who counts as "independent") + +| Level | Definition | +|---|---| +| `repeat run` | The **same operator** re-executes the recipe (catches flakiness, not independence). | +| `second-machine reproduction` | The **same team/org, different machine**. Stronger than a repeat; still not independent of the team. | +| `internal independent reproduction` | **Same org, different person** who did not author the original run. | +| `third-party independent reproduction` | A reproducer with **no org relation** to the maintainers, from a clean environment. **Only this level** may be called "independent reproduction". | + +The same team running the recipe a second time — even on a second machine — is +`second-machine reproduction`, **not** third-party independent reproduction. + +## What was actually run + +**Confirmed from `REPRO.yaml` + the committed run manifests + git history.** The +operator for every run below is the **AIwork4me maintainer** (the repo operator; +git commit author `AIwork4me`, `pyproject.toml` author). No outside reproducer +is recorded. + +### Primary run — Linux ROCm (gfx1100, Radeon PRO W7900) + +| Field | Value | Evidence | +|---|---|---| +| Operator | AIwork4me maintainer (repo operator) | git history; `pyproject.toml` authors | +| Organization relation | Maintainer of this repo | — | +| Hardware | AMD gfx1100, W7900 48 GB, ROCm 7.2, bf16, 1 GPU | `REPRO.yaml` `environment` | +| Recipe | `mineru[pipeline]==3.4.4`; VLM via vLLM-on-ROCm + `mineru_vl_utils` 1.0.5 | `REPRO.yaml` `rocm_recipe`, `environment.venvs` | +| Predictions regenerated by operator | **yes** | `benchmark_run_commits` (`pipeline` e05eec3, `vlm_vllm` b75f788) recorded in each `run_manifest.json` | +| Scores regenerated by operator | **yes** (pinned OmniDocBench scorer commit `2b161d0`) | `REPRO.yaml` `omnidocbench.scorer_commit` | +| Results | pipeline 86.48; vlm-vllm 95.56 (platform CDM) / 95.46 (standalone, same predictions) | `REPRO.yaml` `benchmark.full_1651` | +| Independence level | **`repeat run`** (maintainer self-attested) | — | + +### Second-machine run — Windows HIP (Strix Halo, Radeon 8060S) + +| Field | Value | Evidence | +|---|---|---| +| Operator | AIwork4me maintainer (**same** operator/team) | `REPRO.yaml` `benchmark.windows_hip` | +| Hardware | AMD Ryzen AI MAX+ 395 / Radeon 8060S, Windows ROCm 7.2.1 | `REPRO.yaml` `benchmark.windows_hip.hardware` | +| Predictions regenerated | **yes** (pipeline prediction_source_commit `66e662f`; VLM llama.cpp HIP `274c4fe`) | `REPRO.yaml` `benchmark.windows_hip.full_1651.*.prediction_source_commit` | +| Scores regenerated | **yes** | `results/omnidocbench/v16/windows-hip/*_metric_result.json` | +| Results | pipeline 86.59; vlm-llamacpp 95.46 | `model_card.pipeline.windows-hip.json`, `model_card.windows-hip.json` | +| Independence level | **`second-machine reproduction`** (same team, different machine — **not** independent) | — | + +### What was NOT independently reproduced + +- **No `internal independent reproduction`** (same org, different person) is recorded. +- **No `third-party independent reproduction`** is recorded. +- The VLM 95.46 → 95.56 change is a **re-scoring** of the **same** 1651 + predictions (same team, same prediction commit `b75f788`) under a different CDM + scoring configuration — it is **not** a new inference run and **not** + independent reproduction. +- A clean-environment reproduction (fresh OS, fresh dependency install, fresh + model download) has **not** been automated or recorded by any operator. + +## Consequence for badges + +Every published result in this repo is therefore `community-tested` +(maintainer self-attested), **not** `independently-reproduced`. Promotion of any +row to `independently-reproduced` requires appending a `third-party independent +reproduction` entry to the primary-run table above, with the reproducer's +identity/role, machine, and a clean-environment note. Until then, no doc, model +card, or `REPRO.yaml` field may use the phrase "independently verified / +independent reproduction / verified by another system" — those phrases must link +here, and here they resolve to "not yet". diff --git a/docs/reproducibility.md b/docs/reproducibility.md index d81b30d..9b3e466 100644 --- a/docs/reproducibility.md +++ b/docs/reproducibility.md @@ -1,6 +1,10 @@ # Reproducibility -A score is only meaningful if someone else can reproduce it from the committed repo. The canonical published results are the **OmniDocBench-ROCm platform CDM bundles** under `results/omnidocbench/v16/{linux-rocm,windows-hip}/`; the standalone `mineru-rocm` CLI remains available for developer debugging. `reproducibility.lock.yaml` is the single source of truth (pinned commits, byte-exact weight/GT SHAs, scorer commit, environments, the metric formula, the official anchors, and the ROCm recipes). +A score is only meaningful if someone else can reproduce it from the committed repo. The canonical published results are the **OmniDocBench-ROCm platform CDM bundles** under `results/omnidocbench/v16/{linux-rocm,windows-hip}/`; the standalone `mineru-rocm` CLI remains available for developer debugging. `REPRO.yaml` is the single source of truth (pinned commits, SHA-256-pinned weights/GT **where recorded** — see `reproducibility.level: artifact-manifested` and the `not_recorded` limitations, plus the scorer commit, environments, the metric formula, the official anchors, and the ROCm recipes). + +## MinerU version boundary + +Validated with MinerU **3.4.4** (upstream commit `0dfc9460cd9ab693b9af60ae3fbffd7bc111b062`; `mineru_vl_utils` 1.0.5). This is recorded structurally in `REPRO.yaml` under `validated_software`. **MinerU 4.x / 4.0 alpha has not been validated** and is explicitly marked `not_validated`; results must not be extrapolated to untested MinerU releases. The CLI/backend names used here (`pipeline`, `vlm-vllm`, `vlm-llamacpp`) are the ones validated against MinerU 3.4.4; `vlm-transformers` is sample-only. ## Results (OmniDocBench v1.6, full 1651 pages) @@ -84,9 +88,9 @@ The tested server launch script is tracked in `examples/serve_vlm_vllm.sh`; its ## Non-determinism -- **pipeline**: deterministic within a fixed environment (byte-identical predictions). The Windows-HIP score is +0.11 pp from linux-rocm, within the declared cross-platform tolerance. +- **pipeline**: deterministic within a fixed environment (the recorded reruns produced matching predictions on the same machine). The Windows-HIP score is +0.11 pp from linux-rocm, within the declared cross-platform tolerance. - **VLM (vLLM)**: the platform CDM-scored Overall is **95.56**; the prior standalone-path score was **95.46** (same 1651 predictions, scorer revision `2b161d0`). The Δ +0.10 pp is the Formula-CDM submetric (96.46 → 96.73), attributable to the CDM scoring configuration at scoring time — **not** run-to-run inference drift. Inference-level run-to-run drift was not separately isolated. ## Provenance in the lock -`reproducibility.lock.yaml` records: the `mineru_rocm.release` tag + release commit + annotated-tag object SHA, the **per-run benchmark commits** (`mineru_rocm.benchmark_run_commits`, also recorded in each `run_manifest.json`), the **upstream `mineru`/`mineru_vl_utils` git commits** (resolved via `git ls-remote` against the release tags), byte-exact weight + GT SHAs, the scorer commit, both venvs' full environment, the official anchors, and the metric formula. Deferred fields (`canary_*`, `table_sha256`) are annotated `→ docs/known-gaps.md`. +`REPRO.yaml` records: the `mineru_rocm.release` tag + release commit + annotated-tag object SHA, the **per-run benchmark commits** (`mineru_rocm.benchmark_run_commits`, also recorded in each `run_manifest.json`), the **upstream `mineru`/`mineru_vl_utils` git commits** (resolved via `git ls-remote` against the release tags), SHA-256-pinned weight + GT artifacts **where recorded** (deferred fields `canary_*`, `table_sha256` are `not_recorded` — see `reproducibility.limitations` and `→ docs/known-gaps.md`), the scorer commit, both venvs' full environment, the official anchors, and the metric formula. diff --git a/docs/smoke-test.md b/docs/smoke-test.md new file mode 100644 index 0000000..da9133e --- /dev/null +++ b/docs/smoke-test.md @@ -0,0 +1,57 @@ +# Smoke test + +A fast, portable way to confirm a successful MinerU-ROCm install **without** +running the full OmniDocBench benchmark. It works on CI without an AMD GPU +(`--check-only`) and on a real GPU host (full run). + +```bash +python scripts/smoke_test.py --check-only # CI / no GPU: imports + CLI + test data, no inference +python scripts/smoke_test.py # GPU host: adds the smoke-backend inference (Level 3) +python scripts/smoke_test.py --level 4 --vlm-url http://127.0.0.1:8265/v1 # optional VLM health check +``` + +Exit code is `0` when every run level is `OK` (or `SKIP` with a reason); `1` on +any `FAIL`. Use `--json` for machine-readable output. + +## Levels + +| Level | What it checks | GPU? | +|---|---|---| +| 0 — environment | OS, Python, `mineru_rocm` version, torch + HIP version, GPU name, `torch.cuda.is_available()`, device count, upstream `mineru` version (if installed). Informational. | no | +| 1 — device | `torch.cuda.is_available()` / `device_count()` / `get_device_name(0)`. Fails (full mode) if no AMD GPU is visible. | yes | +| 2 — owned CLI | `mineru-rocm --help` + `mineru-rocm doctor`; advisory `mineru --version` (warns if ≠ the validated 3.4.4). | no | +| 3 — inference | Minimal inference via the **GPU-free `smoke` backend** (the adapter contract: one placeholder `.md` per image). Confirms the install end-to-end. | no | +| 4 — VLM (optional) | Health-checks an OpenAI-compatible VLM endpoint at `--vlm-url/models`. **No assumption** of vLLM or llama.cpp. | no | + +### Why `torch.cuda.*` on an AMD GPU + +A ROCm PyTorch mirrors the CUDA Python surface, so `torch.cuda.is_available()` +and `torch.cuda.get_device_name(0)` report the **AMD** GPU. This is expected — +it is **not** evidence that NVIDIA CUDA is in use. Level 0 prints `hip` (the +ROCm/HIP version) alongside the cuda booleans to make this explicit. + +## Paths + +The smoke test takes **no machine-private path**. Use `--img-dir` (default +`examples/`) and `--out-dir` (default: a fresh temp dir). Nothing is hardcoded +to `/root/...` or any user home. + +## Interpreting results + +- **Green `--check-only`** → the package imports, the CLI works, and the test + data + adapter are present. This is the install-success signal for CI and for + users who do not (yet) have a GPU. It is **not** a GPU runtime verification. +- **Green full run (Level 3)** → additionally, the adapter contract runs + end-to-end and produces non-empty output. The `smoke` backend writes + placeholder text, **not real OCR** — real OCR is the OmniDocBench benchmark + (`reproduce.md`, `reproduce.pipeline.md`), which is the advanced verification. +- **Level 1 FAIL** → no AMD GPU is visible to the installed ROCm PyTorch. Fix + the runtime (ROCm driver, `HSA_OVERRIDE_GFX_VERSION`, `HIP_VISIBLE_DEVICES`) + before running GPU levels. + +## Validated MinerU version + +The smoke test references MinerU **3.4.4** (the validated version, pinned in +`REPRO.yaml` `validated_software`). MinerU 4.x has not been validated; if +`mineru --version` reports something else, the smoke test warns but does not +fail (the boundary is documented, not enforced at install time). diff --git a/docs/upstream-documentation-source.md b/docs/upstream-documentation-source.md new file mode 100644 index 0000000..cc18d82 --- /dev/null +++ b/docs/upstream-documentation-source.md @@ -0,0 +1,98 @@ +# Upstream documentation source — MinerU AMD/ROCm community contribution + +> **Purpose.** This is the **fact source** for a community docs-only contribution +> to upstream [`opendatalab/MinerU`](https://github.com/opendatalab/MinerU) +> (`docs/zh/usage/acceleration_cards/AMD.md`, issue +> [#5288](https://github.com/opendatalab/MinerU/issues/5288)). It is **not** the +> PR itself and is **not** an official-support claim. Every row below traces to +> immutable evidence in [`AIwork4me/MinerU-ROCm`](https://github.com/AIwork4me/MinerU-ROCm). + +## 1. Support nature + +- **community-maintained** — this repo is maintained by AIwork4me, independent of + the MinerU Team / OpenDataLab / AMD. +- **community-tested** — every configuration below was run end-to-end by the + repo maintainers on the listed hardware/software. Status terms are defined in + [`docs/reproducibility-attestation.md`](reproducibility-attestation.md). +- **not** official MinerU support, **not** an AMD product certification, and + **not** extrapolable to unlisted AMD GPUs, ROCm versions, or MinerU releases. +- **Validated with MinerU 3.4.4.** MinerU 4.x / 4.0 alpha has **not** been + validated; results must not be extrapolated to untested MinerU releases. + +## 2. Validated matrix (immutable evidence) + +Every row links to a committed `metric_result.json` at the immutable tag +**`v1.1.0`** (commit `339a5e5`). Independence level is `repeat run` (Linux, +maintainer) or `second-machine reproduction` (Windows, same team) — **not** +third-party independent reproduction (see the attestation). + +| GPU | gfx | OS | ROCm/HIP | PyTorch | MinerU | Backend | Path | Overall | Status | Evidence (v1.1.0) | +|---|---|---|---|---|---|---|---|---:|---|---| +| Radeon PRO W7900 (48 GB) | gfx1100 | Linux | 7.2 | 2.9.1+rocm7.2 | 3.4.4 | pipeline | linux-rocm | 86.48 | community-tested | [metric_result](https://github.com/AIwork4me/MinerU-ROCm/blob/v1.1.0/results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_metric_result.json) | +| Radeon PRO W7900 (48 GB) | gfx1100 | Linux | 7.2 | 2.9.1+rocm7.2 | 3.4.4 (+vl_utils 1.0.5) | vlm-vllm | linux-rocm | 95.56 | community-tested | [metric_result](https://github.com/AIwork4me/MinerU-ROCm/blob/v1.1.0/results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_metric_result.json) | +| Ryzen AI MAX+ 395 / Radeon 8060S (Strix Halo) | gfx1151 | Windows | 7.2.1 (HIP 7.2.53211) | 2.9.1+rocm7.2.1 | 3.4.4 | pipeline | windows-hip | 86.59 | community-tested | [metric_result](https://github.com/AIwork4me/MinerU-ROCm/blob/v1.1.0/results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_metric_result.json) | +| Ryzen AI MAX+ 395 / Radeon 8060S (Strix Halo) | gfx1151 | Windows | 7.2.1 (HIP 7.2.53211) | llama.cpp HIP b9892 | 3.4.4 (+vl_utils 1.0.5) | vlm-llamacpp | windows-hip | 95.46 | community-tested | [metric_result](https://github.com/AIwork4me/MinerU-ROCm/blob/v1.1.0/results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_metric_result.json) | + +Official contextual anchors (upstream README "Local Deployment" table, read from +the `master` branch — the upstream commit was **not** pinned, so these are a +community-verified reference, not an immutable citation): pipeline **86.47**, +vlm-engine **95.30**. + +## 3. Minimal install + smoke test + +```bash +pip install -e ".[dev]" # GPU-free core + tooling +python scripts/smoke_test.py --check-only # imports + CLI + test data (no GPU) +python scripts/smoke_test.py # adds the GPU-free smoke-backend inference (Level 3) +``` + +The GPU runtime (ROCm PyTorch, MinerU models, vLLM-on-ROCm) is a separate, +larger environment pinned in `REPRO.yaml` — see `docs/reproducibility.md` and +`docs/smoke-test.md`. A ROCm PyTorch exposes `torch.cuda.*` for the AMD GPU; that +is expected, not NVIDIA CUDA. + +## 4. Known limitations (must accompany any upstream citation) + +- **MinerU version boundary:** validated with 3.4.4 only; MinerU 4.x not validated. +- **GPU range:** gfx1100 (W7900, RDNA3) and gfx1151 (Strix Halo, RDNA3.5) only. + Other RDNA3/RDNA4 GPUs and ROCm versions are `not-tested`, not "should work". +- **Single GPU:** each benchmark used 1 GPU (no tensor parallel); the W7900 host + has 4 GPUs but `gpu_count_per_benchmark = 1`. +- **Windows/Linux:** the Windows VLM uses llama.cpp HIP (`vlm-llamacpp`), not + vLLM; the Windows pipeline uses Windows ROCm PyTorch + DirectML with a + documented `slanet-plus.onnx` CPU override. +- **Override:** the VLM (vLLM-on-ROCm) build required + `HSA_OVERRIDE_GFX_VERSION=11.0.0` on gfx1100; the pipeline backend did not. +- **Reproducibility level:** `artifact-manifested` + `environment-pinned`, **not** + byte-exact (`table_sha256` and canary GT SHAs are `not_recorded`; see + `REPRO.yaml` `reproducibility.limitations`). +- **VLM determinism:** the VLM (vLLM, bf16) shows run-to-run inference drift that + was not separately isolated; only the scoring delta was attributed. +- **No GPU CI:** public CI runs static + dry-run checks only + (`smoke_test.py --check-only`); GPU runtime verification is a human step. +- **Benchmark scope:** OmniDocBench v1.6 full 1651 pages. Empty outputs: 2/1651 + VLM, 0–1/1651 pipeline. + +## 5. Immutable references + +- **Already-frozen evidence (result bundles, model cards):** linked at the + immutable tag `v1.1.0` in §2. +- **Canonical sources updated by this qualification branch** — pin to the next + release tag after merge: + - [`REPRO.yaml`](../REPRO.yaml) — lock (validated_software, reproducibility.level, reproduction attestation). + - [`docs/reproducibility-attestation.md`](reproducibility-attestation.md) — status terms + independence levels + who-ran-what. + - [`docs/reproducibility.md`](reproducibility.md) — full ROCm recipe. + - [`docs/smoke-test.md`](smoke-test.md) + [`scripts/smoke_test.py`](../scripts/smoke_test.py) — portable smoke test. + - [`scripts/generate_evidence_manifest.py`](../scripts/generate_evidence_manifest.py) + [`evidence/evidence-manifest.json`](../evidence/evidence-manifest.json) — committed-evidence SHA-256 manifest. + +> **TODO (post-merge):** replace the repo-relative links above with +> `https://github.com/AIwork4me/MinerU-ROCm/blob//…` once this +> branch is merged and a new release tag is cut. Do **not** cite a tag that does +> not yet exist. (Current branch HEAD: `244e909`.) + +## 6. Drafted upstream section + +A staged draft of the AMD.md section for the upstream PR lives at +[`docs/upstream-pr/AMD.md.section.zh.md`](upstream-pr/AMD.md.section.zh.md). It +should be refreshed against this document before submission (it currently cites +the standalone 95.46 and pre-Phase-2 wording). diff --git a/docs/upstream-pr/AMD.md.section.zh.md b/docs/upstream-pr/AMD.md.section.zh.md index e3c62ae..2536d02 100644 --- a/docs/upstream-pr/AMD.md.section.zh.md +++ b/docs/upstream-pr/AMD.md.section.zh.md @@ -16,7 +16,7 @@ mineru 3.4.4(pipeline);mineru_vl_utils 1.0.5 + vLLM-on-ROCm 0.16.1(VLM ### 关键配置:HSA_OVERRIDE_GFX_VERSION(仅 gfx1100 已验证) - **pipeline 后端**(进程内 PyTorch):**无需** override —— PyTorch-ROCm 自动识别 gfx1100。 - **VLM 后端经 vLLM**:**必须** `export HSA_OVERRIDE_GFX_VERSION=11.0.0`(实测该 vLLM-on-ROCm 版本所需;仅 gfx1100 验证,非 MinerU 源码要求)。 -- Windows 原生 ROCm 可能不识别此 override(windows-hip 未验证)。 +- Windows:本节配置针对 Linux/vLLM。Windows-HIP 上的 VLM 已单独验证(经 llama.cpp HIP / `vlm-llamacpp`,Strix Halo gfx1151,Overall 95.46;详见 [AIwork4me/MinerU-ROCm](https://github.com/AIwork4me/MinerU-ROCm) 的 `docs/upstream-documentation-source.md`),不走 vLLM,也不依赖此 override。 ### 性能:重要 - **pipeline**:无需补丁,~3–6 s/页,速度正常。 @@ -32,4 +32,4 @@ mineru 3.4.4(pipeline);mineru_vl_utils 1.0.5 + vLLM-on-ROCm 0.16.1(VLM 与上游 README "Local Deployment" 表的公开锚点**一致**(非受控 CUDA-vs-ROCm 对照):pipeline 86.47(Δ+0.01pp)、vlm-engine 95.30(Δ+0.16pp)。 完整可复现锁定(代码 commit、权重 SHA256、评分器 commit、环境)见 -[reproducibility.lock.yaml](https://github.com/AIwork4me/MinerU-ROCm/blob/main/reproducibility.lock.yaml)。 +[REPRO.yaml](https://github.com/AIwork4me/MinerU-ROCm/blob/main/REPRO.yaml)。 diff --git a/docs/upstream-pr/issue-5288-comment.md b/docs/upstream-pr/issue-5288-comment.md index 7bad8ba..97443e9 100644 --- a/docs/upstream-pr/issue-5288-comment.md +++ b/docs/upstream-pr/issue-5288-comment.md @@ -6,4 +6,4 @@ Thanks for the earlier discussion. We've prepared the docs-only contribution and 2. **`README.md`** — extend the GPU-Acceleration **row only** to mention AMD ROCm (gfx1100/RDNA3); the Accuracy row is unchanged. 3. **English mirror** — the acceleration_cards family is currently zh-only; would you like an `en/usage/acceleration_cards/AMD.md` mirror in the same PR, or keep it zh-only for consistency? -Two questions: (a) is this scope welcome as one docs-only PR? (b) any sign-off/DCO convention we should follow (we didn't find a CONTRIBUTING.md)? Full reproducibility lock: https://github.com/AIwork4me/MinerU-ROCm/blob/main/reproducibility.lock.yaml +Two questions: (a) is this scope welcome as one docs-only PR? (b) any sign-off/DCO convention we should follow (we didn't find a CONTRIBUTING.md)? Full reproducibility lock: https://github.com/AIwork4me/MinerU-ROCm/blob/main/REPRO.yaml diff --git a/docs/upstream/mineru-issue-5288-review.md b/docs/upstream/mineru-issue-5288-review.md index 5cf7b54..f1c92b6 100644 --- a/docs/upstream/mineru-issue-5288-review.md +++ b/docs/upstream/mineru-issue-5288-review.md @@ -13,7 +13,7 @@ the repo files/logs/lock, not from the issue's prior wording. | `README.md` | "48 GB ×4" → "1 GPU per benchmark; host has 4× W7900" | | `README.zh-CN.md` | same fix (zh) | | `docs/benchmark-methodology.md` | same fix | -| `reproducibility.lock.yaml` | `rocm_recipe.gpu_arch` comment: removed untested-GPU list (7900 XTX/GRE/7800 XT/7700 XT/7600) → "only gfx1100 tested" | +| `REPRO.yaml` | `rocm_recipe.gpu_arch` comment: removed untested-GPU list (7900 XTX/GRE/7800 XT/7700 XT/7600) → "only gfx1100 tested" | | `scripts/check_repo.py` | added `check_version_consistency` (key-version agreement + ROCm-overclaim guardrail) | | `tests/test_check_repo.py` | +2 tests (clean-on-repo, overclaim-flagged) | @@ -77,4 +77,4 @@ the repo files/logs/lock, not from the issue's prior wording. - `scripts/check_repo.py` → clean (incl. the new `check_version_consistency`). - `git diff --check` → no whitespace errors. - Markdown tables in the draft → 3/3 well-formed (consistent column counts). -- Draft link `…/MinerU-ROCm/blob/main/reproducibility.lock.yaml` → HTTP 200. +- Draft link `…/MinerU-ROCm/blob/main/REPRO.yaml` → HTTP 200. diff --git a/docs/upstream/mineru-issue-5288.md b/docs/upstream/mineru-issue-5288.md index 03dbf1c..17c74e9 100644 --- a/docs/upstream/mineru-issue-5288.md +++ b/docs/upstream/mineru-issue-5288.md @@ -2,7 +2,7 @@ > Staging file for the live GitHub issue. Evidence-first, professional, scoped to what > was actually tested. Every number/version below is sourced from -> `reproducibility.lock.yaml`, `results/omnidocbench/v1.6/**`, the run manifests, or +> `REPRO.yaml`, `results/omnidocbench/v1.6/**`, the run manifests, or > the upstream README. Status: community-validated configuration — **not** an > official-support claim, and the upstream comparison is **not** a controlled > CUDA-vs-ROCm parity measurement. @@ -23,7 +23,7 @@ We ran MinerU 3.4 (pipeline) and MinerU2.5-Pro (VLM served via vLLM) end-to-end ## Verified configuration -All entries are recorded (with source comments) in [`reproducibility.lock.yaml`](https://github.com/AIwork4me/MinerU-ROCm/blob/main/reproducibility.lock.yaml). +All entries are recorded (with source comments) in [`REPRO.yaml`](https://github.com/AIwork4me/MinerU-ROCm/blob/main/REPRO.yaml). | Component | Value (verified) | |---|---| @@ -84,7 +84,7 @@ pip install -e . mineru-rocm --help # sanity: prints the 6 subcommands ``` -`pip install -e .` installs **only** the lightweight `mineru-rocm` CLI package (core dep: PyYAML). It does **not** install the ROCm runtime, ROCm PyTorch, MinerU models, vLLM-on-ROCm, or the OmniDocBench scorer — those are a separate, larger environment pinned in the lock (see [`docs/reproducibility.md`](https://github.com/AIwork4me/MinerU-ROCm/blob/main/docs/reproducibility.md) and [`reproducibility.lock.yaml`](https://github.com/AIwork4me/MinerU-ROCm/blob/main/reproducibility.lock.yaml)). +`pip install -e .` installs **only** the lightweight `mineru-rocm` CLI package (core dep: PyYAML). It does **not** install the ROCm runtime, ROCm PyTorch, MinerU models, vLLM-on-ROCm, or the OmniDocBench scorer — those are a separate, larger environment pinned in the lock (see [`docs/reproducibility.md`](https://github.com/AIwork4me/MinerU-ROCm/blob/main/docs/reproducibility.md) and [`REPRO.yaml`](https://github.com/AIwork4me/MinerU-ROCm/blob/main/REPRO.yaml)). ### 2. Full benchmark dependencies (not provided by the CLI install) diff --git a/evidence/evidence-manifest.json b/evidence/evidence-manifest.json new file mode 100644 index 0000000..728fc9d --- /dev/null +++ b/evidence/evidence-manifest.json @@ -0,0 +1,230 @@ +{ + "algorithm": "sha256", + "tool": "scripts/generate_evidence_manifest.py", + "tool_version": "1.0", + "generated_at": "2026-07-26T23:34:04Z", + "git_commit": "bb64b8bce782632a5e2b35c3d2a4006ab2e5662e", + "file_count": 44, + "entries": [ + { + "path": "REPRO.yaml", + "sha256": "fe08578d2605daf6c77de6465daeb653e9c085489c263b83719992cdd037b8f3", + "size": 10966 + }, + { + "path": "eval/configs/omnidocbench_v16.yaml", + "sha256": "133228816344acf4375ef187cf3656f16fcdef6398d31ad8d7a35c18f3b8e32f", + "size": 363 + }, + { + "path": "model_card.json", + "sha256": "a3da0dcee95ad275d19705a1e5d12be9feee405156dcd183fc0e63e90f653dbe", + "size": 1737 + }, + { + "path": "model_card.pipeline.json", + "sha256": "6897a0e37dc7cf5bc99ae9ec978b7f074b96e37441ecb23da46c4b78efb25b33", + "size": 2108 + }, + { + "path": "model_card.pipeline.windows-hip.json", + "sha256": "2cede7af73adad6aa702fc50965e0b710a2a8befd59ecccb8958a028199885c0", + "size": 1947 + }, + { + "path": "model_card.windows-hip.json", + "sha256": "9850bb358960b3e9f58ba6d79231543e0eb2460bdda7fb3b6b68cea204ebc1f3", + "size": 2543 + }, + { + "path": "results/omnidocbench/v1.6/pipeline/_errors.jsonl", + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "size": 0 + }, + { + "path": "results/omnidocbench/v1.6/pipeline/metric_result.json", + "sha256": "9caebd71d498856661bc6979253226fbe0ce5f26cd8565c19b6ba319be7352aa", + "size": 10153 + }, + { + "path": "results/omnidocbench/v1.6/pipeline/predict.log.tail", + "sha256": "70d77c7c58deb42c1ec877a4ab4e6b952dee5dda38ae16f6ad5df286a4a726fa", + "size": 384 + }, + { + "path": "results/omnidocbench/v1.6/pipeline/run_manifest.json", + "sha256": "b380f73fa66860b85c0e83317cc975602a1631fd76c44fc9f76a64f8da469278", + "size": 933 + }, + { + "path": "results/omnidocbench/v1.6/pipeline/sample_predictions/manifest.json", + "sha256": "7fd99ec978bbba9e80395e847000e975845929249b935c5dadb3a0d90b370e56", + "size": 1472 + }, + { + "path": "results/omnidocbench/v1.6/vlm-vllm/_errors.jsonl", + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "size": 0 + }, + { + "path": "results/omnidocbench/v1.6/vlm-vllm/metric_result.json", + "sha256": "a038b2bad304409fa40d07f1e22eec25548708b717a907b6ce74440d008eeda7", + "size": 10247 + }, + { + "path": "results/omnidocbench/v1.6/vlm-vllm/predict.log.tail", + "sha256": "6cef14a65b2a4f6df06d890b35e63c42c7af2449a7824404210c3d2e9a93537b", + "size": 254 + }, + { + "path": "results/omnidocbench/v1.6/vlm-vllm/run_manifest.json", + "sha256": "8638bdf228476e596a4c8b37dbaf513362c5f553beebe291a8013ac0af181f21", + "size": 954 + }, + { + "path": "results/omnidocbench/v1.6/vlm-vllm/sample_predictions/manifest.json", + "sha256": "ab1428563ac38a082cf53fb0f7429c2087917775109b39d84a9fcd06da776082", + "size": 1472 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_dataset_identity.json", + "sha256": "f29578a108406deb4d173a1db457608d1cee09c85f6b21bef83be1d9dc03861b", + "size": 236 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_metric_result.json", + "sha256": "d5e9ea8e9119496a38d9d8805d0ae0f1373794f0d2d4d976e29cdcd6e821b0ea", + "size": 15886 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_prediction_manifest.json", + "sha256": "65a6ff4ce252906fc134780bcd41367f9fcf76d25b461e7b864d1e858d7976ac", + "size": 335004 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_provenance.json", + "sha256": "5119bacf78406d64e2cb864cd68ed6296384d6ba1a2ed163b419f2b8e5ea8678", + "size": 2287 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_run_stats.json", + "sha256": "5d42d58da4ad1eff39333d2b0a0bd146cf1aeab9a981fd98a553459c9dec3cf6", + "size": 249364 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_run_summary.json", + "sha256": "a382c7299eec7bd5ac6f814984a0a2eccf670da5bde2e23f1f985ed9f1a64bbd", + "size": 1227 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_scoring_config.yaml", + "sha256": "133228816344acf4375ef187cf3656f16fcdef6398d31ad8d7a35c18f3b8e32f", + "size": 363 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_dataset_identity.json", + "sha256": "f29578a108406deb4d173a1db457608d1cee09c85f6b21bef83be1d9dc03861b", + "size": 236 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_metric_result.json", + "sha256": "20a06cddcd5865cecff72933abca8c59b308325b5381c21ca5c724ffe98aef0c", + "size": 15939 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_prediction_manifest.json", + "sha256": "9f4bf28e81e3bd9c981c1e879eafa29bde96594303accbb44e0b871590b097eb", + "size": 334878 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_provenance.json", + "sha256": "41d1390025f149cf2c53b16d7000521aaf7d9e39f139fccf90b01fea483027e8", + "size": 2345 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_run_stats.json", + "sha256": "9699846aac2e23377a28ebfcde90e585c3299385d52547d39436772ef1ffa84a", + "size": 249421 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_run_summary.json", + "sha256": "de46689a8b4b575eda65692cb0db9ca8852e1c8e9dcf868be2d6523381645f63", + "size": 1664 + }, + { + "path": "results/omnidocbench/v16/linux-rocm/mineru2.5_v16_quick_match_cdm_scoring_config.yaml", + "sha256": "133228816344acf4375ef187cf3656f16fcdef6398d31ad8d7a35c18f3b8e32f", + "size": 363 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_dataset_identity.json", + "sha256": "f29578a108406deb4d173a1db457608d1cee09c85f6b21bef83be1d9dc03861b", + "size": 236 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_metric_result.json", + "sha256": "42528062645fff6d43536f05997bd2aa75effe59eb4baf695970bbf1fedbd176", + "size": 15878 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_prediction_manifest.json", + "sha256": "0c36bb569ceb6728c5e805181a0782fbc062fcfc3bd7932dc4af4d3b2c7b2f95", + "size": 335087 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_provenance.json", + "sha256": "73ec8e27ece08d8c4a8ad1380c76a033c7421ec5a121a37cbe5b00b00f479647", + "size": 2322 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_run_stats.json", + "sha256": "f5a922c3d84b085a3079478c810ef9025a60544d1fa84c720cae2d1352ee2224", + "size": 308989 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_run_summary.json", + "sha256": "3d527bb7c1a3ca676bbd48f854322002c77120bdea537a7f1d9bc424bea0048c", + "size": 1014 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru-pipeline_v16_quick_match_cdm_scoring_config.yaml", + "sha256": "b5a95007de166d7e45b857d8278781e87e52a8ec9478d25faea1e2ddb14a57a1", + "size": 752 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_dataset_manifest.json", + "sha256": "a45cd84b04ad8b793e775089640e6b681209abea33ead54c1828ddca35fae496", + "size": 42208096 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_metric_result.json", + "sha256": "b67d8a793c4a8ca74206e594a2aa7221564273a79dd6ed85d6a1ef3becc5cbae", + "size": 16046 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_prediction_manifest.json", + "sha256": "e264a2c77f04bd0ba8d7e9a0ca13f62961d4c3a01c8166290140d092afc39d9b", + "size": 334945 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_provenance.json", + "sha256": "119ec89c53796269320572a3690f47f49cbb07faa4dda93700c299d628317ac4", + "size": 2453 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_run_stats.json", + "sha256": "b97ea338ea7febb6f51674dae9ef2d12dcff34fa38975722eb94f52c7af1ad4c", + "size": 282674 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_run_summary.json", + "sha256": "4f34d7e366cc94ec454518a48c6804271dcdff6fcd7058ae578de64282b8493e", + "size": 1784 + }, + { + "path": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_scoring_config.yaml", + "sha256": "101d5b211aa499fb63e22fe93591d070119581de97cbcf12780ed9a7a068c26b", + "size": 776 + } + ] +} diff --git a/evidence/smoke-real-pipeline-2026-07-26.log b/evidence/smoke-real-pipeline-2026-07-26.log new file mode 100644 index 0000000..b944fd2 --- /dev/null +++ b/evidence/smoke-real-pipeline-2026-07-26.log @@ -0,0 +1,16 @@ +# Real MinerU pipeline smoke — SUCCESS on stable torch (2026-07-26) +# Env: /root/venvs/mineru-pipeline | python 3.12.3 | torch 2.11.0+rocm7.2 | torchvision 0.26.0+rocm7.2 | transformers 4.57.6 | mineru 3.4.4 + mineru[pipeline] +# GPU: AMD gfx1100 (W7900, 48 GB), CUDA_VISIBLE_DEVICES=0, ROCm 7.2 +# Source: MINERU_MODEL_SOURCE=huggingface, HF_ENDPOINT=https://hf-mirror.com (HF direct blocked) +# Input: examples/sample.png (2000x1500) -> read_fn (image->pdf bytes) -> do_parse(backend=pipeline) +# Result: rc=0; DocAnalysis init done; sample.md = 376 bytes of correct OCR; +# full bundle: sample.md + sample_layout.pdf + sample_span.pdf + sample_origin.pdf + content_list.json + middle.json + model.json +# OCR sample (first lines of sample.md): +# ## Who Am I? +# +# ## • Min-Te Sun (Peter) Sun +# +# — An associate professor of Computer Science & Information Engineering, National Central University +# +# NOTE: the validated benchmark torch (REPRO.yaml 2.14.0.dev20260717+rocm7.2, an AMD nightly) is NOT on the +# public pytorch.org rocm7.2 index; this smoke proves the PUBLICLY INSTALLABLE stable torch works. diff --git a/evidence/smoke-run-2026-07-26.log b/evidence/smoke-run-2026-07-26.log new file mode 100644 index 0000000..d575641 --- /dev/null +++ b/evidence/smoke-run-2026-07-26.log @@ -0,0 +1,4 @@ +[L0] OK env: {"device_count": 4, "device_name_0": "AMD Radeon Graphics", "hip": "7.2.53211-e1a6bc5663", "mineru_rocm": "1.0.0", "os": "Linux-6.8.0-79-generic-x86_64-with-glibc2.39", "python": "3.12.3", "torch": "2.9.1+rocm7.2.1.gitff65f5bc", "torch_cuda_available": true, "upstream_mineru": "not installed"} +[L1] OK device: cuda_available=True count=4 name='AMD Radeon Graphics' (ROCm via the torch.cuda API) +[L2] OK cli: mineru-rocm --help + doctor OK. upstream `mineru` not on PATH (optional; the GPU runtime installs it separately) +[L3] OK inference: backend=smoke produced 2 non-empty .md file(s) in /tmp/mineru-rocm-smoke-REDACTED (placeholder text, not real OCR) diff --git a/examples/serve_vlm_vllm.sh b/examples/serve_vlm_vllm.sh index 3b473d5..8275c4c 100755 --- a/examples/serve_vlm_vllm.sh +++ b/examples/serve_vlm_vllm.sh @@ -7,7 +7,19 @@ export VLLM_USE_V1=1 # MANDATORY: v1 logits-processo export HF_ENDPOINT="${HF_ENDPOINT:-https://huggingface.co}" export LD_LIBRARY_PATH="/opt/rocm/lib:${LD_LIBRARY_PATH:-}" -MODEL_DIR="$(ls -d /root/.cache/huggingface/models--opendatalab--MinerU2.5-Pro-2605-1.2B/snapshots/* | head -1)" +# MODEL_DIR may be set explicitly; otherwise derive it from the HF cache +# ($HF_HOME, then the default ~/.cache/huggingface). No host-specific path is +# hardcoded — this works on any machine that has the snapshot cached. +_HF_CACHE="${HF_HOME:-$HOME/.cache/huggingface}" +if [ -z "${MODEL_DIR:-}" ]; then + MODEL_DIR="$(ls -d "${_HF_CACHE}/models--opendatalab--MinerU2.5-Pro-2605-1.2B/snapshots/"*/ 2>/dev/null | head -1)" +fi +if [ -z "${MODEL_DIR:-}" ]; then + echo "[serve_vlm_vllm] MinerU2.5-Pro snapshot not found under ${_HF_CACHE}." >&2 + echo "[serve_vlm_vllm] Set MODEL_DIR explicitly, or pre-cache the repo with:" >&2 + echo "[serve_vlm_vllm] huggingface-cli download opendatalab/MinerU2.5-Pro-2605-1.2B" >&2 + exit 1 +fi REPO="$(cd "$(dirname "$0")/.." && pwd)" PORT="${PORT:-8265}" LOG="${LOG:-/tmp/vlm-vllm.log}" diff --git a/model_card.json b/model_card.json index d281ee1..7eaba4e 100644 --- a/model_card.json +++ b/model_card.json @@ -3,9 +3,10 @@ "model_id": "mineru2.5", "model_version": "2605", "platforms": ["linux-rocm"], - "badge": {"linux-rocm": "community", "windows-hip": "community"}, + "badge": {"linux-rocm": "community-tested", "windows-hip": "community-tested"}, "eval_date": "2026-07-21", "omnidocbench_version": "v1.6", + "mineru_software": {"mineru": "3.4.4", "mineru_commit": "0dfc9460cd9ab693b9af60ae3fbffd7bc111b062", "mineru_vl_utils": "1.0.5", "mineru_4": "not_validated"}, "overall": 95.56, "submetrics": { "text_edit_dist": 0.0359, diff --git a/model_card.pipeline.json b/model_card.pipeline.json index 5fbb74b..ba57953 100644 --- a/model_card.pipeline.json +++ b/model_card.pipeline.json @@ -3,9 +3,10 @@ "model_id": "mineru-pipeline", "model_version": "3.4.4", "platforms": ["linux-rocm", "windows-hip"], - "badge": {"linux-rocm": "community", "windows-hip": "community"}, + "badge": {"linux-rocm": "community-tested", "windows-hip": "community-tested"}, "eval_date": "2026-07-21", "omnidocbench_version": "v1.6", + "mineru_software": {"mineru": "3.4.4", "mineru_commit": "0dfc9460cd9ab693b9af60ae3fbffd7bc111b062", "mineru_4": "not_validated"}, "overall": 86.48, "submetrics": { "text_edit_dist": 0.0566, @@ -19,7 +20,7 @@ "vram": "48 GB per GPU", "rocm_driver": "ROCm 7.2.1" }, - "note": "Supplementary model card for the MinerU 3.4 pipeline. The top-level score and hardware remain the canonical linux-rocm result recorded in reproducibility.lock.yaml; the platform-specific Windows-HIP result is in model_card.pipeline.windows-hip.json. This is not the canonical OmniDocBench-ROCm registry card — see model_card.json (mineru2.5) for the primary registry card.", + "note": "Supplementary model card for the MinerU 3.4 pipeline. The top-level score and hardware remain the canonical linux-rocm result recorded in REPRO.yaml; the platform-specific Windows-HIP result is in model_card.pipeline.windows-hip.json. This is not the canonical OmniDocBench-ROCm registry card — see model_card.json (mineru2.5) for the primary registry card.", "artifacts": { "provenance": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_provenance.json", "run_summary": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_run_summary.json", diff --git a/model_card.pipeline.windows-hip.json b/model_card.pipeline.windows-hip.json index 85bdf0f..4729cc4 100644 --- a/model_card.pipeline.windows-hip.json +++ b/model_card.pipeline.windows-hip.json @@ -3,7 +3,7 @@ "model_id": "mineru-pipeline", "model_version": "3.4.4", "platforms": ["windows-hip"], - "badge": {"windows-hip": "community"}, + "badge": {"windows-hip": "community-tested"}, "eval_date": "2026-07-23", "omnidocbench_version": "v1.6", "overall": 86.59, diff --git a/model_card.windows-hip.json b/model_card.windows-hip.json index d0e730b..0c2c709 100644 --- a/model_card.windows-hip.json +++ b/model_card.windows-hip.json @@ -1,7 +1,14 @@ { + "schema_version": 1, "model_id": "mineru2.5", - "platform": "windows-hip", + "model_version": "2605", + "platforms": ["windows-hip"], + "badge": {"windows-hip": "community-tested"}, + "eval_date": "2026-07-24", + "omnidocbench_version": "v1.6", + "mineru_software": {"mineru": "3.4.4", "mineru_commit": "0dfc9460cd9ab693b9af60ae3fbffd7bc111b062", "mineru_vl_utils": "1.0.5", "mineru_4": "not_validated"}, "backend": "vlm-llamacpp", + "overall": 95.46, "backend_detail": { "server": "llama.cpp HIP (build b9892, commit ee445f93)", "quantization": "Q8_0 (text + mmproj)", @@ -34,11 +41,18 @@ }, "comparison": { "baseline": "Linux vLLM on ROCm (1,651 pages)", - "baseline_overall": 95.46, - "delta_overall": 0.0 + "baseline_overall": 95.56, + "delta_overall": -0.1 }, - "bundle": "results/omnidocbench/v16/windows-hip/", - "source_commit": "274c4fe", - "community_badge": "community", - "_note": "badge remains 'community' pending independent reproduction per release-checklist.md" -} \ No newline at end of file + "artifacts": { + "bundle_readme": "results/omnidocbench/v16/windows-hip/README.md", + "metric_result": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_metric_result.json", + "prediction_manifest": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_prediction_manifest.json", + "provenance": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_provenance.json", + "run_stats": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_run_stats.json", + "run_summary": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_run_summary.json", + "scoring_config": "results/omnidocbench/v16/windows-hip/mineru2.5_v16_quick_match_cdm_scoring_config.yaml" + }, + "source_commit": "274c4fe6ecc9ae2a1d19c5cea75a4a9cdd55e31a", + "note": "Windows-HIP VLM community-tested result (Phase 2). The baseline_overall is the Linux vLLM platform-CDM 95.56 (same prediction set, different platform); the prior standalone-path Linux score was 95.46. Not independently-reproduced; see docs/reproducibility-attestation.md." +} diff --git a/pyproject.toml b/pyproject.toml index aef77e4..b9ecff7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,7 +40,7 @@ windows-pipeline = [ "mineru[pipeline]==3.4.4; platform_system == 'Windows'", "onnxruntime-directml==1.24.4; platform_system == 'Windows'", ] -dev = ["pytest>=8", "ruff>=0.6", "reuse>=1.3", "build", "numpy>=1.26"] +dev = ["pytest>=8", "ruff>=0.6", "reuse>=1.3", "build", "numpy>=1.26", "jsonschema>=4"] [project.urls] Homepage = "https://github.com/AIwork4me/MinerU-ROCm" @@ -61,5 +61,9 @@ mineru_rocm = ["data/*.yaml"] # Production code (src/ + scripts/) is fully ruff-clean. Tests use a legacy # compact style (semicolons, single-letter locals, unused context-manager # bindings) that is tolerated via per-file-ignores rather than churned. +# `select` is pinned to the classic default (E4/E7/E9/F) so `ruff check .` is +# deterministic across ruff versions — newer ruff releases expand the default +# rule set, which would otherwise turn a clean tree red on version drift. [tool.ruff.lint] +select = ["E4", "E7", "E9", "F"] per-file-ignores."tests/**" = ["E701", "E702", "E741", "F841"] diff --git a/reproduce.md b/reproduce.md index 5c0ec56..89faeb1 100644 --- a/reproduce.md +++ b/reproduce.md @@ -1,22 +1,28 @@ --- model_id: mineru2.5 -backend: vllm +backend: vlm-vllm hardware: - gpu: "AMD gfx1100" + gpu: "AMD gfx1100 (Radeon PRO W7900)" vram_min_gb: 48 environment: - type: docker + type: venv rocm: "7.2" + mineru: "3.4.4" + mineru_vl_utils: "1.0.5" + vllm: "vLLM-on-ROCm" +validated_software: + mineru: "3.4.4 (validated)" + mineru_4: "not_validated" command: | - python -m mineru_rocm runner \ - --backend vlm-vllm --platform linux-rocm \ - --server-url http://127.0.0.1:8265/v1 --api-model-name mineru-pro \ - --img-dir /root/datasets/OmniDocBench_data/images \ - --out-dir /tmp/mineru-predictions - - omnidocbench-rocm run --stage score --platform linux-rocm --cdm \ - --predictions-dir /tmp/mineru-predictions \ - --run-stats /tmp/mineru-predictions/_run_stats.json --version v16 + # 1. Serve MinerU2.5-Pro via vLLM-on-ROCm on GPU 0 (needs the VLM venv python). + VLM_VENV_BIN=/path/to/vlm-venv/bin/python bash examples/serve_vlm_vllm.sh + # 2. Predict (infer venv). + mineru-rocm predict --backend vlm-vllm --platform linux-rocm \ + --gt-json "$GT_JSON" --images-dir "$IMAGES_DIR" --pred-dir "$PRED_DIR" \ + --server-url http://127.0.0.1:8265/v1 --api-model-name mineru-pro + # 3. Score (separate scorer venv). + mineru-rocm score --gt-json "$GT_JSON" --pred-dir "$PRED_DIR" --label vlm-vllm \ + --omnidocbench-repo "$OMNIDOCBENCH_REPO" --venv-python "$SCORER_PY" expected_overall: value: 95.56 tolerance: 0.5 @@ -24,35 +30,56 @@ expected_overall: # Reproduce MinerU2.5-Pro VLM 95.56 on AMD ROCm +> Validated with MinerU **3.4.4** (commit `0dfc9460`, pinned in `REPRO.yaml`). +> MinerU 4.x has **not** been validated; results must not be extrapolated to +> untested MinerU releases. See `docs/reproducibility-attestation.md` for who +> ran what. + ## Prerequisites ```bash -rocminfo | grep -E "Name:|VRAM" # must show gfx1100 + ≥48 GB +rocminfo | grep -E "Name:|VRAM" # must show gfx1100 + >=48 GB ls -la /dev/kfd # must exist ``` +Set your own paths (no machine-private defaults are assumed): + +```bash +export GT_JSON=/path/to/OmniDocBench.json +export IMAGES_DIR=/path/to/OmniDocBench/images +export OMNIDOCBENCH_REPO=/path/to/OmniDocBench # the scorer checkout (REPRO.yaml: omnidocbench.scorer_commit) +export SCORER_PY=/path/to/OmniDocBench/.venv/bin/python # separate scorer venv +export PRED_DIR=/path/to/out-vlm +``` + ## Quickstart ```bash -# Start mineru-pro server -python -m mineru_rocm serve-vlm --port 8265 - -# Run adapter -python -m mineru_rocm runner --backend vlm-vllm --platform linux-rocm \ - --server-url http://127.0.0.1:8265/v1 --api-model-name mineru-pro \ - --img-dir /root/datasets/OmniDocBench_data/images \ - --out-dir /tmp/mineru-predictions - -# Score -omnidocbench-rocm run --stage score --platform linux-rocm --cdm \ - --predictions-dir /tmp/mineru-predictions \ - --run-stats /tmp/mineru-predictions/_run_stats.json --version v16 +# 1. Serve MinerU2.5-Pro (vLLM-on-ROCm). The script honors MODEL_DIR / HF_HOME +# overrides and requires VLM_VENV_BIN (the VLM venv python). +VLM_VENV_BIN=/path/to/vlm-venv/bin/python bash examples/serve_vlm_vllm.sh +bash examples/wait_vlm.sh + +# 2. Predict (infer venv). +mineru-rocm predict --backend vlm-vllm --platform linux-rocm \ + --gt-json "$GT_JSON" --images-dir "$IMAGES_DIR" --pred-dir "$PRED_DIR" \ + --server-url http://127.0.0.1:8265/v1 --api-model-name mineru-pro +mineru-rocm validate --gt-json "$GT_JSON" --pred-dir "$PRED_DIR" +mineru-rocm manifest verify --pred-dir "$PRED_DIR" + +# 3. Score (scorer venv). +mineru-rocm score --gt-json "$GT_JSON" --pred-dir "$PRED_DIR" --label vlm-vllm \ + --omnidocbench-repo "$OMNIDOCBENCH_REPO" --venv-python "$SCORER_PY" ``` ## Expected output -Overall **95.56** (±0.5). Text 96.41, Table TEDS 93.54%, Formula CDM 96.73%. +Overall **95.56** (±0.5). Text accuracy (1−EditDist) 96.41, Table TEDS 93.54%, +Formula CDM 96.73%. (`vlm-transformers` is sample-only and not part of this +recipe.) ## If it fails -See [OmniDocBench-ROCm pitfalls](https://github.com/AIwork4me/OmniDocBench-ROCm/docs/pitfalls.md). \ No newline at end of file +Run `mineru-rocm doctor` for an environment probe, then see +[`docs/reproducibility.md`](docs/reproducibility.md) and +[`docs/smoke-test.md`](docs/smoke-test.md). diff --git a/reproduce.pipeline.md b/reproduce.pipeline.md index ff16687..37ad174 100644 --- a/reproduce.pipeline.md +++ b/reproduce.pipeline.md @@ -2,19 +2,20 @@ model_id: mineru-pipeline backend: pipeline hardware: - gpu: "AMD gfx1100" + gpu: "AMD gfx1100 (Radeon PRO W7900)" vram_min_gb: 48 environment: type: venv rocm: "7.2" + mineru: "3.4.4" +validated_software: + mineru: "3.4.4 (validated)" + mineru_4: "not_validated" command: | - mineru-rocm predict pipeline \ - --platform linux-rocm --img-dir /root/datasets/OmniDocBench_data/images \ - --out-dir /tmp/mineru-pipeline-predictions - - omnidocbench-rocm run --stage score --platform linux-rocm --cdm \ - --predictions-dir /tmp/mineru-pipeline-predictions \ - --run-stats /tmp/mineru-pipeline-predictions/_run_stats.json --version v16 + mineru-rocm predict --backend pipeline --platform linux-rocm \ + --gt-json "$GT_JSON" --images-dir "$IMAGES_DIR" --pred-dir "$PRED_DIR" + mineru-rocm score --gt-json "$GT_JSON" --pred-dir "$PRED_DIR" --label pipeline \ + --omnidocbench-repo "$OMNIDOCBENCH_REPO" --venv-python "$SCORER_PY" expected_overall: value: 86.48 tolerance: 0.5 @@ -22,31 +23,51 @@ expected_overall: # Reproduce MinerU 3.4 Pipeline 86.48 on AMD ROCm +> Validated with MinerU **3.4.4** (commit `0dfc9460`, pinned in `REPRO.yaml`). +> MinerU 4.x has **not** been validated; results must not be extrapolated to +> untested MinerU releases. See `docs/reproducibility-attestation.md` for who +> ran what. + ## Prerequisites ```bash -rocminfo | grep -E "Name:|VRAM" # must show gfx1100 + ≥48 GB +rocminfo | grep -E "Name:|VRAM" # must show gfx1100 + >=48 GB ls -la /dev/kfd # must exist ``` +Set your own paths (no machine-private defaults are assumed): + +```bash +export GT_JSON=/path/to/OmniDocBench.json +export IMAGES_DIR=/path/to/OmniDocBench/images +export OMNIDOCBENCH_REPO=/path/to/OmniDocBench # the scorer checkout (REPRO.yaml: omnidocbench.scorer_commit) +export SCORER_PY=/path/to/OmniDocBench/.venv/bin/python # separate scorer venv +export PRED_DIR=/path/to/out-pipeline +``` + ## Quickstart ```bash -# Run pipeline adapter -mineru-rocm predict pipeline \ - --platform linux-rocm --img-dir /root/datasets/OmniDocBench_data/images \ - --out-dir /tmp/mineru-pipeline-predictions - -# Score -omnidocbench-rocm run --stage score --platform linux-rocm --cdm \ - --predictions-dir /tmp/mineru-pipeline-predictions \ - --run-stats /tmp/mineru-pipeline-predictions/_run_stats.json --version v16 +# Predict (infer venv). HSA_OVERRIDE_GFX_VERSION is harmless for the pipeline +# backend; it is required by the VLM (vLLM-on-ROCm) build, not by the pipeline. +mineru-rocm predict --backend pipeline --platform linux-rocm \ + --gt-json "$GT_JSON" --images-dir "$IMAGES_DIR" --pred-dir "$PRED_DIR" +mineru-rocm validate --gt-json "$GT_JSON" --pred-dir "$PRED_DIR" +mineru-rocm manifest verify --pred-dir "$PRED_DIR" + +# Score (scorer venv). +mineru-rocm score --gt-json "$GT_JSON" --pred-dir "$PRED_DIR" --label pipeline \ + --omnidocbench-repo "$OMNIDOCBENCH_REPO" --venv-python "$SCORER_PY" ``` ## Expected output -Overall **86.48** (±0.5). Text 94.31, Table TEDS 82.79%, Formula CDM ~83%. +Overall **86.48** (±0.5). Text accuracy (1−EditDist) 94.34, Table TEDS 82.04%, +Formula CDM 83.07%. (Windows-HIP pipeline Overall 86.59 is a separate platform +result — see `model_card.pipeline.windows-hip.json`.) ## If it fails -See [OmniDocBench-ROCm pitfalls](https://github.com/AIwork4me/OmniDocBench-ROCm/docs/pitfalls.md). \ No newline at end of file +Run `mineru-rocm doctor` for an environment probe, then see +[`docs/reproducibility.md`](docs/reproducibility.md) and +[`docs/smoke-test.md`](docs/smoke-test.md). diff --git a/schemas/model-card.schema.json b/schemas/model-card.schema.json new file mode 100644 index 0000000..b3529fc --- /dev/null +++ b/schemas/model-card.schema.json @@ -0,0 +1,46 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AIwork4me/MinerU-ROCm/schemas/model-card.schema.json", + "title": "MinerU-ROCm model card", + "description": "Unified field model for every model_card*.json. Support-status values are the documented enum (see docs/reproducibility-attestation.md).", + "type": "object", + "required": ["schema_version", "model_id", "platforms", "badge", "omnidocbench_version", "overall", "artifacts"], + "additionalProperties": true, + "properties": { + "schema_version": {"const": 1}, + "model_id": {"type": "string", "enum": ["mineru2.5", "mineru-pipeline"]}, + "model_version": {"type": "string"}, + "platforms": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": {"type": "string", "enum": ["linux-rocm", "windows-hip"]} + }, + "badge": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "string", + "enum": ["community-tested", "independently-reproduced", "experimental", "not-tested", "unsupported"] + } + }, + "eval_date": {"type": "string"}, + "omnidocbench_version": {"const": "v1.6"}, + "mineru_software": { + "type": "object", + "required": ["mineru", "mineru_4"], + "properties": { + "mineru": {"type": "string"}, + "mineru_commit": {"type": "string"}, + "mineru_vl_utils": {"type": "string"}, + "mineru_4": {"type": "string", "enum": ["not_validated", "validated"]} + }, + "additionalProperties": true + }, + "overall": {"type": "number"}, + "submetrics": {"type": "object"}, + "hardware": {"type": "object"}, + "backend": {"type": "string"}, + "artifacts": {"type": "object", "minProperties": 1} + } +} diff --git a/schemas/repro.schema.json b/schemas/repro.schema.json new file mode 100644 index 0000000..254b5c9 --- /dev/null +++ b/schemas/repro.schema.json @@ -0,0 +1,70 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AIwork4me/MinerU-ROCm/schemas/repro.schema.json", + "title": "MinerU-ROCm REPRO.yaml", + "description": "Structural contract for REPRO.yaml. The reproducibility level is a documented enum; MinerU 4.x must be explicitly not_validated. SHA-256 field formatting is enforced by scripts/validate_schemas.py (value is a 64-hex digest or the literal 'not_recorded').", + "type": "object", + "required": ["mineru_rocm", "mineru", "mineru_vl_utils", "validated_software", "model", "omnidocbench", "environment", "benchmark", "reproducibility", "reproduction"], + "additionalProperties": true, + "properties": { + "mineru_rocm": { + "type": "object", + "required": ["repo", "release", "benchmark_run_commits"], + "properties": { + "release": { + "type": "object", + "required": ["tag", "commit", "tag_object_sha"], + "properties": { + "tag": {"type": "string", "pattern": "^v[0-9]+\\.[0-9]+\\.[0-9]+"}, + "commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"}, + "tag_object_sha": {"type": "string", "pattern": "^[0-9a-f]{40}$"} + } + } + } + }, + "mineru": {"type": "object", "required": ["repo", "commit", "version"]}, + "mineru_vl_utils": {"type": "object", "required": ["repo", "commit", "version"]}, + "validated_software": { + "type": "object", + "required": ["mineru", "mineru_4"], + "properties": { + "mineru": { + "type": "object", + "required": ["version", "validated"], + "properties": {"validated": {"type": "boolean"}} + }, + "mineru_4": { + "type": "object", + "required": ["validated", "status"], + "properties": { + "validated": {"type": "boolean"}, + "status": {"type": "string", "enum": ["not_validated", "validated"]} + } + } + } + }, + "omnidocbench": {"type": "object", "required": ["version", "scorer_commit"]}, + "benchmark": {"type": "object"}, + "reproducibility": { + "type": "object", + "required": ["level"], + "properties": { + "level": {"type": "string", "enum": ["environment-pinned", "artifact-manifested", "prediction-reproduced", "score-reproduced", "byte-exact"]} + } + }, + "reproduction": { + "type": "object", + "required": ["terms_doc", "primary_run", "third_party_independent_reproduction"], + "properties": { + "primary_run": { + "type": "object", + "required": ["operator", "independence_level", "regenerated_predictions", "regenerated_scores"] + }, + "third_party_independent_reproduction": { + "type": "object", + "required": ["performed"] + } + } + } + } +} diff --git a/scripts/check_repo.py b/scripts/check_repo.py index 25106e6..c696594 100644 --- a/scripts/check_repo.py +++ b/scripts/check_repo.py @@ -39,8 +39,11 @@ def find_engine_imports(pkg_dir: Path) -> list[str]: return errs +LOCK_FILENAME = "REPRO.yaml" + + def _load_lock(): - p = REPO / "reproducibility.lock.yaml" + p = REPO / LOCK_FILENAME if not p.is_file(): return None import yaml # PyYAML is a core dep @@ -50,7 +53,7 @@ def _load_lock(): def check_lock_sections(lock) -> list[str]: if lock is None: return [] - return [f"reproducibility.lock.yaml missing section: {k}" for k in REQUIRED_LOCK_SECTIONS if k not in lock] + return [f"{LOCK_FILENAME} missing section: {k}" for k in REQUIRED_LOCK_SECTIONS if k not in lock] def check_spdx(repo=REPO) -> list[str]: @@ -165,7 +168,10 @@ def check_modelcard_lock_agreement(lock) -> list[str]: def _current_vlm_overall(lock): """The canonical current VLM Overall, sourced from the lock (single source - of truth). None when not filled.""" + of truth). None when not filled (or when the lock itself is absent, so a + missing lock reports cleanly instead of crashing the gate).""" + if lock is None: + return None return (((lock.get("benchmark") or {}).get("full_1651") or {}).get("vlm_vllm") or {}).get("overall") @@ -417,11 +423,11 @@ def check_score_commands_have_scorer_args(repo=REPO) -> list[str]: if "--omnidocbench-repo" not in block and "OMNIDOCBENCH_REPO" not in block: rel = p.relative_to(repo).as_posix() findings.append(f"{rel}: `mineru-rocm score` example lacks --omnidocbench-repo / OMNIDOCBENCH_REPO") - lockp = repo / "reproducibility.lock.yaml" + lockp = repo / LOCK_FILENAME if lockp.is_file(): for line in lockp.read_text(encoding="utf-8").splitlines(): if "mineru-rocm score" in line and "--omnidocbench-repo" not in line: - findings.append("reproducibility.lock.yaml: rocm_recipe.cli score line lacks --omnidocbench-repo") + findings.append(f"{LOCK_FILENAME}: rocm_recipe.cli score line lacks --omnidocbench-repo") return findings @@ -493,6 +499,164 @@ def check_no_internal_infra(repo=REPO) -> list[str]: return errs +# --- upstream-readiness hardening: structural-drift gates ---------------------- +# These catch the regressions surfaced by the 2026-07-26 audit: a renamed lock +# file with stale consumers, fabricated CLI in docs, /root hardcodes in commands, +# a missing MinerU version boundary, and dead markdown links. + +_DEAD_LOCK_FILENAME = "reproducibility.lock.yaml" + + +def check_lock_present(repo=REPO) -> list[str]: + """REPRO.yaml must exist. The lock was renamed in 6a64af5; a missing lock + silently disables every lock-driven check (they early-return on None).""" + if not (repo / LOCK_FILENAME).is_file(): + return [f"{LOCK_FILENAME} is missing — the reproducibility lock is required (every lock-driven check is a no-op without it)"] + return [] + + +def check_no_dead_lock_filename(repo=REPO) -> list[str]: + """No tracked text file references the pre-rename lock filename (renamed to + REPRO.yaml in 6a64af5). Excludes frozen design records (docs/superpowers/) + and the two scripts that define scan-targets by name.""" + errs = [] + for p in _git_ls_text_files(repo, _LEAK_SUFFIXES): + try: + rel = p.relative_to(repo).as_posix() + except ValueError: + continue + if rel.startswith(".git/") or rel.startswith("docs/superpowers/") or rel.startswith("docs/audit/") or rel.startswith("tests/") or rel in _LEAK_SELF_EXEMPT: + continue + if _DEAD_LOCK_FILENAME in p.read_text(encoding="utf-8", errors="ignore"): + errs.append(f"{rel}: references dead lock filename '{_DEAD_LOCK_FILENAME}' (use '{LOCK_FILENAME}')") + return errs + + +_VALID_MINERU_ROCM_SUBCOMMANDS = {"predict", "score", "doctor", "validate", "manifest", "canary"} +_VALID_MINERU_ROCM_MODULES = {"cli", "driver"} # the only modules with a __main__ guard + + +def _doc_cli_sources(repo: Path) -> list[Path]: + sources = [repo / n for n in ("README.md", "README.zh-CN.md", "reproduce.md", "reproduce.pipeline.md")] + for md in (repo / "docs").rglob("*.md"): + # Exclude frozen design records, audit reports (they quote defects by name), + # and external-contribution staging drafts (they cite upstream paths/commands). + if any(part in {"superpowers", "audit"} for part in md.parts): + continue + if any(part.startswith("upstream") for part in md.parts): + continue + sources.append(md) + return [p for p in sources if p.is_file()] + + +def check_doc_cli_commands_exist(repo=REPO) -> list[str]: + """Documented `mineru-rocm ` / `python -m mineru_rocm.` use real + subcommands/modules — catches fabricated CLI such as the + `python -m mineru_rocm runner` / `serve-vlm` that slipped into reproduce.md.""" + errs = [] + for p in _doc_cli_sources(repo): + rel = p.relative_to(repo).as_posix() + txt = p.read_text(encoding="utf-8", errors="ignore") + for m in re.finditer(r"mineru-rocm\s+([A-Za-z][\w-]*)", txt): + sub = m.group(1) + if sub not in _VALID_MINERU_ROCM_SUBCOMMANDS: + errs.append(f"{rel}: `mineru-rocm {sub}` — '{sub}' is not a known subcommand {sorted(_VALID_MINERU_ROCM_SUBCOMMANDS)}") + for m in re.finditer(r"python\s+-m\s+mineru_rocm[.\s]+([A-Za-z][\w-]*)", txt): + mod = m.group(1) + if mod not in _VALID_MINERU_ROCM_MODULES: + errs.append(f"{rel}: `python -m mineru_rocm {mod}` — module '{mod}' has no __main__ entry (valid: {sorted(_VALID_MINERU_ROCM_MODULES)})") + return errs + + +def check_no_root_in_doc_commands(repo=REPO) -> list[str]: + """No `/root/` in command contexts: bash/sh code blocks in README + reproduce + + docs, plus shell scripts under examples/ and adapter/setup/. Prose and + provenance notes (spike investigation logs, the lock's cross_check_source) + are not command contexts and are excluded.""" + errs = [] + for d in ("examples", "adapter/setup"): + root = repo / d + if not root.is_dir(): + continue + for p in root.rglob("*"): + if p.is_file() and "/root/" in p.read_text(encoding="utf-8", errors="ignore"): + errs.append(f"{p.relative_to(repo)}: shell script hardcodes /root/ (use env vars / $HOME / HF_HOME)") + md_sources = [repo / n for n in ("README.md", "README.zh-CN.md", "reproduce.md", "reproduce.pipeline.md") if (repo / n).is_file()] + for md in (repo / "docs").rglob("*.md"): + if "superpowers" not in md.parts and not md.name.startswith("spike-"): + md_sources.append(md) + for md in md_sources: + rel = md.relative_to(repo).as_posix() + txt = md.read_text(encoding="utf-8", errors="ignore") + for block in re.findall(r"```(?:bash|sh|shell|console)\n(.*?)```", txt, re.DOTALL): + if "/root/" in block: + errs.append(f"{rel}: bash code block hardcodes /root/ (use env vars / placeholders)") + return errs + + +_BOUNDARY_TOKENS = ("4.x", "MinerU 4", "mineru_4", "4.0 alpha", "not been validated", "未经验证", "not_validated") + + +def check_version_boundary_present(lock, repo=REPO) -> list[str]: + """README + docs/reproducibility.md state the validated MinerU version AND an + explicit MinerU-4.x boundary, so results are not silently extrapolated.""" + if lock is None: + return [] + version = (((lock.get("validated_software") or {}).get("mineru") or {}).get("version")) + if not version: + return [] + errs = [] + for name in ("README.md", "README.zh-CN.md", "docs/reproducibility.md"): + p = repo / name + if not p.is_file(): + continue + txt = p.read_text(encoding="utf-8", errors="ignore") + if version not in txt: + errs.append(f"{name}: does not state the validated MinerU version {version}") + if not any(tok in txt for tok in _BOUNDARY_TOKENS): + errs.append(f"{name}: no MinerU 4.x boundary statement (results could be extrapolated to untested releases)") + return errs + + +_LINK_RE = re.compile(r"\[[^\]]+\]\(([^)]+)\)") + + +def check_markdown_internal_links(repo=REPO) -> list[str]: + """Relative markdown links resolve to a tracked file/dir — catches dead links + (the old reproducibility.lock.yaml refs, deleted-file refs). External + http(s)/mailto links and same-file anchors are not checked (network/anchor + checks are out of scope).""" + errs = [] + md_files = [repo / n for n in ("README.md", "README.zh-CN.md") if (repo / n).is_file()] + for md in (repo / "docs").rglob("*.md"): + # Frozen design records, audit reports, and external-contribution staging + # drafts cite upstream/external paths that do not exist in this repo. + if any(part in {"superpowers", "audit"} for part in md.parts): + continue + if any(part.startswith("upstream") for part in md.parts): + continue + md_files.append(md) + for md in md_files: + base = md.parent + rel_doc = md.relative_to(repo).as_posix() + txt = md.read_text(encoding="utf-8", errors="ignore") + for m in _LINK_RE.finditer(txt): + target = m.group(1).split()[0] # drop an optional "title" + if "://" in target or target.startswith(("mailto:", "#")): + continue + path_part = target.split("#")[0].split("?")[0] + if not path_part: + continue + resolved = (base / path_part).resolve() + try: + resolved.relative_to(repo) + except ValueError: + continue # outside the repo (e.g. /tmp); skip + if not resolved.exists(): + errs.append(f"{rel_doc}: dead relative link {target!r} (resolves to missing {path_part})") + return errs + + def check_install_smoke() -> list[str]: """`pip install -e .` succeeds (the PEP 639 / build regression guard).""" cp = subprocess.run([sys.executable, "-m", "pip", "install", "-e", ".", "--quiet"], @@ -507,6 +671,7 @@ def main(argv=None) -> int: lock = _load_lock() readme = (REPO / "README.md").read_text(encoding="utf-8") if (REPO / "README.md").is_file() else "" findings += find_engine_imports(REPO / "src" / "mineru_rocm") + findings += check_lock_present() findings += check_lock_sections(lock) findings += check_spdx() findings += check_readme_scripts_exist(readme) @@ -519,6 +684,11 @@ def main(argv=None) -> int: findings += check_release_and_run_provenance(lock) findings += check_score_commands_have_scorer_args() findings += check_no_internal_infra() + findings += check_no_dead_lock_filename() + findings += check_doc_cli_commands_exist() + findings += check_no_root_in_doc_commands() + findings += check_version_boundary_present(lock) + findings += check_markdown_internal_links() findings += check_install_smoke() if findings: print("check_repo: " + str(len(findings)) + " finding(s):", file=sys.stderr) diff --git a/scripts/generate_evidence_manifest.py b/scripts/generate_evidence_manifest.py new file mode 100755 index 0000000..3a0eccb --- /dev/null +++ b/scripts/generate_evidence_manifest.py @@ -0,0 +1,253 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 AIwork4me +"""Generate / check a SHA-256 evidence manifest for the committed evidence. + +The reproducibility lock (REPRO.yaml) pins the EXTERNAL inputs (upstream commits, +HF weight revisions, GT sha, scorer commit). This manifest pins the COMMITTED +evidence — the result bundles, model cards, lock, and eval config that live in +this repo — so any drift in a load-bearing artifact is detectable. + +Design rules (per the upstream-readiness hardening brief): + - no host-specific paths (no /root/...); paths are repo-relative; + - files are sorted before hashing so the manifest is stable across runs and + machines for identical inputs; + - missing critical files are an error (exit 1), never silently skipped; + - the algorithm, tool, git commit, and generation time are recorded. + +Usage: + python scripts/generate_evidence_manifest.py # write evidence/evidence-manifest.json + python scripts/generate_evidence_manifest.py --check # exit 1 on any drift + python scripts/generate_evidence_manifest.py --dir results/omnidocbench/v16 --out /tmp/m.json + python scripts/generate_evidence_manifest.py --file REPRO.yaml --file model_card.json +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +import yaml # PyYAML is a core dep + +REPO = Path(__file__).resolve().parents[1] +ALGORITHM = "sha256" +TOOL = "scripts/generate_evidence_manifest.py" +TOOL_VERSION = "1.0" + +# Curated canonical evidence set: the committed artifacts that back the +# published scores. Globs are used so newly committed evidence is picked up +# automatically (nothing is silently skipped). +CANONICAL_GLOBS = [ + "REPRO.yaml", + "model_card*.json", + "eval/configs/omnidocbench_v16.yaml", + "results/omnidocbench/v16/**/*.json", + "results/omnidocbench/v16/**/*.yaml", + "results/omnidocbench/v16/**/*.yml", + "results/omnidocbench/v1.6/pipeline/metric_result.json", + "results/omnidocbench/v1.6/pipeline/run_manifest.json", + "results/omnidocbench/v1.6/pipeline/predict.log.tail", + "results/omnidocbench/v1.6/pipeline/_errors.jsonl", + "results/omnidocbench/v1.6/pipeline/sample_predictions/manifest.json", + "results/omnidocbench/v1.6/vlm-vllm/metric_result.json", + "results/omnidocbench/v1.6/vlm-vllm/run_manifest.json", + "results/omnidocbench/v1.6/vlm-vllm/predict.log.tail", + "results/omnidocbench/v1.6/vlm-vllm/_errors.jsonl", + "results/omnidocbench/v1.6/vlm-vllm/sample_predictions/manifest.json", +] +SKIP_NAMES = {".gitignore", ".gitkeep"} + + +def _git_commit(repo: Path) -> str | None: + try: + cp = subprocess.run( + ["git", "-C", str(repo), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=False, + ) + except FileNotFoundError: + return None + return cp.stdout.strip() if cp.returncode == 0 else None + + +def _sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as fh: + for chunk in iter(lambda: fh.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def collect_paths(args) -> list[Path]: + """Resolve the file set from --canonical / --file / --dir. Sorted, deduped, repo-relative.""" + rel: set[str] = set() + if args.canonical or (not args.file and not args.dir): + for pat in CANONICAL_GLOBS: + for p in sorted(REPO.glob(pat)): + if p.is_file() and p.name not in SKIP_NAMES: + rel.add(p.relative_to(REPO).as_posix()) + for f in args.file or []: + p = (REPO / f).resolve() if not Path(f).is_absolute() else Path(f).resolve() + try: + rel.add(p.relative_to(REPO).as_posix()) + except ValueError: + rel.add(str(p)) + for d in args.dir or []: + base = (REPO / d).resolve() if not Path(d).is_absolute() else Path(d).resolve() + for p in sorted(base.rglob("*")): + if p.is_file() and p.name not in SKIP_NAMES and ".git" not in p.parts: + try: + rel.add(p.relative_to(REPO).as_posix()) + except ValueError: + rel.add(str(p)) + return sorted((REPO / r) for r in rel) + + +def build_manifest(paths: list[Path], missing_ok: bool) -> tuple[dict, list[str]]: + entries: list[dict] = [] + errors: list[str] = [] + for p in paths: + if not p.is_file(): + rel = _rel(p) + if missing_ok: + continue + errors.append(f"missing critical evidence file: {rel}") + continue + entries.append( + { + "path": _rel(p), + "sha256": _sha256(p), + "size": p.stat().st_size, + } + ) + manifest = { + "algorithm": ALGORITHM, + "tool": TOOL, + "tool_version": TOOL_VERSION, + "generated_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "git_commit": _git_commit(REPO), + "file_count": len(entries), + "entries": entries, + } + return manifest, errors + + +def _rel(p: Path) -> str: + try: + return p.relative_to(REPO).as_posix() + except ValueError: + return str(p) + + +def _serialize(manifest: dict, fmt: str) -> str: + if fmt == "yaml": + return yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True) + return json.dumps(manifest, indent=2, sort_keys=False) + "\n" + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + ap.add_argument( + "--canonical", + action="store_true", + help="hash the curated canonical evidence set (default)", + ) + ap.add_argument( + "--file", action="append", help="repo-relative file to hash (repeatable)" + ) + ap.add_argument( + "--dir", action="append", help="directory to hash recursively (repeatable)" + ) + ap.add_argument( + "--out", + default="evidence/evidence-manifest.json", + help="output path (default evidence/evidence-manifest.json)", + ) + ap.add_argument( + "--format", + choices=("json", "yaml"), + default=None, + help="output format (inferred from --out suffix by default)", + ) + ap.add_argument( + "--check", + action="store_true", + help="compare against the existing manifest; exit 1 on drift", + ) + ap.add_argument( + "--allow-missing", + action="store_true", + help="do not fail on missing files (default: fail)", + ) + args = ap.parse_args(argv) + + paths = collect_paths(args) + manifest, errors = build_manifest(paths, missing_ok=args.allow_missing) + if errors: + print( + "generate_evidence_manifest: " + str(len(errors)) + " error(s):", + file=sys.stderr, + ) + for e in errors: + print(" - " + e, file=sys.stderr) + return 1 + if not manifest["entries"]: + print("generate_evidence_manifest: no files selected", file=sys.stderr) + return 1 + + out = Path(args.out) + fmt = args.format or ("yaml" if out.suffix.lower() in (".yaml", ".yml") else "json") + + if args.check: + if not out.is_file(): + print( + f"generate_evidence_manifest: --check but {out} does not exist", + file=sys.stderr, + ) + return 1 + existing_text = out.read_text(encoding="utf-8") + existing = ( + yaml.safe_load(existing_text) + if out.suffix.lower() in (".yaml", ".yml") + else json.loads(existing_text) + ) + cur = {e["path"]: e["sha256"] for e in manifest["entries"]} + old = {e["path"]: e["sha256"] for e in existing.get("entries", [])} + drift = [] + for p, sha in cur.items(): + if p not in old: + drift.append(f"new evidence file not in manifest: {p}") + elif old[p] != sha: + drift.append(f"{p}: manifest {old[p]} != current {sha}") + for p in old: + if p not in cur: + drift.append(f"manifest lists {p} but it is no longer present") + if drift: + print( + f"generate_evidence_manifest: {len(drift)} drift finding(s):", + file=sys.stderr, + ) + for d in drift: + print(" - " + d, file=sys.stderr) + return 1 + print(f"generate_evidence_manifest: {len(cur)} files match the manifest") + return 0 + + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(_serialize(manifest, fmt), encoding="utf-8") + print( + f"generate_evidence_manifest: wrote {len(manifest['entries'])} entries to {out} ({ALGORITHM})" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/smoke_test.py b/scripts/smoke_test.py new file mode 100755 index 0000000..cc6c613 --- /dev/null +++ b/scripts/smoke_test.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 AIwork4me +"""Portable MinerU-ROCm smoke test. + +Layered so an ordinary user can confirm a successful install WITHOUT running +the full OmniDocBench benchmark. No host-specific paths (uses --img-dir / +--out-dir / temp dirs). Designed to run on CI without an AMD GPU via +``--check-only``. + +Levels: + 0 environment information (OS, Python, mineru_rocm, torch/HIP, GPU, cuda) + 1 device availability (torch.cuda.is_available / device_count / name) + 2 owned CLI (``mineru-rocm --help`` + ``doctor``); advisory upstream ``mineru`` + 3 minimal inference via the GPU-free ``smoke`` backend (the adapter contract) + 4 (optional) VLM HTTP health check, only when --vlm-url is given + +Note on torch.cuda: a ROCm PyTorch still exposes the ``torch.cuda.*`` API (ROCm +mirrors the CUDA surface), so ``torch.cuda.is_available()`` reports the AMD GPU. +This is expected, not a sign that NVIDIA CUDA is in use. + +Exit code: 0 if every RUN level is OK (SKIP is not a failure); 1 on any FAIL. +``--check-only`` skips GPU-dependent and inference steps and never claims a GPU +run. +""" + +from __future__ import annotations + +import argparse +import json +import platform +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +VALIDATED_MINERU = "3.4.4" + + +def _ok(name, detail=""): + return ("OK", name, detail) + + +def _skip(name, detail=""): + return ("SKIP", name, detail) + + +def _fail(name, detail=""): + return ("FAIL", name, detail) + + +def _run_mineru_rocm(args: list[str]) -> subprocess.CompletedProcess: + """Invoke the owned CLI; prefer the console script, fall back to the module.""" + exe = shutil.which("mineru-rocm") + if exe: + return subprocess.run([exe, *args], capture_output=True, text=True, check=False) + return subprocess.run( + [sys.executable, "-m", "mineru_rocm.cli", *args], + capture_output=True, + text=True, + check=False, + ) + + +def level0_env() -> tuple[str, str, str]: + """Informational only — never fails.""" + import mineru_rocm + + bits = { + "os": platform.platform(), + "python": sys.version.split()[0], + "mineru_rocm": getattr(mineru_rocm, "__version__", "?"), + } + try: + import torch + + bits["torch"] = getattr(torch, "__version__", "?") + bits["hip"] = getattr(torch.version, "hip", None) or "n/a" + bits["torch_cuda_available"] = bool(torch.cuda.is_available()) + bits["device_count"] = torch.cuda.device_count() + if torch.cuda.is_available(): + bits["device_name_0"] = torch.cuda.get_device_name(0) + except Exception as exc: # noqa: BLE001 — torch is optional for the core package + bits["torch"] = f"not importable: {exc}" + try: + import mineru + + bits["upstream_mineru"] = getattr( + mineru, "__version__", "(importable, no __version__)" + ) + except Exception: # noqa: BLE001 + bits["upstream_mineru"] = "not installed" + return _ok("env", json.dumps(bits, sort_keys=True)) + + +def level1_device(check_only: bool) -> tuple[str, str, str]: + try: + import torch + except Exception as exc: # noqa: BLE001 + if check_only: + return _skip( + "device", f"torch not importable ({exc}); dry-run, no GPU check" + ) + return _fail( + "device", + f"torch not importable ({exc}); a ROCm PyTorch is required for GPU levels", + ) + if not torch.cuda.is_available(): + if check_only: + return _skip( + "device", "torch.cuda.is_available() is False; dry-run, no GPU claimed" + ) + return _fail( + "device", + "torch.cuda.is_available() is False — no AMD GPU visible to this ROCm PyTorch", + ) + n = torch.cuda.device_count() + name = torch.cuda.get_device_name(0) if n else "(no device)" + return _ok( + "device", + f"cuda_available=True count={n} name={name!r} (ROCm via the torch.cuda API)", + ) + + +def level2_cli() -> tuple[str, str, str]: + help_cp = _run_mineru_rocm(["--help"]) + if help_cp.returncode != 0: + return _fail( + "cli", + f"`mineru-rocm --help` rc={help_cp.returncode}: {(help_cp.stderr or '').strip()[:200]}", + ) + doc_cp = _run_mineru_rocm(["doctor"]) + if doc_cp.returncode != 0: + return _fail( + "cli", + f"`mineru-rocm doctor` rc={doc_cp.returncode}: {(doc_cp.stderr or '').strip()[:200]}", + ) + note = "" + mu = shutil.which("mineru") + if mu: + v_cp = subprocess.run( + [mu, "--version"], capture_output=True, text=True, check=False + ) + ver = (v_cp.stdout or v_cp.stderr).strip() + note = f"upstream mineru: {ver}" + if VALIDATED_MINERU not in ver: + note += f" (WARN: validated MinerU is {VALIDATED_MINERU}; results must not be extrapolated)" + else: + note = "upstream `mineru` not on PATH (optional; the GPU runtime installs it separately)" + return _ok("cli", "mineru-rocm --help + doctor OK. " + note) + + +def level3_inference( + check_only: bool, img_dir: Path, out_dir: Path | None +) -> tuple[str, str, str]: + adapter = REPO / "adapter" / "run_adapter.py" + if not adapter.is_file(): + return _fail("inference", f"adapter script missing: {adapter}") + if not img_dir.is_dir(): + return _fail("inference", f"--img-dir not found: {img_dir}") + imgs = [ + p for p in img_dir.iterdir() if p.suffix.lower() in (".png", ".jpg", ".jpeg") + ] + if not imgs: + return _fail( + "inference", f"no .png/.jpg images in {img_dir} to run the smoke backend on" + ) + if check_only: + return _skip( + "inference", + f"check-only: adapter + {len(imgs)} test image(s) present; no inference executed (dry-run)", + ) + out = out_dir or Path(tempfile.mkdtemp(prefix="mineru-rocm-smoke-")) + out.mkdir(parents=True, exist_ok=True) + cp = subprocess.run( + [ + sys.executable, + str(adapter), + "--img-dir", + str(img_dir), + "--out-dir", + str(out), + "--platform", + "linux-rocm", + "--backend", + "smoke", + ], + capture_output=True, + text=True, + check=False, + cwd=str(REPO), + ) + if cp.returncode != 0: + return _fail( + "inference", + f"smoke backend rc={cp.returncode}: {(cp.stderr or '').strip()[:300]}", + ) + mds = sorted(out.glob("*.md")) + if not mds: + return _fail("inference", f"smoke backend produced no .md files in {out}") + empty = [p.name for p in mds if p.stat().st_size == 0] + if len(empty) == len(mds): + return _fail( + "inference", f"all {len(mds)} smoke .md outputs are empty in {out}" + ) + return _ok( + "inference", + f"backend=smoke produced {len(mds)} non-empty .md file(s) in {out} (placeholder text, not real OCR)", + ) + + +def level4_vlm(vlm_url: str | None) -> tuple[str, str, str]: + if not vlm_url: + return _skip( + "vlm", "no --vlm-url given (optional; no assumption of vLLM or llama.cpp)" + ) + import urllib.request + + try: + with urllib.request.urlopen(vlm_url.rstrip("/") + "/models", timeout=5) as resp: + body = resp.read(200) + except Exception as exc: # noqa: BLE001 + return _fail("vlm", f"VLM health check failed for {vlm_url}: {exc}") + return _ok( + "vlm", + f"VLM endpoint {vlm_url}/models reachable: {body[:80]!r} (server backend not assumed)", + ) + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + ap.add_argument( + "--check-only", + action="store_true", + help="no GPU / no inference; verify imports, CLI, test data, arg parsing only", + ) + ap.add_argument( + "--level", + type=int, + default=4, + choices=(0, 1, 2, 3, 4), + help="highest level to run (default 4)", + ) + ap.add_argument( + "--img-dir", + default=str(REPO / "examples"), + help="image dir for the smoke backend (default: examples/)", + ) + ap.add_argument( + "--out-dir", + default=None, + help="output dir for smoke predictions (default: a fresh temp dir)", + ) + ap.add_argument( + "--vlm-url", + default=None, + help="VLM OpenAI-compatible base URL for the optional Level 4 health check", + ) + ap.add_argument("--json", action="store_true", help="emit results as JSON") + args = ap.parse_args(argv) + + results = [] + if args.level >= 0: + results.append((0, *level0_env())) + if args.level >= 1: + results.append((1, *level1_device(args.check_only))) + if args.level >= 2: + results.append((2, *level2_cli())) + if args.level >= 3: + results.append( + ( + 3, + *level3_inference( + args.check_only, + Path(args.img_dir), + Path(args.out_dir) if args.out_dir else None, + ), + ) + ) + if args.level >= 4: + results.append((4, *level4_vlm(args.vlm_url))) + + if args.json: + print( + json.dumps( + [ + {"level": lv, "status": st, "name": nm, "detail": d} + for lv, st, nm, d in results + ], + indent=2, + ) + ) + else: + for lv, st, nm, d in results: + print(f"[L{lv}] {st:4} {nm}: {d}") + if args.check_only: + print( + "\nDRY-RUN (--check-only): no GPU inference executed; this is not a GPU runtime verification." + ) + + failed = [r for r in results if r[1] == "FAIL"] + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/validate_schemas.py b/scripts/validate_schemas.py new file mode 100644 index 0000000..4dbd691 --- /dev/null +++ b/scripts/validate_schemas.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 AIwork4me +"""Schema validation gate for REPRO.yaml + model_card*.json. + +Validates each artifact against its JSON Schema (schemas/*.schema.json) and runs +extra cross-field checks the schema language cannot express: + + - every ``*_sha256`` value is a 64-hex lowercase digest OR the literal + ``not_recorded`` (catches typos / truncated hashes); + - every model-card badge key is a known platform. + +Exits 0 clean, 1 on any finding. Errors name the file, the field, the expected +and the actual value, and a fix hint — never just "validation failed". +""" + +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +KNOWN_PLATFORMS = {"linux-rocm", "windows-hip"} + + +def _walk_sha256(obj, path, errs, fname): + """Recurse obj; every key ending in _sha256 must be 64-hex or 'not_recorded'.""" + if isinstance(obj, dict): + for k, v in obj.items(): + p = f"{path}.{k}" if path else k + if k.endswith("_sha256"): + if v is None: + continue # explicit null is allowed (e.g. hsa_override.pipeline) + if not isinstance(v, str) or ( + v != "not_recorded" and not SHA256_RE.match(v) + ): + errs.append( + f"{fname}: {p} = {v!r} is not a 64-hex SHA-256 and not 'not_recorded' " + "(record the digest or set 'not_recorded' with a reason)" + ) + else: + _walk_sha256(v, p, errs, fname) + elif isinstance(obj, list): + for i, v in enumerate(obj): + _walk_sha256(v, f"{path}[{i}]", errs, fname) + + +def _validate_instance(instance, schema, fname, errs, validator_cls): + validator = validator_cls(schema) + for err in sorted(validator.iter_errors(instance), key=lambda e: list(e.path)): + loc = ".".join(str(p) for p in err.path) or "" + errs.append(f"{fname}: {loc}: {err.message}") + + +def main() -> int: + try: + from jsonschema import Draft202012Validator as Validator + except ImportError: + print( + "validate_schemas: jsonschema is required (pip install jsonschema)", + file=sys.stderr, + ) + return 2 + + errs: list[str] = [] + + # --- REPRO.yaml --- + repro_schema = json.loads( + (REPO / "schemas" / "repro.schema.json").read_text(encoding="utf-8") + ) + import yaml + + repro_path = REPO / "REPRO.yaml" + if not repro_path.is_file(): + errs.append("REPRO.yaml: missing (the reproducibility lock is required)") + else: + repro = yaml.safe_load(repro_path.read_text(encoding="utf-8")) + _validate_instance(repro, repro_schema, "REPRO.yaml", errs, Validator) + _walk_sha256(repro, "", errs, "REPRO.yaml") + # MinerU boundary cross-check: if mineru.validated is true, mineru_4 must be not_validated + vs = (repro or {}).get("validated_software") or {} + if (vs.get("mineru") or {}).get("validated") is True and ( + vs.get("mineru_4") or {} + ).get("status") != "not_validated": + errs.append( + "REPRO.yaml: validated_software.mineru.validated=true requires mineru_4.status='not_validated' (4.x must be explicitly out of scope)" + ) + + # --- model cards --- + card_schema = json.loads( + (REPO / "schemas" / "model-card.schema.json").read_text(encoding="utf-8") + ) + cards = sorted(REPO.glob("model_card*.json")) + if not cards: + errs.append("model_card*.json: no model cards found") + for cp in cards: + fname = cp.relative_to(REPO).as_posix() + try: + card = json.loads(cp.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + errs.append(f"{fname}: invalid JSON: {exc}") + continue + _validate_instance(card, card_schema, fname, errs, Validator) + _walk_sha256(card, "", errs, fname) + for key in card.get("badge") or {}: + if key not in KNOWN_PLATFORMS: + errs.append( + f"{fname}: badge key {key!r} is not a known platform {sorted(KNOWN_PLATFORMS)}" + ) + for plat in card.get("platforms") or []: + if plat not in KNOWN_PLATFORMS: + errs.append( + f"{fname}: platform {plat!r} is not a known platform {sorted(KNOWN_PLATFORMS)}" + ) + + if errs: + print(f"validate_schemas: {len(errs)} finding(s):", file=sys.stderr) + for e in errs: + print(f" - {e}", file=sys.stderr) + return 1 + print("validate_schemas: clean") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/mineru_rocm/backends/vlm.py b/src/mineru_rocm/backends/vlm.py index f58e1b8..dbe4404 100644 --- a/src/mineru_rocm/backends/vlm.py +++ b/src/mineru_rocm/backends/vlm.py @@ -104,14 +104,23 @@ class MineruVLRunner: """Wraps mineru_vl_utils.MinerUClient against either a vLLM server (http-client) or an in-process transformers engine. Backend selected by cfg["backend"].""" - # Local snapshot of opendatalab/MinerU2.5-Pro-2605-1.2B (HF_HOME uses the flat - # cache layout, not hub/). This is the transformers model_path default; the HF - # repo id also works (slower first call). Override via cfg["vlm_model_path"]. - _DEFAULT_VLM_MODEL_PATH = ( - "/root/.cache/huggingface/models--opendatalab--MinerU2.5-Pro-2605-1.2B/" - "snapshots/bff20d4ae2bf202df9f45284b4d43681555a97ed" - ) + # Pinned snapshot revision of opendatalab/MinerU2.5-Pro-2605-1.2B (matches + # REPRO.yaml model.vlm.hf_revision). The local snapshot path is derived from + # the HF cache at runtime (see _default_vlm_model_path) — no host-specific + # path is hardcoded. The HF repo id also works (slower first call). Override + # via cfg["vlm_model_path"]. _DEFAULT_VLM_MODEL_ID = "opendatalab/MinerU2.5-Pro-2605-1.2B" + _DEFAULT_VLM_MODEL_REVISION = "bff20d4ae2bf202df9f45284b4d43681555a97ed" + + @classmethod + def _default_vlm_model_path(cls): + """The local snapshot path derived from the HF cache ($HF_HOME, then the + default ~/.cache/huggingface). Portable — no hardcoded home dir.""" + import os + from pathlib import Path + + cache_root = Path(os.environ.get("HF_HOME", str(Path.home() / ".cache" / "huggingface"))) + return cache_root / "models--opendatalab--MinerU2.5-Pro-2605-1.2B" / "snapshots" / cls._DEFAULT_VLM_MODEL_REVISION def __init__(self, platform: str, cfg: dict): self.platform = platform @@ -131,7 +140,7 @@ def _resolve_transformers_model_path(self) -> str: override = self.cfg.get("vlm_model_path") if override: return override - snap = Path(self._DEFAULT_VLM_MODEL_PATH) + snap = Path(self._default_vlm_model_path()) if (snap / "model.safetensors").is_file(): return str(snap) return self._DEFAULT_VLM_MODEL_ID diff --git a/tests/test_check_repo.py b/tests/test_check_repo.py index 2ea6953..b8556d1 100644 --- a/tests/test_check_repo.py +++ b/tests/test_check_repo.py @@ -6,12 +6,14 @@ def test_no_engine_imports_in_core_uses_ast(): """The no-engine scan is AST-based (catches 'import omnidocbench_rocm' anywhere in src/mineru_rocm).""" import scripts.check_repo as cr # noqa + leaks = cr.find_engine_imports(REPO / "src" / "mineru_rocm") assert leaks == [], f"engine imports leaked into core: {leaks}" def test_required_lock_sections_present(): import scripts.check_repo as cr + lock = cr._load_lock() missing = cr.check_lock_sections(lock) assert missing == [], missing @@ -19,11 +21,13 @@ def test_required_lock_sections_present(): def test_spdx_headers_on_src_and_scripts(): import scripts.check_repo as cr + assert cr.check_spdx(REPO) == [] def test_readme_script_references_exist(): import scripts.check_repo as cr + readme = (REPO / "README.md").read_text(encoding="utf-8") assert cr.check_readme_scripts_exist(readme) == [] @@ -31,6 +35,7 @@ def test_readme_script_references_exist(): def test_check_readme_lock_values_pass_when_consistent(): """The README results tables match the lock values (the drift gate).""" import scripts.check_repo as cr + readme = (REPO / "README.md").read_text(encoding="utf-8") lock = cr._load_lock() findings = cr.check_readme_lock_values(readme, lock) @@ -40,7 +45,15 @@ def test_check_readme_lock_values_pass_when_consistent(): def test_check_readme_lock_values_flags_drift(): """README numbers that disagree with the lock are findings (stale-table detection).""" import scripts.check_repo as cr - lock = {"benchmark": {"full_1651": {"pipeline": {"overall": 99.99}, "vlm_vllm": {"overall": 88.88}}}} + + lock = { + "benchmark": { + "full_1651": { + "pipeline": {"overall": 99.99}, + "vlm_vllm": {"overall": 88.88}, + } + } + } readme = "Pipeline Overall 86.48 | VLM 95.46" # neither 99.99 nor 88.88 present findings = cr.check_readme_lock_values(readme, lock) assert len(findings) == 2 # both pipeline + vlm_vllm flagged as drift @@ -53,11 +66,16 @@ def test_check_repo_clean_on_repo(capsys): in CI via `main()`, not here. This test covers engine/lock/SPDX/README + the README↔lock value cross-check.""" import scripts.check_repo as cr + findings = [] findings += cr.find_engine_imports(REPO / "src" / "mineru_rocm") findings += cr.check_lock_sections(cr._load_lock()) findings += cr.check_spdx() - readme = (REPO / "README.md").read_text(encoding="utf-8") if (REPO / "README.md").is_file() else "" + readme = ( + (REPO / "README.md").read_text(encoding="utf-8") + if (REPO / "README.md").is_file() + else "" + ) findings += cr.check_readme_scripts_exist(readme) findings += cr.check_readme_lock_values(readme, cr._load_lock()) findings += cr.check_modelcard_lock_agreement(cr._load_lock()) @@ -74,18 +92,24 @@ def test_check_repo_clean_on_repo(capsys): def test_upstream_commits_pinned_in_lock(): """mineru + mineru_vl_utils carry real git commit SHAs (not 'not_recorded').""" import scripts.check_repo as cr + lock = cr._load_lock() for dep in ("mineru", "mineru_vl_utils"): commit = (lock.get(dep) or {}).get("commit", "") - assert commit != "not_recorded" and len(commit) == 40, f"{dep}.commit not pinned: {commit!r}" + assert commit != "not_recorded" and len(commit) == 40, ( + f"{dep}.commit not pinned: {commit!r}" + ) def test_official_reference_verified(): """The official anchor is sourced from the upstream README (not not_verified).""" import scripts.check_repo as cr + lock = cr._load_lock() ref = (lock.get("benchmark") or {}).get("official_reference") or {} - assert ref.get("source") == "verified", f"official_reference.source not verified: {ref.get('source')!r}" + assert ref.get("source") == "verified", ( + f"official_reference.source not verified: {ref.get('source')!r}" + ) assert ref.get("pipeline_overall") == 86.47 assert ref.get("vlm_overall") == 95.30 @@ -94,23 +118,36 @@ def test_modelcard_lock_agreement(): """Linux/VLM root cards and the Windows pipeline card match their lock entries.""" import json import scripts.check_repo as cr + lock = cr._load_lock() full = (lock.get("benchmark") or {}).get("full_1651") or {} expected = { "model_card.json": (full.get("vlm_vllm") or {}).get("overall"), "model_card.pipeline.json": (full.get("pipeline") or {}).get("overall"), "model_card.pipeline.windows-hip.json": ( - ((((lock.get("benchmark") or {}).get("windows_hip") or {}).get("full_1651") or {}).get("pipeline") or {}).get("overall") + ( + ( + ((lock.get("benchmark") or {}).get("windows_hip") or {}).get( + "full_1651" + ) + or {} + ).get("pipeline") + or {} + ).get("overall") ), } for fname, exp in expected.items(): if exp is None: continue card = json.loads((cr.REPO / fname).read_text(encoding="utf-8")) - assert card["overall"] == exp, f"{fname}.overall {card['overall']} != lock {exp}" + assert card["overall"] == exp, ( + f"{fname}.overall {card['overall']} != lock {exp}" + ) # artefacts must point at an omnidocbench results tree (v1.6 legacy or v16 platform-standard) arts = json.dumps(card.get("artifacts", {})) - assert "omnidocbench/v1.6/" in arts or "omnidocbench/v16/" in arts, f"{fname} artefacts missing omnidocbench path" + assert "omnidocbench/v1.6/" in arts or "omnidocbench/v16/" in arts, ( + f"{fname} artefacts missing omnidocbench path" + ) def test_modelcard_dangling_artefref_flagged(tmp_path, monkeypatch): @@ -119,21 +156,34 @@ def test_modelcard_dangling_artefref_flagged(tmp_path, monkeypatch): leaving URLs + values alone.""" import json import scripts.check_repo as cr + # Build a minimal repo tree: model_card.json with one good ref, one dangling # repo-relative path, one URL, and one . - (tmp_path / "model_card.json").write_text(json.dumps({ - "overall": 95.46, - "artifacts": { - "metric_result": "results/omnidocbench/v1.6/vlm-vllm/metric_result.json", # will NOT exist here - "predict_log": "results/omnidocbench/v1.6/vlm-vllm/predict.log", # dangling - "sample_predictions": "results/omnidocbench/v1.6/vlm-vllm/sample_predictions/", - "upstream_url": "https://example.com/x.json", # URL — skip - "pred_dir": "", # placeholder — skip - }, - }), encoding="utf-8") + (tmp_path / "model_card.json").write_text( + json.dumps( + { + "overall": 95.46, + "artifacts": { + "metric_result": "results/omnidocbench/v1.6/vlm-vllm/metric_result.json", # will NOT exist here + "predict_log": "results/omnidocbench/v1.6/vlm-vllm/predict.log", # dangling + "sample_predictions": "results/omnidocbench/v1.6/vlm-vllm/sample_predictions/", + "upstream_url": "https://example.com/x.json", # URL — skip + "pred_dir": "", # placeholder — skip + }, + } + ), + encoding="utf-8", + ) # Create only sample_predictions/ so exactly one ref resolves; the other two # repo-relative paths (metric_result.json, predict.log) dangle. - (tmp_path / "results" / "omnidocbench" / "v1.6" / "vlm-vllm" / "sample_predictions").mkdir(parents=True) + ( + tmp_path + / "results" + / "omnidocbench" + / "v1.6" + / "vlm-vllm" + / "sample_predictions" + ).mkdir(parents=True) monkeypatch.setattr(cr, "REPO", tmp_path) lock = {"benchmark": {"full_1651": {"vlm_vllm": {"overall": 95.46}}}} findings = cr.check_modelcard_lock_agreement(lock) @@ -145,6 +195,7 @@ def test_modelcard_dangling_artefref_flagged(tmp_path, monkeypatch): def test_current_overall_primary_clean_on_repo(): """The README VLM badge states the lock's current VLM Overall (95.56).""" import scripts.check_repo as cr + findings = cr.check_current_overall_primary(cr._load_lock()) assert findings == [], findings @@ -152,8 +203,11 @@ def test_current_overall_primary_clean_on_repo(): def test_current_overall_primary_flags_drift(tmp_path, monkeypatch): """A README whose VLM badge lags the lock's current Overall is flagged.""" import scripts.check_repo as cr + (tmp_path / "README.md").write_text( - "[![VLM full](https://img.shields.io/badge/VLM(full)-90.00-green)](#x)\n", encoding="utf-8") + "[![VLM full](https://img.shields.io/badge/VLM(full)-90.00-green)](#x)\n", + encoding="utf-8", + ) monkeypatch.setattr(cr, "REPO", tmp_path) lock = {"benchmark": {"full_1651": {"vlm_vllm": {"overall": 95.56}}}} findings = cr.check_current_overall_primary(lock, tmp_path) @@ -163,6 +217,7 @@ def test_current_overall_primary_flags_drift(tmp_path, monkeypatch): def test_prior_overall_contextual_clean_on_repo(): """Every user-facing doc that cites the prior (95.46) also cites the current (95.56).""" import scripts.check_repo as cr + findings = cr.check_prior_overall_contextual(cr._load_lock()) assert findings == [], findings @@ -171,13 +226,27 @@ def test_prior_overall_contextual_flags_orphan(tmp_path, monkeypatch): """A doc citing the prior without the current is flagged; one with both passes.""" import json import scripts.check_repo as cr + # legacy v1.6 metric carries the prior Overall (overall_notebook) - legacy = tmp_path / "results" / "omnidocbench" / "v1.6" / "vlm-vllm" / "metric_result.json" + legacy = ( + tmp_path + / "results" + / "omnidocbench" + / "v1.6" + / "vlm-vllm" + / "metric_result.json" + ) legacy.parent.mkdir(parents=True) - legacy.write_text(json.dumps({"overall_notebook": 95.4635996625401}), encoding="utf-8") + legacy.write_text( + json.dumps({"overall_notebook": 95.4635996625401}), encoding="utf-8" + ) (tmp_path / "docs").mkdir() - (tmp_path / "docs" / "orphan.md").write_text("prior score was 95.46 here\n", encoding="utf-8") - (tmp_path / "docs" / "ok.md").write_text("prior 95.46, current 95.56\n", encoding="utf-8") + (tmp_path / "docs" / "orphan.md").write_text( + "prior score was 95.46 here\n", encoding="utf-8" + ) + (tmp_path / "docs" / "ok.md").write_text( + "prior 95.46, current 95.56\n", encoding="utf-8" + ) lock = {"benchmark": {"full_1651": {"vlm_vllm": {"overall": 95.56}}}} findings = cr.check_prior_overall_contextual(lock, tmp_path) flagged = {f.split(" quotes")[0] for f in findings} @@ -189,6 +258,7 @@ def test_no_internal_infra_in_public_files(): """No committed file under results/ or docs/ (excl. docs/superpowers/) leaks the internal HF mirror IP or host eval-root path.""" import scripts.check_repo as cr + findings = cr.check_no_internal_infra() assert findings == [], findings @@ -196,6 +266,7 @@ def test_no_internal_infra_in_public_files(): def test_no_withdrawn_anchor_claims_in_public_files(): """No user-facing surface re-cites the withdrawn unofficial-anchor story.""" import scripts.check_repo as cr + findings = cr.check_no_withdrawn_anchor_claims() assert findings == [], findings @@ -204,13 +275,18 @@ def test_check_no_withdrawn_anchor_claims_flags_tokens(tmp_path): """The gate flags each withdrawn token in docs/ (excl superpowers) + top-level README/CHANGELOG, and leaves docs/superpowers/ + the lock alone.""" import scripts.check_repo as cr + # docs/*.md (not under superpowers/) — each token flagged once (tmp_path / "docs").mkdir(parents=True) - (tmp_path / "docs" / "a.md").write_text("the old 95.75 was withdrawn\n", encoding="utf-8") + (tmp_path / "docs" / "a.md").write_text( + "the old 95.75 was withdrawn\n", encoding="utf-8" + ) (tmp_path / "docs" / "b.md").write_text("upstream ~95.69 guess\n", encoding="utf-8") # superpowers/ is exempt even if it contains the token (tmp_path / "docs" / "superpowers").mkdir() - (tmp_path / "docs" / "superpowers" / "spec.md").write_text("95.75 design note\n", encoding="utf-8") + (tmp_path / "docs" / "superpowers" / "spec.md").write_text( + "95.75 design note\n", encoding="utf-8" + ) # top-level surfaces scanned (tmp_path / "README.md").write_text("anchor is unverified\n", encoding="utf-8") (tmp_path / "CHANGELOG.md").write_text("status: not_verified\n", encoding="utf-8") @@ -226,6 +302,7 @@ def test_version_consistency_clean_on_repo(): """README + the issue draft state the lock's ROCm version + GPU arch, and no user-facing surface carries a ROCm/version overclaim.""" import scripts.check_repo as cr + findings = cr.check_version_consistency(cr._load_lock()) assert findings == [], findings @@ -234,17 +311,26 @@ def test_version_consistency_flags_overclaim(tmp_path, monkeypatch): """An assertion-form overclaim is flagged; docs/superpowers/ is exempt; the consistency check passes when README states the lock's versions.""" import scripts.check_repo as cr - (tmp_path / "README.md").write_text("tested on ROCm 7.2 + gfx1100\n", encoding="utf-8") + + (tmp_path / "README.md").write_text( + "tested on ROCm 7.2 + gfx1100\n", encoding="utf-8" + ) (tmp_path / "docs").mkdir() - (tmp_path / "docs" / "a.md").write_text("we officially support all RDNA3 GPUs\n", encoding="utf-8") + (tmp_path / "docs" / "a.md").write_text( + "we officially support all RDNA3 GPUs\n", encoding="utf-8" + ) (tmp_path / "docs" / "superpowers").mkdir() - (tmp_path / "docs" / "superpowers" / "s.md").write_text("ROCm 7.2+ design note\n", encoding="utf-8") + (tmp_path / "docs" / "superpowers" / "s.md").write_text( + "ROCm 7.2+ design note\n", encoding="utf-8" + ) monkeypatch.setattr(cr, "REPO", tmp_path) lock = {"environment": {"rocm_hip": "7.2", "gpu_arch": "gfx1100"}} findings = cr.check_version_consistency(lock, tmp_path) joined = "\n".join(findings) # docs/a.md overclaim flagged (officially support + all RDNA3); superpowers exempt - assert "docs/a.md" in joined and ("officially support" in joined or "all RDNA3" in joined), findings + assert "docs/a.md" in joined and ( + "officially support" in joined or "all RDNA3" in joined + ), findings assert not any("superpowers" in f for f in findings), findings @@ -252,6 +338,7 @@ def test_release_and_run_provenance_clean_on_repo(): """The lock's release tag/commit/tag-object SHA + both run commits match git and the manifests on the real repo (Task 1 provenance gate).""" import scripts.check_repo as cr + findings = cr.check_release_and_run_provenance(cr._load_lock()) assert findings == [], findings @@ -259,10 +346,17 @@ def test_release_and_run_provenance_clean_on_repo(): def test_release_and_run_provenance_flags_mismatched_lock(): """A lock whose release/run commits disagree with git + the manifests is flagged.""" import scripts.check_repo as cr - bad = {"mineru_rocm": { - "release": {"tag": "v0.1.0", "commit": "0" * 40, "tag_object_sha": "1" * 40}, - "benchmark_run_commits": {"pipeline": "2" * 40, "vlm_vllm": "3" * 40}, - }} + + bad = { + "mineru_rocm": { + "release": { + "tag": "v0.1.0", + "commit": "0" * 40, + "tag_object_sha": "1" * 40, + }, + "benchmark_run_commits": {"pipeline": "2" * 40, "vlm_vllm": "3" * 40}, + } + } findings = cr.check_release_and_run_provenance(bad, cr.REPO) joined = "\n".join(findings) assert "release.commit" in joined, findings @@ -274,10 +368,13 @@ def test_release_and_run_provenance_flags_mismatched_lock(): def test_provenance_flags_tag_sha_cited_as_commit(tmp_path, monkeypatch): """The annotated-tag object SHA must not appear in a user-facing doc as a commit.""" import scripts.check_repo as cr + tag_sha = "dd591469d009cac246f5090daa7398623d2fd878" d = tmp_path / "docs" / "upstream" d.mkdir(parents=True) - (d / "mineru-issue-5288.md").write_text(f"checkout v0.1.0 = {tag_sha}\n", encoding="utf-8") + (d / "mineru-issue-5288.md").write_text( + f"checkout v0.1.0 = {tag_sha}\n", encoding="utf-8" + ) monkeypatch.setattr(cr, "REPO", tmp_path) lock = {"mineru_rocm": {"release": {"tag": "v0.1.0", "tag_object_sha": tag_sha}}} findings = cr.check_release_and_run_provenance(lock, tmp_path) @@ -287,6 +384,7 @@ def test_provenance_flags_tag_sha_cited_as_commit(tmp_path, monkeypatch): def test_score_commands_have_scorer_args_clean_on_repo(): """Every `mineru-rocm score` example carries the scorer repo on the real repo.""" import scripts.check_repo as cr + findings = cr.check_score_commands_have_scorer_args() assert findings == [], findings @@ -295,14 +393,134 @@ def test_score_commands_flag_missing_scorer_arg(tmp_path, monkeypatch): """A `mineru-rocm score` block without --omnidocbench-repo / OMNIDOCBENCH_REPO is flagged; one with either passes.""" import scripts.check_repo as cr + (tmp_path / "README.md").write_text( - "```bash\nmineru-rocm score --gt-json g --pred-dir p --label x\n```\n", encoding="utf-8") + "```bash\nmineru-rocm score --gt-json g --pred-dir p --label x\n```\n", + encoding="utf-8", + ) (tmp_path / "docs").mkdir() (tmp_path / "docs" / "ok.md").write_text( "```bash\nexport OMNIDOCBENCH_REPO=/x\nmineru-rocm score --gt-json g --pred-dir p --label x\n```\n", - encoding="utf-8") + encoding="utf-8", + ) monkeypatch.setattr(cr, "REPO", tmp_path) findings = cr.check_score_commands_have_scorer_args(tmp_path) joined = "\n".join(findings) - assert "README.md" in joined, findings # the bad block is flagged - assert "docs/ok.md" not in joined, findings # the OMNIDOCBENCH_REPO block passes + assert "README.md" in joined, findings # the bad block is flagged + assert "docs/ok.md" not in joined, findings # the OMNIDOCBENCH_REPO block passes + + +# --- upstream-readiness hardening gates --------------------------------------- + + +def test_lock_present_clean_on_repo(): + import scripts.check_repo as cr + + assert cr.check_lock_present() == [], "REPRO.yaml must be present" + + +def test_lock_present_flags_missing(tmp_path, monkeypatch): + import scripts.check_repo as cr + + monkeypatch.setattr(cr, "REPO", tmp_path) + assert cr.check_lock_present(tmp_path), "missing REPRO.yaml must be flagged" + + +def test_no_dead_lock_filename_clean_on_repo(): + import scripts.check_repo as cr + + assert cr.check_no_dead_lock_filename() == [] + + +def test_no_dead_lock_filename_flags_old_name(tmp_path, monkeypatch): + """A regression that re-introduces the pre-rename lock filename is flagged.""" + import scripts.check_repo as cr + + (tmp_path / "README.md").write_text( + "see reproducibility.lock.yaml for detail\n", encoding="utf-8" + ) + monkeypatch.setattr(cr, "REPO", tmp_path) + monkeypatch.setattr( + cr, "_git_ls_text_files", lambda repo, suf: [tmp_path / "README.md"] + ) + findings = cr.check_no_dead_lock_filename(tmp_path) + assert any("reproducibility.lock.yaml" in f for f in findings), findings + + +def test_doc_cli_clean_on_repo(): + import scripts.check_repo as cr + + assert cr.check_doc_cli_commands_exist() == [] + + +def test_doc_cli_flags_fabricated_command(tmp_path, monkeypatch): + """A fabricated/nonexistent mineru-rocm invocation in a doc is flagged.""" + import scripts.check_repo as cr + + (tmp_path / "README.md").write_text( + "```bash\npython -m mineru_rocm runner --x\nmineru-rocm serve-vlm\n```\n", + encoding="utf-8", + ) + monkeypatch.setattr(cr, "REPO", tmp_path) + findings = cr.check_doc_cli_commands_exist(tmp_path) + joined = "\n".join(findings) + assert "runner" in joined and "serve-vlm" in joined, findings + + +def test_no_root_in_doc_commands_clean_on_repo(): + import scripts.check_repo as cr + + assert cr.check_no_root_in_doc_commands() == [] + + +def test_no_root_in_doc_commands_flags_hardcode(tmp_path, monkeypatch): + """A /root/ path in a bash code block is flagged.""" + import scripts.check_repo as cr + + (tmp_path / "reproduce.md").write_text( + "```bash\nmineru-rocm predict --img-dir /root/data/images\n```\n", + encoding="utf-8", + ) + monkeypatch.setattr(cr, "REPO", tmp_path) + findings = cr.check_no_root_in_doc_commands(tmp_path) + assert any("/root/" in f for f in findings), findings + + +def test_version_boundary_clean_on_repo(): + import scripts.check_repo as cr + + assert cr.check_version_boundary_present(cr._load_lock()) == [] + + +def test_version_boundary_flags_missing_boundary(tmp_path, monkeypatch): + """A README that states the version but omits the 4.x boundary is flagged.""" + import scripts.check_repo as cr + + (tmp_path / "README.md").write_text( + "Validated with MinerU 3.4.4.\n", encoding="utf-8" + ) # no 4.x boundary + monkeypatch.setattr(cr, "REPO", tmp_path) + lock = {"validated_software": {"mineru": {"version": "3.4.4"}}} + findings = cr.check_version_boundary_present(lock, tmp_path) + assert any("boundary" in f for f in findings), findings + + +def test_markdown_internal_links_clean_on_repo(): + import scripts.check_repo as cr + + assert cr.check_markdown_internal_links() == [] + + +def test_markdown_internal_links_flags_dead_link(tmp_path, monkeypatch): + """A relative link to a missing file is flagged; a live link passes.""" + import scripts.check_repo as cr + + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "real.md").write_text("hello\n", encoding="utf-8") + (tmp_path / "README.md").write_text( + "[live](docs/real.md) [dead](docs/missing.md)\n", encoding="utf-8" + ) + monkeypatch.setattr(cr, "REPO", tmp_path) + findings = cr.check_markdown_internal_links(tmp_path) + assert any("missing.md" in f for f in findings), findings + assert not any("real.md" in f for f in findings), findings diff --git a/tests/test_generate_evidence_manifest.py b/tests/test_generate_evidence_manifest.py new file mode 100644 index 0000000..2a6b3f9 --- /dev/null +++ b/tests/test_generate_evidence_manifest.py @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 AIwork4me +"""Tests for the evidence-manifest generator (scripts/generate_evidence_manifest.py).""" + +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] + + +def _load(): + spec = importlib.util.spec_from_file_location( + "gem", REPO / "scripts" / "generate_evidence_manifest.py" + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_canonical_manifest_is_stable(tmp_path): + """Regenerating the canonical manifest yields identical entries (ignoring the timestamp).""" + gem = _load() + paths = gem.collect_paths(argparse_ns()) + assert paths, "canonical evidence set is empty" + m1, e1 = gem.build_manifest(paths, missing_ok=False) + m2, e2 = gem.build_manifest(paths, missing_ok=False) + assert e1 == [] and e2 == [], e1 + # entries are sorted + stable + assert m1["entries"] == m2["entries"] + paths_in_order = [e["path"] for e in m1["entries"]] + assert paths_in_order == sorted(paths_in_order) + + +def test_missing_critical_file_fails(tmp_path, monkeypatch): + """A missing critical file is an error, not silently skipped (without --allow-missing).""" + gem = _load() + fake = tmp_path / "does_not_exist.json" + _, errs = gem.build_manifest([fake], missing_ok=False) + assert errs and "missing critical evidence file" in errs[0] + + +def test_check_detects_drift(tmp_path, monkeypatch): + """--check exits 1 when a committed file's sha no longer matches the manifest.""" + gem = _load() + monkeypatch.setattr(gem, "REPO", tmp_path) + (tmp_path / "REPRO.yaml").write_text("a: 1\n", encoding="utf-8") + (tmp_path / "evidence").mkdir() + # build a manifest against the current content + paths = gem.collect_paths(argtype_with(tmp_path)) + manifest, errs = gem.build_manifest(paths, missing_ok=False) + assert errs == [], errs + out = tmp_path / "evidence" / "evidence-manifest.json" + out.write_text(json.dumps(manifest), encoding="utf-8") + # mutate the file -> drift + (tmp_path / "REPRO.yaml").write_text("a: 2\n", encoding="utf-8") + rc = gem.main(["--check", "--out", str(out)]) + assert rc == 1, "drift must fail --check" + + +class _Ns: + """Minimal argparse Namespace stub for collect_paths.""" + + def __init__(self, **kw): + self.__dict__.update(kw) + + +def argparse_ns(): + return _Ns(canonical=True, file=None, dir=None) + + +def argtype_with(_repo): + return _Ns(canonical=True, file=None, dir=None) diff --git a/tests/test_smoke_test.py b/tests/test_smoke_test.py new file mode 100644 index 0000000..fa19d8a --- /dev/null +++ b/tests/test_smoke_test.py @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 AIwork4me +"""Tests for the portable smoke test (scripts/smoke_test.py).""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] + + +def _load(): + spec = importlib.util.spec_from_file_location( + "smoke_test", REPO / "scripts" / "smoke_test.py" + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_check_only_exits_zero(): + """--check-only is green on the real repo (no GPU / no inference required).""" + st = _load() + assert st.main(["--check-only"]) == 0 + + +def test_help_exits_zero(capsys): + st = _load() + try: + st.main(["--help"]) + except SystemExit as exc: + assert exc.code == 0 + + +def test_level0_env_is_ok_and_informational(): + st = _load() + status, name, detail = st.level0_env() + assert status == "OK" and name == "env" + import json + + info = json.loads(detail) + for key in ("os", "python", "mineru_rocm", "torch"): + assert key in info + + +def test_level4_skips_without_url(): + st = _load() + status, name, _ = st.level4_vlm(None) + assert status == "SKIP" and name == "vlm" + + +def test_level3_check_only_skips_inference(tmp_path): + """In --check-only, Level 3 verifies the adapter + images exist but does not infer.""" + st = _load() + status, name, detail = st.level3_inference( + check_only=True, img_dir=REPO / "examples", out_dir=None + ) + assert status == "SKIP" and "dry-run" in detail diff --git a/tests/test_validate_schemas.py b/tests/test_validate_schemas.py new file mode 100644 index 0000000..70b330b --- /dev/null +++ b/tests/test_validate_schemas.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright 2026 AIwork4me +"""Tests for the schema validation gate (scripts/validate_schemas.py).""" + +from __future__ import annotations + +import json +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] + + +def _validator(): + import importlib.util + + spec = importlib.util.spec_from_file_location( + "validate_schemas", REPO / "scripts" / "validate_schemas.py" + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_validate_schemas_clean_on_repo(): + """REPRO.yaml + all model cards pass schema + cross-field checks on the real repo.""" + assert _validator().main() == 0 + + +def test_bad_sha256_flagged(tmp_path, monkeypatch): + """A *_sha256 value that is neither 64-hex nor 'not_recorded' is flagged.""" + vs = _validator() + errs: list[str] = [] + vs._walk_sha256({"k_sha256": "deadbeef"}, "", errs, "REPRO.yaml") # too short + assert any("k_sha256" in e for e in errs), errs + + +def test_not_recorded_sha256_accepted(tmp_path): + """'not_recorded' is an explicit, accepted placeholder for a missing digest.""" + vs = _validator() + errs: list[str] = [] + vs._walk_sha256({"k_sha256": "not_recorded"}, "", errs, "REPRO.yaml") + assert errs == [], errs + + +def test_valid_sha256_accepted(): + """A real 64-hex digest passes the SHA-256 walker.""" + vs = _validator() + errs: list[str] = [] + vs._walk_sha256({"k_sha256": "a" * 64}, "", errs, "REPRO.yaml") + assert errs == [], errs + + +def test_model_card_rejects_undefined_status(tmp_path, monkeypatch): + """A model card with an undefined badge status is flagged by the schema.""" + from jsonschema import Draft202012Validator as Validator + + schema = json.loads( + (REPO / "schemas" / "model-card.schema.json").read_text(encoding="utf-8") + ) + bad_card = { + "schema_version": 1, + "model_id": "mineru2.5", + "platforms": ["linux-rocm"], + "badge": {"linux-rocm": "verified"}, # 'verified' is NOT in the enum + "omnidocbench_version": "v1.6", + "overall": 95.56, + "artifacts": {"x": "y"}, + } + errs = list(Validator(schema).iter_errors(bad_card)) + assert errs, "undefined status 'verified' must be rejected" + assert "verified" in errs[0].message + + +def test_repro_rejects_unknown_reproducibility_level(): + """REPRO.yaml schema rejects a reproducibility.level outside the documented enum.""" + import json + from jsonschema import Draft202012Validator as Validator + + schema = json.loads( + (REPO / "schemas" / "repro.schema.json").read_text(encoding="utf-8") + ) + errs = list( + Validator(schema).iter_errors( + {"reproducibility": {"level": "totally-reproducible"}} + ) + ) + # The minimal dict is missing other required keys, but the bad enum must be + # among the reported errors. + assert any("totally-reproducible" in e.message for e in errs), [ + e.message for e in errs + ]