Codex/filex layout final score

CHANGELOG.md

@@ -9,7 +9,16 @@
   `aworld-cli` image behavior.
 - Added the asynchronous HTTP service, direct URL/object-reference ingestion,
   bounded queuing, persistent PaddleOCR-VL worker lifecycle, resumable PDF
-  batches, Document IR v2 formatting repair, and video evidence artifacts.
+  batches, Document IR v3 grounding/formatting repair, and video evidence
+  artifacts.
+- Preserved Paddle's detector geometry and confidence before VLM block merging,
+  normalized contextual page labels, and removed HTML/Markdown presentation
+  tokens from grounding attribution text. On a fixed 20-case Layout sample,
+  the official score improved from 62.31% to 76.71% with zero regressions.
+- Completed the clean 500-case Visual Grounding evaluation at 71.6970% and
+  updated the five-dimension equal-weight overall to 65.3520%. The report also
+  distinguishes that leaderboard aggregation from the 65.4367% case-weighted
+  mean across 2,534 numeric results.
 - Added public deployment, hardware, configuration, operations, and security
   documentation plus a pinned ParseBench evaluation report.
 - Removed environment-specific deployment history, internal hosts, repository

aworld-tools/filex/README.md

@@ -13,7 +13,7 @@ The repository ships two deployment shapes:
 
 Supported inputs include PDF, Markdown/text, Word, PowerPoint, Excel/CSV,
 images, audio, video, HTTP(S) files, and YouTube transcript sources. PDF output
-uses Document IR v2; video output can include timestamped keyframes, OCR
+uses Document IR v3; video output can include timestamped keyframes, OCR
 evidence, and a storyboard. Video evidence is not yet a full semantic video
 understanding model.
 
@@ -261,24 +261,26 @@ Useful model variables are:
 
 ## Evaluation
 
-The latest validated component baseline on the pinned ParseBench 2,553-case
-suite is **52.07% five-dimension equal-weight overall**. Strongest performance
-is content faithfulness (82.88%); the main remaining gap is visual grounding
-(5.30%).
+The pinned ParseBench suite contains 2,553 cases and all five dimensions have
+reached a trusted terminal state. The older 5.30% layout number is retired
+because it mixed 458 legacy contract zeros with 42 current Document IR results;
+it is not a valid score for the current runtime. A fixed 20-case current-layout
+A/B improved from **62.31% to 76.71%** after the Document IR v3 repair (11
+improved, 9 unchanged, 0 regressed). The subsequent clean 500-case layout run
+scored **71.70%**.
 
 | Dimension | Cases | FileX score | Official PaddleOCR-VL-1.6 reference |
 | --- | ---: | ---: | ---: |
 | Tables | 503 | 67.64% | 67.77% |
 | Charts | 568 | 56.14% | 54.24% |
 | Content faithfulness | 506 | 82.88% | 82.71% |
 | Semantic formatting | 476 | 48.40% | 54.64% |
-| Visual grounding / layout | 500 | 5.30% | 77.80% |
-| Equal-weight overall | 2,553 | **52.07%** | **67.43%** |
+| Visual grounding / layout | 500 | **71.70%** | 77.80% |
+| Equal-weight overall | 2,553 | **65.35%** | **67.43%** |
 
 Nineteen formatting cases returned `not_scored` and are excluded rather than
-counted as zero. The table combines the newest trusted campaign for each
-dimension, so it is a component baseline rather than a claim that every row was
-produced by one immutable release. See
+counted as zero. The overall row is the equal-weight mean of the five dimension
+means; the case-weighted mean over 2,534 numeric results is 65.44%. See
 [the ParseBench evaluation report](docs/parsebench-evaluation.md) for pinned
 revisions, methodology, limitations, and the optimization roadmap.
 

aworld-tools/filex/docs/parsebench-evaluation.md

@@ -17,9 +17,9 @@ planning, LLM quality, or downstream task completion.
   mean across the five dimensions
 
 System/control-plane errors were tracked separately from parser scores. A
-`not_scored` case was not converted to zero. The final baseline contains 2,534
-numeric scores, 19 `not_scored` formatting cases, and no unresolved execution
-failures.
+`not_scored` case was not converted to zero. All five dimensions reached a
+trusted terminal state with no unresolved execution failures. Nineteen
+formatting cases remain `not_scored` and are excluded from score means.
 
 ## Results
 
@@ -29,13 +29,31 @@ failures.
 | Charts | 568 | 568 | 0 | 56.1396% | 54.24% | +1.8996 pp |
 | Content faithfulness | 506 | 506 | 0 | 82.8802% | 82.71% | +0.1702 pp |
 | Semantic formatting | 476 | 457 | 19 | 48.4013% | 54.64% | -6.2387 pp |
-| Visual grounding / layout | 500 | 500 | 0 | 5.2995% | 77.80% | -72.5005 pp |
-| Equal-weight overall | 2,553 | 2,534 | 19 | **52.0726%** | **67.43%** | **-15.3574 pp** |
-
-The official reference values are the ParseBench published
-PaddleOCR-VL-1.6 Full Pipeline results. FileX is close to the reference on
-tables and content, exceeds it on this chart run, trails on formatting, and
-does not yet satisfy the benchmark's grounding contract.
+| Visual grounding / layout | 500 | 500 | 0 | 71.6970% | 77.80% | -6.1030 pp |
+| Equal-weight overall | 2,553 | 2,534 | 19 | **65.3520%** | **67.43%** | -2.0780 pp |
+
+The official reference values are the ParseBench published PaddleOCR-VL-1.6
+Full Pipeline results. FileX is close to the reference on tables and content,
+exceeds it on this chart run, and trails on formatting and visual grounding.
+The overall value is an equal-weight mean of the five dimension means, matching
+the published leaderboard aggregation. A case-weighted mean over the 2,534
+numeric results is 65.4367%; it answers a different question and should not be
+compared directly with the 67.43% leaderboard overall.
+
+The previously reported 5.2995% layout value is retired: it combined 458
+legacy hard-coded contract zeros with 42 newly scorable Document IR cases. It
+measured a migration state, not the current parser. On the same fixed 20 cases,
+Document IR v3 improved the official mean from 62.3079% to 76.7097% (+14.4018
+percentage points): 11 cases improved, 9 were unchanged, and none regressed.
+The two worst cases moved from 0% to 100% and from 25% to 75%.
+
+The clean 500-case layout result also corrects an evaluation-adapter issue in
+64 PDFs containing both `layout` annotations and legacy `order` rows. The
+upstream JSONL loader otherwise constructs a parse test case instead of a
+layout-detection test case. Only `layout` rows are passed into the official
+layout evaluator; reading order remains scored by that evaluator from each
+annotation's `ro_index`. Those 64 cases all produced numeric scores, averaging
+79.6376%, and the complete 500-case layout mean is 71.6970%.
 
 ## Interpretation
 
@@ -55,32 +73,34 @@ scores do not measure service reliability or throughput.
 
 ### Main gaps
 
-1. **Visual grounding**: FileX does not yet emit reliable element bounding
-   boxes in the coordinate system required by ParseBench. A correct fix needs
-   page geometry, element identity, coordinate normalization, and confidence;
-   guessing boxes would inflate neither correctness nor usefulness.
+1. **Visual grounding**: Document IR v3 now preserves Paddle detector boxes,
+   confidence, page geometry, and parser text/order metadata. Remaining errors
+   are concentrated in detector misses, class ambiguity, and content
+   attribution for image-only controls.
 2. **Semantic formatting**: headings, lists, emphasis, superscript/subscript,
    and reading-order boundaries still lose information in difficult pages.
 3. **Not-scored formatting cases**: these require separate contract diagnosis;
    they are excluded from the mean and must not be presented as parser zeros.
 
 ## Recommended optimization path
 
-1. Extend Document IR v2 with page-relative and pixel-space bounding boxes,
-   source image dimensions, rotation, and stable element IDs.
-2. Carry Paddle layout detections through normalization instead of rebuilding
-   geometry from Markdown.
+1. Keep the fixed 20-case A/B as the release regression set and rerun the full
+   500-case layout suite after detector or mapping changes.
+2. Tune detector thresholds and label aliases against a separate development
+   split, especially small pictures and page furniture, without changing the
+   pinned official scorer.
 3. Add a formatting state machine that reconciles OCR spans with the PDF text
    layer and preserves nested lists, heading levels, emphasis, and scripts.
-4. Create regression sets per formatting rule and grounding object class before
-   another full-suite run.
-5. Re-run only the affected dimensions with pinned data/scorer revisions, then
-   publish a release-level score from one immutable FileX revision.
+4. Create regression sets per formatting rule and grounding object class.
+5. Publish a release-level score only from one immutable FileX revision and
+   complete campaign manifest.
 
 ## Reproducibility boundary
 
 This is the newest trusted result for each component, not a single campaign
-executed from one immutable FileX commit. It should be used as an engineering
-baseline and optimization guide. A release claim should pin the FileX image
-digest, parser configuration, hardware profile, data revision, scorer revision,
-and all five campaign manifests.
+executed from one immutable FileX commit. The 500 layout cases use one repaired
+FileX revision, while the five-dimension overall combines the newest trusted
+campaign per dimension. It should be used as an engineering baseline and
+optimization guide. A release claim should pin the FileX image digest, parser
+configuration, hardware profile, data revision, scorer revision, and all five
+campaign manifests.