Distinct numeric value columns merged into one cell for one row per PDF (customs FDI statistical tables)

[seanpham99]

Bug report: distinct numeric value columns merged in heuristic table extraction

Summary

Two adjacent numeric value columns (Reporting month.Value (USD) and Year to date.Volume) are merged into a single cell for exactly one row per PDF, on Vietnamese Customs FDI statistical reports (Table: 17B/TCHQ). Reproduced on pdf-inspector v1.15.0 (latest release, 2026-08-17) and on master (grid.rs unchanged since the merge pass was introduced).

Environment

• pdf-inspector v1.15.0 (pip pdf-inspector), Python 3.13
• extract_pages_markdown() on a text-based 2-page PDF (78 KB, classify_pdf → text_based, confidence 1.00)

Repro

Fixture: customs FDI preliminary report — July 2026 import, row 6 "Coal" (also reproduced on June 2026 import PDF; both attached). Both are real production PDFs from the Vietnam General Department of Customs.

from pdf_inspector import extract_pages_markdown

r = extract_pages_markdown("july_import.pdf")
md = "\n".join(p.markdown for p in r.pages)
for line in md.splitlines():
    if "|6 Coal|" in line:
        print(line)

Actual (bug):

|6 Coal|Ton|1,709,851|246,515,548 11,734,697||1,555,338,055|

Expected: rep_val and ytd_vol as separate cells:

|6 Coal|Ton|1,709,851|246,515,548|11,734,697|1,555,338,055|

Every other row in the same table extracts correctly (34 rows total; only row 6 merges). Export-direction PDFs (different table layout) extract cleanly — 0 merged rows. The same Coal-row defect reproduces on June 2026 import: |6 Coal|Ton|1,661,481|236,480,212 10,024,724||1,302,660,618|.

Root cause analysis

PDF word geometry (from pdfplumber, x0/x1 coordinates in pt) for the Coal row vs a clean row:

Wheats row value gaps: [22.5, 13.6, 20.6]  min 13.6 pt
Coal   row value gaps: [17.0,  8.1, 12.3]  min  8.1 pt   <-- abnormal

The Coal row's rep_val → ytd_vol gap is 8.1 pt (every other row ≥ 12.3 pt). This is deterministic in the source PDF (same 8.1 pt signature in both June and July; the Coal commodity name is short, shifting the value-column alignment).

In src/tables/grid.rs, merge_numeric_adjacent_clusters:

let merge_dist = threshold * 1.5;   // default threshold 25pt -> 37.5pt
// merge if dense.numeric_frac > 0.50 && sparse.count <= dense.count / 2 && sparse.count <= 5

Both adjacent clusters are predominantly numeric; the cluster centers land within merge_dist (37.5 pt) because the Coal row's 8.1 pt gap pulls the centers together, so the merge pass collapses the two value columns for this row. For all other rows the gap (≥12.3 pt) keeps the centers far enough apart to avoid the merge — but only just (12.3 pt is still well under 37.5 pt; the algorithm happens to survive on center distribution, not on a hard gap check).

Suggested fix

The numeric merge pass exists to fix header-vs-data splits ("header text" next to a dense numeric column). Merging two predominantly numeric clusters into one is never the intended behavior. Propose guarding merge_numeric_adjacent_clusters so it does not merge when both clusters are predominantly numeric:

let should_merge =
    dense.numeric_frac > 0.50
    && sparse.count <= dense.count / 2
    && sparse.count <= 5
    && !(info_a.numeric_frac > 0.50 && info_b.numeric_frac > 0.50); // new guard

This preserves the header-fix behavior while preventing numeric-column collapse. Alternative (weaker): require dist > some_min_gap when both numeric.

Fixtures

• july_import.pdf (78,419 bytes) — July 2026, import, reproduces
• 2026_06_import_preliminary.pdf (78,347 bytes) — June 2026, import, reproduces
• Control (clean): 2026_06_export_preliminary.pdf — export direction, no merge

[seanpham99]

2026_07_import_preliminary.pdf

2026_06_import_preliminary.pdf