Incomplete #227 fix: extract_pages_markdown ignores detected Type3 garble and returns needs_ocr=False

[knealfortegra]

Problem / impact

classify_pdf_bytes() and detect_pdf_bytes() correctly route a custom-encoded Type3 page to OCR, but extract_pages_markdown_bytes() returns the same page's wrong native text with needs_ocr=False.

This is the unsafe direction of the API contradiction reported in #227: a caller using the per-page extraction result silently skips OCR and consumes text that does not match the rendered page.

Environment

• pdf-inspector==1.15.0 from PyPI
• CPython 3.13.13
• macOS arm64
• Reproduced from in-memory bytes; no OCR runtime or model is involved

Reproduction

The fixture is 4,687 bytes, deterministic, stdlib-only, and contains no third-party content.

It draws all page text through a Form XObject using a custom-encoded Type3 font with no ToUnicode map. The visible glyph programs render ordinary fictional business text, while a native extractor that trusts the Encoding names reads a Caesar-shifted string.

Save this as make_repro.py and run python make_repro.py repro.pdf:

make_repro.py

from __future__ import annotations

import sys


def stream(data: bytes) -> bytes:
    return b"<< /Length " + str(len(data)).encode() + b" >>\nstream\n" + data + b"\nendstream"


def caesar(text: str, amount: int) -> str:
    result = []
    for char in text:
        if "A" <= char <= "Z":
            result.append(chr(ord("A") + (ord(char) - ord("A") + amount) % 26))
        else:
            result.append(char)
    return "".join(result)


def build_pdf() -> bytes:
    visible = [
        "POLICY NUMBER SAMPLE COMPANY CLAIM STATUS OPEN",
        "TOTAL INCURRED FIVE THOUSAND TWO HUNDRED",
    ]
    encoded = [caesar(line, 5) for line in visible]
    glyph_names = [chr(code) for code in range(ord("A"), ord("Z") + 1)]
    charprocs = " ".join(f"/{name} {9 + i} 0 R" for i, name in enumerate(glyph_names))
    differences = " ".join(f"/{name}" for name in glyph_names)
    widths = " ".join(["300", *("600" for _ in range(33, 91))])

    form = stream(
        ("BT /F1 14 Tf 28 TL 48 720 Td " f"({encoded[0]}) Tj T* ({encoded[1]}) Tj ET").encode()
    ).replace(
        b"<< /Length ",
        b"<< /Type /XObject /Subtype /Form /FormType 1 /BBox [0 0 612 792] "
        b"/Resources << /Font << /F1 6 0 R >> >> /Length ",
        1,
    )

    objects = [
        b"<< /Type /Catalog /Pages 2 0 R >>",
        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
        b"/Resources << /XObject << /Fm1 5 0 R >> >> /Contents 4 0 R >>",
        stream(b"q /Fm1 Do Q"),
        form,
        (
            "<< /Type /Font /Subtype /Type3 /Name /F1 "
            "/FontBBox [0 -200 700 900] /FontMatrix [0.001 0 0 0.001 0 0] "
            f"/CharProcs << /space 8 0 R {charprocs} >> "
            f"/Encoding << /Type /Encoding /Differences [32 /space 65 {differences}] >> "
            f"/FirstChar 32 /LastChar 90 /Widths [{widths}] "
            "/Resources << /Font << /FB 7 0 R >> >> >>"
        ).encode(),
        b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
        stream(b"300 0 d0"),
    ]

    for encoded_name in glyph_names:
        visible_name = caesar(encoded_name, -5)
        objects.append(stream(f"600 0 d0 BT /FB 700 Tf 0 0 Td ({visible_name}) Tj ET".encode()))

    output = bytearray(b"%PDF-1.4\n%\xe2\xe3\xcf\xd3\n")
    offsets = []
    for number, obj in enumerate(objects, start=1):
        offsets.append(len(output))
        output.extend(f"{number} 0 obj\n".encode())
        output.extend(obj)
        output.extend(b"\nendobj\n")

    xref = len(output)
    output.extend(f"xref\n0 {len(objects) + 1}\n".encode())
    output.extend(b"0000000000 65535 f \n")
    for offset in offsets:
        output.extend(f"{offset:010d} 00000 n \n".encode())
    output.extend(
        f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode()
    )
    return bytes(output)


with open(sys.argv[1] if len(sys.argv) > 1 else "repro.pdf", "wb") as output:
    output.write(build_pdf())

Then run:

import pdf_inspector as p

data = open("repro.pdf", "rb").read()
classification = p.classify_pdf_bytes(data)
detection = p.detect_pdf_bytes(data)
extraction = p.extract_pages_markdown_bytes(data)
page = extraction.pages[0]

print("classify pages_needing_ocr:", list(classification.pages_needing_ocr))
print("detect pages_needing_ocr:", list(detection.pages_needing_ocr))
print("detect reasons:", [(r.page, list(r.reasons)) for r in detection.ocr_reasons_by_page])
print("extract needs_ocr:", page.needs_ocr, repr(page.ocr_reason))
print("extract pages_needing_ocr:", list(extraction.pages_needing_ocr))
print(page.markdown)

Actual

classify pages_needing_ocr: [0]
detect pages_needing_ocr: [1]
detect reasons: [(1, ['suspected_garbled_text'])]
extract needs_ocr: False None
extract pages_needing_ocr: []
UTQNHD SZRGJW XFRUQJ HTRUFSD HQFNR XYFYZX TUJS

## YTYFQ NSHZWWJI KNAJ YMTZXFSI YBT MZSIWJI

The rendered page visibly says:

POLICY NUMBER SAMPLE COMPANY CLAIM STATUS OPEN
TOTAL INCURRED FIVE THOUSAND TWO HUNDRED

With the optional OCR runtime configured, process_pdf_with_ocr_bytes(data, mode="auto", page_numbers=[1]) inherits the extraction verdict: pages_routed_to_ocr=[], ocr_time_ms=0, provenance source='native', and the same wrong Markdown. This is only a consequence check; the minimal reproduction above does not require the OCR feature.

Fixture SHA-256: 5de0b3abe19e648de773520e5e12b074e1241fe321657e74b918c42aec5cccbb.

Expected

The extraction API should preserve the detector's OCR decision for this page instead of returning the decoded Type3 Encoding names as trustworthy native Markdown.

At minimum, page.needs_ocr and extraction.pages_needing_ocr should include the page with ocr_reason='suspected_garbled_text'; suppressing the wrong Markdown or retaining it as an explicitly untrusted OCR candidate is a separate policy choice.

Prior art / scope

This looks like an incomplete edge of #227 rather than the mixed-font classifier problem in #352.

Unlike #352, the classifier and detector are correct here. The disagreement is introduced by the per-page extraction result.

The fixture also resembles the systematic wrong-text symptom from #118, but uses a Type3 custom Encoding with no ToUnicode map rather than a Type0 font with a wrong ToUnicode map.

I tested the latest released Python package, 1.15.0. I did not build current main.

Suggested direction

Merge the detector's per-page OCR reasons into extract_pages_markdown's result, or otherwise ensure both public APIs use the same final trust decision for custom-encoded Type3 pages.

The important contract is that a detector-confirmed suspected_garbled_text page cannot be returned by the extraction API as trusted native text with needs_ocr=False.