Text that is never painted survives extraction: white fill, same-colour fill, and occlusion (Tr 3 is handled correctly)

[hunter-heidenreich]

Summary

content_stream.rs decides whether text is invisible using text rendering mode alone. That is one of several ways a PDF carries text it never paints. Text hidden by a white fill, by a fill matching what is under it, or by being covered by a later opaque object is extracted and returned as ordinary content.

The Tr 3 handling itself is correct, and the gate on the retry (skipped_invisible && !has_visible_text, lib.rs:1093) is a good one. This is about the mechanisms it does not cover.

Borrowing the phrasing from #319: the question is whether the glyph is painted content, and render mode only answers part of it.

Reproduction

probe.pdf attached — 1051 bytes, hand-built so the content stream operators are exactly as intended, six lines on one page each hidden a different way.

make_probe.py (no dependencies, regenerates the attachment byte-for-byte)

# No leading newline: the EOL after the `stream` keyword is a delimiter and is
# NOT part of the stream data, so counting it in /Length overruns by one byte.
CONTENT = b"""BT /F1 14 Tf 0 g 60 740 Td (CASE 1 visible black text on white) Tj ET

BT /F1 14 Tf 1 g 60 700 Td (CASE 2 white fill on unpainted page) Tj ET

1 0 0 rg 50 650 400 26 re f
BT /F1 14 Tf 1 0 0 rg 60 658 Td (CASE 3 red text on a red rectangle) Tj ET

BT /F1 14 Tf 0 g 60 610 Td (CASE 4 black text occluded by a later box) Tj ET
1 g 50 602 400 26 re f

BT /F1 14 Tf 0 g 3 Tr 60 560 Td (CASE 5 black text at render mode 3) Tj ET

BT /F1 14 Tf 0 g 0 Tr 60 520 Td (CASE 6 visible black text, second control) Tj ET
"""


def build(path):
    objs = [
        b"<< /Type /Catalog /Pages 2 0 R >>",
        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
        b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
        b"<< /Length " + str(len(CONTENT)).encode() + b" >>\nstream\n" + CONTENT + b"\nendstream",
        b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
    ]
    out = bytearray(b"%PDF-1.4\n")
    offsets = []
    for i, body in enumerate(objs, start=1):
        offsets.append(len(out))
        out += str(i).encode() + b" 0 obj\n" + body + b"\nendobj\n"
    xref = len(out)
    n = len(objs) + 1
    out += b"xref\n0 " + str(n).encode() + b"\n0000000000 65535 f \n"
    for off in offsets:
        out += ("%010d 00000 n \n" % off).encode()
    out += (b"trailer\n<< /Size " + str(n).encode() + b" /Root 1 0 R >>\nstartxref\n"
            + str(xref).encode() + b"\n%%EOF\n")
    open(path, "wb").write(bytes(out))


build("probe.pdf")

import pdf_inspector as p
for it in p.extract_text_with_positions("probe.pdf"):
    print(repr(it.text))

Actual

'CASE 1 visible black text on white'
'CASE 2 white fill on unpainted page'
'CASE 3 red text on a red rectangle'
'CASE 4 black text occluded by a later box'
'CASE 6 visible black text, second control'

Expected

'CASE 1 visible black text on white'
'CASE 6 visible black text, second control'

Case 5 is correctly dropped. Cases 2, 3 and 4 are returned and none of them is painted.

Each line was checked two ways: what the API returned, and whether the rendered page contains any pixel differing from that line's local background within its band — compared against red for case 3, not against white.

Case	Mechanism	Painted	Extracted
1	black on white	yes	yes
2	white fill (1 g), nothing painted beneath	no	yes
3	red text on a red rectangle	no	yes
4	black text, opaque box painted after it	no	yes
5	render mode 3 (3 Tr)	no	no
6	black on white	yes	yes

Case 4 is the awkward one. Nothing about that text object is unusual: plain black, full opacity, render mode 0. It is invisible only because of paint order, so no property of the glyph or its graphics state can reveal it.

Where it comes from

src/extractor/content_stream.rs tracks text_rendering_mode (declared line 280, set at the "Tr" arm line 416, saved/restored across q/Q, reset at BT). That is the only visibility signal in the text path.

No colour operator is handled there — grep -nE '^\s+"(g|rg|k|sc|scn|cs|G|RG|K|SC|SCN|CS)"' src/extractor/content_stream.rs returns nothing, and the file contains no colour-related identifier at all. The colour operators are implemented in src/extractor/xobjects.rs, so fill colour never reaches the text-extraction path.

Real-world impact

Every bill PDF published by the U.S. Government Publishing Office (the whole congress.gov / govinfo corpus) carries two stamps drawn with a white fill on a white page: a print-tracking line and an operator/workstation identifier. Both are case 2, and both are returned by extract_text_with_positions:

Document	Total items	Stamp-bearing items
BILLS-110hr7337ih.pdf (2 pages, 2008)	62	10
BILLS-119hr10136ih.pdf (5 pages, 2026)	273	25

Five per page in both, for example:

'jbell on PROD1PC69 with BILLS'
'VerDate Aug 31 2005 '
'Jkt 079200'
'Sfmt 6201'
'E:\BILLS\H7337.IH'

Both bills attached. They were chosen 18 years apart because they share no toolchain (ACOMP.exe WinVer 1c15 + Distiller 5.0.5 in 2008; GPO G.S.D.D. + Distiller 26.0 / iText 9.4.0 in 2026) and are affected identically, which suggests a long-standing GPO convention rather than one bad export. I have tested only these two documents and have not swept the corpus.

One thing worth noting in this library's favour: the stamps come back as clean discrete items rather than spliced into neighbouring text, so they are at least filterable downstream by pattern or position.

Relation to existing issues

• Off-page text survives CropBox clipping on sparse and rotated pages #318 (off-page text surviving CropBox clipping) is the same class from a different angle: content present in the stream, never painted, surviving into output. A general "was this painted" test would cover both.
• Blank pages are routed to OCR when they contain no painted content #319 already reasons about painted content for routing; this is the same question asked per glyph rather than per page.
• Dense invisible OCR text layers (Tr 3) are classified TextBased and skip the invisible-text fallback — "OCR is required" on OCRmyPDF output #385 pulls the other way, wanting dense Tr 3 layers kept. Any fix here should suppress text that was never painted, not text that is merely coloured or merely invisible-by-mode, so the two should not collide.
• Overlapping text drawn twice at the same position is not deduplicated, every character is duplicated in markdown output #313 (text drawn twice at the same position not deduplicated) is adjacent to case 4 — both need paint order rather than glyph properties.

Notes toward a fix

Not a prescription: "was this glyph painted into the output" is a compositing question rather than a property of the text object, and case 4 shows it cannot be answered from the glyph alone. The three cases are not equally hard:

• Case 2 is nearly free — a white fill with nothing painted beneath it needs only the non-stroking colour tracked in the text path, which is where the file currently has nothing.
• Case 3 needs the fill compared against whatever occupies that position, which brings in colour spaces, alpha, blend modes and pattern fills.
• Case 4 needs paint order and coverage, which is effectively rasterization.

Case 2 alone would clear the entire congress.gov corpus. White fill is of course not invisible in general — white text on a dark background is legitimate content — so the test has to be colour-against-background rather than a white sentinel.

Environment

• pdf-inspector 1.17.0 (PyPI wheel), Python 3.12.13
• macOS 26.5.1, Darwin arm64
• Pixel verification: Ghostscript 10.07.1 renders at 150 dpi, sampled inside each item's reported bbox against local background

probe.pdf

BILLS-110hr7337ih.pdf
BILLS-119hr10136ih.pdf