fix: extract_pages_markdown no longer splits a word with a mid-word <u> tag

[MADENIYOU]

Fixes #397.

Root cause

TextLine::text_with_formatting (src/types.rs) toggles the <u> tag per positioned item, using each item's own is_underline flag independently. A single visual word can be split across several content-stream text-showing operators, and the source PDF can carry the underline flag on only one fragment — e.g. an underline rectangle whose geometry happens to cover only part of a word's width, a rendering artifact rather than intentional partial-word styling.

Layout-level merging (merge_text_items) already refuses to combine items across a style difference (its own comment explains why: "OR-ing underline instead would stretch <u> spans over neighboring plain text"), so such fragments arrive at TextLine::text_with_formatting as adjacent, unmerged items with zero gap between them. Each toggling its own tag independently produces <u>We</u>alth for a word like "Wealth" — not formatting that was in the source.

Fix

Don't toggle the underline tag between two items when both:

1. Are truly geometrically contiguous — gap under 2% of font size. This is intentionally much tighter than the general prose-spacing threshold (needs_space_between), because the goal here is different: distinguishing "this is the same word, split at the extraction layer" from "these are two separate items a normal layout heuristic happens to join without inserting a space" (many unrelated reasons exist for the latter — punctuation attachment, hyphenation, kerning).
2. Are not separated by closing punctuation — a trailing ./,/;/etc. isn't part of the word before it, even when kerned tightly against it. This condition was needed after a geometry-only version regressed existing snapshot output (td9264.pdf, a real, tightly-kerned legal document) where <u> legitimately opens/closes right at a punctuation boundary.

A continuation meeting both conditions now inherits whichever underline state is already open, instead of evaluating its own (possibly spurious) flag — so the tag wraps the whole word (or none of it), never splitting mid-word.

Testing

• New unit test underline_does_not_toggle_mid_word, reproducing the issue's exact pattern (a word split across 4 items, underline flagged on only one interior fragment). Verified it fails without the fix (reproduces the exact reported <u>We</u>alth split) and passes with it.
• New unit test underline_still_toggles_at_a_real_word_boundary, confirming the fix doesn't suppress underline at genuine word boundaries.
• Iterated against the existing snapshot suite: an early geometry-only version of this fix regressed td9264.pdf (extended <u> across sentence punctuation and into unrelated following prose); the punctuation condition above was added specifically to fix that regression, and the full suite now passes unchanged.
• cargo fmt, cargo test (all passing, 2 new), cargo clippy --all-targets -- -D warnings clean.

🤖 Generated with Claude Code

---

Summary by cubic

Prevents extract_pages_markdown from opening/closing <u> mid-word. Previously underline toggled per item and could emit <u>We</u>alth; now tightly contiguous word fragments are grouped and one underline decision is applied, avoiding order dependence and spans across real word boundaries.

• Precomputes effective underline via TextLine::resolve_underline_by_word_group: groups only when the gap is under 2% of font size, is direction-aware (RTL), agrees with the render loop’s spacing decision (should_join_items), preserves the hyphen no-space exception from needs_space_between, excludes closing punctuation on either side, treats strikeout as a hard boundary, and does not treat ' as a boundary.
• Resolves a run’s underline by majority character count (ties favor plain text); the render loop uses this effective flag. Strikeout still overrides underline; bold/italic behavior unchanged. Tests cover hyphenated joins, order-independence, RTL contiguity, strikeout isolation, symmetric punctuation checks, apostrophe-in-word contractions, CID-font spacing alignment, and the original mid-word and real boundary cases.

^{Written for commit ace4153. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 1 file

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[MADENIYOU]

All five confirmed and fixed — thanks, this was a thorough round.

Order-dependence (P2): Right — serial inheritance made the result depend on fragment order, not just source formatting. Replaced with a two-pass design: group tightly-continuous items into runs first, then decide each run's underline status once via majority character count (ties favor not-underlined). Same source pattern now always resolves identically regardless of which fragment carries the flag.

RTL gap (P2): Confirmed, the gap calc was LTR-only. Made it direction-aware, mirroring should_join_items's own pattern (if prev.x <= curr.x { ... } else { ... }).

Strikeout as hard boundary (P2): Confirmed, and worth noting precisely what this closes: the render loop already excludes <u> on a struck item unconditionally, so an overlapping <u>/<s> span was never actually reachable in this version — but a struck fragment's flag could still skew the majority vote for its non-struck neighbors before this fix. Made strikeout a hard grouping boundary so a struck item's flag can't leak into a neighboring run's tally.

Previous-item trailing punctuation (P2): Confirmed, the check was one-sided. Added the symmetric check on the previous item's own trailing character.

Apostrophe breaks contractions (P2): Confirmed — good catch. Removed ' from the closing-punctuation set; unlike ./,/etc. it can legitimately sit mid-word.

Added one regression test per finding (5 new), each verified to fail without its corresponding piece of the fix and pass with it. Full suite passing, clippy clean.

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[MADENIYOU]

Confirmed and fixed — good catch, and thanks for citing the exact `should_join_items` special cases to check against.

The run-grouping check was making its own independent judgment call using only geometry + punctuation, with no connection to `should_join_items` — the function the render loop's own `needs_space_between` call actually uses. Added it as a required condition for run continuation, so grouping can never disagree with the real spacing decision made in the same render path (CID-font one-word-per-operator gaps, the colon-label case, and anything else `should_join_items` already knows about).

New test `underline_grouping_agrees_with_should_join_items_on_cid_fonts`, reproducing the CID-font case cited in the finding (two distinct C2_-font words with a near-zero gap between them). Verified it fails without the fix (the two words wrongly merge into one underline run: `Alpha Beta` instead of `Alpha Beta`).

Full suite passing, clippy clean.

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[MADENIYOU]

Confirmed and fixed. `needs_space_between`'s hyphen exception is applied on top of `should_join_items`'s result, not inside it — `should_join_items` itself has no hyphen special-case. My previous fix mirrored only `should_join_items`, missing that outer exception.

New test `underline_grouping_preserves_hyphen_no_space_exception`, using the same CID-font shape as the previous finding's test (so `should_join_items` alone says "not joined") with a hyphenated split ("auto-" + "fix"). Verified it fails without the fix (splits into two runs at the hyphen: `auto-fix`) and passes with it.

Full suite passing, clippy clean.

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would require human review. Adds a new heuristic algorithm to group text fragments and decide underline by majority, changing output formatting for all PDFs; several tuning choices (gap threshold, punctuation set) need human judgment to validate against real documents.

Re-trigger cubic</sub>