ci: validate separated ruled-grid fix - #1
Open
Chengyunlai wants to merge 65 commits into
Open
Conversation
`render_line_inline` escaped `*`, `_` and `\` via `escape_inline()` before
handing the text to `apply_style()`, whose mono branch then wrapped the
already-escaped text in backticks. CommonMark: "Backslash escapes do not
work in code blocks, code spans, autolinks, or raw HTML" — so the inserted
backslashes are rendered literally by any conforming renderer.
Prose with a monospace run, driven through `parse_from_pages` with
`output_format = Markdown`:
"Please call the function named" (Helvetica)
+ "get_user_id" (Courier)
+ "before you continue with the setup." (Helvetica)
before: Please call the function named `get\_user\_id` before ...
after: Please call the function named `get_user_id` before ...
"Install the runtime under the folder" (Helvetica)
+ "C:\Program\bin" (Courier)
+ "and then restart the service daemon." (Helvetica)
before: Install the runtime under the folder `C:\\Program\\bin` and ...
after: Install the runtime under the folder `C:\Program\bin` and ...
Adds `style_body()`, which skips escaping for mono spans, and routes the
three escape-then-style call sites through it: the uniform-line fast path
and the per-group path in `render_line_inline`, and the uniform path in
`render_list_item_text`. The mixed-style fallback at the end of
`render_list_item_text` keeps plain `escape_inline` — it has no single
style to consult.
Only markdown output changes, and only for mono spans that contain `*`,
`_` or `\`; the delta is the removal of backslashes that were never
meaningful in that position.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes followups 1 and 2 in TABLE_TEDS_PROGRESS.md. On opendataloader-bench TEDS goes 0.7128 -> 0.7441, cutting the gap to pdf-inspector from 0.101 to 0.070; overall 0.8757 -> 0.8783. Neutral-to-positive on the ParseBench table dimension (0.4032 -> 0.4034). Two-pass last-resort for header + single-data-row tables. try_detect_table_ inferred takes allow_two_row; two_row_second_pass retries only in the index gaps where the normal pass found no table, so it cannot steal a real table's header. The gap restriction alone is not sufficient — fully-justified prose infers clean tracks from its stretched inter-word spaces, so two_row_run_ plausible also requires isolation and header shape. Doc 197: 0.000 -> 0.789, one doc changed, no collateral. Soft-wrap row grouping (merge_continuation_rows), applied where ruled and borderless runs converge. A row folds into its predecessor when it has an empty first cell, its filled columns are a subset of the predecessor's, and it carries no value-like cell. Guarded by a first-column fill test: "empty first cell" only means "wrapped line" when the first column is a label column, and without it sparse-first-column tables (timetables, size charts) collapse to a single row. Doc 150: 0.446 -> 0.861. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013TKUnWQHGdXrcP9KNSsckd
PDFium routinely emits a whole table row as one text run, which is the root of the track chicken-and-egg: the detector needs column tracks to split a merged run, but the merged runs are where the track evidence lives. Word boxes break the cycle — the gutter is visible in the geometry before any table hypothesis exists. split_span_at_gutters splits a run at its internal gutters using a bimodal test on the run's *own* gap distribution (real gutters run 3.4-4.4x the in-cell word gap; fully-justified prose, the one reliable counterfeit, tops out ~1.3x). A fixed threshold cannot work here — the same 4pt gap is a gutter in 6pt type and an ordinary space in 12pt type. Wired into the three places that previously reasoned over raw spans: track inference, strong-row detection, and cell assignment. opendataloader-bench TEDS 0.7441 -> 0.7471, NID +0.0009; ParseBench table composite 0.4034 -> 0.4063. Two docs up, none down on either benchmark. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014H7NdfNMMWsCKZccGXDvsY
…mpty
The ruled-grid density gate counted a rowspan continuation cell as a
failed text assignment, so a table with a merged label column ("1.
Embodying sustainability values" beside three competence rows) died as
mostly-empty and its text spilled into prose.
`rowspan_mask` reads the merge straight off the geometry: a cell whose
top boundary carries no horizontal stroke over its own centre, at a
boundary some vertical rule runs through, is merged with the cell above.
The mask rides on `CellGrid` so it survives the row/column collapses and
the gate forgives the empties that are actually present.
Two guards are load-bearing, each found by a document it broke:
- the vertical-continuation test, without which a page-frame component
shredded body prose into a two-column table;
- "the merged head must hold text", without which a chart's vertical
gridlines produced a 27-column table (-0.0072 NID over 4 docs).
Also prefer real word x's over character-index interpolation when
splitting a run across column anchors. Byte-inert on odl-bench (round
3's gutter split already consumes that class) but geometrically correct
where it does fire.
odl-bench TEDS 0.7471 -> 0.7526, NID/MHS flat, one doc up, none down.
ParseBench table composite 0.4063 -> 0.4074.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRj9HatWAHXGNyLbJVB1k7
…d pass The global ruled pass builds doc 200's table whole, then throws it away because `already_handled` finds that *some* xy_cut leaf tables on its own. The leaf in question was the landscape slide's title band — 3 of the run's 58 lines, which "tables" as a row of gutter-split title fragments. The real table's body rows then spilled out as prose. Require the vetoing leaf to hold at least a quarter of the run's lines. The decorative-frame case the veto defends against is unaffected: there the data region is the majority of the frame's content. odl-bench TEDS 0.7526 -> 0.7579 (doc 200 alone 0.054 -> 0.276, NID +0.097, MHS -0.143 as its titles move into the table), nothing else moves. ParseBench table composite flat at 0.4073. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRj9HatWAHXGNyLbJVB1k7
A table drawn as nothing but a top and bottom hairline has no grid for the ruled detector, and when it has only two columns it is below TABLE_MIN_COLUMNS for the borderless one. Doc 165 scored 0.000 for this reason — round 3's word-geometry splitter recovers both of its column tracks from the merged header run, and they were then thrown away. The rules are the missing evidence. `rule_bands` pairs two horizontal rules with no vertical crossing between them: a band with verticals is a real grid and the ruled detector owns it. Inside such a band, and only in the index gaps the normal pass left empty, retry detection with the column minimum lowered to two and single-piece body rows allowed (a blank second column makes every body row one piece). Detection runs against the band's lines alone, so a run can never reach past the rules that justify the relaxation in the first place. Three further gates: the band's lines must be contiguous, number at least three (a lone rule under a heading and stacked hyperlink underlines are the counterfeits), and the seed must read as a genuine two-cell row. odl-bench TEDS 0.7579 -> 0.7817; doc 165 alone 0.000 -> 1.000, matching pdf-inspector, with NID +0.046 and nothing else moving. ParseBench table composite flat at 0.4074. 308 lib tests pass. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRj9HatWAHXGNyLbJVB1k7
Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRj9HatWAHXGNyLbJVB1k7
A slide deck routinely draws a table as one stroked rect per cell, so a decorative highlight box behind a phrase *inside* a cell contributes a pair of vertical edges that split that column for the whole table. Doc 200 had a 21.6x9.4pt box shred a 210pt `Explanation` column, taking the grid from 4 columns to 6; `collapse_gutter_columns` cannot fuse the sliver back because text centres do land inside it. Add `RULED_VLINE_MIN_COVERAGE`, the mirror of the existing `RULED_HLINE_MIN_COVERAGE`: a vertical rule must span 20% of the component's row extent to count as a column boundary. Applying the filter directly is what does *not* work — dropping evidence can turn a grid the gates rejected into one they accept. Three documents found that the hard way (a bar chart released its gridlines as thematic breaks, a pie chart became a junk 2-column table, and a ParseBench insurance page grew an 11-column table that swallowed the whole page). Raising the fallback threshold patched them one at a time and kept finding new ones. Instead the unfiltered build now runs first and gatekeeps: the filtered grid may only ever *replace* a table that would have been produced anyway, and only when it measurably lowers the straddle census, so a stub sitting on a real column edge in a dense financial table is left alone. odl-bench: TEDS 0.7817 -> 0.7857, NID +0.0006, MHS +0.0000, one doc changed. ParseBench table composite: 0.4074 -> 0.4074 (flat). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KF4eSJHLg1tQmR4iG5ATC6
A table drawn with interior dividers but no outer frame yields `xs` that
stop at the first and last vertical rule, so its outermost columns are
missing entirely and every line in them reads as overhang. Doc 182's 4x4
table came out as the 2x2 grid its interior rules literally describe, and
all 31 lines tripped the overhang guard before the component collapsed.
The evidence is already on the page and nothing read it: the horizontal
rules know how wide the table is, and the verticals know how tall. Extend
each axis to the *median* extent of the perpendicular rules.
Three guards, each load-bearing:
- median, not min/max, so one overshooting stroke cannot widen the grid;
- the new outer band must hold a span centre — the same idiom
`collapse_gutter_columns` uses — or a pen-cap overshoot manufactures an
empty outer column;
- the band must be at least as tall as the shortest existing row. Verticals
routinely overrun the last horizontal by a few points; doc 45 turned a
15.5pt tail on 26pt rows into a phantom row, which was enough for the
ruled grid to outrank a better borderless table and merge five rows into
one. The column axis takes no such floor, because an unruled label column
is both common and legitimately much narrower than its data columns
(doc 182's is 94pt against 259pt).
odl-bench: TEDS 0.7857 -> 0.7980, NID +0.0008, MHS +0.0007, one doc changed
(182: 0.247 -> 0.762), nothing down.
ParseBench table composite: 0.4074 -> 0.4144, the largest single-change gain
so far — four pages of one insurance filing go from ~0.09-0.42 to ~0.99.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF4eSJHLg1tQmR4iG5ATC6
`cluster_v_segments` merges same-x verticals by taking the union of their y-ranges with no gap check. Two tables stacked in one column — each drawing its own short strokes at the same left edge — therefore fuse into a single grid component across a band of blank page. Each table is then evaluated with all of the other's rows empty and dies on the empty-cell fraction, and where the two tables have different layouts their column sets are unioned too, over-segmenting both. Docs 81-84 are one report, all four broken this way by a single 113pt gap; doc 81's `Number | of clauses` split across two columns, recorded earlier as a separate detector-priority problem, is the same union and needs no priority plumbing. Cut the component wherever a band between consecutive horizontals is spanned by no raw, pre-cluster vertical. That is strictly stronger than "a big gap", and unlike a gap rule it cannot fire on a rowspan, where the vertical does continue through the boundary. Each band's verticals are then clipped to the band, so the shared spine cannot carry the neighbour's height back in through the coverage filter or the outer-extent extension. Guards: median row pitch >= 8pt (vector-drawn maths glyphs make components with a 3-4pt pitch), the gap must clear both 2.5x the pitch and 30pt, and every band must keep >= 2 rows or the split is abandoned whole. This is the class the round-4 row-trim experiment failed on: a post-hoc trim keeps the unioned columns, which is why it kept losing docs 81 and 127. Splitting the component leaves 127, 130, 165 and 188 untouched. odl-bench: TEDS 0.7980 -> 0.8134 (pdf-inspector is 0.8141), NID +0.0012, MHS flat. Docs 82/84 reach 1.000, 83 0.995, 81 0.935. ParseBench table composite: 0.4144 -> 0.4171. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KF4eSJHLg1tQmR4iG5ATC6
Three changes, all the same shape: the geometry already answered the question and no code was reading it. TEDS 0.7817 -> 0.8134 against pdf-inspector's 0.8141, so the gap is now -0.0007 and we lead on every metric. ParseBench table composite 0.4074 -> 0.4171. Closes followup 8 (and corrects its diagnosis), rewrites followup 3 now that doc 200's columns are fixed, and adds diagnosed-but-unbuilt entries for docs 119 and 197 including the measurement showing 197's ceiling is a ground-truth idiosyncrasy rather than a defect worth chasing. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KF4eSJHLg1tQmR4iG5ATC6
The `liteparse` skill was documented in two repos with divergent prose and no link between them, and this repo's own landing page never mentioned it — the only way in was a single link from Getting Started. - Expand the guide with requirements (Node 18+, LibreOffice, ImageMagick, uv), the manual SKILL.md install path, the agent plugin, and a summary of the extraction patterns the skill actually teaches - Add the guide to the "Get started" list on the LiteParse index page - Link the guide from the README's Agent Skill section - Link out to the agent tooling hub at developers.llamaindex.ai/for-agents/ Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013vZtD6UvLVCEdWLRGd7Q32
Follow-up to 8ad92a3 / d9ee62e, reworking the flatten-based approach after review. Why flattening at all, rather than reading field values? The parser already exposes `page.form_fields()` with names, values and rects, and synthesizing text items from those would be a fraction of this code. It is wrong in both directions, though: a field can carry a /V that no appearance paints (an unpainted default is not visible text, and would be injected), and a painted appearance carries layout that the value alone loses — line breaks, comb spacing, and the widget's own font metrics. Flattening asks PDFium for what is actually painted, which is the same question the text layer answers everywhere else. Both cases are pinned by fixtures. Correctness: - FPDFPage_Flatten and FPDFAnnot_SetFlags now load via load_fn_opt. Loading them with load_fn! made them mandatory, so any pdfium build without fpdf_flatten.h failed PdfiumBindings::load outright and no document parsed at all. They follow the SignatureApi pattern instead, and a build without them just skips flattening. - The widget appearance walk descends into nested form XObjects. Acrobat and several server-side fillers emit `/Tx BMC q /Fm0 Do Q EMC`, where the top-level object is a form, not text; those filled fields looked empty and were never flattened. - has_annotation_text keeps its original HIDDEN-only semantics. The shared helper had widened it with INVISIBLE and NOVIEW, which silently changed the AnnotationText complexity signal — i.e. OCR routing — for non-widget annotations. The stricter mask now lives only on the form-widget path. - Page text that flattening suppressed is restored. PDFium's text layer emits only one of two runs starting at essentially the same point, so a flattened appearance can knock out page text it lands on. Where the strings match (a producer that wrote the value into both the content stream and the appearance) that is the dedup a partially flattened file needs; where they differ it is data loss, and the pre-flatten copy is put back. Cost: - Documents with no AcroForm catalog cost one form_type() call and never reach the annotation walk. - is_complex no longer reopens the input. Complexity now runs on the flattened document deliberately: AnnotationText means "the text is there, just outside the extractable surface", which stops being true once the value is in the content stream. Routing such a page to OCR re-derives text the parser already returned. - parse reopens the input only when OCR *and* render_form_fields are on, the one consumer that needs live widgets (it initializes the form environment to run document actions and paint computed appearances). Plain OCR rendering does not: a flattened page rasterizes the same. - Recovering suppressed text needs the page's text twice, so it is gated on a bounds-only probe for a text object overlapping a widget rect. The usual form page, whose widgets sit over blank space, pays a page-object walk instead of a second extraction. Also: the 4-tuple return is now ExtractedPages; the flatten/reload dance moved behind Document::flatten_form_widgets so no caller can hold a stale page handle; and the fixture is generated by a checked-in script documenting what each widget exercises, so its exact counts are auditable. Verified each fix fails the suite when reverted individually. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013TXb7LewB9rTYJY1hY9Dbz
- Rename the session primitive open() -> openBatchSession / open_batch_session on every surface (core, napi, wasm, pyo3) so it reads as batch-specific plumbing, and mark it internal in the Node and Python bindings where parseBatches()/parse_batches() is the public API. - Add ParseSession.close() to the napi binding and call it from parseBatches()'s finally block, so the converted-PDF temp file is released when iteration ends (including early break) instead of at GC. - Fix the Python native/wrapper ParseBatch name collision by renaming the pyo3 classes to _ParseBatch/_ParseSession. - Make Python parse_batches() validate and open eagerly so missing-file and target_pages errors raise at call time, not on first iteration; document the equivalent generator laziness on the Node side. - Replace the office-conversion integration test with a unit test that asserts the actual invariant: the converted temp PDF outlives every batch and is removed when the session drops. - Document wasm ParseSession cleanup (free()) and its one-batch-at-a-time constraint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GfJUQBVV5URPba1iRqDE6w
# Conflicts: # crates/liteparse-python/src/lib.rs # crates/liteparse-wasm/src/lib.rs # crates/liteparse/src/parser.rs # packages/node/native.d.ts # packages/node/src/lib.ts # packages/python/liteparse/__init__.py
Review follow-ups on continue_on_page_error: - Roll failed pages out of the image dedup cache so later duplicates never reference a dropped canonical image id; per-page extraction is now a named fn whose side effects (images, error counts, flattened flag) commit only on success instead of via manual rollback - Honor the flag in the OCR render and complexity passes (page keeps its native text, stats attach by page number) and in is_complex() - Warn on stderr when pages are skipped so text/markdown CLI output is not silently missing pages (Rust, Node, and Python CLIs) - Wire --continue-on-page-error into the Node and Python lit CLIs - Serialize PageError page number as `page` to match sibling per-page JSON fields; bindings keep their existing pageNum/page_num convention - Deduplicate Node parsePages result mapping through toParseResult Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAa9HQf765nP4AEMiuUZEk
Resolves config/binding conflicts with run-llama#409 (bounded-memory batch parsing) and run-llama#410 (tolerant page errors). Semantic fixes on top of the textual resolution: - packages/node: parse()/parsePages() use main's toParseResult helper, which now also maps screenshots - render_document_pages takes continue_on_page_error and skips pages that fail to render (mirrors render_pages_for_ocr), so tolerant-mode parses no longer abort on a screenshot render failure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DcsmXwKScQ1cVQksqCXhFy
The render module has no wasm-incompatible dependencies (encode_png was already always-compiled and the OCR path already rasterizes pages on wasm), so compile it for wasm32 and drop the cfg split that silently returned no screenshots there. Only the standalone screenshot()/ screenshot_input() APIs stay native-only — they go through LibreOffice conversion. Exposes extractScreenshots + detectScreenshotRects and the screenshots result field in the wasm binding. Verified end-to-end with a Node smoke run of the wasm-pack build (PNG magic, page count, default-off). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DcsmXwKScQ1cVQksqCXhFy
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbhNeqLgCSjawDk5B8jVfj
…an rendering Review follow-ups to the code-span escaping fix: - apply_style's mono branch used a hardcoded 2-backtick fence, so mono content containing a 2-backtick run closed the span early — and with bodies now unescaped, the spilled tail rendered as live markdown. The fence is now one backtick longer than the longest run in the content (single-backtick output is byte-identical to before). - The style_body + is_plain + apply_style pattern was repeated at all three call sites, leaving apply_style with an undocumented asymmetric precondition (escaped input for non-mono, raw for mono). Folded into render_span(), the single chokepoint call sites should use. The is_plain guard was output-identical to apply_style's plain arm, so this is a pure refactor. Markdown output for apple-10k-2024, long_tiny_text, annotation_text, and sample is byte-identical before and after. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQY3NuFuvKBWXdihAH3tYK
Chengyunlai
force-pushed
the
codex/fix-separated-grid-components
branch
from
August 13, 2026 09:23
514a2ef to
9d740a4
Compare
`reorder_independent_table_lines` sorted table-owned lines by grid rank but wrote them back into their original slot positions, which left two problems. A non-table line interleaved between table rows kept its slot while the table lines moved around it, stranding it mid-table. A page-spanning note between the rows of two side-by-side grids rendered as a bogus table row in the first table and collapsed the second one to loose paragraphs entirely — worse than the fused-table output it replaced. The sort key also omitted `region_path`, so a line could land in a different xy-cut leaf's slot range. `markdown_layout` recovers regions by scanning for maximal runs of equal `region_path` (`classify.rs`), so that shatters a leaf into phantom regions and misaligns the region-relative table runs keyed off it. Reorder one leaf at a time, and only over the span between that leaf's first and last table line, bailing out when anything page-spanning sits inside the span. Leaf contiguity is preserved by construction and the ambiguous pages are left in projection order rather than actively broken. Also resolve table ownership once per item instead of rescanning `table_rects` for every item of every y-band, which made line construction quadratic in band size. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVxkZQtLBRgR6YTLnvdWZB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build and test the existing separated ruled-grid fix on the fork's supported Python wheel targets.
This PR is used only to run the fork CI against the exact patch commit; the upstream contribution remains run-llama#392.