feat(python): expose strip_headers_footers and remove_page_numbers - #458
Open
ddakv wants to merge 1 commit into
Open
feat(python): expose strip_headers_footers and remove_page_numbers#458ddakv wants to merge 1 commit into
ddakv wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Exposes existing Rust markdown options to the Python API. The changes use keyword-only arguments and preserve default behavior, following established library patterns with thorough test coverage.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Exposes two existing
MarkdownOptionsswitches as keyword-only arguments on the Pythonprocess_pdf/process_pdf_bytesbindings:Both default to
True, matchingMarkdownOptions::default()— existing callers see byte-identical behavior.Why
The Rust API has had these switches all along (
PdfOptions::new().markdown(...)), and the WASM binding already exposes a subset ofMarkdownOptions(profile,includePageMarkers,includeImages) — but the Python bindings hardcodePdfOptions::new(), so Python callers cannot opt out of furniture stripping.Our use case: a document-ingestion pipeline (RAG) where header/footer removal must be a per-workflow choice — some corpora need verbatim page text preserved (compliance/eval requirements), others want the noise gone. Today the default (strip) is right for most retrieval workloads, but the pipeline can't offer the choice without this exposure.
Notes on semantics
strip_headers_footers=Falsecleanly bypasses the furniture pass (markdown/mod.rs's gate), keeping running headers/footers.remove_page_numbers=Falsedisables the lexical standalone-page-number filter in postprocess. Folios resolved from positional evidence are still removed — layout analysis depends on them — and the new docstrings/stub/docs say so explicitly, so nobody reads the kwarg as "verbatim output".pages), following theprocess_pdf_with_ocrsignature style. The shared option assembly lives in onebuild_process_optionshelper.Testing
tests/test_python.py: repeated running furniture survives withstrip_headers_footers=False(p1244-1996.pdf) and is stripped by default;Page NNNtext survives withremove_page_numbers=False(multiline_indent_cell_rect_grid.pdf); explicitTruekwargs produce byte-identical markdown to the bare call; kwargs are keyword-only; the bytes variant matches the file variant.cargo fmt,cargo clippy --features python -- -D warnings, andcargo testall pass.pdf_inspector.pyianddocs/python.mdupdated in step.tests/test_python.py::TestMultipleFixtures::test_process_all_fixtures[encrypted-secret123.pdf]fails on unmodifiedmaintoo (pre-existing,process_pdfhas no password parameter); unrelated to this change.Summary by cubic
Expose
strip_headers_footersandremove_page_numbersas keyword-only args inpdf_inspectorforprocess_pdfandprocess_pdf_bytes. This lets Python callers keep running headers/footers and standalone page-number text when needed; defaults preserve previous output.pages:process_pdf(path, pages=None, *, strip_headers_footers=True, remove_page_numbers=True)and same forprocess_pdf_bytes.MarkdownOptions::default()).build_process_optionsthat setsPdfOptions.markdownwith these fields.strip_headers_footers=Falseskips the furniture pass;remove_page_numbers=Falsedisables the standalone page-number filter; folios removed via positional evidence remain removed.pdf_inspector.pyianddocs/python.md; add tests for defaults, keyword-only enforcement, bytes/file parity, and examples where furniture/page numbers are retained.Written for commit 37a9572. Summary will update on new commits.