Skip to content

feat(python): expose strip_headers_footers and remove_page_numbers - #458

Open
ddakv wants to merge 1 commit into
firecrawl:mainfrom
ddakv:feat/python-markdown-kwargs
Open

feat(python): expose strip_headers_footers and remove_page_numbers#458
ddakv wants to merge 1 commit into
firecrawl:mainfrom
ddakv:feat/python-markdown-kwargs

Conversation

@ddakv

@ddakv ddakv commented Aug 24, 2026

Copy link
Copy Markdown

What

Exposes two existing MarkdownOptions switches as keyword-only arguments on the Python process_pdf / process_pdf_bytes bindings:

result = pdf_inspector.process_pdf(
    "document.pdf",
    strip_headers_footers=False,  # keep running headers/footers
    remove_page_numbers=False,    # keep standalone page-number text
)

Both default to True, matching MarkdownOptions::default() — existing callers see byte-identical behavior.

Why

The Rust API has had these switches all along (PdfOptions::new().markdown(...)), and the WASM binding already exposes a subset of MarkdownOptions (profile, includePageMarkers, includeImages) — but the Python bindings hardcode PdfOptions::new(), so Python callers cannot opt out of furniture stripping.

Our use case: a document-ingestion pipeline (RAG) where header/footer removal must be a per-workflow choice — some corpora need verbatim page text preserved (compliance/eval requirements), others want the noise gone. Today the default (strip) is right for most retrieval workloads, but the pipeline can't offer the choice without this exposure.

Notes on semantics

  • strip_headers_footers=False cleanly bypasses the furniture pass (markdown/mod.rs's gate), keeping running headers/footers.
  • remove_page_numbers=False disables the lexical standalone-page-number filter in postprocess. Folios resolved from positional evidence are still removed — layout analysis depends on them — and the new docstrings/stub/docs say so explicitly, so nobody reads the kwarg as "verbatim output".
  • Kwargs are keyword-only (after pages), following the process_pdf_with_ocr signature style. The shared option assembly lives in one build_process_options helper.

Testing

  • Five new tests in tests/test_python.py: repeated running furniture survives with strip_headers_footers=False (p1244-1996.pdf) and is stripped by default; Page NNN text survives with remove_page_numbers=False (multiline_indent_cell_rect_grid.pdf); explicit True kwargs produce byte-identical markdown to the bare call; kwargs are keyword-only; the bytes variant matches the file variant.
  • cargo fmt, cargo clippy --features python -- -D warnings, and cargo test all pass.
  • pdf_inspector.pyi and docs/python.md updated in step.
  • No extraction-path changes and defaults preserved, so the pdf-evals snapshot suite is unaffected by construction (I don't have access to run it).
  • Note: tests/test_python.py::TestMultipleFixtures::test_process_all_fixtures[encrypted-secret123.pdf] fails on unmodified main too (pre-existing, process_pdf has no password parameter); unrelated to this change.

Summary by cubic

Expose strip_headers_footers and remove_page_numbers as keyword-only args in pdf_inspector for process_pdf and process_pdf_bytes. This lets Python callers keep running headers/footers and standalone page-number text when needed; defaults preserve previous output.

  • Add keyword-only args after pages: process_pdf(path, pages=None, *, strip_headers_footers=True, remove_page_numbers=True) and same for process_pdf_bytes.
  • Keep behavior identical for existing callers (defaults match MarkdownOptions::default()).
  • Factor option assembly into a shared build_process_options that sets PdfOptions.markdown with these fields.
  • Semantics: strip_headers_footers=False skips the furniture pass; remove_page_numbers=False disables the standalone page-number filter; folios removed via positional evidence remain removed.
  • Update pdf_inspector.pyi and docs/python.md; add tests for defaults, keyword-only enforcement, bytes/file parity, and examples where furniture/page numbers are retained.

Written for commit 37a9572. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Shadow auto-approve: would auto-approve. Exposes existing Rust markdown options to the Python API. The changes use keyword-only arguments and preserve default behavior, following established library patterns with thorough test coverage.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant