Problem
strip_headers_footers (and the folio filtering) remove text with no trace of what or how much was removed. For ingestion pipelines that persist extracted text (RAG, archival), silent removal is hard to operate: when a customer asks "where did my footer text go" — or worse, when the furniture heuristic misfires on a boundary case — there is nothing to audit.
Our pipeline stamps every page that lost lines with a removed-count in page metadata, so removals are visible without diffing outputs. We currently do this with a layout-model-based extractor and would like parity when processing via pdf-inspector.
Proposal
Add a per-page removed-line count to PdfProcessResult, e.g.:
/// 1-indexed page → number of text lines removed as running headers/footers.
pub removed_header_footer_lines: Vec<(u32, u32)>,
Sketch (happy to implement if the direction sounds right):
- Compute per-page counts at the existing gate in
markdown/mod.rs (diff the per-page line counts around furniture::strip_header_footer_lines) — no change to furniture.rs itself.
- Thread them through
MarkdownConversionOutput (the struct already exists for markdown + detected_tables).
- Surface on
PdfProcessResult and the Python/Node/WASM result wrappers (or Python-only to start, if you prefer keeping the other bindings lean).
Open questions where I'd take your steer:
- Total vs per-page (per-page is what an audit trail needs; total is cheaper to plumb).
- Whether folio removals (
select_items_with_document_folio_context) should be counted in the same field or reported separately — they are removed unconditionally, so an audit entry is arguably even more valuable there.
- Whether
MarkdownConversionOutput is the right vehicle or you'd rather grow the MarkdownDocumentContext with an out-slot.
Related: the sibling PR exposing strip_headers_footers / remove_page_numbers on the Python bindings — this issue is the second half of making furniture handling operable: choice there, visibility here.
Problem
strip_headers_footers(and the folio filtering) remove text with no trace of what or how much was removed. For ingestion pipelines that persist extracted text (RAG, archival), silent removal is hard to operate: when a customer asks "where did my footer text go" — or worse, when the furniture heuristic misfires on a boundary case — there is nothing to audit.Our pipeline stamps every page that lost lines with a removed-count in page metadata, so removals are visible without diffing outputs. We currently do this with a layout-model-based extractor and would like parity when processing via pdf-inspector.
Proposal
Add a per-page removed-line count to
PdfProcessResult, e.g.:Sketch (happy to implement if the direction sounds right):
markdown/mod.rs(diff the per-page line counts aroundfurniture::strip_header_footer_lines) — no change tofurniture.rsitself.MarkdownConversionOutput(the struct already exists formarkdown+detected_tables).PdfProcessResultand the Python/Node/WASM result wrappers (or Python-only to start, if you prefer keeping the other bindings lean).Open questions where I'd take your steer:
select_items_with_document_folio_context) should be counted in the same field or reported separately — they are removed unconditionally, so an audit entry is arguably even more valuable there.MarkdownConversionOutputis the right vehicle or you'd rather grow theMarkdownDocumentContextwith an out-slot.Related: the sibling PR exposing
strip_headers_footers/remove_page_numberson the Python bindings — this issue is the second half of making furniture handling operable: choice there, visibility here.