fix(detector): cap decompressed content-stream size

[abimaelmartell]

Summary

• Decode page and Form content streams incrementally during detection, and stop at 32 MiB so a highly compressible Flate stream cannot materialize an unbounded buffer.
• Ordinary Flate page and Form streams still classify; over-budget streams are skipped for that scan.

Test plan

• cargo test --lib stream_decode
• cargo test --lib detector::tests::flate_page_content_still_finds_text_operators detector::tests::flate_form_xobject_still_finds_text_operators
• detect-pdf on a normal text PDF still reports extractable text

Made with Cursor

---

Summary by cubic

Caps decompressed PDF content-stream size during detector scans at 32 MiB to prevent highly compressible FlateDecode streams from ballooning memory. Previously the detector fully decompressed page and Form streams; now it inflates incrementally and skips any stream that would exceed the cap.

• Adds stream_decode.rs with bounded inflate for FlateDecode; other filters fall back to full decode only if within budget.
• Replaces full decompression with stream_content_for_scan in page and Form XObject scans; over-budget streams return empty content and are skipped for that scan.
• Adds dependency flate2 = "1.1".
• Includes tests ensuring ordinary Flate page/Form streams still yield text operators and that caps are enforced.

^{Written for commit 120f7f5. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/stream_decode.rs">

<violation number="1" location="src/stream_decode.rs:41">
P2: When a Flate content stream has PNG predictor parameters, this branch scans predictor-encoded bytes instead of applying predictor reversal. Apply predictor decoding incrementally or route predictor streams through a correctly bounded decoder, or text pages using them can be misclassified.</violation>

<violation number="2" location="src/stream_decode.rs:48">
P1: When a content stream uses a filter chain, this branch materializes the entire decoded buffer before checking its size. Decode every supported filter incrementally or reject the stream before this call, otherwise a small multi-filter stream bypasses the 32 MiB protection.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When a content stream uses a filter chain, this branch materializes the entire decoded buffer before checking its size. Decode every supported filter incrementally or reject the stream before this call, otherwise a small multi-filter stream bypasses the 32 MiB protection.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/stream_decode.rs, line 48:

<comment>When a content stream uses a filter chain, this branch materializes the entire decoded buffer before checking its size. Decode every supported filter incrementally or reject the stream before this call, otherwise a small multi-filter stream bypasses the 32 MiB protection.</comment>

<file context>
@@ -0,0 +1,145 @@
+    if stream.content.len() > max_bytes {
+        return None;
+    }
+    match stream.decompressed_content() {
+        Ok(data) if data.len() <= max_bytes => Some(data),
+        Ok(_) => None,
</file context>

[cubic-dev-ai]

P2: When a Flate content stream has PNG predictor parameters, this branch scans predictor-encoded bytes instead of applying predictor reversal. Apply predictor decoding incrementally or route predictor streams through a correctly bounded decoder, or text pages using them can be misclassified.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/stream_decode.rs, line 41:

<comment>When a Flate content stream has PNG predictor parameters, this branch scans predictor-encoded bytes instead of applying predictor reversal. Apply predictor decoding incrementally or route predictor streams through a correctly bounded decoder, or text pages using them can be misclassified.</comment>

<file context>
@@ -0,0 +1,145 @@
+    // Plain Flate is the highly compressible case. Detector scans only need
+    // the inflated operator bytes; skip PNG predictors here so inflate can
+    // stop at the budget instead of materializing the full buffer first.
+    if filters.len() == 1 && filters[0] == b"FlateDecode" {
+        return inflate_flate_bounded(&stream.content, max_bytes);
+    }
</file context>