fix(renderer): don't feed binary bodies to the HTML extractor

[rqi14]

http_only.rs decides how to handle a response body from the declared content type alone:

let is_pdf = content_type.as_deref() == Some("application/pdf");
let (html, raw_bytes) = if is_pdf { ... } else { decode_html_bytes(...) };

so anything that isn't literally application/pdf is assumed to be text. Two consequences, both silent:

1. A .docx/.xlsx/.pptx (or any ZIP container) is UTF-8-lossy'd and run through the HTML extractor. The caller gets success: true with markdown beginning PK\u{3}\u{4}...[Content_Types].xml — ~1 MB of binary noise indistinguishable from a real scrape. An LLM consumer downstream will happily summarise it.
2. A real PDF served as application/octet-stream (common for S3 and for Content-Disposition: attachment endpoints) misses the PDF branch entirely and comes back as ~2 MB of %PDF-1.5 ... /FlateDecode ... stream source text instead of the parsed document.

Change

Two cheap checks in the same place:

• %PDF- magic sniff. If the bytes are a PDF but the header disagrees, relabel content_type so the existing PDF path — and the content_type == "application/pdf" gate in crw-crawl/src/single.rs — engages exactly as it already does for correctly-labelled PDFs.
• NUL byte in the first 1 KB (git's binary heuristic) ⇒ the body is not text. Skipped when the header declares a wide charset (utf-16/utf-32/ucs-*), whose HTML legitimately contains NULs.

Binary bodies raise a new CrwError::UnsupportedContentType, mapped to HTTP 422.

It needs to be its own variant rather than HttpError: classify_renderer_error maps HttpError to FailoverErrorKind::NetworkError, which escalates. Measured on a 1.3 MB .docx, that climbs lightpanda → chrome → camoufox and burns 23.06s and a Camoufox session before failing anyway with the generic Near-empty content (61 bytes). No browser turns a .docx into a page, so both escalation gates in lib.rs exclude the variant. Same document after: 0.005s, 422, error naming the actual content type.

crw-server's IntoResponse needed an explicit arm too — the _ => INTERNAL_SERVER_ERROR fallback would have reported a .docx as a server bug.

Measured

Local origin serving a real .docx and arXiv 1706.03762 mislabelled as application/octet-stream:

target	before	after
PDF as octet-stream	success: true, 1,969,949 chars of raw %PDF-1.5 ... /FlateDecode	success: true, 40,701 chars of article text, content_type: application/pdf, 15 pages, 0.286s
.docx	success: true, 1,019,506 chars of ZIP bytes	HTTP 422 in 0.005s, Unsupported content type: application/vnd.openxmlformats-officedocument.wordprocessingml.document (1311881 bytes), no ladder climb

The PDF output is byte-identical to what crw already produces for the same paper served with a correct content type.

Tests

Three for looks_binary (ZIP flagged, HTML and empty bodies passed, declared wide charset respected) and one asserting the server maps the new variant to 422 rather than falling through to _ => 500.

cargo test -p crw-core -p crw-server --lib green (495 / 274). crw-renderer --lib 715 pass; the 4 failures there (connection_failure_catches_connect_timeout, is_retriable_error_false_for_connect_timeout, direct_blackhole_then_{refusing,hanging}_proxy_is_target_unreachable) reproduce identically on unmodified main on this host — Windows refuses blackhole IPs instead of hanging — so they are environmental, not from this change.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[rqi14]

I have read the CLA Document and I hereby sign the CLA

[us]

thanks for this, and for the writeup. the diagnosis is right: http_only.rs routes purely on the declared content type, so a ZIP container gets utf-8-lossy'd into the extractor and comes back under success: true, and a correctly-shaped PDF served as octet-stream never reaches the parser. both are real, and returning a plausible-looking body for something that was never a page is the worse of the two.

the escalation point is a good catch too. burning a camoufox session on a .docx is pure waste, and the ladder swallowing the actual content type behind a generic message is exactly the kind of thing that is painful to debug from a caller's side.

i have the branch checked out and am running it through the full pipeline now, not just the unit boundary. i will come back with findings either way.

[us]

ran this through the full pipeline and it holds up. i reproduced both halves against a local origin first, on unmodified main as a baseline, same binary, same origin:

target	main	your change
real .docx	200, 1,168 chars of PK\x03\x04...[Content_Types].xml	422 in 0.10s
arXiv 1706.03762 as application/octet-stream	200, 1,969,949 chars of raw %PDF-1.5 ... /FlateDecode, 7.51s	200, 40,701 chars of article text, 1.35s
same paper as application/pdf (control)	40,701	40,701

the octet-stream output matches the correctly-labelled control exactly, so the relabel really does hand the body to the same parser rather than to a second path. and reusing the content_type == "application/pdf" seam in crw-crawl/src/single.rs instead of inventing a parallel signal is the right call: the crawl path picks it up for free, which i confirmed with a crawl over a hub page linking to a .docx and to that mislabelled pdf. the pdf gets parsed inside the crawl, the .docx comes back marked and counts as blocked, so it costs the caller nothing.

i pushed a top-up to your branch, rebased onto current main. what it changes and why:

the charset opt-out had to come from the decoder, not from a substring test. looks_binary matched utf-16 / utf-32 / ucs-, but decode_html_bytes resolves the label with encoding_rs::Encoding::for_label, and four labels that resolve to UTF-16 are not substrings of any of those: unicode, csunicode, unicodefeff, unicodefffe. charset=unicode is what classic IIS emits. measured, same bytes, only the label differing:

main:  /u16-unicode  200  '# Wide\n\n# Legacy IIS page...'
yours: /u16-unicode  422  unsupported_content_type

so a page that scrapes correctly today became a hard failure. the opt-out now asks for_label directly, which is also shorter than what it replaced and cannot drift from the decoder. utf-32 came out of the list, by the way: WHATWG has no UTF-32, for_label("utf-32") is None, so that arm was protecting a body the decoder cannot decode either.

the NUL test needed to stay off declared-HTML bodies. an html page with a stray NUL in its first 1 KB renders fine in a real browser, the HTML5 tokenizer maps NUL to U+FFFD. more importantly, because the escalation gates exclude the variant, a single \x00 injected by an nginx sub_filter would have shut the whole ladder down for that origin at zero cost to its human visitors. it now only applies when the origin did not declare an html-ish type, using the is_html_like_content_type helper already in the crate. an absent or empty Content-Type still gets sniffed, since that is the case a byte check exists for. this is also how firecrawl draws the line: a content-type denylist, never a body sniff on a declared text/html.

crw scrape still climbed the ladder. the cli runs its own phase-2 escalation outside FallbackRenderer, so crw scrape <docx-url> printed info: HTTP fetch failed (...), trying JS renderer... and then spawned LightPanda and Chrome anyway. same guard applied there, so the measurement in your commit message now holds on the cli too.

tests. the three looks_binary tests stay green if you delete the call site, delete the %PDF- relabel, or delete both escalation exclusions, so i added ones that fail. the ladder ones count renderer invocations rather than checking the error variant, because the variant alone survives deleting the guard. there is a wiremock file driving FallbackRenderer::fetch against a real origin for the relabel, the refusal, an undeclared type and a utf-16 page. every fix above is mutation-checked in both directions.

smaller: is_pdf is computed after the relabel so sniffed_pdf and the two || sniffed_pdf disjunctions collapse into one branch; the refusal path gets a log line, since it is the only path that errors without escalating and we would otherwise never see it in production; and the new code is documented in error-codes.md and troubleshooting.md, with pdf-parsing.md corrected, it said parsing needs an application/pdf response and that is no longer true thanks to your sniff.

one thing i should mention plainly: i reworded three characters in your commit message body. the repo has a hard no-em-dash rule and there were three in there; everything else, including authorship, is untouched.

last piece, since merging here reaches production within the hour: i evaluated the guard predicate against the gate dataset itself, firecrawl/scrape-content-dataset-v1, all 1,000 urls fetched live. two of them flip from a 200 to a 422, an .xls and an octet-stream document, and neither carries ground truth, so recall is unchanged and the raw success count moves by at most 2. that is a fair price for not returning PK\x03\x04 under success: true.

cargo test --workspace 4,218 green, clippy and fmt clean. merging once ci finishes. thanks for a genuinely well-diagnosed report, the writeup with the before and after numbers made this much faster to verify.

[us]

merged in ab5e93e (yours) and 4f741d5 (the top-up), rebased onto main, no merge commit.

worth saying since it is not obvious from outside: opencore is the engine and a merge to main reaches production on its own, so this is already on its way to the live api rather than waiting for a release. i will confirm it there.

thanks again. two things about this report that made it easy to act on: you traced the actual branch that made the decision rather than describing the symptom, and you attached a before and after with real byte counts, which meant i could reproduce your exact claim on a baseline build instead of taking it on trust. the octet-stream case in particular is the kind of thing that hides forever, since it returns a confident 200 and megabytes of text that look like content until you read them.

if you are looking for another one: /v1/batch/scrape advances its completed counter for a url whose scrape returned an error, without advancing blocked and without putting a document in the results array, so the caller pays for a url that came back as nothing. the crawl path already solves the identical problem with the failed_page / push_failed_page helpers in crw-crawl/src/crawl.rs, they are just private. the catch is that the job-completion gate keys off completed, so it needs care rather than a straight lift. happy to open an issue with the detail if you would rather have it written up first.

[us]

wrote it up as #487, with the code path, the crawl-side helpers to reuse, and the one part that needs care (the job-completion gate keys off completed, so that counter cannot be changed without checking the gate). all yours if you want it.