Skip to content

fix(sanitizer): redact PEM key bodies and no-prefix AWS secrets - #6678

Merged
bug-ops merged 1 commit into
mainfrom
fix/6592-pem-secret-scrubber
Jul 28, 2026
Merged

fix(sanitizer): redact PEM key bodies and no-prefix AWS secrets#6678
bug-ops merged 1 commit into
mainfrom
fix/6592-pem-secret-scrubber

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

scrub_secret_shapes and its consumers missed PEM private-key bodies and raw (no-prefix) AWS secret access keys. No existing pattern spanned a PEM key's multi-line body at all, since the -----BEGIN entry in SECRET_PREFIXES only ever matches a literal single-line token.

  • Added PEM_PRIVATE_KEY_PATTERN (multi-line, bounded, covers RSA/EC/DSA/OPENSSH/ENCRYPTED/PGP-block/RFC4716-SSH2 label variants) and a PEM_PRIVATE_KEY_UNTERMINATED_PATTERN fallback for footerless/truncated headers, to zeph_common::secrets.
  • Wired both into zeph-sanitizer::scrub_secret_shapes, zeph-core::redact::redact_secrets (transitively), and zeph-memory::compression_guidelines::redact_sensitive (previously not PEM-aware at all).
  • Fixed zeph-subagent's live-transcript streaming path (forward.rs): its fixed-size holdback window could split a PEM header/footer across separately-sanitized deltas, letting an unmarked middle slice of key body through. The holdback now widens for an unterminated header and extends past an already-closed block's footer, so a whole PEM span always flushes as one contiguous unit.
  • Added AWS_SECRET_KEY_PATTERN, a context-anchored heuristic that only flags a 40+ char base64-ish run when immediately preceded by an AWS-secret-style marker (aws_secret_access_key, secret_access_key, aws_session_token, etc., tolerant of separator style, so it also matches camelCase JSON keys like SecretAccessKey/SessionToken).

Closes #6592

Review process

This went through 4 rounds of developer/adversarial-critic iteration before code review:

  • Round 1: initial PEM + AWS detectors — critic found the streaming path bypassed detection entirely (256-byte flat holdback split PEM blocks across deltas) and a footerless PEM leaked its whole body.
  • Round 2: fixed both — critic then reproduced a real leak on the most common real-world PEM layout (key immediately followed by a cert in one delta) plus an over-redaction regression in the footerless fallback.
  • Round 3: fixed both via an iterative backward-walk holdback algorithm and a character-class-bounded fallback — critic's rewrite-diff review then caught a reachable panic on multibyte (CJK/emoji) input in the new algorithm.
  • Round 4: one-line alignment fix for the panic, verified against an 863-case multibyte sweep — critic approved.

Code review then caught two additional CI-gate issues none of the four rounds had checked (a broken rustdoc intra-doc link, and 12 gitleaks hits on the new PEM-shaped test fixtures) plus one doc-wording nit, all fixed in a follow-up round and re-verified.

Two minor tradeoffs are accepted and documented in-code rather than fixed, per team discretion during review:

  • A terminated PEM block whose body exceeds the 8,192-byte cap leaks its tail (real keys are well under this cap; the cap exists to bound how much a forged/unterminated block can hide).
  • An unterminated -----BEGIN with no footer yet holds back up to 8,192 bytes in the live transcript with no status indicator (delayed, never dropped).

Test plan

  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings — clean
  • cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins — 15174/15177 passed; the 3 failures are pre-existing wall-clock concurrency-timing tests in files this diff does not touch, reproduced independently under confirmed heavy concurrent build load on this machine (uptime load average ~126)
  • RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler" — clean
  • cargo test --doc --workspace --features "desktop,ide,server,chat,pdf,scheduler" — all passing
  • gitleaks protect --staged — clean (added scoped allowlist entries for the new fake PEM/AWS test fixtures, matched on fixture-body content rather than path, so real secrets in these files would still be caught)
  • New regression tests cover: full PEM block redaction, multiple blocks in one string, mismatched header/footer label over-match, footerless/truncated PEM, chunk-boundary streaming (key+cert bundle and bare trailing -----BEGIN split across deltas), multibyte (CJK) input near a holdback boundary, AWS true/false positives, and prose-survival around a truncated PEM mention

@github-actions github-actions Bot added bug Something isn't working size/XL Extra large PR (500+ lines) documentation Improvements or additions to documentation memory zeph-memory crate (SQLite) rust Rust code changes core zeph-core crate labels Jul 28, 2026
@bug-ops
bug-ops enabled auto-merge (squash) July 28, 2026 02:09
@bug-ops
bug-ops force-pushed the fix/6592-pem-secret-scrubber branch from 0b60e3f to 15cd605 Compare July 28, 2026 02:09
@bug-ops
bug-ops force-pushed the fix/6592-pem-secret-scrubber branch from 15cd605 to 6dc6da1 Compare July 28, 2026 11:36
scrub_secret_shapes and its consumers missed PEM private-key bodies and
raw AWS secret access keys: no existing pattern spanned a PEM key's
multi-line body, since the -----BEGIN prefix entry only ever matched a
single-line token. Add bounded PEM (terminated + unterminated-fallback)
and context-anchored AWS-secret detectors to zeph-common, wire them into
zeph-sanitizer, zeph-core's transitive redact_secrets, and zeph-memory's
previously non-PEM-aware compression_guidelines.

The subagent live-transcript streaming path needed a dedicated fix: its
fixed-size holdback window could split a PEM header and footer across
separately-sanitized deltas, letting an unmarked middle slice of key
body through. The holdback now widens for an unterminated header and
extends past an already-closed block's footer so the whole span always
flushes as one unit.

Closes #6592
@bug-ops
bug-ops force-pushed the fix/6592-pem-secret-scrubber branch from 6dc6da1 to d391bd1 Compare July 28, 2026 12:21
@bug-ops
bug-ops merged commit 3e75865 into main Jul 28, 2026
43 checks passed
@bug-ops
bug-ops deleted the fix/6592-pem-secret-scrubber branch July 28, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working core zeph-core crate documentation Improvements or additions to documentation memory zeph-memory crate (SQLite) rust Rust code changes size/XL Extra large PR (500+ lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zeph-sanitizer secret-shape scrubber misses PEM key bodies and no-prefix secret formats

1 participant