Skip to content

debug_dump: read_request_dump test helper races non-atomic write_private, flaky under parallel load #6327

Description

@bug-ops

Description

Found during code review of PR #6326 (issue #6315, debug dump redaction fix). DebugDumper::write (crates/zeph-core/src/debug_dump/mod.rs) calls zeph_common::fs_secure::write_private, which opens the target file with truncate and then calls write_all — not atomic, unlike the crate's own atomic_write_private sibling (which writes to a .tmp file and renames into place). This predates PR #6326, originating from #6029.

The shared test helper read_request_dump (crates/zeph-core/src/debug_dump/mod.rs) polls std::fs::read_to_string and, on the first successful open, immediately calls serde_json::from_str(&content).unwrap() instead of treating a parse failure as "not ready yet, keep polling." Under parallel test execution, a reader can open the file inside the truncate-but-not-yet-written window, get an empty string, and panic immediately rather than retrying.

PR #6326 added a separate read_dump_file helper with a non-empty-content guard specifically to avoid this race for its own new tests, but left read_request_dump itself unchanged (explicitly out of scope per code review guidance).

Reproduction Steps

  1. Run, several times in a row: cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler,testing" -E 'package(zeph-core) and (test(redact) or test(debug_dump) or test(dump_))'
  2. Observe: roughly 1 in 5-7 runs, a test using read_request_dump (e.g. raw_dump_request_includes_request_metadata, raw_dump_request_redacts_secrets_in_provider_request_messages, json_dump_request_includes_request_metadata, memcot_state_null_when_absent) fails with:
    called `Result::unwrap()` on an `Err` value: Error("EOF while parsing a value", line: 1, column: 0)
    
  3. Isolated reruns of the failing test alone consistently pass — the failure is a parallel-execution race, not a deterministic bug in the redaction/dump logic itself.

Expected Behavior

Either write() uses an atomic write (atomic_write_private, matching the crate's own established pattern for this exact problem), or read_request_dump retries on JSON parse failure instead of panicking on the first successful-but-possibly-empty read (mirroring read_dump_file's non-empty-content guard added in PR #6326).

Actual Behavior

Intermittent test panic under parallel nextest execution, non-deterministic, low frequency (~1 in 5-7 full-filter runs observed).

Environment

Logs / Evidence

Found and independently reproduced twice during PR #6326's code review cycle (once by the reviewer, once by the developer during re-verification) — see .local/handoff/2026-07-16T19-58-50-review.md and .local/handoff/2026-07-16T20-07-05-developer.md in that PR's branch for full detail.

Metadata

Metadata

Assignees

Labels

P3Research — medium-high complexitybugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions