Skip to content

fix: remove wall-clock-timing races from 3 flaky zeph-core tests - #6685

Merged
bug-ops merged 1 commit into
mainfrom
fix/6679-flaky-timing-tests
Jul 28, 2026
Merged

fix: remove wall-clock-timing races from 3 flaky zeph-core tests#6685
bug-ops merged 1 commit into
mainfrom
fix/6679-flaky-timing-tests

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

Three tests inferred concurrency or non-blocking behavior from fixed wall-clock thresholds, which spuriously failed under CI load (confirmed on two consecutive GitHub Actions runs of the same commit on PR #6678):

  • after_tool_hooks_run_concurrently_across_tier_indices (apply_tier_results_tests.rs) — replaced the elapsed-time-vs-N*delay comparison with an atomic max-in-flight high-water mark around the in-process RuntimeLayer::after_tool calls. Serial execution can never exceed 1, so max_in_flight == n is a sound structural proof.
  • pre_tool_use_hooks_fire_concurrently_across_tier_indices (pre_tool_use_concurrency_tests.rs, #[cfg(unix)]) — PreToolUse hooks run as real subprocesses with no shared in-process memory, so this uses a two-phase rendezvous/witness barrier instead: every hook must individually observe all N markers present before creating a witness file, which serial dispatch can never satisfy. This replaced an earlier marker-file-poller draft that still raced a fixed sleep against subprocess-start spread (caught in review — the original CI failure showed ~980ms spread for 4x60ms hooks, exactly the kind of assumption this rewrite removes).
  • dump_request_smoke_returns_promptly_and_write_still_lands -> dump_request_returns_before_blocking_write_completes (debug_dump/mod.rs) — replaced a 50ms wall-clock budget with a test-only TestWriteGate channel handshake (30s bound, so a regression to an inline write path fails with a clear panic instead of hanging the CI shard) that proves dump_request returns before its spawn_blocking write completes, with no timing race window at all.

No production behavior changed — only test assertions/instrumentation, plus a #[cfg(test)]-only field on DebugDumper.

Closes #6679

Test plan

  • Targeted 10x+ repeated loop for the 3 rewritten tests, multiple rounds across the iteration cycle (final count: 0 flakes across 100+ combined runs)
  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings
  • cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins (15154 passed, 0 failed)
  • Rustdoc gate (RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler")
  • Independent adversarial review confirmed the barrier logic is sound (no false-pass path under serial dispatch, no hang path on regression) and the #[cfg(unix)] gating is complete

after_tool_hooks_run_concurrently_across_tier_indices and
pre_tool_use_hooks_fire_concurrently_across_tier_indices inferred hook
concurrency from elapsed time vs N * per-task-delay, which spuriously
failed under CI load. Replaced with structural proofs: an atomic
max-in-flight high-water mark for the in-process hook path, and a
two-phase rendezvous/witness barrier for the subprocess-based hook path
(serial dispatch can never satisfy the barrier, so it fails
deterministically instead of racing a fixed sleep against subprocess-
start spread).

dump_request_smoke_returns_promptly_and_write_still_lands raced a 50ms
wall-clock budget to prove dump_request returns before its
spawn_blocking write completes. Replaced with a test-only TestWriteGate
channel handshake (30s bound, so a regression to an inline write path
fails with a clear panic instead of hanging the CI shard).

Closes #6679
@github-actions github-actions Bot added bug Something isn't working size/L Large PR (201-500 lines) documentation Improvements or additions to documentation rust Rust code changes core zeph-core crate labels Jul 28, 2026
@bug-ops
bug-ops merged commit c9ccb39 into main Jul 28, 2026
43 checks passed
@bug-ops
bug-ops deleted the fix/6679-flaky-timing-tests branch July 28, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working core zeph-core crate documentation Improvements or additions to documentation rust Rust code changes size/L Large PR (201-500 lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 wall-clock timing tests are flaky on CI, blocking the required CI Status check

1 participant