Skip to content

fix(tracing): flush local trace on SIGTERM and fix span-duration fragmentation - #6692

Merged
bug-ops merged 1 commit into
mainfrom
fix/6683-local-trace-truncation
Jul 28, 2026
Merged

fix(tracing): flush local trace on SIGTERM and fix span-duration fragmentation#6692
bug-ops merged 1 commit into
mainfrom
fix/6683-local-trace-truncation

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

  • Local trace files written by the local telemetry backend were left as unterminated JSON when the process was killed via pkill's default SIGTERM, since foreground CLI/TUI mode had no SIGTERM handling and the Chrome trace FlushGuard never dropped. init_tracing now shares the guard between the normal drop path and a new SIGTERM listener via a take-once cell, installed only on invocation paths that don't already own SIGTERM (--daemon, serve-sessions, scheduler serve each have their own graceful shutdown).
  • Switches build_chrome_layer to TraceStyle::Async, since the previous Threaded style fired on_enter/on_exit on every poll of an async span, fragmenting any span with an internal .await into hundreds of short on-CPU slices instead of one continuous wall-clock duration.
  • Reworked the jq trace-analysis recipes in .claude/rules/continuous-improvement.md to pair Async-style b/e events by id with a LIFO stack (that file is untracked/gitignored, so this change does not appear in the diff).

Closes #6683
Closes #6682

Test plan

  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings
  • cargo clippy --profile ci --workspace --all-targets --features full -- -D warnings
  • cargo nextest run --config-file .github/nextest.toml --features full -E 'binary(zeph) and test(tracing_init)' (19/19 passed, 4 new/rewritten tests including a genuine concurrent-take race test and first coverage for TraceStyle::Async)
  • Rustdoc gate on both CI-curated and full feature sets
  • gitleaks protect --staged
  • Live-tested: ran zeph in CLI foreground mode with telemetry.backend = "local", killed with pkill -TERM -f "target/.*zeph" (the issue's exact repro) — resulting trace file parses cleanly with jq '.', no truncation
  • Live-tested: confirmed a span with an internal .await no longer fragments into hundreds of on-CPU slices under TraceStyle::Async

@github-actions github-actions Bot added documentation Improvements or additions to documentation rust Rust code changes dependencies Dependency updates bug Something isn't working size/L Large PR (201-500 lines) labels Jul 28, 2026
@bug-ops
bug-ops enabled auto-merge (squash) July 28, 2026 13:05
…mentation

Local trace files written by the local telemetry backend were left as
unterminated JSON when the process was killed via pkill's default SIGTERM,
since foreground CLI/TUI mode had no SIGTERM handling and the Chrome trace
FlushGuard never dropped. init_tracing now shares the guard between the
normal drop path and a new SIGTERM listener via a take-once cell, installed
only on invocation paths that don't already own SIGTERM (--daemon,
serve-sessions, scheduler serve each have their own graceful shutdown).

Also switches build_chrome_layer to TraceStyle::Async, since the previous
Threaded style fired on_enter/on_exit on every poll of an async span,
fragmenting any span with an internal await into hundreds of short on-CPU
slices instead of one continuous wall-clock duration.

Closes #6683
Closes #6682
@bug-ops
bug-ops force-pushed the fix/6683-local-trace-truncation branch from 307ec50 to 6b9aec3 Compare July 28, 2026 13:06
@bug-ops
bug-ops merged commit 4b67c7e into main Jul 28, 2026
43 checks passed
@bug-ops
bug-ops deleted the fix/6683-local-trace-truncation branch July 28, 2026 13:17
bug-ops added a commit that referenced this pull request Jul 28, 2026
log_guard (the rolling-file WorkerGuard) relied on Drop to flush
buffered log writes, the same as chrome_guard before #6692 fixed it.
A SIGTERM without a handler skips Drop entirely, discarding whatever
lines were still in the async writer's channel (reproduced 6/6 with a
tight self-kill race, losing 127-939 lines per run).

Generalizes the take-once-cell + SIGTERM-listener pattern from #6692
into an unconditional LogGuardCell, since file logging (unlike Chrome
tracing) is not gated behind the profiling feature. Both
TracingGuards::drop and spawn_sigterm_flush_task now take-and-flush
whichever cell(s) are still Some. Extracts should_install_sigterm_flush_task
as a pure, unit-tested gate so a mis-wire of owns_sigterm_elsewhere no
longer silently regresses in default builds.

Closes #6693
bug-ops added a commit that referenced this pull request Jul 28, 2026
log_guard (the rolling-file WorkerGuard) relied on Drop to flush
buffered log writes, the same as chrome_guard before #6692 fixed it.
A SIGTERM without a handler skips Drop entirely, discarding whatever
lines were still in the async writer's channel (reproduced 6/6 with a
tight self-kill race, losing 127-939 lines per run).

Generalizes the take-once-cell + SIGTERM-listener pattern from #6692
into an unconditional LogGuardCell, since file logging (unlike Chrome
tracing) is not gated behind the profiling feature. Both
TracingGuards::drop and spawn_sigterm_flush_task now take-and-flush
whichever cell(s) are still Some. Extracts should_install_sigterm_flush_task
as a pure, unit-tested gate so a mis-wire of owns_sigterm_elsewhere no
longer silently regresses in default builds.

Closes #6693
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dependencies Dependency updates documentation Improvements or additions to documentation rust Rust code changes size/L Large PR (201-500 lines)

Projects

None yet

1 participant