Skip to content

Local trace files truncated when zeph is killed via pkill (FlushGuard never dropped) #6683

Description

@bug-ops

Description

Follow-up from #6676. Local trace files written by the local telemetry backend (tracing-chrome 0.7.2) are left as unterminated JSON — missing the closing ] — when the process is killed via pkill -f "target/.*zeph" (the documented teardown method, see .claude/rules/continuous-improvement.md "Kill Zeph after live test"). tracing-chrome only writes the trailing ] when its FlushGuard is dropped during a normal/graceful shutdown; a SIGKILL/SIGTERM-without-handler from pkill skips that drop entirely.

This is not rare: 2 of 7 sampled trace files in .local/traces/ (the two largest — 392MB and 404MB, i.e. the two longest live-testing sessions) were found in this truncated state during #6676's verification.

Impact

Any truncated trace file fails to parse with any JSON tool (jq reports Unfinished JSON term at EOF, no partial output) unless manually repaired. Since the truncation correlates with the longest sessions, this systematically loses trace data for exactly the sessions most likely to contain interesting bottleneck data. A jq-side recovery command now exists (documented in .claude/rules/continuous-improvement.md after #6676) but it's a manual workaround, not a fix — every long-running local-tracing session risks losing its trace file unless the operator remembers to shut the process down gracefully.

Reproduction Steps

  1. Run a live session with telemetry.backend = "local" for long enough to write a sizeable trace file.
  2. Kill the process with pkill -f "target/.*zeph" (as documented for live-testing teardown).
  3. Attempt to parse the resulting .local/traces/<file>.json with jq '.'.
  4. Observe: jq: error: Unfinished JSON term at EOF.

Expected Behavior

Either the trace file should be valid JSON regardless of how the process is terminated, or the documented teardown procedure should ensure a graceful shutdown that flushes the trace writer.

Suggested Fix

Options to evaluate (not mutually exclusive):

  1. Install a signal handler (SIGTERM/SIGINT) in the telemetry init path that explicitly drops the FlushGuard before process exit.
  2. Periodically flush/checkpoint the trace writer during a long session rather than only at clean shutdown, so a kill loses at most the tail rather than the whole file.
  3. Update the documented teardown procedure (.claude/rules/continuous-improvement.md "Kill Zeph after live test") to prefer a graceful signal (SIGTERM with a handler, or a clean CLI exit path) over pkill's default SIGTERM-without-handler-guarantee, if that alone would fix it.

Environment

  • Version: unreleased (main, as of 2026-07-28)
  • Features: telemetry local backend (tracing-chrome 0.7.2)

Logs / Evidence

Found during verification of #6676 (local trace jq recipe fix) — recovery one-liner documented there but the root cause (no graceful flush on kill) is unaddressed.

Metadata

Metadata

Assignees

Labels

P3Research — medium-high complexitybugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions