Skip to content

fix(durable): reclaim crash-orphaned running executions via staleness sweep - #6300

Merged
bug-ops merged 2 commits into
mainfrom
fix/6254-durable-crash-orphan-sweep
Jul 14, 2026
Merged

fix(durable): reclaim crash-orphaned running executions via staleness sweep#6300
bug-ops merged 2 commits into
mainfrom
fix/6254-durable-crash-orphan-sweep

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds Journal::sweep_orphans, a crash-orphan staleness sweep folded into the existing supervised durable-retention tick (no new spawn site), running before the TTL prune. A status='running' row whose updated_at exceeds the new [durable.retention] stale_running_after_secs (default 3600s, 0 disables) becomes a candidate; it is hard-aborted only after a non-blocking try-acquire of its INV-15 advisory ExecutionLock succeeds — a live owner short-circuits to a skip, since staleness of updated_at alone never proves the owner is dead.
  • Converts zeph-orchestration's journal_budget and zeph-scheduler's fire_with_durable to open_execution_exclusive, so their DagRun/ScheduledJob rows are observable to the sweep's liveness check; ExecutionLocked degrades to a graceful Ok(()) skip in both adapters, never a task failure.
  • Extends open_execution's reopen path to un-finalize aborted rows too (previously only completed/failed), since the sweep makes aborted the common outcome of a resumable crash.
  • zeph durable prune now runs the sweep before the TTL prune; --dry-run reports both counts separately. Postgres/:memory:/non-Unix backends (no advisory-lock dir) are a documented no-op with a warn-once log — never a staleness-only abort, which would reintroduce false-aborts of live executions.
  • Spec updated first (specs/064-durable-execution/spec.md, INV-16/INV-17 + Crash-Orphan Sweep subsection) via architect → critic (2 rounds) → sdd → reviewer, then implementation followed the committed spec.

Closes #6254

Process notes

  • Design went through 2 rounds of architect/critic iteration before the spec was written: round 1 found a critical flaw (the flock liveness signal only covered AgentTurn executions, would have caused deterministic false-aborts of long-running DagRun/ScheduledJob executions); round 2 approved the fix (converting those two call sites to open_execution_exclusive).
  • Implementation went through a critical fix cycle: impl-critic found sweep_orphans_in_batches could hang indefinitely when the count of live-but-stale (lock-held) executions reached prune_batch_size, because the batch query had no keyset cursor and re-selected the same locked rows forever. Fixed via keyset pagination on (updated_at, execution_id); re-reviewed and approved.

Test plan

  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings
  • cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins (13751 passed)
  • RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler"
  • RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --features desktop,ide,server,chat,pdf,scheduler --locked
  • cargo test --doc --workspace --features "desktop,ide,server,chat,pdf,scheduler"
  • cargo check -p zeph-durable --no-default-features --features postgres (compile-correctness for the no-op Postgres path)
  • gitleaks protect --staged --no-banner --redact
  • Real-DB/real-flock regression tests: sweep termination under all-lock-held candidates exceeding batch size, batch-boundary straddling, concurrent sweep-vs-reopen race, ExecutionLocked → Ok(()) for both adapters, INV-16 reopen-unifies-aborted
  • CLI subprocess integration tests for durable prune/durable prune --dry-run
  • Testing playbook (.local/testing/playbooks/durable.md) and coverage-status.md updated with 3 new scenarios (main-repo path, not tracked in this diff)
  • Live kill -9 mid-turn session test — not run in this environment (documented as a coverage gap, not silently skipped); real-DB/real-flock unit tests cover the same mechanism at the function level

bug-ops added 2 commits July 14, 2026 23:22
Follow-up from #6251/PR #6253 (finalize() wiring): an ungraceful process
exit (SIGKILL, panic, OOM) never runs a finalize() path, leaving
durable_executions rows status='running' invisible to the TTL prune
forever if never resumed.

Adds INV-16 (reopen un-finalizes all terminal statuses, not just
completed/failed) and INV-17 (flock-verified staleness sweep, never
staleness alone) plus the Crash-Orphan Sweep subsection: a
RetentionPolicy.stale_running_after_secs knob and a sweep pass folded
into the existing retention tick that hard-aborts orphaned running rows
after a non-blocking INV-15 advisory-lock probe confirms no live owner.

Scopes the flock-liveness signal to be universal across production
execution kinds by converting the two call sites that previously opened
without the exclusive lock (zeph-scheduler's ScheduledJob,
zeph-orchestration's DagRun) to open_execution_exclusive. Documents
Postgres/:memory:/non-Unix (lock_dir=None) as an explicit non-goal with
a hard NEVER against shipping a staleness-only abort there.

Design only; implementation tracked as the next step on this branch.
… sweep

An ungraceful process exit (SIGKILL, panic, OOM) never runs a finalize()
path, leaving durable_executions rows status='running' invisible to the
TTL prune (which only ever considers finalized_at IS NOT NULL rows)
forever if never resumed.

Adds Journal::sweep_orphans, folded into the existing supervised
retention tick (no new spawn site), running before prune(). A
status='running' row whose updated_at exceeds the new
stale_running_after_secs (default 3600s, 0 disables) becomes a
candidate; it is hard-aborted only after a non-blocking try-acquire of
its INV-15 advisory ExecutionLock succeeds — a live owner short-circuits
to skip, since staleness alone never proves the owner is dead.

Converts zeph-orchestration's journal_budget (P2) and zeph-scheduler's
fire_with_durable (P3) to open_execution_exclusive so their DagRun/
ScheduledJob rows are observable to the sweep; ExecutionLocked degrades
to a graceful Ok(()) skip in both, never a task failure.

Extends open_execution's reopen path to un-finalize aborted rows too
(previously only completed/failed), since the sweep makes aborted the
common outcome of a resumable crash and a resumed execution whose row
kept finalized_at set was prunable out from under the active resume.

zeph durable prune now runs the sweep before the TTL prune; --dry-run
reports both counts separately. Postgres/:memory:/non-Unix backends
(no advisory-lock dir) are a documented no-op with a warn-once log,
never a staleness-only abort.

Closes #6254
@github-actions github-actions Bot added documentation Improvements or additions to documentation rust Rust code changes tests Test-related changes config Configuration file changes bug Something isn't working size/XL Extra large PR (500+ lines) labels Jul 14, 2026
@bug-ops
bug-ops enabled auto-merge (squash) July 14, 2026 21:40
@bug-ops
bug-ops merged commit d9b9ff4 into main Jul 14, 2026
43 checks passed
@bug-ops
bug-ops deleted the fix/6254-durable-crash-orphan-sweep branch July 14, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working config Configuration file changes documentation Improvements or additions to documentation rust Rust code changes size/XL Extra large PR (500+ lines) tests Test-related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

durable execution: crash-orphaned running executions have no reclamation path (staleness sweep needed)

1 participant