fix(durable): reclaim crash-orphaned running executions via staleness sweep - #6300
Merged
Conversation
Follow-up from #6251/PR #6253 (finalize() wiring): an ungraceful process exit (SIGKILL, panic, OOM) never runs a finalize() path, leaving durable_executions rows status='running' invisible to the TTL prune forever if never resumed. Adds INV-16 (reopen un-finalizes all terminal statuses, not just completed/failed) and INV-17 (flock-verified staleness sweep, never staleness alone) plus the Crash-Orphan Sweep subsection: a RetentionPolicy.stale_running_after_secs knob and a sweep pass folded into the existing retention tick that hard-aborts orphaned running rows after a non-blocking INV-15 advisory-lock probe confirms no live owner. Scopes the flock-liveness signal to be universal across production execution kinds by converting the two call sites that previously opened without the exclusive lock (zeph-scheduler's ScheduledJob, zeph-orchestration's DagRun) to open_execution_exclusive. Documents Postgres/:memory:/non-Unix (lock_dir=None) as an explicit non-goal with a hard NEVER against shipping a staleness-only abort there. Design only; implementation tracked as the next step on this branch.
… sweep An ungraceful process exit (SIGKILL, panic, OOM) never runs a finalize() path, leaving durable_executions rows status='running' invisible to the TTL prune (which only ever considers finalized_at IS NOT NULL rows) forever if never resumed. Adds Journal::sweep_orphans, folded into the existing supervised retention tick (no new spawn site), running before prune(). A status='running' row whose updated_at exceeds the new stale_running_after_secs (default 3600s, 0 disables) becomes a candidate; it is hard-aborted only after a non-blocking try-acquire of its INV-15 advisory ExecutionLock succeeds — a live owner short-circuits to skip, since staleness alone never proves the owner is dead. Converts zeph-orchestration's journal_budget (P2) and zeph-scheduler's fire_with_durable (P3) to open_execution_exclusive so their DagRun/ ScheduledJob rows are observable to the sweep; ExecutionLocked degrades to a graceful Ok(()) skip in both, never a task failure. Extends open_execution's reopen path to un-finalize aborted rows too (previously only completed/failed), since the sweep makes aborted the common outcome of a resumable crash and a resumed execution whose row kept finalized_at set was prunable out from under the active resume. zeph durable prune now runs the sweep before the TTL prune; --dry-run reports both counts separately. Postgres/:memory:/non-Unix backends (no advisory-lock dir) are a documented no-op with a warn-once log, never a staleness-only abort. Closes #6254
bug-ops
enabled auto-merge (squash)
July 14, 2026 21:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Journal::sweep_orphans, a crash-orphan staleness sweep folded into the existing supervised durable-retention tick (no new spawn site), running before the TTL prune. Astatus='running'row whoseupdated_atexceeds the new[durable.retention] stale_running_after_secs(default 3600s,0disables) becomes a candidate; it is hard-aborted only after a non-blocking try-acquire of its INV-15 advisoryExecutionLocksucceeds — a live owner short-circuits to a skip, since staleness ofupdated_atalone never proves the owner is dead.zeph-orchestration'sjournal_budgetandzeph-scheduler'sfire_with_durabletoopen_execution_exclusive, so theirDagRun/ScheduledJobrows are observable to the sweep's liveness check;ExecutionLockeddegrades to a gracefulOk(())skip in both adapters, never a task failure.open_execution's reopen path to un-finalizeabortedrows too (previously onlycompleted/failed), since the sweep makesabortedthe common outcome of a resumable crash.zeph durable prunenow runs the sweep before the TTL prune;--dry-runreports both counts separately. Postgres/:memory:/non-Unix backends (no advisory-lock dir) are a documented no-op with a warn-once log — never a staleness-only abort, which would reintroduce false-aborts of live executions.specs/064-durable-execution/spec.md, INV-16/INV-17 + Crash-Orphan Sweep subsection) via architect → critic (2 rounds) → sdd → reviewer, then implementation followed the committed spec.Closes #6254
Process notes
AgentTurnexecutions, would have caused deterministic false-aborts of long-runningDagRun/ScheduledJobexecutions); round 2 approved the fix (converting those two call sites toopen_execution_exclusive).sweep_orphans_in_batchescould hang indefinitely when the count of live-but-stale (lock-held) executions reachedprune_batch_size, because the batch query had no keyset cursor and re-selected the same locked rows forever. Fixed via keyset pagination on(updated_at, execution_id); re-reviewed and approved.Test plan
cargo +nightly fmt --checkcargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warningscargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins(13751 passed)RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler"RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --features desktop,ide,server,chat,pdf,scheduler --lockedcargo test --doc --workspace --features "desktop,ide,server,chat,pdf,scheduler"cargo check -p zeph-durable --no-default-features --features postgres(compile-correctness for the no-op Postgres path)gitleaks protect --staged --no-banner --redactExecutionLocked → Ok(())for both adapters, INV-16 reopen-unifies-aborteddurable prune/durable prune --dry-run.local/testing/playbooks/durable.md) and coverage-status.md updated with 3 new scenarios (main-repo path, not tracked in this diff)