Skip to content

durable execution: DurableRetentionService background prune sweep is never spawned in production #6264

Description

@bug-ops

Description

zeph-durable::retention::RetentionService::run() (crates/zeph-durable/src/retention.rs) implements
the periodic background sweep documented in specs/064-durable-execution/spec.md ("Retention &
Compaction"): every prune_interval_secs, it calls Journal::prune(policy) to delete finalized
executions past ttl_completed_secs/ttl_failed_secs.

This service is never instantiated or spawned anywhere in production. Grepping every reference to
RetentionService/DurableRetention outside its defining crate:

crates/zeph-durable/README.md
crates/zeph-durable/src/backend.rs   (backend trait method used by the *manual* CLI path only)
crates/zeph-durable/src/retention.rs (definition + doctest showing `tokio::spawn(service.run())`)
crates/zeph-durable/src/lib.rs       (re-export)

Zero call sites in zeph-core, zeph-orchestration, zeph-scheduler, or src/ (daemon.rs,
runner.rs, acp.rs, cli.rs) actually spawn service.run(). Only the manual zeph durable prune
CLI subcommand (src/commands/durable.rs:330-339) reaches backend.prune(policy) — a one-shot,
operator-triggered call, unrelated to the periodic service.

This is a distinct residual gap from #6251/#6254: PR #6253 fixed finalize(Completed/Failed) wiring
so executions correctly transition to a terminal status, and #6254 tracks crash-orphaned running
rows that never reach a terminal status at all. This issue is about the third leg: even executions
that correctly reach finalized_at IS NOT NULL are never automatically reclaimed
, because the
periodic sweep that would query and delete them is dead code. #6254's own reproduction steps say
"run zeph durable prune (or the background sweep)" as if the background sweep already runs — it
does not.

Reproduction Steps

  1. Config: [durable.retention] ttl_completed_secs = 3, prune_interval_secs = 5 (short TTL to
    observe within a live session), durable.enabled = true, agent_turns = true.
  2. Run a single agent turn to completion (finalizes Completed per PR fix(durable): wire finalize(Completed/Failed) into production execution paths #6253's shutdown/detach
    wiring — confirmed working live in CI-1377).
  3. Keep the session alive >20s past TTL + interval, with RUST_LOG covering zeph_durable::retention=debug.
  4. Observe: no supervised_task span for any retention service, no "durable retention prune sweep
    completed" debug log line (the only log line the service ever emits, retention.rs:233) — the
    sweep never ran.
  5. Confirmed statically: no call site outside crates/zeph-durable/ ever calls RetentionService::run()
    or spawns it via TaskSupervisor.

Expected Behavior

Per spec.md's "Retention & Compaction" section, terminal executions past their TTL should be
automatically reclaimed by a periodic background sweep, without requiring an operator to manually
run zeph durable prune.

Actual Behavior

durable_executions (and their journal rows) accumulate unboundedly regardless of
ttl_completed_secs/ttl_failed_secs config, unless an operator manually invokes zeph durable prune.
The config knobs (prune_interval_secs, prune_batch_size) that only make sense for a periodic
service are entirely inert.

Environment

Logs / Evidence

  • crates/zeph-durable/src/retention.rs:209-236RetentionService::run(), the dead sweep loop
  • crates/zeph-durable/src/retention.rs:195-203 — doctest itself uses tokio::spawn(service.run()),
    i.e. even the crate's own example never claims a production caller does this
  • src/commands/durable.rs:330-339 — the only production caller of backend.prune(policy), manual
    CLI only
  • Live reproduction this cycle (CI-1380): .local/testing/config/durable-ttl-test.toml,
    .local/testing/data/ci1380-durable/ — dedicated scratch DB, no shared-state contamination

Metadata

Metadata

Assignees

Labels

P1High ROI, low complexity — do next sprintbugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions