Skip to content

orchestration: idle_timeout progress-signal plumbing (Alt A) — eviction-safe coalescing timestamp #6245

Description

@bug-ops

Description

Deferred from specs/075-orchestration-node-control-parity/ (PR #6242, source issue #6021). v1 ships idle_timeout_secs as a defined, config-surfaced, but documented no-op — no progress-signal plumbing exists anywhere in the codebase today (confirmed: zero heartbeat/liveness/progress signals across zeph-subagent and zeph-orchestration). This issue tracks wiring the real idle-detection mechanism.

Why this is deferred, not implemented directly

The obvious approach — route a per-task progress signal through the existing completion-event channel (mpsc::channel(64) in crates/zeph-orchestration/src/scheduler/mod.rs) — is unsafe: wait_event() already drops buffered events under saturation (logged as "completion event dropped — task may remain Running until timeout"). Multiplexing high-frequency progress signals (one per agent-loop iteration or tool-call) onto that same channel risks evicting a real terminal completion, hanging a healthy task until run_timeout. This eviction hazard was judged to exceed a P3's risk budget for v1.

Target design (already spec'd, not yet implemented)

  • Preferred mechanism: a coalescing per-task progress timestamp, NOT a queued message. Each running task holds a shared Arc<AtomicU64> (or a watch::channel<Instant>) that the executor writes on each progress boundary; the scheduler reads it in check_timeouts() as last_progress_at. A timestamp write coalesces (latest wins) and cannot evict anything — O(1) memory, no queue, lock-free with no torn reads on tier-1 platform targets.
  • If a channel is used instead, it MUST be separate from the completion channel, and progress MUST be consumed inline at recv (update last_progress_at, never pushed to any buffered-events structure).
  • Bounded emission granularity must be defined (e.g. one signal per agent-loop iteration or per tool-call completion — coarse enough not to spin).
  • RunInline tasks cannot participate in this model as-is (the tick loop is blocked during inline execution) — idle_timeout should remain a documented no-op for RunInline specifically, or (as an extension) the inline loop could self-check idle between its own iterations.
  • Document precisely what emits "progress" and warn in config docs / --init wizard help text that idle_timeout must be set above the longest expected single-operation duration (a legitimate long tool call — build, large download — crosses no progress boundary and would otherwise trip idle detection on a healthy task).

Environment

  • Source: specs/075-orchestration-node-control-parity/srs.md §9 (FR-D-02), spec.md
  • Design-review handoff: .local/handoff/2026-07-13T21-05-40-critic.md (S3 — channel-eviction risk finding)

Priority

P3 — no demonstrated urgent need; run_timeout alone covers the majority of the LangGraph TimeoutPolicy parity value.

Metadata

Metadata

Assignees

Labels

P3Research — medium-high complexityorchestrationTask orchestration / DAG schedulingresearchResearch-driven improvement

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions