Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,9 +243,18 @@ Safest local verification sequence after non-trivial changes:
This repository dogfoods an empty `commands.test` so the agent-driven targeted path is the default; do not reintroduce `go test -race ./...` as a local Test override.
Process-group reaping on clean/error exit (#357) and Unix WaitDelay remain the lifecycle safety net when agents spawn test workers - restoring the agent-driven path must not revive the daemon OOM leak.
Those agent turns are bounded by `test_agent_timeout` (default 30m, global-only): a stalled evidence or repair agent is cancelled and the run fails instead of waiting forever. Native adapters already honor that deadline through `CommandContext`; the missing piece was the Test step never setting one. Docs owner is `docs/src/content/docs/reference/global-config.md`.
Every other pipeline agent invocation is bounded by `agent_timeout` (default 30m, global-only) at `pipeline.RunAgent` / the executor `timeoutAgent` seam, so a new agent-spawning step cannot hang a run by forgetting a deadline. Review keeps `review_agent_timeout` as a per-round budget; an existing sooner deadline is honored rather than capped. The invocation context is scoped only to `Agent.Run`; a late successful return after the deadline is rejected. Docs owner is `docs/src/content/docs/reference/global-config.md`.
Every other pipeline agent invocation is bounded by `agent_timeout` (default 30m, global-only) at `pipeline.RunAgent` / the executor `timeoutAgent` seam, so a new agent-spawning step cannot hang a run by forgetting a deadline. Review keeps `review_agent_timeout` as a per-turn budget (the fix turn and the rereview turn each get a fresh full budget); an existing sooner deadline is honored rather than capped. The invocation context is scoped only to `Agent.Run`; a late successful return after the deadline is rejected. Docs owner is `docs/src/content/docs/reference/global-config.md`.
Regressions: `TestTestStep_InitialAgent_TargetedValidationContract`, `TestTestStep_FixMode_TargetedVerificationContract`, `TestTestStep_FixMode_DriverFullSuiteInstructionDoesNotOverrideContract`, `TestTestStep_InitialAgent_NoTargetedEvidenceRequiresHonestFinding`, `TestTestStep_HangingEvidenceAgentFailsRunAfterTimeout`, `TestCodexAgent_RunCancelsSilentHang`, `TestDogfoodConfig_NoBroadLocalTestCommand`, `TestCIWorkflow_RetainsFullRaceSuiteAsBroadRegressionOwner`, plus the existing #357 reap/WaitDelay tests, `TestRunAgent_*`, `TestExecutor_DirectAgentRunIsDeadlineBounded`, `TestDocumentStep_HangingAgentFailsRunAfterTimeout`, `TestLintStep_HangingAgentFailsRunAfterTimeout`, `TestCIStep_HangingFixAgentFailsAfterTimeout`, `TestRebaseStep_HangingConflictAgentFailsAfterTimeout`.

**Agent Liveness, Not stdout Bytes (`internal/pipeline/liveness.go`, `internal/agent/pi_liveness.go`)**

- Agent timeouts are SILENCE budgets, not hard wall-clock caps: one monotonic last-activity clock per invocation, cancelled with the whole process tree only after the full configured budget (`agent_timeout` / `review_agent_timeout` / `test_agent_timeout`) passes with no activity. The fix turned the old fixed deadline into this watchdog because pi in `--mode json` buffers stdout when piped, so a healthy fixer was killed as "silent 30m" 23s after its last real session event, and a review rereview inherited the fix turn's spent budget (killed ~10 minutes into its own turn).
- Activity folds three kinds into the one clock (`agent.ActivityKind`): stdout bytes (all native adapters, wired at `startNativeAgentCommand`), native process lifecycle (a start re-arms the clock, which is what makes every new process's budget fresh), and bound pi-session JSONL advancement. Managed-server adapters (opencode, rovodev) report nothing and behave exactly as the legacy fixed deadline.
- The pi session binding is narrow and fail-closed (`startPiSessionWatcher`): `--no-session` turns have no file and stay stdout/lifecycle-only; a resume binds only to the one existing `_<uuid>.jsonl` in the launched cwd's session dir (pi's `--<resolved cwd with / \\ : -> -->--` encoding under `PI_CODING_AGENT_DIR` or `~/.pi/agent`, cwd resolved physical like pi's `process.cwd()`); a fresh durable session binds only when exactly ONE new file appears after launch, and unbinds if the late stdout session header names a different id. Relocated storage (`--session-dir` / `PI_CODING_AGENT_SESSION_DIR`), multiple matches, or an unresolvable dir all mean NO binding - never credit unrelated activity.
- The timeout diagnostic (`AgentTimeoutError`, re-labeled per step) names per-kind last-activity ages ("last activity: pi session events 30m0s ago" / "no stdout, lifecycle, or session-event activity observed") with no content or paths; bind/unbind transitions also land in the step log via lifecycle `liveness` notes.
- Nested seams share one owner: `RunAgent` installs the watchdog and stores it in the ctx; the executor `timeoutAgent` backstop reuses it via `livenessFromContext` instead of stacking a second clock. A parent with an existing hard deadline still wins unchanged (legacy callers).
- Regressions: `internal/pipeline/liveness_test.go`, `internal/agent/pi_liveness_test.go`, `TestRunAgent_PiSessionActivityKeepsBufferedStdoutInvocationAlive`, `TestRunAgent_FrozenPiSessionIsKilledAfterTheFullBudget`, `TestRunAgent_SessionFreePiQuietStdoutIsKilledAtBudget` (`internal/pipeline/agent_run_pi_test.go`), `TestReviewStep_EachAgentTurnGetsItsOwnFreshBudget`.

**Intent Provenance & Conformance (`internal/pipeline/steps/intent_prompt.go`)**

- Intent carries provenance: an explicit `axi run --intent` persists `Source==db.RunIntentSourceAgent` ("agent", score 1); a transcript match persists the agent name ("claude"/"codex"/...). The executor propagates it as `StepContext.IntentSource` alongside `UserIntent` (`executor.go`).
Expand Down
20 changes: 11 additions & 9 deletions docs/src/content/docs/reference/global-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -391,10 +391,12 @@ For older active runs that do not yet have activity rows, AXI falls back to the

### agent_timeout

Maximum wall-clock time for one pipeline agent invocation that does not already have a more specific deadline.
Maximum silence time for one pipeline agent invocation that does not already have a more specific deadline.
This is the default-by-construction budget: Document, Lint, Rebase conflict repair, PR drafting, CI auto-fix, and any future agent-spawning step are bounded even if they forget to install their own timer.
Review still uses [`review_agent_timeout`](#review_agent_timeout) as a per-round budget, Test still uses [`test_agent_timeout`](#test_agent_timeout) per invocation, and Intent keeps its five-minute extraction cap; any existing deadline is honored rather than capped.
When this deadline expires, the agent is cancelled and the invocation returns a timeout diagnostic instead of remaining active indefinitely. Agent-driven mutation steps fail the run, while PR drafting follows its existing agent-error fallback and continues with deterministic content.
The budget is a liveness budget, not a hard wall-clock cap: the invocation is cancelled only after this long with no reported activity. Activity means bytes on the agent's stdout pipe, native process lifecycle events, and - for the Pi adapter - advancement of the exact pi session file bound to that invocation (pi buffers stdout when piped, so a quiet pipe alone does not prove a quiet agent). Each newly launched agent process starts with a fresh full budget.
When no pi session can be bound unambiguously (session-free `--no-session` turns, relocated session storage, or ambiguous files), only stdout and lifecycle evidence govern the invocation, matching the previous behavior.
Review uses [`review_agent_timeout`](#review_agent_timeout) per agent turn, Test uses [`test_agent_timeout`](#test_agent_timeout) per invocation, and Intent keeps its five-minute extraction cap; any existing deadline is honored rather than capped.
When the silence budget expires with no activity from any source, the agent's whole process tree is terminated and the invocation returns a timeout diagnostic naming the last observed activity instead of remaining active indefinitely. Agent-driven mutation steps fail the run, while PR drafting follows its existing agent-error fallback and continues with deterministic content.
A late successful return after the deadline is rejected, so post-agent commits and PR content cannot use work from a timed-out turn.

| | |
Expand All @@ -409,9 +411,9 @@ It is global-only: repository config and environment variables cannot override i

### review_agent_timeout

Maximum wall-clock time for the Review step's agent turns in one review round.
The budget starts at that round's first agent turn and covers its optional review-fix turn plus the rereview turn together; every later auto-fix round starts a fresh budget.
When the deadline expires, the review agent is cancelled and the run fails with a diagnostic naming the timeout instead of remaining active indefinitely.
Maximum silence time for one Review-step agent turn.
The optional review-fix turn and the rereview turn each get their own fresh full budget measured from that turn's own activity: a long fix turn cannot leave the rereview only the remainder. Liveness evidence is the same as [`agent_timeout`](#agent_timeout) (stdout bytes, process lifecycle, bound pi session advancement).
When the silence budget expires with no activity, the review agent's process tree is terminated and the run fails with a diagnostic naming the timeout and the last observed activity instead of remaining active indefinitely.

| | |
| ------- | ---------------------- |
Expand All @@ -424,9 +426,9 @@ Raise it for repositories whose reviews legitimately run long; it bounds only th

### test_agent_timeout

Maximum wall-clock time for one Test-step agent invocation.
The budget covers the post-test evidence-gathering turn, and a Test-repair turn gets its own budget of the same length.
When the deadline expires, the test agent is cancelled and the run fails with a diagnostic naming the timeout instead of remaining active indefinitely.
Maximum silence time for one Test-step agent invocation.
The post-test evidence-gathering turn and a Test-repair turn each get their own budget of this size. Liveness evidence is the same as [`agent_timeout`](#agent_timeout).
When the silence budget expires with no activity, the test agent's process tree is terminated and the run fails with a diagnostic naming the timeout and the last observed activity instead of remaining active indefinitely.

| | |
| ------- | ---------------------- |
Expand Down
2 changes: 1 addition & 1 deletion internal/agent/acpx.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ func (a *acpxAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error)
if err != nil {
return nil, fmt.Errorf("acpx stdin pipe: %w", err)
}
started, err := startNativeAgentCommand(cmd)
started, err := startNativeAgentCommand(cmd, opts.OnActivity)
if err != nil {
_ = stdin.Close()
return nil, fmt.Errorf("acpx start: %w", err)
Expand Down
43 changes: 43 additions & 0 deletions internal/agent/activity.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package agent

// ActivityKind identifies one source of liveness evidence for a running agent
// invocation. The pipeline's invocation watchdog folds every reported kind
// into a single monotonic last-activity clock: an invocation is killed only
// after its full configured silence budget elapses with no activity from any
// source. Kinds are deliberately coarse - they carry no content, no paths,
// and no session payloads - so timeout diagnostics can name the evidence
// class without leaking prompt or session data.
type ActivityKind string

const (
// ActivityStdout reports bytes visible on the agent's stdout pipe. This
// is the wrapper's traditional evidence source; some harnesses (pi in
// JSON mode) buffer stdout when piped, so a quiet pipe does not prove a
// quiet agent.
ActivityStdout ActivityKind = "stdout"
// ActivityLifecycle reports native process lifecycle transitions
// (process start, process exit). A start also resets the silence clock,
// which is what gives every freshly launched process its own full
// budget. Process existence alone is never activity.
ActivityLifecycle ActivityKind = "lifecycle"
// ActivitySession reports advancement of the exact adapter-native
// session file bound to this invocation (pi's session JSONL). It is
// credited only when the binding is unambiguous; otherwise the
// invocation falls back to stdout/lifecycle evidence only.
ActivitySession ActivityKind = "session"
)

// ActivityKindLabel renders a kind for operator-facing diagnostics. The
// labels name evidence classes, never content.
func ActivityKindLabel(kind ActivityKind) string {
switch kind {
case ActivityStdout:
return "stdout bytes"
case ActivityLifecycle:
return "process lifecycle"
case ActivitySession:
return "pi session events"
default:
return string(kind)
}
}
7 changes: 7 additions & 0 deletions internal/agent/agent.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ type RunOpts struct {
JSONSchema json.RawMessage // structured output schema (optional)
OnChunk func(text string) // streaming text callback (optional)
OnLifecycle func(LifecycleEvent) // native agent lifecycle callback (optional)
// OnActivity reports coarse liveness evidence (bytes on the stdout pipe,
// native process lifecycle, bound adapter-native session advancement) to
// the pipeline's per-invocation silence watchdog. It carries kinds only,
// never content. Adapters that report nothing behave exactly as before:
// the watchdog then measures silence from invocation start, matching the
// legacy fixed-deadline behavior.
OnActivity func(ActivityKind)
// Session, when non-nil, asks a session-capable adapter (see
// SessionResumer) to start or resume a durable native session. Adapters
// without session support ignore it and run cold; the caller detects the
Expand Down
2 changes: 1 addition & 1 deletion internal/agent/antigravity.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ func (a *antigravityAgent) runOnce(ctx context.Context, opts RunOpts) (*Result,
cmd.Env = a.gitSafeEnv(opts.CWD)
shellenv.ConfigureShellCommand(cmd)

started, err := startNativeAgentCommand(cmd)
started, err := startNativeAgentCommand(cmd, opts.OnActivity)
if err != nil {
return nil, fmt.Errorf("antigravity start: %w", err)
}
Expand Down
2 changes: 1 addition & 1 deletion internal/agent/claude.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ func (a *claudeAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error

var stderrBuf []byte
var stderrWG sync.WaitGroup
started, err := startNativeAgentCommand(cmd)
started, err := startNativeAgentCommand(cmd, opts.OnActivity)
if err != nil {
return nil, fmt.Errorf("claude start: %w", err)
}
Expand Down
2 changes: 1 addition & 1 deletion internal/agent/codex.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ func (a *codexAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error)

var stderrBuf []byte
var stderrWG sync.WaitGroup
started, err := startNativeAgentCommand(cmd)
started, err := startNativeAgentCommand(cmd, opts.OnActivity)
if err != nil {
return nil, fmt.Errorf("codex start: %w", err)
}
Expand Down
2 changes: 1 addition & 1 deletion internal/agent/copilot.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ func (a *copilotAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, erro

var stderrBuf []byte
var stderrWG sync.WaitGroup
started, err := startNativeAgentCommand(cmd)
started, err := startNativeAgentCommand(cmd, opts.OnActivity)
if err != nil {
return nil, fmt.Errorf("copilot start: %w", err)
}
Expand Down
2 changes: 1 addition & 1 deletion internal/agent/grok.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ func (a *grokAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error)

var stderrBuf []byte
var stderrWG sync.WaitGroup
started, err := startNativeAgentCommand(cmd)
started, err := startNativeAgentCommand(cmd, opts.OnActivity)
if err != nil {
return nil, fmt.Errorf("grok start: %w", err)
}
Expand Down
23 changes: 21 additions & 2 deletions internal/agent/native_command.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,17 @@ type nativeAgentPipe struct {
file *os.File
done func()
doneOnce sync.Once
// onData, when set, is called after every successful read. It feeds the
// invocation silence watchdog: bytes visible on the pipe are liveness
// evidence. Kinds only - the bytes themselves never leave the parse path.
onData func()
}

func (p *nativeAgentPipe) Read(b []byte) (int, error) {
n, err := p.file.Read(b)
if n > 0 && p.onData != nil {
p.onData()
}
if err != nil {
p.markDone()
}
Expand All @@ -58,7 +65,12 @@ func (p *nativeAgentPipe) markDone() {
p.doneOnce.Do(p.done)
}

func startNativeAgentCommand(cmd *exec.Cmd) (*nativeAgentCommand, error) {
// startNativeAgentCommand launches cmd with piped stdout/stderr and reports
// coarse liveness evidence through onActivity: one lifecycle signal at process
// start (which resets the silence clock for the fresh process) and one stdout
// signal per successful pipe read. onActivity may be nil, in which case the
// caller's watchdog keeps its legacy invocation-start-only behavior.
func startNativeAgentCommand(cmd *exec.Cmd, onActivity func(ActivityKind)) (*nativeAgentCommand, error) {
stdoutR, stdoutW, err := os.Pipe()
if err != nil {
return nil, fmt.Errorf("stdout pipe: %w", err)
Expand Down Expand Up @@ -88,8 +100,15 @@ func startNativeAgentCommand(cmd *exec.Cmd) (*nativeAgentCommand, error) {
remainingPipes: 2,
pipesDone: make(chan struct{}),
}
started.stdout = &nativeAgentPipe{file: stdoutR, done: started.markPipeDone}
var onStdoutData func()
if onActivity != nil {
onStdoutData = func() { onActivity(ActivityStdout) }
}
started.stdout = &nativeAgentPipe{file: stdoutR, done: started.markPipeDone, onData: onStdoutData}
started.stderr = &nativeAgentPipe{file: stderrR, done: started.markPipeDone}
if onActivity != nil {
onActivity(ActivityLifecycle)
}
go func() {
err := cmd.Wait()
started.terminate()
Expand Down
Loading
Loading