Skip to content

fix(agents): recover empty final responses once - #198

Open
jasonqlwilliams-alt wants to merge 13 commits into
kunchenguid:mainfrom
jasonqlwilliams-alt:fm/gnhf-empty-response-recovery-o1
Open

fix(agents): recover empty final responses once#198
jasonqlwilliams-alt wants to merge 13 commits into
kunchenguid:mainfrom
jasonqlwilliams-alt:fm/gnhf-empty-response-recovery-o1

Conversation

@jasonqlwilliams-alt

@jasonqlwilliams-alt jasonqlwilliams-alt commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Intent

Resolve issue #163 with the preserved committed implementation. When an agent turn otherwise completes but its final response is empty, send exactly one continuation nudge before recording the iteration as failed. If that one retry is also empty, fail normally; never loop or add another retry. Make the continuation nudge visible in the run log. Exclude GitHub Copilot from recovery because its resume contract is unverified. Exclude Pi and document the exclusion because Pi has no recoverable context under --no-session. Do not broaden session behavior, add new native adapters, or change unrelated recovery contracts. Revalidate and deliver the committed work through the existing standard no-mistakes contribution path and PR #198; maintainers alone may merge.

What Changed

  • Recover completed empty final responses exactly once for Claude, Codex, OpenCode, Rovo Dev, and ACP by continuing the same session, logging the nudge, and accumulating usage across both turns.
  • Preserve normal failure behavior when recovery is unsafe or unsuccessful, including incomplete turns, unavailable sessions, a second empty response, Copilot, and Pi.
  • Document the recovery contract and add unit and end-to-end coverage for retries, exclusions, aborts, usage, and log handling.

Risk Assessment

🚨 High: The main recovery flow is well bounded, but a supported Claude configuration still violates the authoritative exact-session and no-session-broadening requirements, requiring explicit approval or correction before merge.

Testing

After correcting preliminary test-only expectations to the actual persisted output contracts, the focused adapter tests, production build, normal CLI path, successful recovery, and exhausted single-retry path all passed. Manual CLI evidence confirms one visible same-session nudge, no third attempt, correct success/failure recording, and expected git outcomes; this non-UI change is demonstrated with transcripts and persisted logs rather than screenshots.

Evidence: Recovered CLI run

Successful run: 1 good iteration and 1 commit.

╭────────────────────────────────────────────────────────────╮
│ × gnhf stopped                                             │
│   opencode ran for 3s before: max iterations reached (1)   │
╰────────────────────────────────────────────────────────────╯

  iterations      1 total       1 good       0 failed
  tokens          13 in         5 out        
  branch diff     1 commit      +1           -0
  files           0 added       1 updated    0 deleted

  notes           /home/jason/.no-mistakes/worktrees/c0b016bcb19b/01KZTFDNYMQ72NTA7BVDZ8KSYX/.evidence-run/repo/.gnhf/runs/recover-the-empty-re-dbf6e0/notes.md
  debug log       /home/jason/.no-mistakes/worktrees/c0b016bcb19b/01KZTFDNYMQ72NTA7BVDZ8KSYX/.evidence-run/repo/.gnhf/runs/recover-the-empty-re-dbf6e0/gnhf.log

  next steps      git log --oneline f03782b584be..HEAD
                  git diff --stat f03782b584be..HEAD
                  gh pr create

  too much        git push no-mistakes:
  to review?      https://github.com/kunchenguid/no-mistakes
Evidence: Successful agent turns

Exactly two turns used session-1; the second contained only the continuation nudge.

{"event":"message:start","sessionId":"session-1","prompt":"You are working autonomously towards an objective given below.\nThis is iteration 1. Each iteration aims to make an incremental step forward, not to complete the entire objective.\n\n## Instructions\n\n1. Read .gnhf/runs/recover-the-empty-re-dbf6e0/notes.md first to understand what has been done in previous iterations. Do NOT write to or modify notes.md - it is maintained automatically by the gnhf orchestrator\n2. Identify the next smallest logical unit of work that's individually verifiable and would make incremental progress towards the objective, and treat that as the scope of this iteration\n3. If you attempted a solution and it didn't end up moving the needle on the objective, document learnings and record success=false, then conclude the iteration rather than continuously pivoting\n4. If you made code changes, run build/tests/linters/formatters if available to validate your work. Do NOT make any git commits - that will be handled automatically by the gnhf orchestrator\n5. If you started any long-running background processes (dev servers, browsers, watchers, Electron, etc.), stop them before finishing the iteration\n6. Only submit the final JSON object after the result is final: your work is complete, validation is done, and you have stopped any background processes you started\n\n## Output\n\n- success: whether you were able to make a meaningful contribution that got us closer towards the objective. setting this to false means any code change you made should be discarded. A complete no-op iteration (no file changes AND no new meaningful learnings worth recording) is not a success - set success=false so the run can halt rather than spin on no-op iterations\n- summary: a concise one-sentence summary of the accomplishment in this iteration\n- key_changes_made: an array of descriptions for key changes you made. don't group this by file - group by logical units of work. don't describe activities - describe material outcomes\n- key_learnings: an array of new learnings that were surprising, weren't captured by previous notes and would be informative for future iterations\n\n## Objective\n\nrecover the empty response\n\nWhen you finish, reply with only valid JSON.\nDo not wrap the JSON in markdown fences.\nDo not include any prose before or after the JSON.\nThe JSON must match this schema exactly: {\"type\":\"object\",\"additionalProperties\":false,\"properties\":{\"success\":{\"type\":\"boolean\"},\"summary\":{\"type\":\"string\"},\"key_changes_made\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"key_learnings\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}},\"required\":[\"success\",\"summary\",\"key_changes_made\",\"key_learnings\"]}"}
{"event":"message:start","sessionId":"session-1","prompt":"You did not produce a final answer. Continue and provide your final summary now."}
Evidence: Successful recovery events

One continuation event followed by a successful iteration.

{"timestamp":"2026-08-12T08:24:45.571Z","pid":17139,"event":"opencode:output:continuation","sessionId":"session-1","attempt":1,"prompt":"You did not produce a final answer. Continue and provide your final summary now."}
{"timestamp":"2026-08-12T08:24:47.256Z","pid":17139,"event":"iteration:end","iteration":1,"elapsedMs":2011,"success":true,"summary":"mocked objective complete","keyChanges":1,"keyLearnings":1,"consecutiveFailures":0,"totalInputTokens":13,"totalOutputTokens":5,"tokensEstimated":false,"commitCount":1}
Evidence: Exhausted CLI run

Exhausted run: 1 failed iteration and 0 commits.

╭─────────────────────────────────────────────────────────────────╮
│ × gnhf stopped                                                  │
│   opencode ran for 1s before: OpenCode produced no final answer │
╰─────────────────────────────────────────────────────────────────╯

  iterations      1 total       0 good       1 failed
  tokens          6 in          0 out        
  branch diff     0 commits     +0           -0
  files           0 added       0 updated    0 deleted

  notes           /home/jason/.no-mistakes/worktrees/c0b016bcb19b/01KZTFDNYMQ72NTA7BVDZ8KSYX/.evidence-run-failure/repo/.gnhf/runs/stop-after-one-empty-02c13d/notes.md
  debug log       /home/jason/.no-mistakes/worktrees/c0b016bcb19b/01KZTFDNYMQ72NTA7BVDZ8KSYX/.evidence-run-failure/repo/.gnhf/runs/stop-after-one-empty-02c13d/gnhf.log

  next steps      git log --oneline 8b0bff497560..HEAD
                  git diff --stat 8b0bff497560..HEAD
                  gh pr create

  too much        git push no-mistakes:
  to review?      https://github.com/kunchenguid/no-mistakes
Evidence: Exhausted agent turns

Exactly two empty turns used session-1, proving no third retry occurred.

{"event":"message:start","sessionId":"session-1","prompt":"You are working autonomously towards an objective given below.\nThis is iteration 1. Each iteration aims to make an incremental step forward, not to complete the entire objective.\n\n## Instructions\n\n1. Read .gnhf/runs/stop-after-one-empty-02c13d/notes.md first to understand what has been done in previous iterations. Do NOT write to or modify notes.md - it is maintained automatically by the gnhf orchestrator\n2. Identify the next smallest logical unit of work that's individually verifiable and would make incremental progress towards the objective, and treat that as the scope of this iteration\n3. If you attempted a solution and it didn't end up moving the needle on the objective, document learnings and record success=false, then conclude the iteration rather than continuously pivoting\n4. If you made code changes, run build/tests/linters/formatters if available to validate your work. Do NOT make any git commits - that will be handled automatically by the gnhf orchestrator\n5. If you started any long-running background processes (dev servers, browsers, watchers, Electron, etc.), stop them before finishing the iteration\n6. Only submit the final JSON object after the result is final: your work is complete, validation is done, and you have stopped any background processes you started\n\n## Output\n\n- success: whether you were able to make a meaningful contribution that got us closer towards the objective. setting this to false means any code change you made should be discarded. A complete no-op iteration (no file changes AND no new meaningful learnings worth recording) is not a success - set success=false so the run can halt rather than spin on no-op iterations\n- summary: a concise one-sentence summary of the accomplishment in this iteration\n- key_changes_made: an array of descriptions for key changes you made. don't group this by file - group by logical units of work. don't describe activities - describe material outcomes\n- key_learnings: an array of new learnings that were surprising, weren't captured by previous notes and would be informative for future iterations\n\n## Objective\n\nstop after one empty-response retry\n\nWhen you finish, reply with only valid JSON.\nDo not wrap the JSON in markdown fences.\nDo not include any prose before or after the JSON.\nThe JSON must match this schema exactly: {\"type\":\"object\",\"additionalProperties\":false,\"properties\":{\"success\":{\"type\":\"boolean\"},\"summary\":{\"type\":\"string\"},\"key_changes_made\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"key_learnings\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}},\"required\":[\"success\",\"summary\",\"key_changes_made\",\"key_learnings\"]}"}
{"event":"message:start","sessionId":"session-1","prompt":"You did not produce a final answer. Continue and provide your final summary now."}
Evidence: Exhausted recovery events

One continuation event followed by the normal empty-response failure.

{"timestamp":"2026-08-12T08:31:48.543Z","pid":8863,"event":"opencode:output:continuation","sessionId":"session-1","attempt":1,"prompt":"You did not produce a final answer. Continue and provide your final summary now."}
{"timestamp":"2026-08-12T08:31:48.550Z","event":"agent:run:error","iteration":1,"error":{"name":"EmptyAgentResponseError","message":"OpenCode produced no final answer"}}
{"timestamp":"2026-08-12T08:31:48.560Z","pid":8863,"event":"iteration:end","iteration":1,"elapsedMs":336,"success":false,"summary":"OpenCode produced no final answer","keyChanges":0,"keyLearnings":0,"consecutiveFailures":1,"totalInputTokens":6,"totalOutputTokens":0,"tokensEstimated":false,"commitCount":0}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 3 issues (1 error, 2 warnings)
  • 🚨 src/core/agents/claude.ts:188 - When agentArgsOverride.claude contains -c or --continue, the retry keeps that recency-based flag and suppresses --resume <captured-session-id>. Another Claude invocation can therefore become “most recent,” causing the nudge to enter an unrelated conversation and fabricate the iteration summary; an ID-less turn is also incorrectly considered recoverable in this configuration. For continuation turns, require the captured session ID and replace the broad flag with --resume <id>; otherwise skip recovery.
  • ⚠️ src/core/agents/acp.ts:488 - The zero-length check treats whitespace-only ACP output as a final response. A completed turn emitting only spaces or newlines therefore reaches the JSON parse error instead of receiving the required one-shot continuation. Trim the output candidates when deciding whether the response is empty.
  • ⚠️ src/core/agents/codex.ts:351 - A whitespace-only Codex agent_message is truthy, so a completed turn with blank text bypasses empty-response recovery and fails JSON parsing. Normalize with trim() before the empty check so blank completed turns receive the one allowed nudge.

🔧 Fix: fix exact-session and whitespace empty-response recovery
3 issues (1 error, 2 warnings) still open:

  • 🚨 src/core/agents/claude.ts:172 - buildClaudeArgs removes -c/--continue for recovery but preserves --fork-session. With the supported override --continue --fork-session, the retry adds --resume <captured-id> while retaining --fork-session, so Claude creates another session instead of resuming the exact one. This contradicts the required “Do not broaden session behavior” criterion and README’s exact-session claim. Decide whether recovery must strip --fork-session or whether this exception is intentional and should be documented.
  • ⚠️ src/core/agents/codex.ts:89 - The unsupported-argument check misses compact short-option values accepted by codex exec, such as -C/tmp, -sworkspace-write, and -pwork. codex exec resume rejects these, so an otherwise recoverable empty turn launches a continuation that fails before delivering the nudge. Detect attached values for the denylisted short options.
  • ⚠️ src/core/agents/copilot.ts:313 - A whitespace-only Copilot assistant.message is truthy, so it bypasses the documented empty-response exclusion and reports a generic JSON parse failure instead of naming the unverified resume contract. Trim when checking for an empty final message; recovery should remain disabled.
✅ **Test** - passed

✅ No issues found.

  • Inspected the target diff and authoritative intent, including the documented Pi and Copilot exclusions.
  • pnpm exec vitest run src/core/agents/empty-response.test.ts src/core/agents/acp.test.ts src/core/agents/claude.test.ts src/core/agents/codex.test.ts src/core/agents/copilot.test.ts src/core/agents/opencode.test.ts src/core/agents/pi.test.ts src/core/agents/rovodev.test.ts src/core/agents/stream-utils.test.ts src/core/config.test.ts
  • pnpm run build
  • Added executable E2E coverage in e2e/e2e.test.ts and the mock OpenCode fixture.
  • pnpm exec vitest run e2e/e2e.test.ts -t "runs one iteration from an argv prompt and cleans up the mock opencode server|recovers one completed empty turn in the same session and records the nudge|fails after one continuation when both completed turns are empty"
  • Manually ran the built CLI with GNHF_MOCK_OPENCODE_EMPTY_ONCE=1 and inspected its stdout summary, agent turns, commit, gnhf.log, and notes.md.
  • Manually ran the built CLI with GNHF_MOCK_OPENCODE_ALWAYS_EMPTY=1 and verified two same-session turns, one logged nudge, normal failure, and no commit.
  • Removed transient fixture repositories and verified only the intentional E2E test changes remain in the worktree.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6883c80514

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/core/agents/claude.ts Outdated
Comment on lines +189 to +191
...(resumeSessionId && !userSpecifiedSessionContinuation(userArgs)
? ["--resume", resumeSessionId]
: []),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resume the captured Claude session explicitly

When agentArgsOverride.claude contains -c or --continue, this condition suppresses --resume <captured-session-id> during the empty-response recovery, while the original broad continuation flag remains in userArgs. If another Claude invocation becomes the most recent session between turns, the nudge can enter an unrelated conversation and return a fabricated summary; either override the continuation flag with the captured ID or skip recovery for this configuration. This also contradicts the documented guarantee that Claude resumes the exact session.

AGENTS.md reference: AGENTS.md:L10-L10

Useful? React with 👍 / 👎.

@jasonqlwilliams-alt
jasonqlwilliams-alt force-pushed the fm/gnhf-empty-response-recovery-o1 branch from 6883c80 to f024004 Compare August 12, 2026 08:38

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74cdae9790

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


setupChildProcessHandlers(child, "copilot", logStream, reject, () => {
setupChildProcessHandlers(child, "copilot", reject, () => {
if (!lastAgentMessage) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Classify whitespace-only Copilot replies as empty

When Copilot emits an assistant.message whose data.content contains only spaces or newlines, lastAgentMessage is truthy, so this guard falls through to parseAgentOutput and reports a generic JSON parse failure instead of COPILOT_EMPTY_RESPONSE_MESSAGE. Copilot should remain non-retryable, but the newly documented failure contract says a skipped nudge names the unverified-resume reason, so trim the message when performing this emptiness check.

AGENTS.md reference: AGENTS.md:L10-L10

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant