Skip to content

feat(reliability): graceful degradation for non-critical agent failures #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
snipcodeit merged 3 commits into from
Mar 6, 2026
Merged

feat(reliability): graceful degradation for non-critical agent failures #243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aaa061b
feat(reliability): add agent criticality classification module
Mar 6, 2026
c66ba94
feat(reliability): add criticality annotations to all Task() spawn po…
Mar 6, 2026
3c1e8e5
Merge branch 'main' into issue/234-implement-graceful-degradation-for-n
Mar 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 115 additions & 28 deletions commands/run/execute.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ ic.buildGSDPromptContext({
" 2>/dev/null || echo "")
```

<!-- mgw:criticality=critical spawn_point=planner -->
<!-- Critical: plan creation is required for pipeline to proceed.
On failure: retry via existing retry loop, then dead-letter. -->

**Pre-spawn diagnostic hook (planner):**
```bash
DIAG_PLANNER=$(node -e "
Expand All @@ -90,7 +94,6 @@ const id = dh.beforeAgentSpawn({
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down Expand Up @@ -176,6 +179,23 @@ PLAN_CHECK=$(node ~/.claude/get-shit-done/bin/gsd-tools.cjs verify plan-structur
```
Parse the JSON result. If structural issues found, include them in the plan-checker prompt below so it has concrete problems to evaluate rather than searching from scratch.

6. **(If --full) Spawn plan-checker, handle revision loop (max 2 iterations):**

<!-- mgw:criticality=advisory spawn_point=plan-checker -->
<!-- Advisory: plan checking is a quality gate, not a pipeline blocker.
If this agent fails, log a warning and proceed with the unchecked plan.
The plan was already verified structurally by gsd-tools verify plan-structure.

Graceful degradation pattern:
```
PLAN_CHECK_RESULT=$(wrapAdvisoryAgent(Task(...), 'plan-checker', {
issueNumber: ISSUE_NUMBER,
fallback: '## VERIFICATION PASSED (plan-checker unavailable — structural check only)'
}))
# If fallback returned, skip the revision loop and proceed to execution
```
-->

6. **(If --full) Spawn plan-checker (with diagnostic capture), handle revision loop (max 2 iterations):**

**Pre-spawn diagnostic hook (plan-checker):**
Expand All @@ -191,7 +211,6 @@ const id = dh.beforeAgentSpawn({
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down Expand Up @@ -248,6 +267,13 @@ dh.afterAgentSpawn({
If issues found and iteration < 2: spawn planner revision, then re-check.
If iteration >= 2: offer force proceed or abort.

7. **Spawn executor (task agent):**

<!-- mgw:criticality=critical spawn_point=executor -->
<!-- Critical: execution produces the code changes. Without it, there is
nothing to commit or PR. On failure: retry via existing retry loop,
then dead-letter. -->

7. **Spawn executor (task agent, with diagnostic capture):**

**Pre-spawn diagnostic hook (executor):**
Expand All @@ -263,7 +289,6 @@ const id = dh.beforeAgentSpawn({
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down Expand Up @@ -321,6 +346,25 @@ VERIFY_RESULT=$(node ~/.claude/get-shit-done/bin/gsd-tools.cjs verify-summary "$
```
Parse JSON result. Use `passed` field for go/no-go. Checks summary existence, files created, and commits.

9. **(If --full) Spawn verifier:**

<!-- mgw:criticality=advisory spawn_point=verifier -->
<!-- Advisory: verification is quality assurance after execution is complete.
The code changes and commits already exist. If verification fails,
log a warning and proceed to PR creation with a note that verification
was skipped.

Graceful degradation pattern:
```
VERIFY_RESULT=$(wrapAdvisoryAgent(Task(...), 'verifier', {
issueNumber: ISSUE_NUMBER,
fallback: null
}))
# If fallback returned (null), create a minimal VERIFICATION.md:
# "## VERIFICATION SKIPPED\nVerifier agent unavailable. Manual review recommended."
```
-->

9. **(If --full) Spawn verifier (with diagnostic capture):**

**Pre-spawn diagnostic hook (verifier):**
Expand All @@ -336,7 +380,6 @@ const id = dh.beforeAgentSpawn({
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down Expand Up @@ -404,8 +447,13 @@ If any step above fails (executor or verifier agent returns error, summary missi

```bash
# On failure — classify and decide whether to retry
# CRITICALITY-AWARE: check if the failing agent is advisory first.
# Advisory agent failures are logged and skipped (pipeline continues).
# Critical agent failures follow the existing retry/dead-letter flow.

FAILURE_CLASS=$(node -e "
const { classifyFailure, canRetry, incrementRetry, getBackoffMs } = require('./lib/retry.cjs');
const { isAdvisory } = require('./lib/agent-criticality.cjs');
const { loadActiveIssue } = require('./lib/state.cjs');
const fs = require('fs'), path = require('path');

Expand All @@ -415,29 +463,43 @@ const file = files.find(f => f.startsWith('${ISSUE_NUMBER}-') && f.endsWith('.js
const filePath = path.join(activeDir, file);
let issueState = JSON.parse(fs.readFileSync(filePath, 'utf-8'));

// Classify the failure from the error context
const error = { message: '${EXECUTION_ERROR_MESSAGE}' };
const result = classifyFailure(error);
console.error('Failure classified as: ' + result.class + ' — ' + result.reason);

// Persist failure class to state
issueState.last_failure_class = result.class;

if (result.class === 'transient' && canRetry(issueState)) {
const backoff = getBackoffMs(issueState.retry_count || 0);
issueState = incrementRetry(issueState);
fs.writeFileSync(filePath, JSON.stringify(issueState, null, 2));
// Output: backoff ms so shell can sleep
console.log('retry:' + backoff + ':' + result.class);
// Check if the failing agent is advisory
const failingSpawnPoint = '${FAILING_SPAWN_POINT}';
if (failingSpawnPoint && isAdvisory(failingSpawnPoint)) {
// Advisory agent failure — log warning and continue pipeline
console.error('MGW WARNING: Advisory agent \"' + failingSpawnPoint + '\" failed. Continuing pipeline.');
console.log('advisory_degraded:' + failingSpawnPoint);
} else {
// Permanent failure or retries exhausted — dead-letter
issueState.dead_letter = true;
fs.writeFileSync(filePath, JSON.stringify(issueState, null, 2));
console.log('dead_letter:' + result.class);
// Critical agent failure — classify and apply retry/dead-letter logic
const error = { message: '${EXECUTION_ERROR_MESSAGE}' };
const result = classifyFailure(error);
console.error('Failure classified as: ' + result.class + ' — ' + result.reason);

// Persist failure class to state
issueState.last_failure_class = result.class;

if (result.class === 'transient' && canRetry(issueState)) {
const backoff = getBackoffMs(issueState.retry_count || 0);
issueState = incrementRetry(issueState);
fs.writeFileSync(filePath, JSON.stringify(issueState, null, 2));
// Output: backoff ms so shell can sleep
console.log('retry:' + backoff + ':' + result.class);
} else {
// Permanent failure or retries exhausted — dead-letter
issueState.dead_letter = true;
fs.writeFileSync(filePath, JSON.stringify(issueState, null, 2));
console.log('dead_letter:' + result.class);
}
}
")

case "$FAILURE_CLASS" in
advisory_degraded:*)
DEGRADED_AGENT=$(echo "$FAILURE_CLASS" | cut -d':' -f2)
echo "MGW: Advisory agent '${DEGRADED_AGENT}' failed — gracefully degraded, continuing pipeline."
# Do NOT set EXECUTION_SUCCEEDED=false — pipeline continues
# Skip to next step (advisory output is optional)
;;
retry:*)
BACKOFF_MS=$(echo "$FAILURE_CLASS" | cut -d':' -f2)
BACKOFF_SEC=$(( (BACKOFF_MS + 999) / 1000 ))
Expand Down Expand Up @@ -642,7 +704,11 @@ fi
" 2>/dev/null || echo "")
```

**Pre-spawn diagnostic hook (milestone planner):**
<!-- mgw:criticality=critical spawn_point=milestone-planner -->
<!-- Critical: phase planning is required — cannot execute without a plan.
On failure: retry via milestone retry loop, then dead-letter. -->

**Pre-spawn diagnostic hook (milestone planner):**
```bash
DIAG_MS_PLANNER=$(node -e "
const dh = require('${REPO_ROOT}/lib/diagnostic-hooks.cjs');
Expand All @@ -655,7 +721,6 @@ fi
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down Expand Up @@ -733,7 +798,12 @@ fi
# Parse EXEC_INIT JSON for: executor_model, verifier_model, phase_dir, plans, incomplete_plans, plan_count
```

**Pre-spawn diagnostic hook (milestone executor):**
<!-- mgw:criticality=critical spawn_point=milestone-executor -->
<!-- Critical: phase execution produces the code changes. Without it,
there is nothing to commit or PR. On failure: retry via milestone
retry loop, then dead-letter. -->

**Pre-spawn diagnostic hook (milestone executor):**
```bash
DIAG_MS_EXECUTOR=$(node -e "
const dh = require('${REPO_ROOT}/lib/diagnostic-hooks.cjs');
Expand All @@ -746,7 +816,6 @@ fi
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down Expand Up @@ -804,7 +873,26 @@ fi
done
```

**e. Spawn verifier agent (gsd:verify-phase, with diagnostic capture):**
**e. Spawn verifier agent (gsd:verify-phase):**

<!-- mgw:criticality=advisory spawn_point=milestone-verifier -->
<!-- Advisory: phase verification is quality assurance after execution
completes. The code changes and commits already exist. If verification
fails, log a warning and proceed with a note that verification was
skipped for this phase.

Graceful degradation pattern:
```
VERIFY_RESULT=$(wrapAdvisoryAgent(Task(...), 'milestone-verifier', {
issueNumber: ISSUE_NUMBER,
fallback: null
}))
# If fallback returned, create minimal VERIFICATION.md:
# "## VERIFICATION SKIPPED\nVerifier agent unavailable for phase ${PHASE_NUMBER}."
```
-->

**e. Spawn verifier agent (gsd:verify-phase, with diagnostic capture):**

**Pre-spawn diagnostic hook (milestone verifier):**
```bash
Expand All @@ -819,7 +907,6 @@ fi
process.stdout.write(id);
" 2>/dev/null || echo "")
```

```
Task(
prompt="
Expand Down
6 changes: 6 additions & 0 deletions commands/run/pr-create.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,12 @@ ic.assembleIssueContext(${ISSUE_NUMBER})

Read issue state for context.

<!-- mgw:criticality=critical spawn_point=pr-creator -->
<!-- Critical: PR creation is the pipeline's final output. Without it,
the entire pipeline run produces no deliverable. On failure: the
pipeline marks the issue as failed (no retry — PR creation errors
are typically permanent: branch conflicts, permissions, etc.). -->

**Pre-spawn diagnostic hook (PR creator):**
```bash
DIAG_PR_CREATOR=$(node -e "
Expand Down
14 changes: 14 additions & 0 deletions commands/run/triage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,20 @@ NEW_COMMENTS=$(gh issue view $ISSUE_NUMBER --json comments \

2. **Spawn classification agent (with diagnostic capture):**

<!-- mgw:criticality=advisory spawn_point=comment-classifier -->
<!-- Advisory: comment classification failure does not block the pipeline.
If this agent fails, log a warning and treat all new comments as
informational (safe default — pipeline continues with stale data).

Graceful degradation pattern:
```
CLASSIFICATION_RESULT=$(wrapAdvisoryAgent(Task(...), 'comment-classifier', {
issueNumber: ISSUE_NUMBER,
fallback: '{"classification":"informational","reasoning":"comment classifier unavailable","new_requirements":[],"blocking_reason":""}'
}))
```
-->

**Pre-spawn diagnostic hook:**
```bash
CLASSIFIER_PROMPT="<full classifier prompt assembled above>"
Expand Down
Loading
Loading