Skip to content

Surface actionable guidance when LLM extraction fallback has no API key (no_api_key_for_fallback) #262

@dcramer

Description

@dcramer

Problem

When the agent SDK returns malformed output for a hunk (no {"findings": ...} block, unbalanced JSON, prose-wrapped output, etc.), Warden falls back to a Haiku-based extractor in extractFindingsWithLLM (src/sdk/extract.ts:174). That fallback uses the raw @anthropic-ai/sdk Anthropic client and requires an API key from getAnthropicApiKey() (src/utils/index.ts:57).

Users running Warden via Claude Code OAuth/subscription never set WARDEN_ANTHROPIC_API_KEY or ANTHROPIC_API_KEY because the Agent SDK doesn't need them — but the Haiku fallback does. Result: the fallback returns error: 'no_api_key_for_fallback' (src/sdk/extract.ts:182) and the CLI surfaces it verbatim:

⚠  Extraction failed: homeassistant/components/config/area_registry.py:1-177 — no_api_key_for_fallback
⚠  Extraction failed: homeassistant/components/frontend/storage.py:248-322 — no_api_key_for_fallback

Two problems:

  1. The message is unactionable. A user reading no_api_key_for_fallback has no idea what to set, why, or whether the run is degraded. Today they have to grep the source to find out.
  2. OAuth users silently lose recovery. Tier-1 regex parses most output, but the malformed-output cases — large complex hunks where the model wandered into prose — are exactly where the fallback matters most. OAuth users get zero findings on those hunks instead of recovered ones.

Repro

  1. unset WARDEN_ANTHROPIC_API_KEY ANTHROPIC_API_KEY
  2. Authenticate via claude login (OAuth).
  3. Run a skill against a large file whose hunk spans 100+ lines (any framework code with deep nesting).
  4. Expect the warning above whenever the model emits prose around its JSON.

Suggested fix

Two layers, smallest first:

1. Better error surface (low-effort, high value)

In the CLI warning path (src/cli/output/tasks.ts — wherever the extraction-failed line is rendered), map no_api_key_for_fallback to a single-line hint:

⚠  Extraction failed: <loc> — output unparseable; set WARDEN_ANTHROPIC_API_KEY to enable LLM fallback

mapExtractionErrorCode in src/sdk/errors.ts is the natural seam — it already turns no_api_key_for_fallback into extraction_no_api_key. Extend it (or its caller) to provide a human string keyed by code.

Apply the same treatment to the other extraction failure codes (no_findings_json, unbalanced_json, invalid_json, missing_findings_key, findings_not_array, no_findings_to_extract) so all of them say something useful.

2. (Optional, larger) OAuth-credential reuse for the fallback

Investigate whether the Haiku fallback can reuse the Agent SDK's OAuth credentials instead of needing a separate API key. If feasible, this removes the failure mode entirely for subscription users. If not, document explicitly that an API key is required for fallback recovery, and consider emitting a single one-time warning at run start instead of one per failed hunk.

Acceptance criteria

  • Each extraction-failure warning includes either a remediation hint or a clear human description (not the raw code).
  • Documentation note in the CLI/init flow that mentions LLM fallback requires WARDEN_ANTHROPIC_API_KEY independently of how the Agent SDK authenticates.
  • (Stretch) Fallback works under OAuth without a separate API key.

References

  • Origin of error code: src/sdk/extract.ts:174-185
  • Tier dispatch: src/sdk/analyze.ts:48-82
  • Code-to-key mapping: src/sdk/errors.ts:142
  • API key resolution: src/utils/index.ts:57

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions