Skip to content

troubleshooting.md

Latest commit

 

History

History
263 lines (181 loc) · 7.82 KB

troubleshooting.md

File metadata and controls

263 lines (181 loc) · 7.82 KB
Raw

Troubleshooting Guide

Common issues and their fixes when working with Animus.

Environment Diagnostics

Run the built-in doctor command first:

animus doctor

This checks for required tools, API keys, configuration files, and common misconfigurations. Use --fix to attempt automatic repairs:

animus doctor --fix

Daemon Won't Start

Symptoms: animus daemon start exits immediately or animus daemon status shows not running.

Steps:

  1. Check daemon status:

    animus daemon status

  2. Read the daemon log:

    animus daemon logs

  3. Try running in foreground for immediate error output:

    animus daemon run

  4. Check if another daemon instance is already running (port conflict).

CLAUDECODE Environment Variable Blocking claude CLI

Symptoms: Inside a Claude Code session, the daemon fails to start agents with an error like "Cannot be launched inside another Claude Code session".

Cause: Claude Code sets CLAUDECODE=1 in the environment. The claude CLI refuses to run when it detects this variable.

Fix: Unset the variable before starting the daemon:

unset CLAUDECODE
animus daemon start

Or use an env prefix:

env -u CLAUDECODE animus daemon start

Agent Runtime Config Overriding Compiled Defaults

Symptoms: The daemon uses unexpected models. For example, all phases use the same model instead of routing research to gemini.

Cause: The resolved agent runtime has explicit model and tool fields that override the compiled routing table. Those values can come from authored workflow YAML or from animus workflow agent-runtime set.

Fix: Inspect the resolved runtime first:

animus workflow agent-runtime get    # Inspect current config

Then either remove the YAML override from .animus/workflows.yaml / .animus/workflows/*.yaml, or replace the compiled runtime with explicit null values:

animus workflow agent-runtime set --input-json '{"agents":{"default":{"model":null,"tool":null}}}'

Or read the config cascade documentation in the Model Routing Guide.

Paused / Ghost Task State

Symptoms: A task shows as blocked or paused but should be ready. The daemon skips it.

Cause: When tasks are blocked, paused=true is set internally. Direct JSON edits or incomplete status transitions can leave ghost state. (Pausing a workflow only writes an informational paused by workflow <id> annotation to the task's blocked_reason — it never sets the paused flag — and workflow cancel syncs the task to cancelled, so workflow lifecycle controls no longer leave ghosts on their own.)

Fix: Always reset via animus subject status --kind task:

animus subject status --kind task --id task:TASK-XXX --status ready

This clears paused, blocked_at, blocked_reason, blocked_phase, and blocked_by, and the command prints an unstuck: cleared ... line naming exactly which flags it cleared. Never hand-edit Animus-managed runtime JSON or SQLite state.

Runner Connection Issues

Symptoms: Workflows fail with runner connection errors. Agents do not start.

Steps:

  1. Check provider plugin health:

    animus plugin status
animus daemon health

    If every provider shows installed: false, verify the plugin binaries still exist under ~/.animus/plugins/ and still have their execute bit. A present but non-executable animus-provider-* binary is treated as unhealthy on purpose.

  2. Detect orphaned CLI processes:

    animus doctor --check orphan_cli_processes

  3. Clean up stale tracker entries if found (live PIDs get a manual kill suggestion instead of being terminated automatically):

    animus doctor --fix

  4. Verify API keys are set:

    animus doctor --check api_keys

Build Cache Stale

Symptoms: After editing protocol types or model routing, cargo build does not pick up changes.

Cause: Cargo's incremental compilation may not detect changes in certain files, especially when the edit lands in a shared crate or freshly updated git dependency.

Fix: Touch the changed file to force recompilation:

touch crates/orchestrator-config/src/agent_runtime_config.rs
cargo build -p orchestrator-cli

Daemon Log Location

Use Animus to inspect or clear daemon logs:

animus daemon logs
animus daemon clear-logs

Animus stores runtime state under ~/.animus/<repo-scope>/, and log plumbing is managed by the runtime binaries rather than a project-local .animus/daemon.log contract.

Current paths:

  • daemon process log: ~/.animus/<repo-scope>/daemon/daemon.log
  • structured runtime event mirror: ~/.animus/<repo-scope>/logs/events.jsonl

Daemon Status Looks Slow With Many Plugins

Symptoms: animus daemon status used to feel slow on machines with many installed subject or transport plugins.

Cause: Provider-health reporting previously walked the full plugin discovery pipeline, which could manifest-probe binaries unrelated to daemon/provider readiness.

Current behavior: daemon status now answers the provider-health portion by checking the installed animus-provider-* binaries directly and verifying that at least one is executable. If status is still slow, the remaining cost is elsewhere in daemon state loading rather than plugin manifest probes.

Notification Delivery Missing Credentials

Symptoms: The daemon keeps running, but notification deliveries fail, or a notifier plugin cannot see the webhook URL / auth header env vars you expected.

Cause: Notification connectors do not inherit the daemon's full environment. The notifier subprocess only receives env var names explicitly referenced by the persisted notification_config block in ~/.animus/<repo-scope>/daemon/pm-config.json.

Fix: Inspect the persisted daemon config and ensure the connector uses the right *_env fields:

animus daemon config

For a generic webhook connector, use url_env for the destination URL and headers_env for per-header env lookups, for example:

{
  "type": "webhook",
  "id": "ops-webhook",
  "enabled": true,
  "url_env": "ANIMUS_NOTIFY_WEBHOOK_URL",
  "headers_env": {
    "Authorization": "ANIMUS_NOTIFY_BEARER_TOKEN"
  }
}

If you change shell env vars without updating the referenced names, restart the daemon so the new process environment is available to the notifier plugin.

Workflow Stuck or Failed

Steps:

  1. List workflows to find the problematic one:

    animus workflow list

  2. Inspect the workflow:

    animus workflow get --id WF-001

  3. Check decisions made during execution:

    animus workflow decisions --id WF-001

  4. View the agent output:

    animus output read --run-id RUN-001

  5. If the workflow is stuck, you can cancel and retry:

    animus workflow cancel --id WF-001
animus subject status --kind task --id task:TASK-XXX --status ready

Missing API Keys

Symptoms: Agents fail to start with authentication errors.

Check:

animus doctor --check api_keys

Required keys by tool:

Tool Environment Variable
claude ANTHROPIC_API_KEY
codex OPENAI_API_KEY
gemini GEMINI_API_KEY or GOOGLE_API_KEY
oai-runner MINIMAX_API_KEY, ZAI_API_KEY, or OPENAI_API_KEY

Gemini Redirected on Write Phases

Symptoms: Research phases that use gemini get redirected to claude even though they do not need write access.

Cause: enforce_write_capable_phase_target redirects non-write-capable tools by default.

Fix: route read-only phases to Gemini in workflow YAML, and use a write-capable tool such as claude, codex, or oai-runner for implementation phases that modify the repository.