diff --git a/modules/hosts/204-agent.nix b/modules/hosts/204-agent.nix index c430fc8..7499f7e 100644 --- a/modules/hosts/204-agent.nix +++ b/modules/hosts/204-agent.nix @@ -303,211 +303,6 @@ deps = [ ]; }; - # Custom Hermes skill: the alert-autofix pipeline (see - # plans/hermes-alert-autofix.md). A `hermes cron` job runs this skill on an - # interval; it polls the Mimir Alertmanager for newly-firing alerts, and for - # each one it judges fixable in this repo it spawns a Claude Code instance - # (`claude -p`, via the bundled autonomous-ai-agents/claude-code skill) that - # opens a DRAFT PR against nixconfig, then presents the PR link back in - # Discord. It NEVER merges, deploys, or pushes to main — a human does that. - # Installed the same way as the mimir-alerting skill above (activation-time - # install into $HERMES_HOME/skills/, no changes to hermes-agent itself). The - # cron job itself is runtime state (~/.hermes/cron/jobs.json) created once by - # hand — see the plan's "One-time setup" — like the spotify/copilot auth. - system.activationScripts.hermesAutofixSkill = - let - skillFile = pkgs.writeText "SKILL.md" '' - --- - name: hermes-autofix - description: "React to a firing homelab alert by opening a draft nixconfig PR: triage the alert, and if it is fixable in the nixconfig repo, spawn Claude Code to write the fix and open a draft PR, then present the PR in Discord. Never merges or deploys." - version: 1.0.0 - author: phonkd homelab - license: Unlicense - platforms: [linux, macos] - metadata: - hermes: - tags: [monitoring, alerting, incident-response, nixconfig, autofix, homelab] - category: devops - requires_toolsets: [terminal] - related_skills: [mimir-alerting, autonomous-ai-agents/claude-code] - --- - - # Hermes Alert Autofix - - React to a firing homelab alert by proposing a fix as a **draft pull - request** against the `nixconfig` repo, then presenting it in Discord. - You **propose, you never apply**: the pipeline ends at a draft PR + a - Discord summary. A human reviews, merges, and runs `deploy `. - - ## When to Use - - - You are invoked by the `alert-autofix` cron job (every ~10m) to check - for newly-firing alerts and act on them. - - A human explicitly asks you to "look at the alerts and open a fix". - - Do **not** use this to silence or create alert rules — that's the - separate `mimir-alerting` skill. - - ## Hard rules (read first) - - - **Draft PRs only.** `gh pr create --draft`. Never `gh pr merge`, - never push to `main`, never `deploy`, never touch a running host. - - **One PR per alert.** Dedupe on the alert fingerprint via the - watermark file (below). If an alert is already handled (has an open - PR recorded), skip it — do not open a second PR. - - **Only act on alerts fixable in nixconfig.** A recurring failed - systemd unit, a wrong alert threshold, a missing firewall port, a bad - service option: fixable — open a PR. A dead disk, an upstream outage, - anything needing a human decision or a hardware fix: **not** fixable — - emit a one-line triage note and record it so you don't re-triage it - every tick. - - **Distrust generated nix.** Claude Code has invented nonexistent - options before (e.g. a `services.foo.settings` that doesn't exist). - The task you hand it MUST require verifying option names against the - real module source and `nix-instantiate --parse`, never a full eval. - - If you have nothing to report, end your response with the literal - token `[SILENT]` so the cron delivery is suppressed (no Discord spam). - - ## Before you start - - - **Alerts API:** `GET http://10.9.0.1:9009/alertmanager/api/v2/alerts` - — reachable over the home site-to-site VPN, which this host already - has (same endpoint the `mimir-alerting` skill uses). No auth, - single-tenant, no `X-Scope-OrgID` needed. - - **Repo:** `phonkd/nixconfig` (this repo). `gh` is already - authenticated via `GITHUB_TOKEN`; `claude` is authenticated via - `CLAUDE_CODE_OAUTH_TOKEN`. Clone/PR that repo explicitly — this is an - unattended pipeline, so never leave the owner as a placeholder that - could clone or PR the wrong repo. - - **Watermark:** `$HERMES_HOME/autofix-state/handled.json` — a single - JSON object mapping alert fingerprint to what you did (`{"pr":}` or - `{"skipped":""}`). Create the dir if missing. This is the - dedupe memory; treat a fingerprint present here as already handled. - **It must stay valid JSON — never append raw text to it.** Record a - result by *merging* one key in, atomically via a temp file: - - ```bash - wm="$HERMES_HOME/autofix-state/handled.json" - # $fp = fingerprint, $entry = a JSON value like '{"pr":42}' - tmp=$(mktemp) - jq --arg fp "$fp" --argjson e "$entry" '. + {($fp): $e}' "$wm" > "$tmp" && mv "$tmp" "$wm" - ``` - - **The nixconfig skill.** The fix itself must follow this repo's wiring - rules. Instruct Claude Code to read the repo's `.claude/skills/` - `nixconfig` and `nixconfig-ops` skills before changing anything. - - ## Procedure - - 1. **Poll and dedupe.** Fetch active (firing) alerts and their - fingerprints, then subtract anything already in the watermark: - - ```bash - mkdir -p "$HERMES_HOME/autofix-state" - wm="$HERMES_HOME/autofix-state/handled.json" - [ -f "$wm" ] || echo '{}' > "$wm" - curl -sS http://10.9.0.1:9009/alertmanager/api/v2/alerts \ - | jq -c '.[] | select(.status.state=="active") - | {fp: .fingerprint, name: .labels.alertname, - severity: .labels.severity, instance: .labels.instance, - summary: .annotations.summary, - description: .annotations.description}' \ - | while read -r a; do - fp=$(echo "$a" | jq -r .fp) - jq -e --arg fp "$fp" 'has($fp)' "$wm" >/dev/null && continue - echo "$a" # NEW, unhandled — triage it below - done - ``` - - If nothing new prints, respond `[SILENT]` and stop. - - 2. **Triage each new alert.** Read its labels/annotations. If useful, - cross-check context per `nixconfig-ops` (e.g. Loki at - `http://10.9.0.1:3100` for the failing unit's logs, or the ALERTS - series in Mimir). Decide **fixable in nixconfig?** - - - **No** → merge `{"skipped": ""}` for this - fingerprint into the watermark (the atomic `jq` merge above — never - append raw text) and add a one-line note to your Discord summary. - Move on. - - **Yes** → go to step 3. - - 3. **Spawn Claude Code** (via the `autonomous-ai-agents/claude-code` - skill). Clone fresh per run so there's no stale checkout to drift, - and hand it a tightly-scoped task. Example: - - ```bash - work=$(mktemp -d) - gh repo clone phonkd/nixconfig "$work" -- --depth 1 - claude -p "You are fixing a homelab NixOS/nix-darwin config in $work. - A monitoring alert is firing: on — - . Root cause and fix it in this repo. - RULES: read .claude/skills nixconfig + nixconfig-ops first; make the - MINIMAL change; verify every NixOS option name against the real - module source (do NOT invent options); verify with - 'nix-instantiate --parse ' + grep for dangling refs, NEVER a - full nixosConfigurations eval; match the repo's commit style; create - a new branch 'autofix/-'; commit; then - 'gh pr create --draft --title ... --body ...' explaining the alert, - the root cause, and how to verify. Do NOT merge, deploy, or push to - main. Output the PR URL as the last line." \ - --output-format json \ - --allowedTools 'Read,Edit,Bash(git *),Bash(gh *),Bash(nix-instantiate *),Bash(grep *),Bash(rg *)' \ - --max-turns 40 --max-budget-usd 2 \ - --add-dir "$work" - ``` - - Parse the JSON result (`.result`, `.subtype`). On `subtype == - "success"` extract the draft PR URL/number; on error, record the - failure in the watermark and report it rather than retrying in a - loop. - - 4. **Record + present.** Merge `{"pr": }` for this fingerprint into - the watermark (the atomic `jq` merge above — never append raw text), - then include a line in your Discord response: - - - Fixable + PR opened: - `🔧 on → draft PR #: ` - - Not fixable: - `⚠️ on : ` - - End normally (this delivers to Discord). Only use `[SILENT]` when - there was genuinely nothing new. - - ## Pitfalls - - - **Don't loop on a flapping alert.** The watermark is keyed by - fingerprint; a re-firing alert with the same fingerprint stays - handled. If an alert legitimately recurs after its PR is merged and - you want to reconsider it, that's a manual watermark edit, not - automatic. - - **Don't widen `--allowedTools`.** Claude Code must not need network - writes beyond `git`/`gh`; never grant it deploy or merge tools. - - **Budget the spawn.** Keep `--max-turns`/`--max-budget-usd` set so a - confused run can't burn unbounded cost. - - **Owner/remote.** If `gh` can't infer the repo, set it explicitly; - don't open a PR against the wrong fork. - - ## Verification - - Before finishing a tick, confirm for each alert you acted on: - - 1. The watermark now contains its fingerprint (so it won't re-trigger). - 2. If you opened a PR, it exists and is a **draft** - (`gh pr view --json isDraft,url`). - 3. Your Discord summary names the alert and links the PR (or states why - it wasn't fixable). Nothing was merged or deployed. - ''; - in - { - text = '' - install -D -m 0644 \ - -o ${config.services.hermes-agent.user} -g ${config.services.hermes-agent.group} \ - ${skillFile} \ - ${config.services.hermes-agent.stateDir}/.hermes/skills/devops/hermes-autofix/SKILL.md - ''; - deps = [ ]; - }; - # vdirsyncer + khal client config for the caldav skill below. Written # into the hermes state dir (HOME for the hermes services) so both # tools find it at their default paths. The remote is radicale on @@ -786,7 +581,7 @@ tags: [nixconfig, tasks, automation, homelab] category: devops requires_toolsets: [terminal] - related_skills: [task-notes, hermes-autofix] + related_skills: [task-notes] --- # Claude Code Work-State Sync @@ -832,8 +627,7 @@ line. draft / approved / in-progress count; done/superseded do not. - **Open PRs:** `gh pr list --repo phonkd/nixconfig --state open - --json number,title,headRefName,isDraft` (drafts included — - hermes-autofix's own PRs are work items too). + --json number,title,headRefName,isDraft` (drafts included). ## Procedure diff --git a/plans/hermes-alert-autofix.md b/plans/hermes-alert-autofix.md deleted file mode 100644 index a2a2a74..0000000 --- a/plans/hermes-alert-autofix.md +++ /dev/null @@ -1,160 +0,0 @@ -# Hermes alert-autofix pipeline - -**Repo(s):** `nixconfig` (one new Hermes skill on `204-agent`, plus a one-time -`hermes cron` setup step) and, at runtime, **draft PRs against `nixconfig` -itself**. **Status:** draft. - -## Goal - -Close the loop from *alert fires* → *fix proposed*. Today an alert goes -Mimir ruler → Mimir Alertmanager → Discord webhook (see -`plans/alert-overview.md`) and then just sits there as a red line a human has to -read, diagnose, and hand-fix. The want: **Hermes reacts to a firing alert, -spawns a Claude Code instance that opens a (draft) PR against this repo fixing -the cause, and Hermes presents that PR back in Discord** for the human to review, -merge, and `deploy`. - -Autonomy envelope (the safe default this plan builds): - -- Hermes **proposes**, never applies. The pipeline ends at a **draft PR** + a - Discord summary. A human merges and runs `deploy `. No auto-merge, no - auto-deploy, no push to `main`. This matches the repo rule that *the user opens - and merges PRs themselves*, and the scar tissue that GitHub-style autofix - commits have invented nonexistent nix options before (see the `nixconfig` - skill). -- **One PR per distinct alert**, deduped by alert fingerprint so a - chronically-firing unit doesn't spawn a PR every poll. -- Hermes only opens a PR for alerts it judges **fixable in `nixconfig`**; - everything else gets a one-line triage note (or stays `[SILENT]`). - -## What already exists (nothing new to build in Hermes) - -Confirmed against `NousResearch/hermes-agent` @ `daedf4f` (the flake-locked rev): - -| Primitive | How | Where | -|---|---|---| -| Scheduled unattended prompt | `hermes cron create "every 10m" "" --skills … --deliver …`; in-process 60s ticker runs a full agent turn and delivers the result | `cron/`, jobs in `~/.hermes/cron/jobs.json` | -| Inbound HTTP webhook → agent run | `POST :8644/webhooks/`, HMAC-authed, templates the payload into a prompt | `gateway/platforms/webhook.py` | -| Read live alerts | `GET http://10.9.0.1:9009/alertmanager/api/v2/alerts` (no auth, single-tenant) | already documented in the `mimir-alerting` skill on this host | -| Claude Code coding task → PR | claude-code skill shells out to `claude -p '' --output-format json` in a workdir; can branch/commit/`gh pr create` | `skills/autonomous-ai-agents/claude-code/SKILL.md`, already installed | -| Deliver back to Discord | cron job's final agent response is delivered to `--deliver` target | `cron/scheduler.py` (`[SILENT]` sentinel suppresses delivery) | - -The `204-agent` host already ships everything the coding step needs: -`pkgs.gh` + `GITHUB_TOKEN` (fine-grained PAT for the `gh` CLI), `pkgs.claude-code` -+ `CLAUDE_CODE_OAUTH_TOKEN`, `tmux`, `jq`, and the bundled claude-code skill -(`modules/hosts/204-agent.nix`). So the pipeline is **one new skill + one cron -job**, no new packages, no new inbound network path. - -## Trigger: cron-poll, not webhook (decision) - -Two ways to wake Hermes on an alert. This plan picks **poll**: - -- **Poll (chosen).** A `hermes cron` job every ~10m curls - `GET 10.9.0.1:9009/alertmanager/api/v2/alerts`, diffs against a watermark, and - triages only *new* firing alerts. Hermes already reaches `10.9.0.1` outbound - (it ships metrics there via Alloy and the `mimir-alerting` skill hits this exact - endpoint), so **no new network path and no Alertmanager config change** — the - Alertmanager receiver/template is UI-managed on the obs host and out of repo - scope (see `plans/alert-overview.md`). Watermarking gives free dedup. Cost: - up-to-poll-interval latency, which is irrelevant for a "open a PR for a human" - loop. -- **Webhook (rejected for now).** `POST 204:8644/webhooks/alert-triage` from - Alertmanager is lower-latency and HMAC-clean, but needs (a) obs→`192.168.3.204` - inbound reachability that doesn't exist today, and (b) a new Alertmanager - webhook receiver added through the Grafana UI on the obs host — reintroducing - exactly the UI-managed, non-repo-tracked config the poll avoids. Keep as a - future upgrade if poll latency ever matters. -- **Discord passthrough (rejected).** The existing Alertmanager→Discord message - could trigger Hermes, but only with `DISCORD_ALLOW_BOTS=all|mentions` (default - `none` drops other-bot/webhook posts), and routing an agent off a bot message - in a channel is brittle vs. a structured API poll. - -## The pipeline, end to end - -1. **Cron fires** (`every 10m`). The skill itself (no `--script` preprocessor) - curls the Alertmanager alerts API via the terminal toolset, filters - `status.state == "active"` (firing), and keeps only fingerprints not already in - the watermark; nothing new → the agent responds `[SILENT]` and delivery is - suppressed. -2. **Triage** (`hermes-autofix` skill, below). For each new firing alert Hermes: - pulls the alert's labels/annotations, optionally cross-checks Loki/Mimir for - context (the `nixconfig-ops` conventions), and decides: **fixable in - nixconfig?** Recurring failed-unit alerts, a wrong threshold, a missing - firewall port, a misconfigured service option — yes. A dead disk, upstream - outage, or something needing a human decision — no (triage note only). -3. **Spawn Claude Code** for a fixable alert. The skill invokes - `claude -p '' --output-format json` in a **fresh clone** of the repo - (`gh repo clone` to a temp dir per run — no persistent checkout to drift), - instructing it to: read the `nixconfig` skill, make the minimal change, verify - with `nix-instantiate --parse` + grep (never a full eval — repo rule), branch, - commit in the repo's style, and `gh pr create --draft`. -4. **Present.** Hermes' final response — delivered to the Discord ops channel — - leads with the alert, one-line root-cause, and the **draft PR link**, e.g. - `🔧 crowdsec-firewall-bouncer failing on 201-mono → draft PR #NN: pin bouncer - to … / mask the unit`. If not fixable: `⚠️ : `. -5. **Human** reviews the draft PR, merges (or closes), and runs `deploy `. - The watermark keeps the same alert from re-triggering while the PR is open. - -## nixconfig changes (this PR) - -**One thing, declarative:** a new `hermes-autofix` skill installed into -`$HERMES_HOME/skills/devops/hermes-autofix/SKILL.md` via an activation script on -`204-agent`, mirroring the existing `hermesMimirAlertingSkill` block in -`modules/hosts/204-agent.nix` (same install pattern, same owner/group). The -skill body encodes the whole procedure above, including the hard guardrails: -draft-only, never merge/deploy/push-to-main, `nix-instantiate --parse` not full -evals, match commit style, and *distrust invented options — diff against a known -rule/module before trusting a generated change*. - -Nothing else in nix is required: no new packages (all present), no new secret -(`GITHUB_TOKEN` already authorizes `gh`), no firewall change (poll is outbound), -no new systemd service (the cron ticker lives inside the already-running -`hermes-agent` gateway process). - -## One-time setup (documented, user runs once — like the spotify/copilot auth) - -After `deploy 204-agent` installs the skill, create the cron job once on the host -(cron jobs are runtime state in `~/.hermes/cron/jobs.json`, not declarative): - -```bash -# as the hermes user on 204-agent -hermes cron create "every 10m" \ - "Run the hermes-autofix triage procedure for any newly-firing alerts." \ - --name alert-autofix \ - --skills hermes-autofix,autonomous-ai-agents/claude-code \ - --deliver discord: -``` - -The skill is self-contained (it does the poll + watermark inline via the terminal -toolset), so no `--script` preprocessor is needed. Verify with -`hermes cron run alert-autofix` (one manual tick) and `hermes cron status`, and -sanity-check the first PR it opens by hand before trusting the loop. - -## Guardrails / risks - -- **Blast radius is a draft PR.** The pipeline never mutates a running host; the - worst case is a bad draft PR the human ignores or closes. No auto-merge, no - deploy, no push to `main`. -- **Cost / runaway.** Each fixable alert spends one `claude -p` run. The - watermark caps this to one run per new alert; recurring alerts don't re-spawn - while their PR is open. Optionally cap with `--max-budget-usd` on the - `claude -p` call and a per-tick alert limit in the poll script. -- **Bad-fix risk.** Generated nix has invented options before — the skill forces - `nix-instantiate --parse` + grep and a diff-against-known-good check, and the - draft/human-merge gate is the backstop. A merged bad branch is diffable against - the prior commit (repo rule). -- **Noise.** If it ever PRs the same known-broken units repeatedly, either fix or - silence those at the source (`plans/alert-overview.md` Track C) or add an - allowlist/denylist of alert names the skill will act on. - -## Open decisions - -- **Poll interval** — `every 10m` proposed; tighten to `5m` or loosen to `30m` - freely (cron edit, no redeploy). -- **Fixable-alert scope** — start with Hermes' own judgement ("fixable in - nixconfig?"); add an explicit allowlist of alert names if it acts on things it - shouldn't. **Recommend** starting judgement-based and watching the first few. -- **Latency upgrade** — move poll → webhook only if 10-minute latency ever - matters; requires the obs→204 path + Alertmanager receiver (above). -- **Deliver target** — which Discord channel the presentation lands in (the ops - channel that already gets the Alertmanager webhook is the natural home).