Skip to content

Remove alert-autofix pipeline from nixconfig #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
phonkd merged 1 commit into from
Jul 24, 2026
+2 −368
Merged

Remove alert-autofix pipeline from nixconfig #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 2 additions & 208 deletions modules/hosts/204-agent.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,211 +303,6 @@
deps = [ ];
};

# Custom Hermes skill: the alert-autofix pipeline (see
# plans/hermes-alert-autofix.md). A `hermes cron` job runs this skill on an
# interval; it polls the Mimir Alertmanager for newly-firing alerts, and for
# each one it judges fixable in this repo it spawns a Claude Code instance
# (`claude -p`, via the bundled autonomous-ai-agents/claude-code skill) that
# opens a DRAFT PR against nixconfig, then presents the PR link back in
# Discord. It NEVER merges, deploys, or pushes to main — a human does that.
# Installed the same way as the mimir-alerting skill above (activation-time
# install into $HERMES_HOME/skills/, no changes to hermes-agent itself). The
# cron job itself is runtime state (~/.hermes/cron/jobs.json) created once by
# hand — see the plan's "One-time setup" — like the spotify/copilot auth.
system.activationScripts.hermesAutofixSkill =
let
skillFile = pkgs.writeText "SKILL.md" ''
---
name: hermes-autofix
description: "React to a firing homelab alert by opening a draft nixconfig PR: triage the alert, and if it is fixable in the nixconfig repo, spawn Claude Code to write the fix and open a draft PR, then present the PR in Discord. Never merges or deploys."
version: 1.0.0
author: phonkd homelab
license: Unlicense
platforms: [linux, macos]
metadata:
hermes:
tags: [monitoring, alerting, incident-response, nixconfig, autofix, homelab]
category: devops
requires_toolsets: [terminal]
related_skills: [mimir-alerting, autonomous-ai-agents/claude-code]
---

# Hermes Alert Autofix

React to a firing homelab alert by proposing a fix as a **draft pull
request** against the `nixconfig` repo, then presenting it in Discord.
You **propose, you never apply**: the pipeline ends at a draft PR + a
Discord summary. A human reviews, merges, and runs `deploy <host>`.

## When to Use

- You are invoked by the `alert-autofix` cron job (every ~10m) to check
for newly-firing alerts and act on them.
- A human explicitly asks you to "look at the alerts and open a fix".

Do **not** use this to silence or create alert rules — that's the
separate `mimir-alerting` skill.

## Hard rules (read first)

- **Draft PRs only.** `gh pr create --draft`. Never `gh pr merge`,
never push to `main`, never `deploy`, never touch a running host.
- **One PR per alert.** Dedupe on the alert fingerprint via the
watermark file (below). If an alert is already handled (has an open
PR recorded), skip it — do not open a second PR.
- **Only act on alerts fixable in nixconfig.** A recurring failed
systemd unit, a wrong alert threshold, a missing firewall port, a bad
service option: fixable — open a PR. A dead disk, an upstream outage,
anything needing a human decision or a hardware fix: **not** fixable —
emit a one-line triage note and record it so you don't re-triage it
every tick.
- **Distrust generated nix.** Claude Code has invented nonexistent
options before (e.g. a `services.foo.settings` that doesn't exist).
The task you hand it MUST require verifying option names against the
real module source and `nix-instantiate --parse`, never a full eval.
- If you have nothing to report, end your response with the literal
token `[SILENT]` so the cron delivery is suppressed (no Discord spam).

## Before you start

- **Alerts API:** `GET http://10.9.0.1:9009/alertmanager/api/v2/alerts`
— reachable over the home site-to-site VPN, which this host already
has (same endpoint the `mimir-alerting` skill uses). No auth,
single-tenant, no `X-Scope-OrgID` needed.
- **Repo:** `phonkd/nixconfig` (this repo). `gh` is already
authenticated via `GITHUB_TOKEN`; `claude` is authenticated via
`CLAUDE_CODE_OAUTH_TOKEN`. Clone/PR that repo explicitly — this is an
unattended pipeline, so never leave the owner as a placeholder that
could clone or PR the wrong repo.
- **Watermark:** `$HERMES_HOME/autofix-state/handled.json` — a single
JSON object mapping alert fingerprint to what you did (`{"pr":<n>}` or
`{"skipped":"<reason>"}`). Create the dir if missing. This is the
dedupe memory; treat a fingerprint present here as already handled.
**It must stay valid JSON — never append raw text to it.** Record a
result by *merging* one key in, atomically via a temp file:

```bash
wm="$HERMES_HOME/autofix-state/handled.json"
# $fp = fingerprint, $entry = a JSON value like '{"pr":42}'
tmp=$(mktemp)
jq --arg fp "$fp" --argjson e "$entry" '. + {($fp): $e}' "$wm" > "$tmp" && mv "$tmp" "$wm"
```
- **The nixconfig skill.** The fix itself must follow this repo's wiring
rules. Instruct Claude Code to read the repo's `.claude/skills/`
`nixconfig` and `nixconfig-ops` skills before changing anything.

## Procedure

1. **Poll and dedupe.** Fetch active (firing) alerts and their
fingerprints, then subtract anything already in the watermark:

```bash
mkdir -p "$HERMES_HOME/autofix-state"
wm="$HERMES_HOME/autofix-state/handled.json"
[ -f "$wm" ] || echo '{}' > "$wm"
curl -sS http://10.9.0.1:9009/alertmanager/api/v2/alerts \
| jq -c '.[] | select(.status.state=="active")
| {fp: .fingerprint, name: .labels.alertname,
severity: .labels.severity, instance: .labels.instance,
summary: .annotations.summary,
description: .annotations.description}' \
| while read -r a; do
fp=$(echo "$a" | jq -r .fp)
jq -e --arg fp "$fp" 'has($fp)' "$wm" >/dev/null && continue
echo "$a" # NEW, unhandled — triage it below
done
```

If nothing new prints, respond `[SILENT]` and stop.

2. **Triage each new alert.** Read its labels/annotations. If useful,
cross-check context per `nixconfig-ops` (e.g. Loki at
`http://10.9.0.1:3100` for the failing unit's logs, or the ALERTS
series in Mimir). Decide **fixable in nixconfig?**

- **No** → merge `{"skipped": "<one-line reason>"}` for this
fingerprint into the watermark (the atomic `jq` merge above — never
append raw text) and add a one-line note to your Discord summary.
Move on.
- **Yes** → go to step 3.

3. **Spawn Claude Code** (via the `autonomous-ai-agents/claude-code`
skill). Clone fresh per run so there's no stale checkout to drift,
and hand it a tightly-scoped task. Example:

```bash
work=$(mktemp -d)
gh repo clone phonkd/nixconfig "$work" -- --depth 1
claude -p "You are fixing a homelab NixOS/nix-darwin config in $work.
A monitoring alert is firing: <alertname> on <instance> —
<summary>. Root cause and fix it in this repo.
RULES: read .claude/skills nixconfig + nixconfig-ops first; make the
MINIMAL change; verify every NixOS option name against the real
module source (do NOT invent options); verify with
'nix-instantiate --parse <file>' + grep for dangling refs, NEVER a
full nixosConfigurations eval; match the repo's commit style; create
a new branch 'autofix/<alertname>-<short>'; commit; then
'gh pr create --draft --title ... --body ...' explaining the alert,
the root cause, and how to verify. Do NOT merge, deploy, or push to
main. Output the PR URL as the last line." \
--output-format json \
--allowedTools 'Read,Edit,Bash(git *),Bash(gh *),Bash(nix-instantiate *),Bash(grep *),Bash(rg *)' \
--max-turns 40 --max-budget-usd 2 \
--add-dir "$work"
```

Parse the JSON result (`.result`, `.subtype`). On `subtype ==
"success"` extract the draft PR URL/number; on error, record the
failure in the watermark and report it rather than retrying in a
loop.

4. **Record + present.** Merge `{"pr": <n>}` for this fingerprint into
the watermark (the atomic `jq` merge above — never append raw text),
then include a line in your Discord response:

- Fixable + PR opened:
`🔧 <alertname> on <instance> → draft PR #<n>: <one-line what it does>`
- Not fixable:
`⚠️ <alertname> on <instance>: <why it needs a human>`

End normally (this delivers to Discord). Only use `[SILENT]` when
there was genuinely nothing new.

## Pitfalls

- **Don't loop on a flapping alert.** The watermark is keyed by
fingerprint; a re-firing alert with the same fingerprint stays
handled. If an alert legitimately recurs after its PR is merged and
you want to reconsider it, that's a manual watermark edit, not
automatic.
- **Don't widen `--allowedTools`.** Claude Code must not need network
writes beyond `git`/`gh`; never grant it deploy or merge tools.
- **Budget the spawn.** Keep `--max-turns`/`--max-budget-usd` set so a
confused run can't burn unbounded cost.
- **Owner/remote.** If `gh` can't infer the repo, set it explicitly;
don't open a PR against the wrong fork.

## Verification

Before finishing a tick, confirm for each alert you acted on:

1. The watermark now contains its fingerprint (so it won't re-trigger).
2. If you opened a PR, it exists and is a **draft**
(`gh pr view <n> --json isDraft,url`).
3. Your Discord summary names the alert and links the PR (or states why
it wasn't fixable). Nothing was merged or deployed.
'';
in
{
text = ''
install -D -m 0644 \
-o ${config.services.hermes-agent.user} -g ${config.services.hermes-agent.group} \
${skillFile} \
${config.services.hermes-agent.stateDir}/.hermes/skills/devops/hermes-autofix/SKILL.md
'';
deps = [ ];
};

# vdirsyncer + khal client config for the caldav skill below. Written
# into the hermes state dir (HOME for the hermes services) so both
# tools find it at their default paths. The remote is radicale on
Expand Down Expand Up @@ -786,7 +581,7 @@
tags: [nixconfig, tasks, automation, homelab]
category: devops
requires_toolsets: [terminal]
related_skills: [task-notes, hermes-autofix]
related_skills: [task-notes]
---
Comment on lines 583 to 585

# Claude Code Work-State Sync
Expand Down Expand Up @@ -832,8 +627,7 @@
line. draft / approved / in-progress count; done/superseded do
not.
- **Open PRs:** `gh pr list --repo phonkd/nixconfig --state open
--json number,title,headRefName,isDraft` (drafts included —
hermes-autofix's own PRs are work items too).
--json number,title,headRefName,isDraft` (drafts included).

## Procedure

Expand Down
Loading