Swarm control plane + AgenticOS hardening (lifecycle, model router, permissions, perf, Obsidian)#690
Open
estevanjim03-cyber wants to merge 40 commits into
Conversation
…e sub-64K models Three failures kept the swarm looping and blocked: 1. Dead-session loop. ensureLiveTmuxSession() returned ok as soon as a tmux session *name* existed, without checking the agent was alive. When `hermes chat --tui` had exited (bare shell pane), dispatch pasted the prompt into a dead shell, no checkpoint ever arrived, and the orchestrator re-dispatched forever. Add agentPaneIsAlive() (foreground process + exit sentinel check); a dead pane is killed and the session rebuilt. 2. Missing launcher wrappers. 7 of 10 workers had no ~/.local/bin/<id> wrapper, so the lifecycle renew path returned "Wrapper not found" and never spawned. Add scripts/swarm-generate-wrappers.sh to generate wrappers for every worker in swarm.yaml, under both the <workerId> and <wrapper-field> names the two code paths expect. Wrappers forward args to hermes (oneshot dispatch) and default to the persistent TUI when invoked bare (lifecycle start). 3. Sub-64K models. qa (qwen2.5:7b) and maintainer (mistral:7b) ran 32K-context models, below Hermes Agent's 64K minimum, so `agent init` failed and every dispatch was recorded BLOCKED. Reassign both to llama3.1:8b (128K). Verified end-to-end: planted a dead session, dispatched, pane rebuilt to a live agent; dispatch returned delivery:tmux ok:true and the worker executed tools. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sts, 4 new agents Phase 1 — unified agent file space: - Swarm oneshot dispatch now runs in the shared workspace root (~/workspace or HERMES_WORKSPACE_DIR) instead of $HOME, and wrapper launchers cd there too, so agent output always lands where the Files page looks. Phase 2 — all-cloud model fleet (swarm.yaml): - builder -> qwen3-coder:480b, strategist -> kimi-k2-thinking, researcher -> qwen3.5:397b, ops-watch -> ministral-3:8b, qa/maintainer/km-agent/inbox-triage -> deepseek-v4-flash. All verified live against ollama-cloud with per-session model stickiness checks. Phase 3 — performance: - Code-split 12 eager routes (three.js/recharts/HermesWorld no longer in first paint); throttle 2s/3s chat pollers to 5s/8s. Phase 4 — repairs: - Fix zsh-reserved `status` variable in the tmux launch command (it aborted every persistent worker launch under zsh) and export a PATH with ~/.local/bin so the Hermes TUI finds node under launchd. - /api/memory: migrate off the nonexistent gateway endpoint to the local filesystem memory browser (hermes-agent 0.17.0 has no /api/memory). - /api/commands: return an empty command list on gateway 404 instead of erroring, so the composer falls back to built-in slash commands. - Repair the full test suite (33 failed -> 0): align MCP store tests with the getStateDir() workspace subdirectory via HERMES_WORKSPACE_STATE_DIR, update hub-search tests for the offset param and 500 limit clamp, update chat-message-list tests to post-outsourc-e#543 attach behavior and drop tests for a never-merged helper, refresh i18n zh label, exclude e2e/ from vitest. Phase 5 — roster grows to 14: - New agents: release-agent (merge-to-ship), security-auditor (scheduled sweeps), quant-agent (trading pipeline steward), concierge (daily digest, greenlight queue). Profiles, wrappers, and greenlight gates included. Verified end-to-end: dispatch -> persistent tmux delivery -> agent file write lands in ~/workspace and appears in the Files page; 725/725 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…weep cron, cost meter, per-profile creds Six control-plane upgrades toward a self-managing AgenticOS: A. Blocked-item management — unblockMissionAssignment (retry/dismiss) and clearAllBlocked in swarm-missions, POST actions on /api/swarm-missions, Retry/Dismiss buttons and a Clear-all control in the swarm reports view. Blocked assignments no longer accumulate forever. B. Permission modes — per-worker approvals.mode control (ask/smart/auto/yolo) via new /api/swarm-permissions + swarm-permissions server module; badge selector on worker cards and a global all-workers control. Oneshot dispatch honors it: only explicit 'ask' keeps Hermes approval prompts. C. Model router — dispatch-time tier classification (light/standard/heavy/ reasoning -> ministral-3:8b / deepseek-v4-flash / qwen3-coder:480b / kimi-k2-thinking) with per-worker modelTiers bands in swarm.yaml, one automatic tier escalation on oneshot failure, and /model switching for live TUI sessions. Verified live: "ping" ran on ministral-3:8b, an implementation task ran on qwen3-coder:480b. Disable with HERMES_SWARM_MODEL_ROUTER=0. D. Lifecycle sweep — swarm-lifecycle-sweep.sh fixed (port, cookie auth from workspace-sessions.json) and scheduled via launchd every 10 minutes (com.hermes.swarm-sweep), so token-pressure handoffs and worker renewal run unattended. E. Cost meter — /api/swarm-usage aggregates per-worker token usage (today + 7d, per model) from profile state.db files; usage panel on the swarm screen. F. Credential split — swarm-split-creds.sh replaces the shared .env symlink in every worker profile with a minimal per-profile .env (mode 600): inference key everywhere, provider extras only where needed, no Discord tokens in any worker. 755/755 tests green (30 new). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, desktop orphan reaper Knowledge/Memory "slow to show up" was a frontend issue — the APIs respond in <3ms, but the global QueryClient had no staleTime so every page refetched on every mount and re-flashed a loading state. Set sane defaults (staleTime 30s, gcTime 5m, no refetch-on-focus) so revisits render from cache instantly. Obsidian: the knowledge base already reads a local markdown folder, which is exactly what an Obsidian vault is. Created ~/workspace/vault (seeded, Obsidian- recognizable), pointed the knowledge source at it (now shows pages + wikilink graph), and baked OBSIDIAN_VAULT_PATH/HERMES_KNOWLEDGE_VAULT into the km-agent and concierge profiles via swarm-split-creds.sh so agents read/write it. Graceful API states: - /api/swarm-missions: implement the mark_ready_for_eric action the reports UI already calls (was rejected as "Unsupported action") — flags a blocked item as escalated to the operator. - /api/sessions/:key/status: return 404 for a missing session instead of 500. - /api/external-memory candidates+search: return an empty result set when no provider is configured instead of 500. Desktop orphan reaper: swarm-lifecycle-sweep.sh now kills leaked `hermes ... dashboard --port 0` profile backends when the Hermes Desktop app isn't running (works around the upstream force-quit process-orphan bug). tsc unchanged (no new errors); mcp + swarm-missions suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The core "session timed out" + false-blocked symptom: dispatchBlockReason marked an assignment BLOCKED whenever the checkpoint poll timed out, even when the dispatch itself succeeded (result.ok === true, process exited 0, files written). Workers were running tasks to completion but the mission recorded them as blocked because the agent didn't emit a structured checkpoint block. - dispatchBlockReason now only blocks genuine failures (!result.ok). A successful-but-uncheckpointed run is no longer a block. - Oneshot success without a parsed checkpoint synthesizes a DONE checkpoint from the worker output, so the assignment records as completed instead of hanging in limbo. - Live-tmux timeout is reframed as "delivered; worker still running" — the agent is working async in the pane, not failed. Also fixes a corrupted reviewer profile (provider had reverted to `deepseek` + api.deepseek.com with no key, causing silent fallback to llama3.1:8b) back to ollama-cloud. Verified: a real file-writing task now runs to completion (file created), reports ok, and records zero blocked assignments where the old path recorded a false block. 755/755 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ovider button; ops playbook Full workspace sweep (UI buttons + all 14 workers + backend APIs). All 14 workers dispatch and respond; 0 blocked. Fixes for the issues found: Provider drift (the recurring one) — hermes-agent's TUI rewrites a worker's model.provider from the bare model name to deepseek/custom/openrouter (none keyed), causing a silent fallback to llama3.1:8b. Killed at the root, three layers: - swarm-model-resolver: map the ollama-cloud catalog (deepseek-v4-*, qwen3-*, kimi-*, ministral-*, glm-*, gpt-oss) to provider `ollama-cloud`. - swarm-profile-config.syncSwarmProfileModel: now clears a stale provider- specific base_url (e.g. api.deepseek.com) and heals even when provider/ default already match. - swarm-dispatch: re-assert the roster model's canonical provider on EVERY dispatch (not just tmux-start), so drift self-heals. - swarm-model-router: emit provider-qualified ids `ollama-cloud/<model>` for -m and /model so the CLI can't re-guess the provider. Verified: corrupt a config to provider:deepseek -> next dispatch heals it to ollama-cloud, TUI loads the correct model, no llama fallback. Other fixes from the sweep: - /api/models: upstream /v1/models 401 no longer discards the local catalog and 503s the endpoint (now 200). - Settings remove-provider button: server rejected the `remove-provider` patch action -> added the action, store handler, and auth-profile cleanup. Docs: new SWARM-OPERATIONS-PLAYBOOK.md — durable handoff for cloud sessions (recurring bugs, exact fix commands, build/verify/ship). Linked from AGENTS.md. 755/755 tests, tsc unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… wipe blocked board Adds a Clear All control in the orchestrator card header, between Router and the Settings gear. One click (with confirm) gives a clean slate: - cancelAllSwarmMissions: cancel every non-terminal assignment across all missions. - clearAllBlocked: wipe the blocked/needs-input board. - reset each affected worker's runtime + kill its tmux session, so nothing keeps executing. New POST action `cancel-all` on /api/swarm-missions. Verified live: 14 missions/assignments cancelled, 14 tmux sessions killed (14 -> 0), 0 blocked, 0 active afterward. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sh session The cancel-all action now clears every worker profile's chat/session store, not just cancelling tasks. For each of the 14 worker profiles it moves state.db aside to a single recoverable `state.db.cleared.bak` (overwriting any prior backup) and removes the WAL/SHM sidecars — hermes recreates an empty store on next launch. Moving rather than deleting in-place avoids FTS5 corruption and keeps one recovery point. Verified live: Clear All cancelled tasks, killed all tmux sessions, and cleared chats for all 14 workers; the backup retained prior history, the next dispatch started a fresh session, and the worker responded normally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ssions Tier-1 completeness pass. TUI completion detection — a live-TUI dispatch that finished without emitting a structured checkpoint no longer hangs as `executing`. On checkpoint-poll timeout the pane is read (readIdleTuiReply): if the TUI is back at an idle `ready` status line with no active-work markers, a DONE checkpoint is synthesized from the reply text; if still busy, the soft timeout stays. Verified: tmux-delivered task flipped to DONE with the agent's real reply. Spend cap — evaluateSpendCap gates the dispatch POST: once the day's total tokens (summed from the usage aggregator) cross HERMES_SWARM_DAILY_TOKEN_CAP, dispatch returns 429 and pauses. Disabled by default (cap 0/unset). Status is exposed on /api/swarm-usage as `spendCap` for the UI. Scheduled agents — the new roles now do recurring work: scripts/swarm-scheduled-mission.sh dispatches a mission to a worker (cookie auth from workspace-sessions.json); scripts/swarm-install-schedules.sh installs launchd timers — security-auditor 02:00 (dep/secret/header sweep), quant-agent 07:00 (market brief), concierge 08:00 (digest + greenlight queue). Installed and smoke-tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…scord digest Backups — scripts/hermes-backup.sh + com.hermes.backup launchd (03:00) tar the irreplaceable runtime state not in git (.runtime missions/kanban, the Obsidian vault, agent memory) to ~/hermes-backups, keeping the newest 14. Smoke-tested. Health strip — swarm-health summary.warnings now surfaces the recurring operational issues before they bite: provider drift (workers on deepseek/ openrouter/custom), missing launcher wrappers, missing profile dirs. The strip already renders warnings. Verified: corrupting a provider raised the drift warning; a dispatch self-healed it. Discord digest — scripts/hermes-discord-digest.sh + com.hermes.discord-digest launchd (08:05) posts a swarm digest to Discord via the hermes bot: active / blocked counts, the greenlight queue (items awaiting the operator), health warnings, and today's token spend vs cap. Channel comes from ~/.hermes/.env with auto-discovery when the configured home channel is stale (it was). Secrets never printed. Delivered live (HTTP 200). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s to the swarm
Echo Studio was a fake demo (handleCreate incremented a counter; nothing
persisted). Now it's a working natural-language tool builder wired to the swarm:
- New echo-pages store (.runtime/echo-pages.json, atomic) + /api/echo-pages
route (GET list, POST create/update/delete).
- Create: persists the page spec, then dispatches a build mission to the
builder worker ("build a workspace tool page that implements: <prompt>") and
marks the page `building`. The page id is now optional (derived from title).
- Manage tab: lists real persisted pages with live status badges
(draft/building/ready/failed) and a delete action, replacing the empty
placeholder.
- Stats cards now reflect real counts.
Verified end-to-end: create → persisted (draft) + builder dispatch returned →
update → building → delete → gone.
Note: the skills/operations "coming soon" buttons are left as-is — they are
honestly labeled unbuilt features (skill install/hub/export), not dead-ends.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e Greek gods Agora was a local mock (fake drifting users + scripted chatter). Now it's a real multiplayer lobby: - agora-presence-server.mjs: a WebSocketServer attached to the workspace http server on /api/agora-ws (in server-entry.js). In-memory room, no DB. Relays presence (position/facing) and chat between connected peers, broadcasts join/leave, sends the roster to newcomers. Cookie-auth on upgrade, mirroring the app's password rule (open when no HERMES_PASSWORD/CLAUDE_PASSWORD). - use-agora-room.ts: replaced the fake drift/chatter with a live WS client — broadcasts self position (~16/s) + chat, applies incoming peers/moves/chat. Same return interface, so the world/chat/avatar components are unchanged. Auto-reconnects; degrades to "you alone" if the WS can't connect (e.g. vite dev) instead of crashing. - use-agora-profile.ts: new arrivals spawn as a random Greek god (Hermes, Athena, Apollo, Artemis, Iris, Nike, Eros, Pan, Chronos) with a matching name + portrait, instead of everyone defaulting to Hermes. Verified live: two WS peers saw each other join and chat crossed between them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two gaps in the Clear All (cancel-all) flow: 1. In-flight oneshot dispatches kept running. cancel-all cancelled the mission records + killed tmux, but a spawned `hermes chat -q … --source swarm-dispatch` child ran to completion. It now pkills those. The marker `--source swarm-dispatch` was moved to the FRONT of the argv (before the multi-thousand-char -q prompt) because macOS pkill -f only matches within a ~4KB command-line window — the trailing marker was invisible, so the kill silently matched nothing. Verified: a real dispatch process goes 1 → 0. 2. Agent chat logs were not cleared. Clear All moved state.db aside but left logs/agent.log (the log surfaced in the runtime/TUI view) intact, so old chat stayed visible. clearWorkerChatHistory now also truncates agent.log, errors.log, tui_gateway_crash.log, and swarm-dispatch-startup.log. Verified: qa agent.log 260K → 4K. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ecurring timeout and chat-comeback bugs
Two intermittent failures with the same signature ("works the first time,
breaks the next"):
1. Dispatch timeout after Clear All. ensureLiveTmuxSession waited a fixed
1.2s after launching the TUI, but a cold TUI start can spend 30-60s on
"Installing TUI dependencies…". The prompt was pasted into a still-booting
pane, vanished, no checkpoint ever arrived → "session timed out". Warm
sessions masked it, which is why the first use after boot worked and the
first use after Clear All (all sessions cold) didn't. The launch path now
POLLS the pane for the ready prompt (or exit sentinel) up to 90s and only
then allows delivery; if not ready it returns an explicit error and the
dispatch falls back to oneshot.
2. Chats coming back after Clear All. `tmux kill-session` returns before the
hermes process inside dies, and hermes FLUSHES/RECREATES state.db on
shutdown — confirmed live: state.db reappeared 4s after being moved aside.
Clearing before death silently undid the clear. cancel-all now kills all
tmux sessions AND in-flight oneshots first, then waits for the worker
processes to actually die (pgrep poll, SIGKILL escalation after 8s), and
only then clears state.db + logs. Also removes the backup's WAL/SHM
sidecars.
Verified: 3 consecutive use → Clear All → use cycles; every dispatch
delivered + checkpointed, every clear left 0 processes, 0 tmux sessions,
0 chat sessions, empty logs. Also fixed the 3 stale test fixtures missing
modelTiers (tsc 98 → 95).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four API routes were reachable without the shared local-or-auth gate: - /api/swarm-kanban (GET/POST/PATCH) — board read/write - /api/events — SSE stream leaked live chat events - /api/playground-npc — open LLM proxy via the gateway - /api/playground-admin — relied on the client-controlled Host header All now use requireLocalOrAuth; playground-admin's spoofable Host check replaced with the same gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er learning, skill harvesting, nightly self-improvement - Outcome memory: every dispatch appends a JSONL record (.runtime/swarm-outcomes.jsonl) via finalizeDispatch; recent failure lessons are injected into the worker's next prompt. - Scoreboard: /api/swarm-scoreboard + dashboard card with per-worker success rates, blocks, and average duration (per-tier stats included). - Router learning: a tier with <40% success over 3+ recent attempts for a worker is escalated up-front, clamped to the worker's allowed band. - Skill harvesting: DONE checkpoints with concrete evidence become vault skill notes (~/workspace/vault/skills); keyword-matched skills are injected into future dispatch prompts. - Nightly self-improvement: com.hermes.swarm.self-improve (01:00) gathers tsc/test/health/scoreboard evidence and dispatches the maintainer to stage one reviewable fix — never pushes or merges; operator greenlights via the morning digest. All hooks are best-effort and never fail a dispatch. 765 tests pass (10 new); tsc unchanged at the 95-error baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first live run of the nightly job had the maintainer run 'git checkout -b' inside the live workspace repo, switching the operator's working branch out from under an in-flight session. The mission prompt now requires 'git worktree add' under ~/workspace/nightly-fixes/ and forbids checkout/switch in the live repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Discord bot (scripts/hermes-discord-bot.mjs, launchd KeepAlive com.hermes.discord-bot): - REST polling, zero new dependencies; auto-discovers a reachable text channel when the configured one is stale. - Read-only commands for anyone in the channel: !status, !blocked, !help. - Mutating commands (!dispatch, !retry, !dismiss, !clearblocked) require DISCORD_OPERATOR_ID in ~/.hermes/.env to match the message author; without it the bot stays read-only. - Talks only to the local workspace API with the claude-auth session cookie; secrets never printed. Voice → orchestrator: mic button in the swarm RouterChat records speech and appends the transcript to the mission prompt (server /api/transcribe when a remote STT provider is configured, browser SpeechRecognition otherwise). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…po branch A worker twice ran 'git checkout -b' inside the live workspace repo, switching the operator's branch under running sessions. Layers: 1. Every dispatched worker prompt now forbids state-changing git (checkout/switch/reset/rebase/branch -f) in the live repo and mandates isolated worktrees under ~/workspace/worker-trees/. 2. swarm-self-improve.sh pre-creates the nightly worktree + branch itself, so the maintainer has no reason to run branch commands at all. 3. The 10-minute lifecycle sweep gained a branch guard: if the live repo's branch differs from .runtime/expected-branch, it auto-restores via 'checkout -m' when the found branch is worker-made (nightly/*, worker/*) and posts a Discord alert either way. Operators switching intentionally update .runtime/expected-branch. Verified live: created nightly/test-guard, ran the sweep, branch was restored automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Watchdog (scripts/hermes-watchdog.sh, launchd com.hermes.watchdog every 5 min): - Probes workspace/gateway/dashboard HTTP and the discord-bot process; restarts a down service once via launchctl kickstart and posts a Discord alert; escalates if it is still down the next cycle. - Low-disk alert (<10 GB free, at most daily) and wedged-worker alert (state executing with no output for >2h, at most hourly) — never auto-kills in-flight work. - Verified live: killed the dashboard, watchdog detected it, attempted restart and posted the alert; healthy runs keep counters at zero. Unified timeline: - /api/swarm-timeline merges mission events, dispatch outcomes, scheduled agent runs, and lifecycle-sweep/branch-guard rows into one newest-first feed (filter by worker or source). All readers best-effort; a corrupt file never breaks the feed. - Timeline panel on the swarm cards view with per-source filter chips. - Branch-guard log rows now carry an epoch timestamp for accurate ordering. Multi-machine workers intentionally deferred. 766 tests pass; tsc at the 95-error baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ette upgrade, vault drop zone - Local semantic memory: ollama nomic-embed-text index over vault/playbook/ outcomes/handoffs; dispatch injects relevant memories into worker prompts; /api/rag search + reindex; keyword fallback when ollama is down - Priority task queue (.runtime/swarm-queue.json): P1-P3, auto-assign to idle workers, sweep drains 2/cycle; UI panel, Discord !queue/!queuelist, API - Morning briefing: digest script now includes overnight timeline stats, queue depth, weak workers, watchdog tail, disk, calendar; moved to 07:30 - Command palette: Swarm/Dashboard/Tasks screens, queue-task and RAG-reindex actions, live semantic search results - Vault drop zone: drag files anywhere -> vault/inbox + companion note + async reindex (/api/vault-ingest) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…loud backup, weekly report - Router demotes a worker one model tier when its record at the lower tier is deep and strong (>=5 attempts, >=80% success); escalation still wins - server-entry gzips compressible static assets with an in-memory cache: 2.26MB main chunk -> 684KB over the wire (note: manualChunks crashes rollup under TanStack Start multi-env build — documented in playbook) - /api/swarm-pipeline: staged dispatch (research->build->QA) threading each stage's checkpoint results into the next stage's prompts - hermes-backup.sh mirrors archives to iCloud Drive HermesBackups/ (keep 7) - hermes-weekly-report.sh: Sunday 18:00 self-retro to vault/reports + Discord summary (com.hermes.weekly-report) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… reset, dispatch pause, per-worker Stop
Operator-reported: worker stuck 'reviewing' for 5h and Clear All seemed to
do nothing. Three root causes fixed:
- Zombie runtimes: tmux worker dies without checkpoint -> runtime.json says
busy forever. New reapZombieSwarmRuntimes() resets busy workers with no
tmux session and no output for 30+ min; lifecycle sweep calls it every
cycle via POST /api/swarm-runtime/reset {action:'reap'}
- Reset undone by live session: per-worker runtime reset now kills the
swarm-<id> tmux session BEFORE writing runtime.json, so the TUI monitor
can't rewrite the state back to executing
- Board refills after Clear All: cancel-all now writes a 10-minute
.runtime/dispatch-pause-until; queue drain and scheduled missions honor
it (manual dispatches unaffected)
- New per-worker Stop (x) button on each active-agent card to kill+reset a
single stuck worker without clearing the whole board
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oposals The brain-quality tier: the swarm now verifies its own work, decomposes goals into pipelines autonomously, and proposes its own tasks. - Auto-verification (swarm-verify.ts): every non-trivial DONE checkpoint auto-queues a P1 adversarial [verify] task for the reviewer, who must end with VERDICT: CONFIRMED/REFUTED. Refuted -> Discord alert. Loop-safe, deduped per claim, HERMES_SWARM_AUTO_VERIFY=0 to disable - Standing goals (swarm-goals.ts, /api/swarm-goals): operator states a goal; each sweep cycle the engine advances one goal: strategist plans a JSON pipeline (oneshot, planner-not-doer prompt) -> pipeline runs -> strategist assesses -> done or next iteration, bounded by maxIterations. Discord !goal/!goals. Live-tested end-to-end: plan -> pipeline -> assess -> DONE - Proactive proposals (swarm-suggest.ts): deterministic scanners (recurring block reasons, weak workers, frequent zombie reaps) file queue items as 'proposed'; approve via queue panel, !approve, or morning briefing. Max 5 open, fingerprint-deduped, never auto-dispatched - Dispatch gains per-assignment oneshot flag (live TUI first-checkpoint is an ack — planners need the full synchronous answer) - Queue 'proposed' status; briefing shows proposals + active goals Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tions watcher Phone push (important events only): new src/server/notify.ts sends via ntfy.sh to a private topic (HERMES_NTFY_TOPIC in ~/.hermes/.env). Wired into refuted verifications, goal done/failed, blocked workers (throttled), and watchdog alerts. Unset topic disables cleanly. GitHub reach: scripts/hermes-github-watch.sh polls the notifications API with the keychain token each lifecycle sweep, dedupes via state file, posts all new items to Discord and pushes only review requests, mentions, security alerts, and assignments to the phone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One-time consent via scripts/hermes-gmail-auth.mjs (fixed-port loopback, token stored mode-600 in ~/.hermes/google-token.json). Sweep-driven scripts/hermes-gmail-watch.mjs posts new unread to Discord and pushes Gmail-important/starred mail to the phone. Deduped via state file; HERMES_GMAIL_WATCH=0 disables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tall Every dispatch now saves its full transcript to .runtime/swarm-replays.jsonl (capped). New /replay screen lists recent dispatches with a click-through transcript view and a per-day activity chart aggregated from outcome records. scripts/hermes-install.sh bootstraps or repairs a machine: prereq checks, build, launchd fleet load, and an operator checklist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New /api/swarm-events streams state-change events (1s file-mtime watcher over runtime/queue/goals/pipelines/missions/outcomes). The swarm board subscribes via useSwarmEvents and invalidates react-query caches on push, so updates land in ~1s; the old 30-60s polls remain as fallback. Queue approve/cancel now update the UI optimistically and reconcile on settle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Busy cloneable roles (builder, researcher, qa, maintainer, km-agent) now spawn clone workers (builder-2, builder-3) on demand during queue drain: a filtered copy of the base profile plus a roster entry whose wrapper and profile point back at the base. Caps: HERMES_MAX_CLONES per role (default 3) and HERMES_MAX_PARALLEL busy workers globally (default 4). Verification exemption now resolves clone ids to their base role. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t, scout) scribe: notes/transcripts to vault docs + action items (km-agent profile). data-analyst: weekly outcomes/timeline trend report, Sundays 17:30. archivist: nightly vault gardening + RAG reindex, 02:30, max 20 changes. scout: daily 07:00 web briefing with sources to vault/reports/. Profiles cloned from km-agent/researcher; launchd schedules loaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lates
POST /api/nl-command routes plain English to the right primitive:
heuristics catch 'goal:', 'ask <worker> to …', and questions; everything
else is classified by a strategist oneshot into queue/goal/answer (queue
verbatim as the safe fallback). Workflow templates are saved parameterized
pipelines ({{input}} substitution) seeded with ship-feature,
security-audit, and deep-research; runnable from the API, the ⌘K palette
(Do:/Ship feature/Deep research), and Discord (!do, !templates,
!template <id> <input>).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Failed goals (iteration cap) and failed goal pipelines now queue a researcher autopsy that writes a root-cause note to vault/autopsies/, feeding RAG so the swarm learns from its own failures. The lifecycle sweep runs a daily anomaly check (failure-rate regression or dispatch volume spike vs the trailing week) with at most one phone push per day. The weekly report gains a fixed 3-task self-benchmark through the real dispatch path to catch silent worker-quality regressions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every mutating API call (dispatch, queue actions, goal create/pause, runtime reset, template run, nl-command) appends to an HMAC hash-chained audit log (~/.hermes/logs/audit.jsonl, key file 600); editing, truncating or reordering entries breaks the chain, detectable via GET /api/audit-log?verify=1. scripts/hermes-security-scan.sh runs Mondays 08:00: secret file permission enforcement (auto-fix to 600), unexpected 0.0.0.0 listeners, token-shaped strings in tracked files and logs, audit chain integrity, and Tailscale Funnel exposure — findings go to Discord + phone. Route auth sweep confirmed every API route checks auth or is an intentionally public rate-limited endpoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eline checkpoints Queue drain: resolveWorkerForRole now takes an exclude set so two tasks for the same role in one drain go to base + clone instead of the second being skipped (live-tested: builder + builder-2 ran in parallel). Pipelines: stage assignments dispatch oneshot — a live TUI session can return a stale checkpoint file from a previous task instantly, making a stage look complete with old results (reviewer verdict from an earlier run surfaced in a fresh audit). Stage summaries also prefer full output over the clip-prone checkpoint parse, matching the goals-engine fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A pipeline run executes in the dispatching process's memory; restarting the server mid-run leaves it in 'running' forever with no consumer for its checkpoints (hit during shakedown when a template POST raced a kickstart). listPipelineRuns now fails-out runs older than the theoretical max runtime with an explicit orphaned error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'⏸ Stop everything' on the dashboard disables every com.hermes.* launchd job except the workspace UI (disable persists across reboots), kills all swarm tmux sessions, and pauses automated dispatch for 7 days. The button flips to '💤 Hibernating — Wake', which re-enables and restarts the whole fleet and clears the pause. Backed by scripts/hermes-hibernate.sh and an audited /api/hibernate endpoint. Live-tested both directions: 17 services down to 1 and back to 17, all worker sessions killed on stop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the agent Swarm into a self-managing control plane and fixes a broad set of workspace issues found across a full audit. Four stacked commits.
1. Swarm worker lifecycle (
53ed7822)statusreserved-word + PATH sohermes chat --tuilaunches under launchd.agent init→ permanent BLOCKED).2. Unified file space + cloud fleet + new agents (
f1d6c892)~/workspace) instead of scattering across$HOME.base_urloverrides; fallback chains.3. Control-plane upgrades (
e304cc3c)4. Perf + Obsidian + graceful states (
dd2b4686)mark_ready_for_ericaction implemented; session-status 404 not 500; external-memory graceful empty; desktop orphan-process reaper.Testing
tsc --noEmitintroduces no new errors.~/workspace).🤖 Generated with Claude Code