From 0deccf08de8540c023cafe3bc22de9fca71b26e4 Mon Sep 17 00:00:00 2001 From: ruby-dlee Date: Fri, 24 Jul 2026 13:00:48 -0400 Subject: [PATCH 1/9] feat: launch crews from direct account directories (#24) * Launch crews from account directories * Make bootstrap backend tests hermetic * no-mistakes(review): Fix direct account routing and secondmate recovery * no-mistakes(review): Complete direct account routing cutover fixes * no-mistakes(review): Preserve direct recovery context and actionable diagnostics * no-mistakes(review): Harden direct recovery identity and retained endpoint cleanup * no-mistakes(review): Harden direct recovery worktree identity validation * no-mistakes(review): Harden direct routing recovery identity and retained state * no-mistakes(review): Finalize direct recovery identity and cleanup lifecycle * no-mistakes(review): Defer direct secondmate sync to guarded recovery * no-mistakes(review): Restore legacy secondmate routing scope * no-mistakes(test): Fix legacy continuation missing-brief fixture * Fix account launch shell lint * no-mistakes(review): Preserve retained direct recovery state during teardown * no-mistakes(document): Captain: document direct account routing cutover --- .agents/skills/bootstrap-diagnostics/SKILL.md | 2 +- AGENTS.md | 28 +- README.md | 4 +- bin/fm-account-directory.sh | 398 +++++++ bin/fm-account-routing-lib.sh | 12 +- bin/fm-bootstrap.sh | 39 +- bin/fm-dispatch-select.sh | 54 +- bin/fm-spawn.sh | 1061 +++++++++++++++-- bin/fm-teardown.sh | 78 ++ docs/architecture.md | 10 +- docs/configuration.md | 71 +- docs/scripts.md | 5 +- tests/fm-account-directory.test.sh | 915 ++++++++++++++ tests/fm-account-routing.test.sh | 97 +- tests/fm-bootstrap.test.sh | 92 +- tests/fm-dispatch-select.test.sh | 35 +- tests/fm-teardown.test.sh | 74 ++ 17 files changed, 2679 insertions(+), 296 deletions(-) create mode 100755 bin/fm-account-directory.sh create mode 100755 tests/fm-account-directory.test.sh diff --git a/.agents/skills/bootstrap-diagnostics/SKILL.md b/.agents/skills/bootstrap-diagnostics/SKILL.md index 584ec6022de..b2644e0f110 100644 --- a/.agents/skills/bootstrap-diagnostics/SKILL.md +++ b/.agents/skills/bootstrap-diagnostics/SKILL.md @@ -22,7 +22,7 @@ The inline rules in `AGENTS.md` section 3 still bind: detect, then consent, then For `quota-axi`, bootstrap requires it because crew-dispatch `quota-balanced` may call it; `bin/fm-dispatch-select.sh` still degrades at runtime when quota data is unavailable. - `MISSING_MANUAL: (instructions: )` - tell the captain why the tool is required and give them the printed instructions URL, but do not pass the tool to `bin/fm-bootstrap.sh install`; wait for the captain to complete the manual installation, then rerun session start to confirm the dependency is present. - `BACKEND_INVALID: (known: )` - the resolved runtime backend has no verified dependency or lifecycle contract, so do not dispatch work until the invalid `FM_BACKEND` or `config/backend` value is corrected to one of the listed backends. -- `ACCOUNT_ROUTING: invalid routing policy - ` - the environment or `config/account-routing-mode` cannot resolve to exactly one of `off`, `observe`, or `enforce`; fix the reported source before dispatch because managed spawns will fail closed on the same policy error. +- `ACCOUNT_ROUTING: invalid routing policy - ` - the environment or `config/account-routing-mode` cannot resolve to exactly one of `off`, `observe`, or `enforce`; fix the reported source before dispatch because routed spawns will fail closed on the same policy error. - `NEEDS_GH_AUTH` - ask the captain to run `! gh auth login` (interactive; you cannot run it for them). - `TANGLE: ` - the primary checkout is stranded on a feature branch instead of its default branch; `AGENTS.md` section 8 explains why this guard exists and what it protects. The work is safe on that branch ref; restore the primary to its default branch with the printed `git -C checkout `, then re-validate that branch in a proper worktree. diff --git a/AGENTS.md b/AGENTS.md index 9427247d769..cfd3369ea01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,8 +79,9 @@ tools/ independently versioned provider-neutral components, commit config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "default" = same as firstmate. Inherited as the literal file: a concrete primary adapter value also controls a secondmate home's own crewmates (section 4) config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) -config/account-routing-mode optional Agent Fleet routing policy (`off`, `observe`, or `enforce`); LOCAL, gitignored; default off; inherited by secondmate homes (docs/configuration.md "Agent Fleet account routing") +config/account-routing-mode optional account routing policy (`off`, `observe`, or `enforce`); new ship/scout observe/enforce launches use direct account directories, secondmate launches retain legacy Agent Fleet routing, and existing managed metadata retains legacy Agent Fleet recovery; LOCAL, gitignored; default off; inherited by secondmate homes (docs/configuration.md "Agent Fleet account routing") config/secondmate-account-pool optional Agent Fleet pool the PRIMARY uses for SECONDMATE launches when routing is enabled; LOCAL, gitignored; selection-only and NOT inherited +Direct account-directory launch currently covers ship/scout crewmates only; secondmate integration is deferred to a dedicated follow-up. config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = default tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend, herdr/zellij/cmux are experimental new-task spawn backends, and Orca is legacy-recovery-only (docs/tmux-backend.md, docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and Orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; not inherited into secondmate homes config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") @@ -102,7 +103,12 @@ state/ volatile runtime signals; gitignored .status appended by crewmates: ": " wake-event lines, not current-state truth .turn-ended touched by turn-end hooks .grok-turnend-token firstmate-owned grok hook registry token for the task; removed by teardown - .meta written by fm-spawn: window=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, generation_id=, report_required=; managed tasks add account_pool=, account_profile=, account_task=, account_attempt=, and provider_session_id= (docs/configuration.md "Agent Fleet account routing"); kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, appends pr= and GitHub's pr_head= when available; fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) + .meta written by fm-spawn: window=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, generation_id=, report_required= + Direct ship and scout launches also own account_home=, worktree_git_dir=, worktree_git_dir_identity=, and exactly one authoritative final-state field: worktree_git_ref= for an attached branch or worktree_git_head= for an intentional detached HEAD. Their metadata may temporarily carry worktree_git_setup_ref= and worktree_git_setup_head= while the brief's required `fm/` branch transition is pending; recovery accepts only that exact setup state or the authoritative `fm/` ref and removes the setup fields after adoption. + Direct recovery validates the canonical worktree path, exact physical Git-dir identity, and authoritative final state before account preparation and again immediately before endpoint creation; any drift fails closed without launching. + If endpoint removal after a failed new direct spawn cannot be confirmed, direct_spawn_cleanup=pending and rollback_pending=1 retain the endpoint and worktree identity for explicit teardown. + Secondmate Agent Fleet routing and legacy managed recovery own account_pool=, account_profile=, account_task=, account_attempt=, and provider_session_id= (docs/configuration.md "Agent Fleet account routing"). + kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, appends pr= and GitHub's pr_head= when available; fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) .check.sh optional slow poll you write per task (e.g. merged-PR check) x-watch.check.sh generated X-mode relay poll shim; present only when opted in (section 14) x-inbox/ generated X-mode pending mention payloads; fmx-respond drains it (section 14) @@ -207,8 +213,9 @@ That refusal is the consultation backstop, so the rules are never silently skipp The requirement is gated only on the file's presence; when the file is absent, `fm-spawn.sh` keeps resolving the crewmate harness from `config/crew-harness` as before. Secondmate launches are exempt because they resolve through `fm-harness.sh secondmate`, not the crewmate dispatch-profile rules. -`quota-balanced` selection is deterministic and owned by `bin/fm-dispatch-select.sh`; its header documents the general-window rules, Agent Fleet pool-summary path, freshness margin, and every fallback, and it degrades to the first array element whenever quota data is unusable. -When candidates carry account pools, the selector uses only Agent Fleet summaries and never falls through to quota-axi's default-account data; pass its selected pool to spawn for the atomic concrete-profile lease. +`quota-balanced` selection is deterministic and owned by `bin/fm-dispatch-select.sh`; every real new ship/scout launch uses direct account-directory selection. +When candidates carry account pools, the selector chooses the ordered first profile and passes its pool to spawn only as a compatibility activation input, never as an account choice or lease request. +The now-unreachable Agent Fleet pool-summary implementation and inactive new-lease fixtures are legacy code deferred to follow-up task `remove-fleet-routing-deadcode`. Quota trouble must never block dispatch. Precedence, highest first: @@ -255,6 +262,8 @@ Reconcile reality with your records before doing anything else, working from the Do not sweep every `fm-*` tmux window, herdr tab, zellij tab, Orca terminal, or cmux workspace across all sessions during recovery; another firstmate home's child endpoints may share that namespace and are not this home's orphans. 5. If the digest reports a recorded direct-report's endpoint as `dead` (or a meta has no `window=`), reconcile it through its meta as described below. 6. For meta with no window, or an endpoint the digest reported dead, reconcile by kind. + If ship/scout meta records `account_home=`, relaunch with `bin/fm-spawn.sh --recover-direct-account`. + That mode reloads the recorded task context and performs fresh direct account-directory selection. If meta records `account_profile=`, first try the exact sticky session with `bin/fm-spawn.sh --resume-account`; when that session/profile is unavailable, use `--continue-account` for the task-owned provider-neutral handoff after re-verifying live and repository state, with `bin/fm-account-continuation.sh` owning the fail-closed packet contract. For ordinary crewmates, check the recorded backend metadata first; use `treehouse status` for treehouse-backed tasks, and the recorded `orca_worktree_id=`/`terminal=` for Orca tasks. For an unmanaged `kind=secondmate`, load `secondmate-provisioning`; its "Recovery" section owns the explicit routing decision and respawn procedure. @@ -429,13 +438,14 @@ Load `harness-adapters` before spawning or recovering any direct report so trust ```sh bin/fm-spawn.sh projects/ # uses the active crewmate harness only when no crew-dispatch.json is active bin/fm-spawn.sh projects/ --harness codex --model gpt-5.5 --effort high # explicit profile axes -bin/fm-spawn.sh projects/ --harness codex --account-pool codex-crew # dynamic Agent Fleet account pool -bin/fm-spawn.sh projects/ --harness claude --account-profile claude-2 # explicit Agent Fleet profile -bin/fm-spawn.sh --resume-account # sticky managed recovery; never a fresh prompt -bin/fm-spawn.sh --continue-account # fresh managed session from verified task-owned continuation state +bin/fm-spawn.sh projects/ --harness codex --account-pool codex-crew # compatibility flag activating direct account-directory selection +bin/fm-spawn.sh projects/ --harness claude --account-profile claude-2 # compatibility flag activating direct account-directory selection +bin/fm-spawn.sh --recover-direct-account # metadata-preserving ship/scout direct account recovery with fresh selection +bin/fm-spawn.sh --resume-account # sticky legacy managed recovery; never a fresh prompt +bin/fm-spawn.sh --continue-account # fresh legacy managed session from verified task-owned continuation state bin/fm-spawn.sh projects/ --backend # explicit new-task runtime backend (docs/configuration.md "Runtime backend") bin/fm-spawn.sh projects/ --scout # scout task; records kind=scout in meta -bin/fm-spawn.sh [] --secondmate # launch or recover a persistent secondmate in its home +bin/fm-spawn.sh [] --secondmate # launch a persistent secondmate in its home bin/fm-spawn.sh =projects/ =projects/ [--scout] # batch: one call, several tasks ``` diff --git a/README.md b/README.md index 064bd811845..d97dbc49ab2 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ Launching a supported harness inside it instantiates your first mate - and makes - **Two task shapes** - ship tasks deliver a change; scout tasks investigate, plan, reproduce, or audit and leave a report. - **Explicit project modes** - each project ships via `no-mistakes`, `direct-PR`, or `local-only`, with an optional `+yolo` autonomy flag. - **Optional secondmates** - opt in to persistent domain supervisors that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, supervising project clones or a project-less firstmate-repo domain, kept on the primary firstmate version by guarded local fast-forwards and checked for live agent processes at session start. -- **Optional multi-account routing** - route Claude and Codex crews through explicit or pooled [Agent Fleet](tools/agent-fleet/) profiles while preserving sticky recovery and provider-neutral continuation. +- **Optional multi-account routing** - select Claude and Codex account directories directly for new ship and scout crews, while retaining [Agent Fleet](tools/agent-fleet/) for secondmates and legacy recovery. - **Durable completion reports** - every new ship and scout task publishes a detailed, visual, searchable report to one machine-global stack through fail-closed teardown. - **Event-driven, zero-token supervision** - a bash watcher sleeps on the fleet and wakes the first mate only when something needs you; verified primary harnesses also get a turn-end backstop that blocks or follows up on a blind stop when work is in flight and supervision is not live. - **Optional X mode** - opt in with one local `.env` token so firstmate can answer your public `@myfirstmate` mentions, act on normal reversible mention requests through the same lifecycle as chat requests, acknowledge spawned work, and post up to three public-safe completion follow-ups within seven days for genuine milestones and the final outcome without changing non-X behavior; dry-run preview records would-be replies and dismissals locally before go-live. @@ -157,7 +157,7 @@ Setup guides for tmux (the default) and the new-task-capable herdr, zellij, and You chat with the first mate. It routes each request to a crewmate in its own session endpoint and git worktree, supervises the fleet with a zero-token event-driven watcher, and brings you finished PRs, approved local merges, or investigation reports. Optional secondmates extend this to persistent domain supervisors, dispatch profiles let you steer which harness handles which task, and an opt-in X mode lets the same fleet answer public mentions. -Optional Agent Fleet routing distributes Claude and Codex work across account profiles, while the completion report stack keeps the result independent of any account or Firstmate home. +Optional account routing selects Claude and Codex account directories directly for new ship and scout crews, while the completion report stack keeps the result independent of any account or Firstmate home. `codex-app` is not a runtime backend yet; [docs/codex-app-backend.md](docs/codex-app-backend.md) owns the Codex App boundary. Full architecture - the supervision engine, worktree isolation, secondmates, dispatch and account routing, completion reports, project modes, optional X mode, fleet sync, and self-update - is in [docs/architecture.md](docs/architecture.md). diff --git a/bin/fm-account-directory.sh b/bin/fm-account-directory.sh new file mode 100755 index 00000000000..3ba66b5462c --- /dev/null +++ b/bin/fm-account-directory.sh @@ -0,0 +1,398 @@ +#!/usr/bin/env bash +# Select and prepare direct Claude or Codex account-directory launches. +# Usage: +# fm-account-directory.sh select +# fm-account-directory.sh install-herdr-hook +# fm-account-directory.sh prepare +# +# This header is the single owner of the direct account-directory contract. +# FM_ACCOUNT_DIRECTORY_CUTOVER: direct-observe-passwd-home-v2 +# Account homes are discovered under the current passwd user's +# .local/share/agent-fleet/accounts// tree without fixed counts. +# Codex selection removes that account's quota-axi window cache immediately +# before every read, sets CODEX_HOME plus the account-isolated XDG_CACHE_HOME, +# accepts only a fresh result with at least one numeric five_hour or weekly +# window, and picks the account with the highest minimum remaining percentage. +# A Codex account with no such freshly readable window is skipped as unhealthy. +# Claude quota is not currently distinguishable per config directory because +# quota-axi cannot non-interactively resolve Claude's config-dir-specific macOS +# Keychain credential. +# Claude therefore never treats a missing usage window as account failure and +# selects the first real account directory in stable bytewise sort order. +# Selection prints only the chosen absolute account home on stdout and logs +# health, fallback, and choice diagnostics on stderr. +# prepare selects the account and idempotently runs Herdr's own integration +# installer with CODEX_HOME or CLAUDE_CONFIG_DIR set to the chosen home. +# It verifies the installed per-profile hook before printing the chosen home. +# +# Credential state is read-only. +# This script never logs in, imports credentials, or invokes a provider model. +# Test-only command, root, passwd-home, Perl, and timeout overrides require +# FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1. +set -u + +TEST_LAB_TOKEN=firstmate-account-directory-test-lab-v1 + +usage() { + sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//' >&2 +} + +log() { + printf 'fm-account-directory: %s\n' "$*" >&2 +} + +test_lab_enabled() { + [ "${FM_ACCOUNT_DIRECTORY_TEST_LAB:-}" = "$TEST_LAB_TOKEN" ] +} + +system_perl() { + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_PERL_BIN:-}" ]; then + printf '%s\n' "$FM_ACCOUNT_DIRECTORY_PERL_BIN" + else + printf '%s\n' /usr/bin/perl + fi +} + +passwd_home() { + local home perl_bin + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_PASSWD_HOME:-}" ]; then + home=$FM_ACCOUNT_DIRECTORY_PASSWD_HOME + else + perl_bin=$(system_perl) || return 1 + [ -x /usr/bin/env ] && [ -x "$perl_bin" ] || { + echo "error: /usr/bin/env and /usr/bin/perl are required to resolve the current passwd home" >&2 + return 1 + } + # shellcheck disable=SC2016 # Perl source is intentionally single-quoted. + home=$(/usr/bin/env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin "$perl_bin" -e ' + my @p = getpwuid($<); + exit 1 unless @p && defined $p[7] && $p[7] =~ m{^/}; + exit 1 if $p[7] =~ /[\x00-\x1f\x7f]/; + print $p[7]; + ' 2>/dev/null) || { + echo "error: cannot resolve the current passwd home" >&2 + return 1 + } + fi + case "$home" in + *$'\n'*|*$'\r'*) + echo "error: passwd home contains a line break" >&2 + return 1 + ;; + /*) ;; + *) + echo "error: passwd home must be absolute: $home" >&2 + return 1 + ;; + esac + [ -d "$home" ] && [ ! -L "$home" ] || { + echo "error: passwd home is not a real directory: $home" >&2 + return 1 + } + printf '%s\n' "$home" +} + +account_root() { + local root home + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_ROOT:-}" ]; then + root=$FM_ACCOUNT_DIRECTORY_ROOT + else + home=$(passwd_home) || return 1 + root=$home/.local/share/agent-fleet/accounts + fi + case "$root" in + *$'\n'*|*$'\r'*) + echo "error: account-directory root contains a line break" >&2 + return 1 + ;; + /*) ;; + *) + echo "error: account-directory root must be absolute: $root" >&2 + return 1 + ;; + esac + printf '%s\n' "$root" +} + +quota_command() { + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_QUOTA_AXI:-}" ]; then + printf '%s\n' "$FM_ACCOUNT_DIRECTORY_QUOTA_AXI" + return 0 + fi + command -v quota-axi 2>/dev/null || { + echo "error: quota-axi is required for fresh Codex account selection" >&2 + return 1 + } +} + +herdr_command() { + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_HERDR:-}" ]; then + printf '%s\n' "$FM_ACCOUNT_DIRECTORY_HERDR" + return 0 + fi + command -v herdr 2>/dev/null || { + echo "error: herdr is required to install the selected account's integration hook" >&2 + return 1 + } +} + +quota_timeout_seconds() { + local timeout=15 + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_QUOTA_TIMEOUT_SECONDS:-}" ]; then + timeout=$FM_ACCOUNT_DIRECTORY_QUOTA_TIMEOUT_SECONDS + fi + case "$timeout" in + ''|*[!0-9]*|0) + echo "error: Codex quota timeout must be a positive integer" >&2 + return 1 + ;; + esac + printf '%s\n' "$timeout" +} + +run_bounded() { + local timeout=$1 perl_bin + shift + perl_bin=$(system_perl) || return 1 + [ -x "$perl_bin" ] || return 127 + # shellcheck disable=SC2016 # Perl source is intentionally single-quoted. + PERL5LIB='' PERL5OPT='' "$perl_bin" -e ' + use POSIX qw(setpgid WNOHANG); + my ($timeout, @command) = @ARGV; + my $pid = fork(); + exit 125 unless defined $pid; + if ($pid == 0) { + setpgid(0, 0); + exec {$command[0]} @command; + exit 127; + } + setpgid($pid, $pid); + my $deadline = time() + $timeout; + while (1) { + my $waited = waitpid($pid, WNOHANG); + if ($waited == $pid) { + my $status = $?; + exit(($status & 127) ? 128 + ($status & 127) : ($status >> 8)); + } + exit 125 if $waited == -1; + if (time() >= $deadline) { + kill "TERM", -$pid; + for (1 .. 4) { + select undef, undef, undef, 0.05; + exit 124 if waitpid($pid, WNOHANG) == $pid; + } + kill "KILL", -$pid; + waitpid($pid, 0); + exit 124; + } + select undef, undef, undef, 0.05; + } + ' "$timeout" "$@" +} + +valid_account_home() { # + local vendor_dir=$1 candidate=$2 name + [ -d "$candidate" ] && [ ! -L "$candidate" ] || return 1 + case "$candidate" in + "$vendor_dir"/*) ;; + *) return 1 ;; + esac + name=${candidate##*/} + case "$name" in + ''|.*|*[!A-Za-z0-9._-]*) return 1 ;; + esac +} + +first_account_home() { # + local vendor=$1 root vendor_dir candidate + root=$(account_root) || return 1 + vendor_dir=$root/$vendor + [ -d "$vendor_dir" ] && [ ! -L "$vendor_dir" ] || { + echo "error: no account-directory root for $vendor at $vendor_dir" >&2 + return 1 + } + LC_ALL=C + export LC_ALL + for candidate in "$vendor_dir"/*; do + valid_account_home "$vendor_dir" "$candidate" || continue + printf '%s\n' "$candidate" + return 0 + done + echo "error: no account directories found for $vendor under $vendor_dir" >&2 + return 1 +} + +fresh_codex_usage_json() { # + local account_home=$1 quota_bin=$2 cache_home cache_file environment_name timeout status + timeout=$(quota_timeout_seconds) || return 1 + cache_home=$account_home/.agent-fleet-quota-cache + cache_file=$cache_home/quota-axi/quotas.json + if { [ -e "$cache_home" ] || [ -L "$cache_home" ]; } \ + && { [ ! -d "$cache_home" ] || [ -L "$cache_home" ]; }; then + log "codex account $account_home skipped: its quota cache root is not a real directory" + return 1 + fi + if { [ -e "$cache_home/quota-axi" ] || [ -L "$cache_home/quota-axi" ]; } \ + && { [ ! -d "$cache_home/quota-axi" ] || [ -L "$cache_home/quota-axi" ]; }; then + log "codex account $account_home skipped: its quota-axi cache directory is not a real directory" + return 1 + fi + if [ -e "$cache_file" ] || [ -L "$cache_file" ]; then + rm -f "$cache_file" || { + log "codex account $account_home skipped: could not clear its quota cache for a fresh health read" + return 1 + } + fi + ( + while IFS='=' read -r environment_name _; do + case "$environment_name" in + XDG_*|QUOTA_AXI_*|AGENT_FLEET_*) unset "$environment_name" ;; + esac + done < <(/usr/bin/env) + CODEX_HOME=$account_home + XDG_CACHE_HOME=$cache_home + export CODEX_HOME XDG_CACHE_HOME + if run_bounded "$timeout" "$quota_bin" --provider codex --json 2>/dev/null; then + return 0 + else + status=$? + fi + if [ "$status" -eq 124 ]; then + log "codex account $account_home skipped: quota read timed out after ${timeout}s" + fi + return "$status" + ) +} + +codex_score() { # + jq -er ' + [.providers[]? + | select(.provider == "codex" and .state.status == "fresh") + | (.windows // [])[]? + | select((.id == "five_hour" or .id == "weekly") + and (.kind // "") != "model" + and (.percentRemaining | type) == "number") + | .percentRemaining] + | if length == 0 then empty else min end + ' 2>/dev/null <&2 + return 1 + } + command -v jq >/dev/null 2>&1 || { + echo "error: jq is required for Codex account usage selection" >&2 + return 1 + } + quota_bin=$(quota_command) || return 1 + LC_ALL=C + export LC_ALL + for candidate in "$vendor_dir"/*; do + valid_account_home "$vendor_dir" "$candidate" || continue + usage=$(fresh_codex_usage_json "$candidate" "$quota_bin") || usage= + score=$(codex_score "$usage") || score= + if [ -z "$score" ]; then + log "codex account $candidate skipped: no freshly readable usage window" + continue + fi + log "codex account $candidate fresh remaining score=$score" + if [ -z "$best_home" ] || awk -v candidate_score="$score" -v current_score="$best_score" \ + 'BEGIN { exit !(candidate_score > current_score) }'; then + best_home=$candidate + best_score=$score + fi + done + [ -n "$best_home" ] || { + echo "error: no healthy Codex account has a freshly readable usage window" >&2 + return 1 + } + log "selected codex account $best_home with fresh remaining score=$best_score" + printf '%s\n' "$best_home" +} + +select_claude() { + local selected + selected=$(first_account_home claude) || return 1 + log "CLAUDE USAGE UNREADABLE: quota-axi cannot non-interactively resolve Claude's config-dir-specific macOS Keychain credential today; selecting the first account directory by stable sort: $selected" + printf '%s\n' "$selected" +} + +select_account() { # + case "$1" in + codex) select_codex ;; + claude) select_claude ;; + *) + echo "error: direct account-directory selection supports only claude or codex, not '$1'" >&2 + return 1 + ;; + esac +} + +install_herdr_hook() { # + local vendor=$1 account_home=$2 root vendor_dir herdr_bin expected_hook + root=$(account_root) || return 1 + vendor_dir=$root/$vendor + valid_account_home "$vendor_dir" "$account_home" || { + echo "error: unsafe $vendor account home for Herdr hook installation: $account_home" >&2 + return 1 + } + herdr_bin=$(herdr_command) || return 1 + case "$vendor" in + codex) + CODEX_HOME=$account_home "$herdr_bin" integration install codex >/dev/null || { + echo "error: Herdr Codex integration install failed for $account_home" >&2 + return 1 + } + expected_hook=$account_home/herdr-agent-state.sh + ;; + claude) + CLAUDE_CONFIG_DIR=$account_home "$herdr_bin" integration install claude >/dev/null || { + echo "error: Herdr Claude integration install failed for $account_home" >&2 + return 1 + } + expected_hook=$account_home/hooks/herdr-agent-state.sh + ;; + *) + echo "error: Herdr account hook installation supports only claude or codex, not '$vendor'" >&2 + return 1 + ;; + esac + [ -f "$expected_hook" ] && [ ! -L "$expected_hook" ] || { + echo "error: Herdr installer did not create the expected $vendor hook at $expected_hook" >&2 + return 1 + } + log "Herdr $vendor hook ready at $expected_hook" +} + +case "${1:-}" in + -h|--help) + usage + exit 0 + ;; + select) + [ "$#" -eq 2 ] || { usage; exit 2; } + select_account "$2" + ;; + install-herdr-hook) + [ "$#" -eq 3 ] || { usage; exit 2; } + install_herdr_hook "$2" "$3" + ;; + prepare) + [ "$#" -eq 2 ] || { usage; exit 2; } + selected_home=$(select_account "$2") || exit 1 + install_herdr_hook "$2" "$selected_home" || exit 1 + printf '%s\n' "$selected_home" + ;; + *) + usage + exit 2 + ;; +esac diff --git a/bin/fm-account-routing-lib.sh b/bin/fm-account-routing-lib.sh index b046a3c15cb..3fb31539b61 100644 --- a/bin/fm-account-routing-lib.sh +++ b/bin/fm-account-routing-lib.sh @@ -15,12 +15,10 @@ # FM_ACCOUNT_ROUTING and executable overrides are accepted only when the # unmistakable FM_ACCOUNT_ROUTING_TEST_LAB opt-in is active. # Valid modes are off, observe, and enforce. -# Off does not invoke Agent Fleet. -# Observe performs only `choose --dry-run`, never creates a lease, never wraps -# the provider launch, and never writes managed account metadata. -# Enforce atomically reserves one profile after endpoint and worktree setup, -# immediately before provider launch, and fails closed on every Agent Fleet or -# validation error. +# New observe and enforce launches use bin/fm-account-directory.sh and do not +# invoke Agent Fleet. +# Agent Fleet selection and lease helpers remain as legacy recovery support and +# isolated dead-code fixtures pending remove-fleet-routing-deadcode. # # FM_AGENT_FLEET_BIN may name a deterministic fake only in tests/labs. # Production always opens the current passwd user's fixed regular front door at @@ -1012,7 +1010,7 @@ fm_account_safe_lineage_value() { fm_account_meta_key_owned() { # case "$1" in - window|worktree|project|harness|kind|mode|yolo|tasktmp|model|effort|report_required|generation_id|backend|tmux_window_id|tmux_session_target|account_pool|account_profile|account_task|account_attempt|account_predecessor_task|account_predecessor_attempt|account_predecessor_provider|account_predecessor_profile|account_predecessor_pool|account_predecessor_session|account_predecessor_cleanup|account_rollback_cleanup|account_rollback_backup|account_rollback_artifacts|account_rollback_preserve_session|continuation_packet|provider_session_id|herdr_session|herdr_workspace_id|herdr_tab_id|herdr_pane_id|zellij_session|zellij_tab_id|zellij_pane_id|orca_worktree_id|terminal|cmux_workspace_id|cmux_surface_id|home|projects|rollback_pending) return 0 ;; + window|worktree|worktree_git_dir|worktree_git_dir_identity|worktree_git_ref|worktree_git_head|worktree_git_setup_ref|worktree_git_setup_head|project|harness|kind|mode|yolo|tasktmp|model|effort|report_required|generation_id|backend|tmux_window_id|tmux_session_target|account_home|direct_spawn_cleanup|direct_spawn_backup|direct_spawn_artifacts|direct_recovery_cleanup|direct_recovery_backup|direct_recovery_artifacts|account_pool|account_profile|account_task|account_attempt|account_predecessor_task|account_predecessor_attempt|account_predecessor_provider|account_predecessor_profile|account_predecessor_pool|account_predecessor_session|account_predecessor_cleanup|account_rollback_cleanup|account_rollback_backup|account_rollback_artifacts|account_rollback_preserve_session|continuation_packet|provider_session_id|herdr_session|herdr_workspace_id|herdr_tab_id|herdr_pane_id|zellij_session|zellij_tab_id|zellij_pane_id|orca_worktree_id|terminal|cmux_workspace_id|cmux_surface_id|home|projects|rollback_pending) return 0 ;; *) return 1 ;; esac } diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index 17388bf7ff5..a849158afe5 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -35,8 +35,8 @@ # distinct from the endpoint pane-presence check): outcomes distinguish # no-op, successful respawn, explicit-routing deferral, skipped # recovery, and failed recovery. A confirmed-dead unmanaged generation -# is deferred until an operator chooses whether to preserve unmanaged -# routing or convert it to managed routing. Session-start scope only; +# is deferred until an operator chooses whether to preserve +# unmanaged routing or convert it to managed routing. Session-start scope only; # see AGENTS.md "Session start" and docs/tmux-backend.md / # docs/herdr-backend.md "Agent liveness probe" for the empirical basis. # A TANGLE line means the firstmate primary checkout (FM_ROOT) is stranded @@ -603,6 +603,7 @@ EOF BOOTSTRAP_JQ_REPORTED=0 ACCOUNT_ROUTING_MODE=off +ACCOUNT_ROUTING_NEEDS_DIRECT_TOOLS=0 ACCOUNT_ROUTING_NEEDS_AGENT_FLEET=0 CREW_DISPATCH_ROUTING_VALID=unknown @@ -614,26 +615,48 @@ account_routing_preflight() { mode_error=${mode#error: } echo "ACCOUNT_ROUTING: invalid routing policy - $mode_error" fi + [ "$ACCOUNT_ROUTING_MODE" = off ] || ACCOUNT_ROUTING_NEEDS_DIRECT_TOOLS=1 [ "$ACCOUNT_ROUTING_MODE" != enforce ] || ACCOUNT_ROUTING_NEEDS_AGENT_FLEET=1 } account_routing_dependency_preflight() { - local needs_agent_fleet=$ACCOUNT_ROUTING_NEEDS_AGENT_FLEET dispatch + local needs_direct=$ACCOUNT_ROUTING_NEEDS_DIRECT_TOOLS needs_agent_fleet=$ACCOUNT_ROUTING_NEEDS_AGENT_FLEET dispatch meta direct_perl dispatch="$CONFIG/crew-dispatch.json" if [ -f "$dispatch" ]; then if [ "$CREW_DISPATCH_ROUTING_VALID" = 1 ]; then - jq -e '.. | objects | select(has("account_pool") or has("account_profile"))' "$dispatch" >/dev/null 2>&1 && needs_agent_fleet=1 + jq -e '.. | objects | select(has("account_pool") or has("account_profile"))' "$dispatch" >/dev/null 2>&1 && needs_direct=1 elif [ "$CREW_DISPATCH_ROUTING_VALID" = unknown ] \ && grep -Eq '"account_(pool|profile)"[[:space:]]*:' "$dispatch" 2>/dev/null; then - needs_agent_fleet=1 + needs_direct=1 fi fi - [ "$needs_agent_fleet" = 1 ] || return 0 - fm_account_fleet_bin >/dev/null 2>&1 || missing_tool_diagnostic agent-fleet - if ! command -v jq >/dev/null 2>&1; then + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + if grep -q '^account_home=.' "$meta" 2>/dev/null \ + && ! grep -qx 'kind=secondmate' "$meta" 2>/dev/null; then + needs_direct=1 + fi + if grep -q '^account_profile=' "$meta" 2>/dev/null \ + || grep -qx 'account_rollback_cleanup=pending' "$meta" 2>/dev/null; then + needs_agent_fleet=1 + fi + done + if [ "$needs_direct" = 1 ] && ! command -v jq >/dev/null 2>&1; then echo "MISSING: jq (install: $(install_cmd jq))" BOOTSTRAP_JQ_REPORTED=1 fi + if [ "$needs_direct" = 1 ] && ! command -v herdr >/dev/null 2>&1; then + missing_tool_diagnostic herdr + fi + direct_perl=/usr/bin/perl + if [ "${FM_ACCOUNT_DIRECTORY_TEST_LAB:-}" = firstmate-account-directory-test-lab-v1 ] \ + && [ -n "${FM_ACCOUNT_DIRECTORY_PERL_BIN:-}" ]; then + direct_perl=$FM_ACCOUNT_DIRECTORY_PERL_BIN + fi + if [ "$needs_direct" = 1 ] && [ ! -x "$direct_perl" ]; then + missing_tool_diagnostic perl + fi + [ "$needs_agent_fleet" = 0 ] || fm_account_fleet_bin >/dev/null 2>&1 || missing_tool_diagnostic agent-fleet } crew_dispatch_validate() { diff --git a/bin/fm-dispatch-select.sh b/bin/fm-dispatch-select.sh index f2dc5278ebb..a95fe07d336 100755 --- a/bin/fm-dispatch-select.sh +++ b/bin/fm-dispatch-select.sh @@ -11,15 +11,11 @@ # contract: # - Any candidate carrying account_profile is invalid because pinned profiles # are direct per-spawn overrides, never inputs to quota-balanced selection. -# - A candidate set carrying account_pool uses only Agent Fleet's no-secret -# `pool status` summaries. Every candidate must then carry account_pool and -# use claude/codex. Only a non-degraded, quota-fresh provider summary backed -# by at least one freshly proven eligible profile is available. The best -# adjusted headroom wins; exact ties use the first array element. Stale or -# otherwise degraded summaries are diagnostics only. Agent Fleet trouble -# degrades to the first element and never falls through to default-account -# quota-axi data; enforced spawn still obtains the real fresh lease before -# any provider launch. +# - A candidate set carrying account_pool uses the first array element and +# passes the pool only as a compatibility activation input. Spawn performs +# the real direct account-directory selection. New dispatch never calls +# Agent Fleet. The unreachable pool-summary implementation remains only for +# deferred removal under remove-fleet-routing-deadcode. # - Enforced account routing rejects quota-balanced candidates without pools. # Off and observe retain the legacy no-pool quota-axi path. # - Per candidate vendor it takes the minimum percentRemaining across that @@ -40,8 +36,8 @@ # # quota-balanced uses quota-axi --json unless --quota-json supplies a fixture. # FM_DISPATCH_QUOTA_AXI overrides the quota command. -# FM_DISPATCH_AGENT_FLEET and FM_AGENT_FLEET_BIN are test/lab-only. Production -# uses the fixed passwd-home ~/.local/bin/agent-fleet front door. +# FM_DISPATCH_AGENT_FLEET and FM_AGENT_FLEET_BIN belong only to the unreachable +# legacy pool-summary implementation pending remove-fleet-routing-deadcode. # FM_DISPATCH_STALE_CLEAR_MARGIN overrides the default 20 point stale margin. set -u @@ -87,18 +83,6 @@ log() { printf 'fm-dispatch-select: %s\n' "$*" >&2 } -if [ -n "${FM_DISPATCH_AGENT_FLEET_TIMEOUT:-}" ]; then - AGENT_FLEET_TIMEOUT=$FM_DISPATCH_AGENT_FLEET_TIMEOUT - case "$AGENT_FLEET_TIMEOUT" in - ''|*[!0-9]*|0) - echo "error: FM_DISPATCH_AGENT_FLEET_TIMEOUT must be a positive integer" >&2 - exit 2 - ;; - esac -else - AGENT_FLEET_TIMEOUT=$(fm_account_selection_timeout) || exit 2 -fi - while [ "$#" -gt 0 ]; do case "$1" in --select) @@ -220,6 +204,30 @@ if [ "$routing_mode" = enforce ]; then fi fi if [ "$pooled_count" -gt 0 ]; then + if [ "$pooled_count" -ne "$profile_count" ] || ! printf '%s\n' "$profiles_json" | "$FM_DISPATCH_JQ_BIN" -e 'all(.[]; (.account_pool | length) > 0 and (.harness == "claude" or .harness == "codex"))' >/dev/null 2>&1; then + log "account_pool quota-balanced candidates must all name claude/codex pools; using first profile" + first_profile + exit 0 + fi + log "account_pool is a direct-routing compatibility input; using the first profile and deferring account selection to spawn" + first_profile + exit 0 +fi + +# Legacy Agent Fleet pool summaries are unreachable and retained only until +# remove-fleet-routing-deadcode removes the wider selection and lease machinery. +if [ "$pooled_count" -gt 0 ]; then + if [ -n "${FM_DISPATCH_AGENT_FLEET_TIMEOUT:-}" ]; then + AGENT_FLEET_TIMEOUT=$FM_DISPATCH_AGENT_FLEET_TIMEOUT + case "$AGENT_FLEET_TIMEOUT" in + ''|*[!0-9]*|0) + echo "error: FM_DISPATCH_AGENT_FLEET_TIMEOUT must be a positive integer" >&2 + exit 2 + ;; + esac + else + AGENT_FLEET_TIMEOUT=$(fm_account_selection_timeout) || exit 2 + fi if [ "$pooled_count" -ne "$profile_count" ] || ! printf '%s\n' "$profiles_json" | "$FM_DISPATCH_JQ_BIN" -e 'all(.[]; (.account_pool | length) > 0 and (.harness == "claude" or .harness == "codex"))' >/dev/null 2>&1; then log "account_pool quota-balanced candidates must all name claude/codex pools; using first profile" first_profile diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index d78eb85e70a..30d2e5ac098 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -1,8 +1,10 @@ #!/usr/bin/env bash +# FM_ACCOUNT_DIRECTORY_CUTOVER: direct-observe-passwd-home-v2 # Spawn a direct report: a new crewmate in a treehouse worktree, an eligible # pre-cutover Orca respawn, or a secondmate in its isolated firstmate home. # Usage: fm-spawn.sh [--harness |harness|launch-command] [--model ] [--effort ] [--backend ] [--account-pool ] [--account-profile ] [--no-account-routing] [--scout] # fm-spawn.sh [] [--harness |harness|launch-command] [--model ] [--effort ] [--backend ] [--account-pool ] [--account-profile ] [--no-account-routing] --secondmate +# fm-spawn.sh --recover-direct-account # fm-spawn.sh (--resume-account|--continue-account) [--harness ] [--account-pool ] [--account-profile ] # --harness is the explicit per-spawn harness/profile adapter. The old # positional harness arg still works for back-compat. @@ -49,27 +51,30 @@ # respawn exactly like the harness axis, and explicit --model/--effort flags # still win over the file's tokens. # Account routing is independently default-off. Its precedence and off/observe/ -# enforce behavior are owned by fm-account-routing-lib.sh. --account-pool asks -# Agent Fleet to atomically select one concrete profile; --account-profile pins -# a concrete profile (and optionally validates it belongs to --account-pool). -# Either explicit account flag enforces routing for this spawn even when the -# global mode is off/observe. --no-account-routing is the emergency per-spawn -# opt-out and cannot be combined with either account flag. Enforced routing is -# supported only for claude/codex and fails closed; it never silently launches -# the default account. The resolved profile wraps the provider command before -# it is submitted to the selected backend, so every backend receives the same -# launch string. -# config/secondmate-account-pool is the primary's durable, non-inherited pool -# for secondmate AGENTS. An explicit account flag overrides it. A secondmate's -# own crewmates use inherited crew dispatch/routing policy, not this pool. -# --resume-account is an internal recovery path. It requires existing sticky -# account/profile/session metadata plus Agent Fleet's matching SessionStart -# mapping, reuses the recorded worktree/home, and executes `agent-fleet resume -# --task`; any missing or mismatched recovery truth blocks before pane creation. -# --continue-account is the provider-neutral recovery path. It verifies a dead -# endpoint and current repository state, builds a task-owned continuation packet, -# launches a fresh provider session through a new namespaced Agent Fleet attempt, -# and releases the predecessor only after the new SessionStart mapping is bound. +# enforce resolution is owned by fm-account-routing-lib.sh. Direct account- +# directory launch currently covers ship/scout crewmates only; secondmate +# integration is deferred and retains legacy Agent Fleet routing. +# For a NEW routed Claude or Codex ship/scout, fm-account-directory.sh discovers +# the current user's account homes, chooses one through its direct per-vendor +# usage contract, installs that profile's Herdr hook, and prefixes the provider +# command with CLAUDE_CONFIG_DIR or CODEX_HOME. +# Existing --account-pool and --account-profile inputs remain compatibility +# activation signals for new direct launches; their aliases do not constrain +# the direct usage choice. --no-account-routing remains the emergency per-spawn +# opt-out and cannot be combined with either account flag. Off launches retain +# their existing default-identity behavior. +# config/secondmate-account-pool remains the primary's durable, non-inherited +# Agent Fleet selection input for secondmate agents when routing is enabled. A +# secondmate's own crewmates use inherited crew dispatch/routing policy, not +# this setting. +# --resume-account and --continue-account are legacy recovery paths only for +# existing account_profile metadata. They retain the sealed Agent Fleet +# session/lease behavior needed to recover those already-managed generations; +# ship/scout launches never create that metadata. +# --recover-direct-account is the ship/scout account_home recovery path. It reloads kind, +# project, worktree, harness, backend, model, effort, mode, yolo, and report +# requirements from metadata, selects a fresh account directory, and creates +# only a replacement endpoint in the recorded worktree. # A --secondmate spawn also propagates the primary's declared inheritable config # into the secondmate home's config/, so the secondmate's OWN crewmates, # dispatch profiles, and backlog backend inherit the primary's settings @@ -158,10 +163,164 @@ spawn_managed_endpoint_state() { #