diff --git a/docker/.env.example.AGENTS.md b/docker/.env.example.AGENTS.md new file mode 100644 index 000000000..40ab25957 --- /dev/null +++ b/docker/.env.example.AGENTS.md @@ -0,0 +1,3 @@ +# .env.example — index + +Container env knobs, each commented. API keys, `DASHBOARD_PORT`, `PI_GATEWAY_PORT`, `PI_GATEWAY_BIND`, `TUNNEL_ENABLED`, `ZROK_TOKEN`, `PI_SPAWN_STRATEGY`, `MEM_LIMIT`, `PI_WORKSPACES`, `PI_DASHBOARD_PIN_DIRS`, `PI_GATEWAY_TCP` (gateway TCP listener; ON here because external pi sessions cannot reach the in-container socket). See change: docker-packaging. See change: add-pi-gateway-transport-identity. diff --git a/docker/AGENTS.md b/docker/AGENTS.md index 36783cea7..3f4c094a7 100644 --- a/docker/AGENTS.md +++ b/docker/AGENTS.md @@ -4,20 +4,20 @@ Files in this directory. One row per file. Non-source area. Subdir files owned b | File | Purpose | |------|---------| -| `.env.example` | Container env knobs, each commented. API keys, `DASHBOARD_PORT`, `PI_GATEWAY_PORT`, `PI_GATEWAY_BIND`, `TUNNEL_ENABLED`, `ZROK_TOKEN`, `PI_SPAWN_STRATEGY`, `MEM_LIMIT`, `PI_WORKSPACES`, `PI_DASHBOARD_PIN_DIRS`, `PI_GATEWAY_TCP` (gateway TCP listener; ON here because external pi sessions cannot reach the in-container socket). See change: docker-packaging. See change: add-pi-gateway-transport-identity. | +| `.env.example` | Container env knobs, each commented. API keys, `DASHBOARD_PORT`, `PI_GATEWAY_PORT`, `PI_GATEWAY_BIND`,… → see `.env.example.AGENTS.md` | | `.gitignore` | Excludes `.env` + `compose.override.yml` (user-specific). See change: docker-packaging. | | `compose.dev.yml` | Dev overlay. Bind-mounts source, anonymous node_modules volume keeps Linux node-pty, exposes 5173, `NODE_ENV=development`, command `start --dev`. See change: docker-packaging. | | `compose.override.yml.example` | Power-user template. Path-identical workspace binds, `:ro` example, `PI_DASHBOARD_PIN_DIRS`. Copy to `compose.override.yml`. See change: docker-packaging. | -| `compose.test.cap.yml` | Overlay-mode capability layer. Grants `cap_add: [SYS_ADMIN]` for `mount -t overlay` in test-entrypoint.sh. test-up.sh layers it only in overlay mode; omits when `TEST_COPY_MODE=1` (copy mode runs no added capability). See change: docker-test-harness. | -| `compose.test.yml` | Test overlay on `compose.yml`. Sets `PI_DASHBOARD_NO_MDNS=1`, `DASHBOARD_PORT="${DASHBOARD_PORT:-18000}"`, `PI_GATEWAY_PORT="${PI_GATEWAY_PORT:-18999}"` (de-hardcoded, env-driven), `PI_GATEWAY_BIND=127.0.0.1`, `TUNNEL_ENABLED=false`. Per-worktree image tag `pi-dashboard:${TEST_IMAGE_TAG:-local}`, `restart: "unless-stopped"` (matches base compose.yml; was "no": dashboard daemon is container PID 1, so `POST /api/restart` exited the container and nothing revived the harness. unless-stopped still honours explicit `docker stop`; `test-down.sh` runs `compose down` regardless. Trade-off: crash-looping container respawns instead of staying dead; `test-up.sh` health gate still fails the run when dashboard never answers). No `cap_add` here — CAP_SYS_ADMIN layered by `compose.test.cap.yml` in overlay mode only. Overrides `pi-state` volume to tmpfs (ephemeral). Mounts `HOST_CWD` ro lower at `/mnt/test-lower`, tmpfs upper/work, fixtures source at `/fixtures-src`, `test-entrypoint.sh` bind. entrypoint `test-entrypoint.sh`. Runs `user: root` + `HOME=/home/pi` so entrypoint can `mkdir`/`mount` overlay at `/` while state lands in pi-state tmpfs. Passes `PI_E2E_SEED` env to container (`PI_E2E_SEED: "${PI_E2E_SEED:-}"`); default empty keeps harness UI-only. Bind-mounts `../qa/fixtures:/app/qa/fixtures:ro` so test-entrypoint stages faux fixture against warm image without rebuild. Passes `PI_E2E_TRUSTED_NETWORKS: "${PI_E2E_TRUSTED_NETWORKS:-}"` (optional comma-separated override for the PI_E2E_SEED trustedNetworks seed; empty => trust any peer). Passes `PI_TEST_PEERS: "${PI_TEST_PEERS:-}"` (flow-plugin e2e peer selector: both\|no-am\|legacy\|bad-registration; empty => no flow-peer wiring). See change: docker-test-harness. See change: parallelize-test-harness. See change: add-e2e-spawn-scenarios. See change: add-e2e-faux-model-roundtrip. See change: add-flow-plugin-e2e-tests. Passes `PI_E2E_OAUTH` + `PI_E2E_OAUTH_BASE` env through to container (PI_E2E_OAUTH=1 seeds resolvable github provider + bypassUrls before boot so /auth/* routes exist for oauth-redirect-base.spec.ts; requires PI_E2E_SEED=1; unset = harness unchanged). See change: config-override-oauth-redirect-base. Passes `PI_SYNTH_AGENT_TICKS: "${PI_SYNTH_AGENT_TICKS:-}"` (throttle-cadence arm: 1 => test-entrypoint stages the synthetic Agent-tick producer and SKIPS the subagents producer; requires PI_E2E_SEED=1; unset = real subagents). See change: reduce-bridge-tick-bandwidth. | -| `compose.yml` | Base compose. Service `pi-dashboard`, `init:true`, env-driven ports, named volumes `pi-state`→`/home/pi/.pi` + `zrok-state`→`/home/pi/.zrok2`, tmpfs `/tmp`, healthcheck `/api/health`, mem limit. Three commented volume perf profiles (default/performance/ephemeral). Sets `PI_GATEWAY_TCP: "${PI_GATEWAY_TCP:-1}"` — the TCP listener is opt-in since the transport change; the container keeps it with bridge auth mandatory (D10b). See change: docker-packaging. See change: add-pi-gateway-transport-identity. | -| `Dockerfile` | Multi-stage. base: `node:24-bookworm-slim` + tmux/jq/git/curl/ripgrep/fd-find/poppler-utils/build-essential/python3 + zrok (ARG `ZROK_VERSION`). app: non-root pi UID 1000, global `@earendil-works/pi-coding-agent@0.84.1` (pinned; old `@mariozechner` scope frozen). See change: update-pi-core-0-84-adopt-apis, pnpm install+build, npm link, purge build-essential/python3. EXPOSE 8000 9999. VOLUME `/home/pi/.pi` `/home/pi/.zrok2` — preceded by an explicit `mkdir` + `chown pi:pi` of both, because the `VOLUME` instruction creates them as ROOT whatever `USER` is in force and a fresh named volume inherits that ownership, which made `docker compose up` on a clean machine EACCES-restart-loop (masked in E2E by `user: "root"`). COPYs `supervise-daemon.sh` alongside `entrypoint.sh`. ENTRYPOINT `entrypoint.sh`. base also installs `gh` (GitHub CLI, ARG `GH_VERSION`) + `jj` (jujutsu, ARG `JJ_VERSION`, musl static binary, no checksum manifest); app installs `openspec` global alongside pi. base also `COPY tsconfig.base.json tsconfig.json scripts patches` before `pnpm install` (packages extend `../../tsconfig.base.json`; postinstall runs `scripts/maybe-patch-package.cjs` + `scripts/fix-pty-permissions.cjs`; patch-package replays `patches/`). zrok checksum grep uses `grep -E "[ /]${asset}$"` (checksum lines prefix asset with `./`). zrok v2 (`ZROK_VERSION=2.0.4`): tarball ships `zrok2`; extract `zrok2` + `ln -s zrok2 zrok`; smoke `zrok2 version` && `zrok version`. See change: support-zrok-v2. `COPY qa/fixtures ./qa/fixtures` so test-entrypoint stages faux extension. app also installs the flow-plugin e2e peers globally (ARG `PI_FLOWS_VERSION` `@blackbelt-technology/pi-flows`, ARG `PI_ANTHROPIC_MESSAGES_VERSION` `@blackbelt-technology/pi-anthropic-messages`); test-entrypoint.sh selectively wires them per PI_TEST_PEERS. See change: add-flow-plugin-e2e-tests. `corepack enable` pins `pnpm@11.15.1` (root `packageManager`); build `COPY package.json pnpm-lock.yaml pnpm-workspace.yaml` then `pnpm install --frozen-lockfile` + `pnpm run build` (pnpm re-resolves the platform optional-dep tree from the lockfile, so the old npm #4828 `rm -f package-lock.json` lockfile-nuke hack is gone); global tool installs (`npm install -g`) stay npm. See change: docker-packaging. See change: adopt-pnpm-for-dev-ci. See change: docker-test-harness. See change: add-e2e-faux-model-roundtrip. See change: selectable-tool-output-links. See change: default-node-24-keep-22-floor. See change: align-pi-080.6-docker. code-server install layer removed (ARG `CODE_SERVER_VERSION` gone). See change: remove-external-editor-integration. app also installs ARG `PI_DASHBOARD_SUBAGENTS_VERSION` `@blackbelt-technology/pi-dashboard-subagents` globally — the ONLY producer of `tool_execution_update` (its `extensions/agent.ts` drives `createProgressEmitter` → `progress.schedule(details)` into the tool `onProgress`). Absent it, the collapse scenarios P2/F4 have no workload and pass VACUOUSLY. See change: collapse-superseded-tool-execution-updates. | -| `entrypoint.sh` | Runs `seed-auth.js`, seeds spawnStrategy into config.json first-run-only, starts tmux server, enrolls zrok v2 headless when `ZROK_TOKEN` set + not already enrolled (`zrok2 enable "$ZROK_TOKEN" --headless`, idempotent via `~/.zrok2/environment.json`), runs pi-dashboard with env→flag mapping (`DASHBOARD_PORT`→`--port`, `PI_GATEWAY_PORT`→`--pi-port`, `TUNNEL_ENABLED=0`→`--no-tunnel`), then SUPERVISES the detached daemon via `supervise-daemon.sh` instead of `exec`ing it: `pi-dashboard start` returns as soon as the server detaches, so PID 1 exited and the container cycled every ~30s under `restart: unless-stopped` (invisible in E2E, which runs `test-entrypoint.sh` and supervises itself). `PI_ENTRYPOINT_NO_SUPERVISE=1` restores the old exec-and-return for a wrapping caller. See change: docker-packaging, support-zrok-v2, add-pi-gateway-transport-identity. | -| `lib-ports.sh` | Sourced pure-helper lib. Exports `derive_hash(cwd)` (cksum), `derive_project(cwd)` -> `pi-dash-test-`, `is_free(port)` (bash /dev/tcp check), `find_free_in_window(start,lo,hi)` (wrap scan, cap=window size). Windows: dashboard 18000-18999, gateway 19000-19999. See change: parallelize-test-harness. | +| `compose.test.cap.yml` | Overlay-mode capability layer. Grants `cap_add: [SYS_ADMIN]` for `mount -t overlay` in test-entrypoint.sh. → see `compose.test.cap.yml.AGENTS.md` | +| `compose.test.yml` | Test overlay on `compose.yml`. Sets `PI_DASHBOARD_NO_MDNS=1`, `DASHBOARD_PORT="${DASHBOARD_PORT:-18000}"`,… → see `compose.test.yml.AGENTS.md` | +| `compose.yml` | Base compose. Service `pi-dashboard`, `init:true`, env-driven ports, named volumes `pi-state`→`/home/pi/.pi`… → see `compose.yml.AGENTS.md` | +| `Dockerfile` | Multi-stage. base: `node:24-bookworm-slim` +… → see `Dockerfile.AGENTS.md` | +| `entrypoint.sh` | Runs `seed-auth.js`, seeds spawnStrategy into config.json first-run-only, starts tmux server, enrolls zrok v2… → see `entrypoint.sh.AGENTS.md` | +| `lib-ports.sh` | Sourced pure-helper lib. Exports `derive_hash(cwd)` (cksum), `derive_project(cwd)` -> `pi-dash-test-`,… → see `lib-ports.sh.AGENTS.md` | | `README.md` | User guide. Quick-start, config table, path-identical workspace mounts, volume perf profiles, gateway access, dev mode, Electron remote mode. See change: docker-packaging. | -| `supervise-daemon.sh` | `supervise_daemon [label]` — keeps PID 1 alive for a DETACHED dashboard daemon, sourced by BOTH entrypoints so the deployment and the harness cannot drift (the loop living only in `test-entrypoint.sh` is what let the container ship unsupervised). Waits for the pidfile (a cold jiti start outruns the launcher's 30s readiness window), re-reads it every 5s so `POST /api/restart` survives under a new pid, tolerates `RESTART_GRACE_TICKS` (24 x5s) of downtime, and forwards TERM/INT to the CURRENT owner. Deliberately not a foreground server: PID 1 being the server makes `/api/restart` kill the container. See change: add-pi-gateway-transport-identity. | -| `test-down.sh` | Teardown. Re-derives `COMPOSE_PROJECT_NAME` from `$PWD` via lib-ports.sh cksum. `docker compose -p -f compose.yml -f compose.test.yml down -v`. Warns on malformed `.pi-test-harness.json` but continues. Removes `.pi-test-harness.json` after down. Drops tmpfs `pi-state` + overlay upper. Host pristine. See change: docker-test-harness. See change: parallelize-test-harness. | -| `test-entrypoint.sh` | Test entrypoint wrapper. `HOST_CWD` set → mounts overlayfs (lower `/mnt/test-lower` ro, upper/work tmpfs) at identical `${HOST_CWD}`; `TEST_COPY_MODE=1` falls back to `cp -a` onto tmpfs (no CAP_SYS_ADMIN). Copies `/fixtures-src` to `/fixtures` tmpfs, git init + commit each VCS fixture via a loop (`sample-git sample-hook-ok sample-hook-fail` — the hook fixtures added for change friendlier-worktree-init). Launches base `entrypoint.sh` in background. Fail-fast smoke: curl `/api/health` 200 + one `/ws` WebSocket connect via Node 24 global WebSocket; exits non-zero on failure. Runs base `entrypoint.sh` (launches DETACHED `pi-dashboard start` daemon), then smoke check, then keeps PID 1 alive by polling `~/.pi/dashboard/server.pid` until daemon exits; SIGTERM/INT forwarded to daemon. Tolerates base launcher readiness-timeout non-zero exit (detached daemon survives); polls `/api/health` up to ~90s itself. curl probes bounded `--connect-timeout 1 --max-time 2`. Copy-mode workspace `cp` fail-fast (no error swallow). `PI_E2E_SEED=1` → before base entrypoint, seeds fake never-valid anthropic oauth credential to `~/.pi/agent/auth.json` (flips providersReady, clears LandingPage onboarding gate), seeds trustedNetworks to `~/.pi/dashboard/config.json` so the browser's non-loopback source IP clears createNetworkGuard for `/api/browse` directory listing + `/api/providers`. NOW DEFAULTS TO `0.0.0.0/0`, narrowable via `PI_E2E_TRUSTED_NETWORKS` (comma-separated): the old RFC1918-only list (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) assumed published-port traffic is always SNAT'd through a private bridge gateway, which is FALSE on a VPN'd host — behind Cloudflare WARP the observed peer was the PUBLIC 172.67.221.13, outside 172.16.0.0/12 (spans only 172.16-172.31), so every browser request 403'd (“Network not allowed” + “Server offline”) and every scenario spec died in `pinDirectory`. Trust boundary is the disposable, localhost-published, RAM-backed container, not an IP range. See change: compact-warm-replay-stream (harness fix); both no-op if files exist so base seed-auth.js + config seed skip. `PI_E2E_SEED=1` also stages faux fixture as global extension `~/.pi/agent/extensions/faux-provider/index.ts` + sibling `faux-scenarios.ts`; seeds `defaultModel=faux/faux-1` into `~/.pi/dashboard/config.json` + `~/.pi/agent/settings.json` (merge, no clobber); symlinks `/app/node_modules` into staged extension dir so fixture resolves `@earendil-works/pi-ai`; no-op when present; default OFF. See change: docker-test-harness. See change: add-e2e-spawn-scenarios. See change: add-e2e-faux-model-roundtrip. PI_E2E_SEED=1 also seeds config.json#modelProxy (enabled + one apiKey, hash=sha256 of fixed pi-proxy-e2e-oauth-filter-* key, scopes ["all"]) so /v1/* reachable by tests/e2e/model-proxy-oauth-filter.spec.ts. See change: filter-oauth-incompatible-models. PI_E2E_SEED=1 also seeds the faux role-preset (`/app/qa/fixtures/faux-roles.json`, `_comment` stripped) to `~/.pi/agent/providers.json` (all roles -> faux/faux-1). When `PI_TEST_PEERS` set (both\|no-am\|legacy\|bad-registration): registers the baked pi-flows engine into settings.json packages[] (+ bare `pi-flows` node_modules symlink under /fixtures/sample-git for the bridge tier-1 probe) so flows discover/run, and symlinks the baked anthropic-messages peer into /fixtures/sample-git/node_modules under the scoped name (both/bad-registration), the legacy `@pi/anthropic-messages` (legacy), or not at all (no-am); bad-registration also exports PI_DASHBOARD_DISABLE_PLUGIN_BRIDGE_PACKAGES_WRITE=1 so the bridge stays out of packages[]. When PI_TEST_PEERS set it ALSO: pre-trusts SESSION_CWD in ~/.pi/agent/trust.json (the .pi/flows project resources make /fixtures/sample-git trust-requiring; a headless RPC session would block on pi's "Trust project folder?" prompt -> REGISTER_TIMEOUT); bumps config.json spawnRegisterTimeoutMs=90000 (pi-flows engine jiti-compile on a cold spawn can exceed the 30s default); and warms the pi-flows jiti compile cache in the BACKGROUND (`pi --list-models &`, never delays health). See change: add-flow-plugin-e2e-tests. `PI_E2E_SEED=1` also seeds a decoration-mismatched local install at `/fixtures/local-pkg/image-fit-extension` (`package.json#name` = `@blackbelt-technology/pi-image-fit-extension`, dir basename decorated differently so the pure-string basename rule cannot match) and registers that PATH in settings.json `packages[]` → lands in `activeSources`, exercising the fs-aware name fallback that drives the card's Active/Remove button. Consumed by tests/e2e/recommended-local-name-match.spec.ts. See change: match-local-installs-by-package-name. Fixture MUST be a LOADABLE extension: `type:module` + `pi.extensions:["index.js"]` (not `main`) + an inert `index.js` default export — pi loads every `packages[]` entry at session start and an entry-less manifest is fatal. package.json + index.js written only when absent; settings registration always ensured. See change: restore-ask-user-tool-state-on-reconnect. `PI_E2E_SEED=1` also seeds two plugin-page fixture states into `~/.pi/dashboard/plugins/`: `e2e-broken` (`server` entry throws → `PluginStatus.error`) and `e2e-needs-req` (`requires.piExtensions: ["pi-e2e-absent-extension"]` → `missingRequirements`); and `e2e-dependent` (`dependsOn: ["e2e-needs-req"]` → disabling the dependency raises the cascade-confirm dialog; no monorepo plugin declares `dependsOn`, so the cascade path is otherwise unreachable at L3). All three claim `settings-section` so they earn a plugin page + nav child. Each FILE is guarded by its own `-f` check so a partial fixture heals on the next run. Idempotent (each skips when its package.json is present). Consumed by tests/e2e/plugin-settings-pages.spec.ts. See change: plugin-settings-pages. PI_E2E_SEED also stages `qa/fixtures/e2e-notify.ext.ts` → `~/.pi/agent/extensions/e2e-notify/` (the `ctx.ui.notify` L3 driver). `register_flows` now pins `/app/packages/extension` FIRST in `settings.json#packages` — tool-name precedence is FIRST-registration-wins, and pi-flows' own `ask_user` ({question,type}) otherwise shadows the dashboard's ({method,title}), failing every faux ask scenario. See change: split-notify-from-prompt-request. `PI_E2E_OAUTH=1` (opt-in; unset = harness unchanged): before base entrypoint + daemon launch, seeds `auth.providers.github` (`e2e-client-id`/`e2e-client-secret`) + `auth.secret` + `bypassUrls:["/"]` + `redirectBaseUrl` (`PI_E2E_OAUTH_BASE`, default `https://pi-e2e-a.example.com`) into config.json. MUST be before launch: `registerAuthPlugin` returns early on empty boot provider registry → no /auth/* route, no `_reloadAuth` (design D6). Spec cannot seed itself: pi-state is RAM-backed tmpfs, every container start hands server fresh empty ~/.pi and discards `PUT /api/config` writes. github = only built-in provider resolving with NO network I/O (static endpoints, no OIDC discovery) → works offline. `bypassUrls:["/"]` mandatory: Playwright-host requests arrive NON-loopback; prefix matches every URL so armed gate denies nothing on SHARED harness. Consumed by tests/e2e/oauth-redirect-base.spec.ts; set by tests/e2e/global-setup.ts (`PI_E2E_OAUTH: process.env.PI_E2E_OAUTH ?? "1"`). See change: config-override-oauth-redirect-base. `register_subagents` registers the baked `pi-dashboard-subagents` into `settings.json#packages` + a `node_modules` symlink under SESSION_CWD. Runs UNCONDITIONALLY inside the PI_TEST_PEERS block (not per-arm): collapse scenarios need the tick producer in every peer shape, and it registers no `ask_user`, so it cannot disturb the first-registration-wins precedence `register_flows` guards (bridge stays pinned FIRST). Adds one more TS extension to the per-session jiti compile set — on a COLD cache (e.g. after `docker volume prune`) the first turn can exceed a spec's 30 s budget; the existing background `pi --list-models` warm covers it, so let the warm finish before driving the UI. See change: collapse-superseded-tool-execution-updates. `PI_SYNTH_AGENT_TICKS=1` stages `qa/fixtures/faux-agent-ticks.ext.ts` → `~/.pi/agent/extensions/faux-agent-ticks/` (synthetic `Agent`-tick producer for the throttle cadence L3 rows) and SKIPS `register_subagents` — the synthetic `Agent` tool owns the tool name (first-registration-wins) and the two never coexist. See change: reduce-bridge-tick-bandwidth. | -| `test-up.sh` | Spin-up. Sources lib-ports.sh. Exports `HOST_CWD=$PWD`. Derives stable port pair + `COMPOSE_PROJECT_NAME` from HOST_CWD hash. Honours `DASHBOARD_PORT`+`PI_GATEWAY_PORT` override as pair (exactly one set = exit 1). Idempotent re-up reuses ports when project containers run. Exports `TEST_IMAGE_TAG=$COMPOSE_PROJECT_NAME` (per-worktree baked image tag; falls back `:local`). Writes `${HOST_CWD}/.pi-test-harness.json`. Sets `PI_GATEWAY_BIND=127.0.0.1`, `TUNNEL_ENABLED=false`. Runs `docker compose -p "$COMPOSE_PROJECT_NAME" -f compose.yml -f compose.test.yml up`, wrapped in a bounded bind-collision retry (MAX_BIND_RETRIES=5, re-derives port pair on `port is already allocated`; skipped when ports pinned verbatim). Layers `compose.test.cap.yml` only in overlay mode; omits when `TEST_COPY_MODE=1`. Banner shows chosen URL + project. Exports `PI_TEST_PEERS="${PI_TEST_PEERS:-}"` (flow-plugin e2e peer selector) passed through to the container. See change: docker-test-harness. See change: parallelize-test-harness. See change: add-flow-plugin-e2e-tests. | -| `TESTING.md` | Runbook for disposable isolated test harness. Quick start. Isolation-guarantee table: home-lock, mDNS, ports, `~/.pi` state. Path-parity explanation. Overlay-vs-copy-mode tradeoff. Fixtures-vs-mount usage. UI-only-vs-e2e key seeding. agent-browser manual-QA entry. See change: docker-test-harness. | -| `up.sh` | Workspace launcher. Parses `PI_WORKSPACES` path-separator list, one path-identical `-v dir:dir` RW bind per entry, exports `PI_DASHBOARD_PIN_DIRS`=same list, `docker compose run --service-ports`. See change: docker-packaging. | +| `supervise-daemon.sh` | `supervise_daemon [label]` — keeps PID 1 alive for a DETACHED dashboard daemon, sourced by BOTH… → see `supervise-daemon.sh.AGENTS.md` | +| `test-down.sh` | Teardown. Re-derives `COMPOSE_PROJECT_NAME` from `$PWD` via lib-ports.sh cksum. → see `test-down.sh.AGENTS.md` | +| `test-entrypoint.sh` | Test entrypoint wrapper. `HOST_CWD` set → mounts overlayfs (lower `/mnt/test-lower` ro, upper/work tmpfs) at… → see `test-entrypoint.sh.AGENTS.md` | +| `test-up.sh` | Spin-up. Sources lib-ports.sh. Exports `HOST_CWD=$PWD`. Derives stable port pair + `COMPOSE_PROJECT_NAME`… → see `test-up.sh.AGENTS.md` | +| `TESTING.md` | Runbook for disposable isolated test harness. Quick start. → see `TESTING.md.AGENTS.md` | +| `up.sh` | Workspace launcher. Parses `PI_WORKSPACES` path-separator list, one path-identical `-v dir:dir` RW bind per… → see `up.sh.AGENTS.md` | diff --git a/docker/Dockerfile.AGENTS.md b/docker/Dockerfile.AGENTS.md new file mode 100644 index 000000000..5774bbbd1 --- /dev/null +++ b/docker/Dockerfile.AGENTS.md @@ -0,0 +1,3 @@ +# Dockerfile — index + +Multi-stage. base: `node:24-bookworm-slim` + tmux/jq/git/curl/ripgrep/fd-find/poppler-utils/build-essential/python3 + zrok (ARG `ZROK_VERSION`). app: non-root pi UID 1000, global `@earendil-works/pi-coding-agent@0.84.1` (pinned; old `@mariozechner` scope frozen). See change: update-pi-core-0-84-adopt-apis, pnpm install+build, npm link, purge build-essential/python3. EXPOSE 8000 9999. VOLUME `/home/pi/.pi` `/home/pi/.zrok2` — preceded by an explicit `mkdir` + `chown pi:pi` of both, because the `VOLUME` instruction creates them as ROOT whatever `USER` is in force and a fresh named volume inherits that ownership, which made `docker compose up` on a clean machine EACCES-restart-loop (masked in E2E by `user: "root"`). COPYs `supervise-daemon.sh` alongside `entrypoint.sh`. ENTRYPOINT `entrypoint.sh`. base also installs `gh` (GitHub CLI, ARG `GH_VERSION`) + `jj` (jujutsu, ARG `JJ_VERSION`, musl static binary, no checksum manifest); app installs `openspec` global alongside pi. base also `COPY tsconfig.base.json tsconfig.json scripts patches` before `pnpm install` (packages extend `../../tsconfig.base.json`; postinstall runs `scripts/maybe-patch-package.cjs` + `scripts/fix-pty-permissions.cjs`; patch-package replays `patches/`). zrok checksum grep uses `grep -E "[ /]${asset}$"` (checksum lines prefix asset with `./`). zrok v2 (`ZROK_VERSION=2.0.4`): tarball ships `zrok2`; extract `zrok2` + `ln -s zrok2 zrok`; smoke `zrok2 version` && `zrok version`. See change: support-zrok-v2. `COPY qa/fixtures ./qa/fixtures` so test-entrypoint stages faux extension. app also installs the flow-plugin e2e peers globally (ARG `PI_FLOWS_VERSION` `@blackbelt-technology/pi-flows`, ARG `PI_ANTHROPIC_MESSAGES_VERSION` `@blackbelt-technology/pi-anthropic-messages`); test-entrypoint.sh selectively wires them per PI_TEST_PEERS. See change: add-flow-plugin-e2e-tests. `corepack enable` pins `pnpm@11.15.1` (root `packageManager`); build `COPY package.json pnpm-lock.yaml pnpm-workspace.yaml` then `pnpm install --frozen-lockfile` + `pnpm run build` (pnpm re-resolves the platform optional-dep tree from the lockfile, so the old npm #4828 `rm -f package-lock.json` lockfile-nuke hack is gone); global tool installs (`npm install -g`) stay npm. See change: docker-packaging. See change: adopt-pnpm-for-dev-ci. See change: docker-test-harness. See change: add-e2e-faux-model-roundtrip. See change: selectable-tool-output-links. See change: default-node-24-keep-22-floor. See change: align-pi-080.6-docker. code-server install layer removed (ARG `CODE_SERVER_VERSION` gone). See change: remove-external-editor-integration. app also installs ARG `PI_DASHBOARD_SUBAGENTS_VERSION` `@blackbelt-technology/pi-dashboard-subagents` globally — the ONLY producer of `tool_execution_update` (its `extensions/agent.ts` drives `createProgressEmitter` → `progress.schedule(details)` into the tool `onProgress`). Absent it, the collapse scenarios P2/F4 have no workload and pass VACUOUSLY. See change: collapse-superseded-tool-execution-updates. diff --git a/docker/TESTING.md.AGENTS.md b/docker/TESTING.md.AGENTS.md new file mode 100644 index 000000000..a4de2122a --- /dev/null +++ b/docker/TESTING.md.AGENTS.md @@ -0,0 +1,3 @@ +# TESTING.md — index + +Runbook for disposable isolated test harness. Quick start. Isolation-guarantee table: home-lock, mDNS, ports, `~/.pi` state. Path-parity explanation. Overlay-vs-copy-mode tradeoff. Fixtures-vs-mount usage. UI-only-vs-e2e key seeding. agent-browser manual-QA entry. See change: docker-test-harness. diff --git a/docker/compose.test.cap.yml.AGENTS.md b/docker/compose.test.cap.yml.AGENTS.md new file mode 100644 index 000000000..6f749ba06 --- /dev/null +++ b/docker/compose.test.cap.yml.AGENTS.md @@ -0,0 +1,3 @@ +# compose.test.cap.yml — index + +Overlay-mode capability layer. Grants `cap_add: [SYS_ADMIN]` for `mount -t overlay` in test-entrypoint.sh. test-up.sh layers it only in overlay mode; omits when `TEST_COPY_MODE=1` (copy mode runs no added capability). See change: docker-test-harness. diff --git a/docker/compose.test.yml b/docker/compose.test.yml index a95fee3a0..c4b64fdc0 100644 --- a/docker/compose.test.yml +++ b/docker/compose.test.yml @@ -116,6 +116,14 @@ services: # coexist (first-registration-wins). Requires PI_E2E_SEED=1. Unset = # harness unchanged (real subagents). PI_SYNTH_AGENT_TICKS: "${PI_SYNTH_AGENT_TICKS:-}" + # Bridge subagent-timeline strip (change: reduce-subagent-details-payload, + # D2). `0` forwards every frame UNSTRIPPED — the push arm of the + # pull-vs-push A/B, and the anti-vacuity arm proving the strip is what + # makes the pushes thin. Read per call by the bridge, but a container env + # var is fixed for the life of the harness, so each arm is a SEPARATE + # test-up. Unset = strip ON (production default). + # See change: verify-subagent-pull-under-load. + PI_DASHBOARD_SUBAGENT_STRIP: "${PI_DASHBOARD_SUBAGENT_STRIP:-}" volumes: # Path-parity: host CWD as the read-only overlay lower layer. The diff --git a/docker/compose.test.yml.AGENTS.md b/docker/compose.test.yml.AGENTS.md new file mode 100644 index 000000000..ef1c72d80 --- /dev/null +++ b/docker/compose.test.yml.AGENTS.md @@ -0,0 +1,3 @@ +# compose.test.yml — index + +Test overlay on `compose.yml`. Sets `PI_DASHBOARD_NO_MDNS=1`, `DASHBOARD_PORT="${DASHBOARD_PORT:-18000}"`, `PI_GATEWAY_PORT="${PI_GATEWAY_PORT:-18999}"` (de-hardcoded, env-driven), `PI_GATEWAY_BIND=127.0.0.1`, `TUNNEL_ENABLED=false`. Per-worktree image tag `pi-dashboard:${TEST_IMAGE_TAG:-local}`, `restart: "unless-stopped"` (matches base compose.yml; was "no": dashboard daemon is container PID 1, so `POST /api/restart` exited the container and nothing revived the harness. unless-stopped still honours explicit `docker stop`; `test-down.sh` runs `compose down` regardless. Trade-off: crash-looping container respawns instead of staying dead; `test-up.sh` health gate still fails the run when dashboard never answers). No `cap_add` here — CAP_SYS_ADMIN layered by `compose.test.cap.yml` in overlay mode only. Overrides `pi-state` volume to tmpfs (ephemeral). Mounts `HOST_CWD` ro lower at `/mnt/test-lower`, tmpfs upper/work, fixtures source at `/fixtures-src`, `test-entrypoint.sh` bind. entrypoint `test-entrypoint.sh`. Runs `user: root` + `HOME=/home/pi` so entrypoint can `mkdir`/`mount` overlay at `/` while state lands in pi-state tmpfs. Passes `PI_E2E_SEED` env to container (`PI_E2E_SEED: "${PI_E2E_SEED:-}"`); default empty keeps harness UI-only. Bind-mounts `../qa/fixtures:/app/qa/fixtures:ro` so test-entrypoint stages faux fixture against warm image without rebuild. Passes `PI_E2E_TRUSTED_NETWORKS: "${PI_E2E_TRUSTED_NETWORKS:-}"` (optional comma-separated override for the PI_E2E_SEED trustedNetworks seed; empty => trust any peer). Passes `PI_TEST_PEERS: "${PI_TEST_PEERS:-}"` (flow-plugin e2e peer selector: both\|no-am\|legacy\|bad-registration; empty => no flow-peer wiring). See change: docker-test-harness. See change: parallelize-test-harness. See change: add-e2e-spawn-scenarios. See change: add-e2e-faux-model-roundtrip. See change: add-flow-plugin-e2e-tests. Passes `PI_E2E_OAUTH` + `PI_E2E_OAUTH_BASE` env through to container (PI_E2E_OAUTH=1 seeds resolvable github provider + bypassUrls before boot so /auth/* routes exist for oauth-redirect-base.spec.ts; requires PI_E2E_SEED=1; unset = harness unchanged). See change: config-override-oauth-redirect-base. Passes `PI_SYNTH_AGENT_TICKS: "${PI_SYNTH_AGENT_TICKS:-}"` (throttle-cadence arm: 1 => test-entrypoint stages the synthetic Agent-tick producer and SKIPS the subagents producer; requires PI_E2E_SEED=1; unset = real subagents). See change: reduce-bridge-tick-bandwidth. Passes `PI_DASHBOARD_SUBAGENT_STRIP: "${PI_DASHBOARD_SUBAGENT_STRIP:-}"` (bridge subagent-timeline strip; `0` = forward every frame UNSTRIPPED = the push arm + anti-vacuity arm of the pull-vs-push A/B. Bridge reads it per call, but a container env var is fixed for the harness life, so each arm is a SEPARATE test-up; unset/empty = strip ON). See change: verify-subagent-pull-under-load. diff --git a/docker/compose.yml.AGENTS.md b/docker/compose.yml.AGENTS.md new file mode 100644 index 000000000..39b93b1a2 --- /dev/null +++ b/docker/compose.yml.AGENTS.md @@ -0,0 +1,3 @@ +# compose.yml — index + +Base compose. Service `pi-dashboard`, `init:true`, env-driven ports, named volumes `pi-state`→`/home/pi/.pi` + `zrok-state`→`/home/pi/.zrok2`, tmpfs `/tmp`, healthcheck `/api/health`, mem limit. Three commented volume perf profiles (default/performance/ephemeral). Sets `PI_GATEWAY_TCP: "${PI_GATEWAY_TCP:-1}"` — the TCP listener is opt-in since the transport change; the container keeps it with bridge auth mandatory (D10b). See change: docker-packaging. See change: add-pi-gateway-transport-identity. diff --git a/docker/entrypoint.sh.AGENTS.md b/docker/entrypoint.sh.AGENTS.md new file mode 100644 index 000000000..effe1178d --- /dev/null +++ b/docker/entrypoint.sh.AGENTS.md @@ -0,0 +1,3 @@ +# entrypoint.sh — index + +Runs `seed-auth.js`, seeds spawnStrategy into config.json first-run-only, starts tmux server, enrolls zrok v2 headless when `ZROK_TOKEN` set + not already enrolled (`zrok2 enable "$ZROK_TOKEN" --headless`, idempotent via `~/.zrok2/environment.json`), runs pi-dashboard with env→flag mapping (`DASHBOARD_PORT`→`--port`, `PI_GATEWAY_PORT`→`--pi-port`, `TUNNEL_ENABLED=0`→`--no-tunnel`), then SUPERVISES the detached daemon via `supervise-daemon.sh` instead of `exec`ing it: `pi-dashboard start` returns as soon as the server detaches, so PID 1 exited and the container cycled every ~30s under `restart: unless-stopped` (invisible in E2E, which runs `test-entrypoint.sh` and supervises itself). `PI_ENTRYPOINT_NO_SUPERVISE=1` restores the old exec-and-return for a wrapping caller. See change: docker-packaging, support-zrok-v2, add-pi-gateway-transport-identity. diff --git a/docker/lib-ports.sh.AGENTS.md b/docker/lib-ports.sh.AGENTS.md new file mode 100644 index 000000000..9ded8865b --- /dev/null +++ b/docker/lib-ports.sh.AGENTS.md @@ -0,0 +1,3 @@ +# lib-ports.sh — index + +Sourced pure-helper lib. Exports `derive_hash(cwd)` (cksum), `derive_project(cwd)` -> `pi-dash-test-`, `is_free(port)` (bash /dev/tcp check), `find_free_in_window(start,lo,hi)` (wrap scan, cap=window size). Windows: dashboard 18000-18999, gateway 19000-19999. See change: parallelize-test-harness. diff --git a/docker/supervise-daemon.sh.AGENTS.md b/docker/supervise-daemon.sh.AGENTS.md new file mode 100644 index 000000000..00a43666b --- /dev/null +++ b/docker/supervise-daemon.sh.AGENTS.md @@ -0,0 +1,3 @@ +# supervise-daemon.sh — index + +`supervise_daemon [label]` — keeps PID 1 alive for a DETACHED dashboard daemon, sourced by BOTH entrypoints so the deployment and the harness cannot drift (the loop living only in `test-entrypoint.sh` is what let the container ship unsupervised). Waits for the pidfile (a cold jiti start outruns the launcher's 30s readiness window), re-reads it every 5s so `POST /api/restart` survives under a new pid, tolerates `RESTART_GRACE_TICKS` (24 x5s) of downtime, and forwards TERM/INT to the CURRENT owner. Deliberately not a foreground server: PID 1 being the server makes `/api/restart` kill the container. See change: add-pi-gateway-transport-identity. diff --git a/docker/test-down.sh.AGENTS.md b/docker/test-down.sh.AGENTS.md new file mode 100644 index 000000000..b4ffe2652 --- /dev/null +++ b/docker/test-down.sh.AGENTS.md @@ -0,0 +1,3 @@ +# test-down.sh — index + +Teardown. Re-derives `COMPOSE_PROJECT_NAME` from `$PWD` via lib-ports.sh cksum. `docker compose -p -f compose.yml -f compose.test.yml down -v`. Warns on malformed `.pi-test-harness.json` but continues. Removes `.pi-test-harness.json` after down. Drops tmpfs `pi-state` + overlay upper. Host pristine. See change: docker-test-harness. See change: parallelize-test-harness. diff --git a/docker/test-entrypoint.sh.AGENTS.md b/docker/test-entrypoint.sh.AGENTS.md new file mode 100644 index 000000000..dbd52ee46 --- /dev/null +++ b/docker/test-entrypoint.sh.AGENTS.md @@ -0,0 +1,3 @@ +# test-entrypoint.sh — index + +Test entrypoint wrapper. `HOST_CWD` set → mounts overlayfs (lower `/mnt/test-lower` ro, upper/work tmpfs) at identical `${HOST_CWD}`; `TEST_COPY_MODE=1` falls back to `cp -a` onto tmpfs (no CAP_SYS_ADMIN). Copies `/fixtures-src` to `/fixtures` tmpfs, git init + commit each VCS fixture via a loop (`sample-git sample-hook-ok sample-hook-fail` — the hook fixtures added for change friendlier-worktree-init). Launches base `entrypoint.sh` in background. Fail-fast smoke: curl `/api/health` 200 + one `/ws` WebSocket connect via Node 24 global WebSocket; exits non-zero on failure. Runs base `entrypoint.sh` (launches DETACHED `pi-dashboard start` daemon), then smoke check, then keeps PID 1 alive by polling `~/.pi/dashboard/server.pid` until daemon exits; SIGTERM/INT forwarded to daemon. Tolerates base launcher readiness-timeout non-zero exit (detached daemon survives); polls `/api/health` up to ~90s itself. curl probes bounded `--connect-timeout 1 --max-time 2`. Copy-mode workspace `cp` fail-fast (no error swallow). `PI_E2E_SEED=1` → before base entrypoint, seeds fake never-valid anthropic oauth credential to `~/.pi/agent/auth.json` (flips providersReady, clears LandingPage onboarding gate), seeds trustedNetworks to `~/.pi/dashboard/config.json` so the browser's non-loopback source IP clears createNetworkGuard for `/api/browse` directory listing + `/api/providers`. NOW DEFAULTS TO `0.0.0.0/0`, narrowable via `PI_E2E_TRUSTED_NETWORKS` (comma-separated): the old RFC1918-only list (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) assumed published-port traffic is always SNAT'd through a private bridge gateway, which is FALSE on a VPN'd host — behind Cloudflare WARP the observed peer was the PUBLIC 172.67.221.13, outside 172.16.0.0/12 (spans only 172.16-172.31), so every browser request 403'd (“Network not allowed” + “Server offline”) and every scenario spec died in `pinDirectory`. Trust boundary is the disposable, localhost-published, RAM-backed container, not an IP range. See change: compact-warm-replay-stream (harness fix); both no-op if files exist so base seed-auth.js + config seed skip. `PI_E2E_SEED=1` also stages faux fixture as global extension `~/.pi/agent/extensions/faux-provider/index.ts` + sibling `faux-scenarios.ts`; seeds `defaultModel=faux/faux-1` into `~/.pi/dashboard/config.json` + `~/.pi/agent/settings.json` (merge, no clobber); symlinks `/app/node_modules` into staged extension dir so fixture resolves `@earendil-works/pi-ai`; no-op when present; default OFF. See change: docker-test-harness. See change: add-e2e-spawn-scenarios. See change: add-e2e-faux-model-roundtrip. PI_E2E_SEED=1 also seeds config.json#modelProxy (enabled + one apiKey, hash=sha256 of fixed pi-proxy-e2e-oauth-filter-* key, scopes ["all"]) so /v1/* reachable by tests/e2e/model-proxy-oauth-filter.spec.ts. See change: filter-oauth-incompatible-models. PI_E2E_SEED=1 also seeds the faux role-preset (`/app/qa/fixtures/faux-roles.json`, `_comment` stripped) to `~/.pi/agent/providers.json` (all roles -> faux/faux-1). When `PI_TEST_PEERS` set (both\|no-am\|legacy\|bad-registration): registers the baked pi-flows engine into settings.json packages[] (+ bare `pi-flows` node_modules symlink under /fixtures/sample-git for the bridge tier-1 probe) so flows discover/run, and symlinks the baked anthropic-messages peer into /fixtures/sample-git/node_modules under the scoped name (both/bad-registration), the legacy `@pi/anthropic-messages` (legacy), or not at all (no-am); bad-registration also exports PI_DASHBOARD_DISABLE_PLUGIN_BRIDGE_PACKAGES_WRITE=1 so the bridge stays out of packages[]. When PI_TEST_PEERS set it ALSO: pre-trusts SESSION_CWD in ~/.pi/agent/trust.json (the .pi/flows project resources make /fixtures/sample-git trust-requiring; a headless RPC session would block on pi's "Trust project folder?" prompt -> REGISTER_TIMEOUT); bumps config.json spawnRegisterTimeoutMs=90000 (pi-flows engine jiti-compile on a cold spawn can exceed the 30s default); and warms the pi-flows jiti compile cache in the BACKGROUND (`pi --list-models &`, never delays health). See change: add-flow-plugin-e2e-tests. `PI_E2E_SEED=1` also seeds a decoration-mismatched local install at `/fixtures/local-pkg/image-fit-extension` (`package.json#name` = `@blackbelt-technology/pi-image-fit-extension`, dir basename decorated differently so the pure-string basename rule cannot match) and registers that PATH in settings.json `packages[]` → lands in `activeSources`, exercising the fs-aware name fallback that drives the card's Active/Remove button. Consumed by tests/e2e/recommended-local-name-match.spec.ts. See change: match-local-installs-by-package-name. Fixture MUST be a LOADABLE extension: `type:module` + `pi.extensions:["index.js"]` (not `main`) + an inert `index.js` default export — pi loads every `packages[]` entry at session start and an entry-less manifest is fatal. package.json + index.js written only when absent; settings registration always ensured. See change: restore-ask-user-tool-state-on-reconnect. `PI_E2E_SEED=1` also seeds two plugin-page fixture states into `~/.pi/dashboard/plugins/`: `e2e-broken` (`server` entry throws → `PluginStatus.error`) and `e2e-needs-req` (`requires.piExtensions: ["pi-e2e-absent-extension"]` → `missingRequirements`); and `e2e-dependent` (`dependsOn: ["e2e-needs-req"]` → disabling the dependency raises the cascade-confirm dialog; no monorepo plugin declares `dependsOn`, so the cascade path is otherwise unreachable at L3). All three claim `settings-section` so they earn a plugin page + nav child. Each FILE is guarded by its own `-f` check so a partial fixture heals on the next run. Idempotent (each skips when its package.json is present). Consumed by tests/e2e/plugin-settings-pages.spec.ts. See change: plugin-settings-pages. PI_E2E_SEED also stages `qa/fixtures/e2e-notify.ext.ts` → `~/.pi/agent/extensions/e2e-notify/` (the `ctx.ui.notify` L3 driver). `register_flows` now pins `/app/packages/extension` FIRST in `settings.json#packages` — tool-name precedence is FIRST-registration-wins, and pi-flows' own `ask_user` ({question,type}) otherwise shadows the dashboard's ({method,title}), failing every faux ask scenario. See change: split-notify-from-prompt-request. `PI_E2E_OAUTH=1` (opt-in; unset = harness unchanged): before base entrypoint + daemon launch, seeds `auth.providers.github` (`e2e-client-id`/`e2e-client-secret`) + `auth.secret` + `bypassUrls:["/"]` + `redirectBaseUrl` (`PI_E2E_OAUTH_BASE`, default `https://pi-e2e-a.example.com`) into config.json. MUST be before launch: `registerAuthPlugin` returns early on empty boot provider registry → no /auth/* route, no `_reloadAuth` (design D6). Spec cannot seed itself: pi-state is RAM-backed tmpfs, every container start hands server fresh empty ~/.pi and discards `PUT /api/config` writes. github = only built-in provider resolving with NO network I/O (static endpoints, no OIDC discovery) → works offline. `bypassUrls:["/"]` mandatory: Playwright-host requests arrive NON-loopback; prefix matches every URL so armed gate denies nothing on SHARED harness. Consumed by tests/e2e/oauth-redirect-base.spec.ts; set by tests/e2e/global-setup.ts (`PI_E2E_OAUTH: process.env.PI_E2E_OAUTH ?? "1"`). See change: config-override-oauth-redirect-base. `register_subagents` registers the baked `pi-dashboard-subagents` into `settings.json#packages` + a `node_modules` symlink under SESSION_CWD. Runs UNCONDITIONALLY inside the PI_TEST_PEERS block (not per-arm): collapse scenarios need the tick producer in every peer shape, and it registers no `ask_user`, so it cannot disturb the first-registration-wins precedence `register_flows` guards (bridge stays pinned FIRST). Adds one more TS extension to the per-session jiti compile set — on a COLD cache (e.g. after `docker volume prune`) the first turn can exceed a spec's 30 s budget; the existing background `pi --list-models` warm covers it, so let the warm finish before driving the UI. See change: collapse-superseded-tool-execution-updates. `PI_SYNTH_AGENT_TICKS=1` stages `qa/fixtures/faux-agent-ticks.ext.ts` → `~/.pi/agent/extensions/faux-agent-ticks/` (synthetic `Agent`-tick producer for the throttle cadence L3 rows) and SKIPS `register_subagents` — the synthetic `Agent` tool owns the tool name (first-registration-wins) and the two never coexist. See change: reduce-bridge-tick-bandwidth. diff --git a/docker/test-up.sh.AGENTS.md b/docker/test-up.sh.AGENTS.md new file mode 100644 index 000000000..1b5cc7d24 --- /dev/null +++ b/docker/test-up.sh.AGENTS.md @@ -0,0 +1,3 @@ +# test-up.sh — index + +Spin-up. Sources lib-ports.sh. Exports `HOST_CWD=$PWD`. Derives stable port pair + `COMPOSE_PROJECT_NAME` from HOST_CWD hash. Honours `DASHBOARD_PORT`+`PI_GATEWAY_PORT` override as pair (exactly one set = exit 1). Idempotent re-up reuses ports when project containers run. Exports `TEST_IMAGE_TAG=$COMPOSE_PROJECT_NAME` (per-worktree baked image tag; falls back `:local`). Writes `${HOST_CWD}/.pi-test-harness.json`. Sets `PI_GATEWAY_BIND=127.0.0.1`, `TUNNEL_ENABLED=false`. Runs `docker compose -p "$COMPOSE_PROJECT_NAME" -f compose.yml -f compose.test.yml up`, wrapped in a bounded bind-collision retry (MAX_BIND_RETRIES=5, re-derives port pair on `port is already allocated`; skipped when ports pinned verbatim). Layers `compose.test.cap.yml` only in overlay mode; omits when `TEST_COPY_MODE=1`. Banner shows chosen URL + project. Exports `PI_TEST_PEERS="${PI_TEST_PEERS:-}"` (flow-plugin e2e peer selector) passed through to the container. See change: docker-test-harness. See change: parallelize-test-harness. See change: add-flow-plugin-e2e-tests. diff --git a/docker/up.sh.AGENTS.md b/docker/up.sh.AGENTS.md new file mode 100644 index 000000000..b65f6890c --- /dev/null +++ b/docker/up.sh.AGENTS.md @@ -0,0 +1,3 @@ +# up.sh — index + +Workspace launcher. Parses `PI_WORKSPACES` path-separator list, one path-identical `-v dir:dir` RW bind per entry, exports `PI_DASHBOARD_PIN_DIRS`=same list, `docker compose run --service-ports`. See change: docker-packaging. diff --git a/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/design.md b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/design.md new file mode 100644 index 000000000..3f6f0d09a --- /dev/null +++ b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/design.md @@ -0,0 +1,422 @@ +## Context + +`reduce-subagent-details-payload` (archived 2026-08-15) moved the running +subagent timeline from push to pull: the bridge strips `details.entries` from +every non-terminal frame on both carriers, keeps the FAT frame in +`SubagentFrameBuffer`, and serves it back on a `subagent_resync_request` as a +synthetic `subagents:started` `event_forward`. A mounted inspector re-fires that +request on a cadence (D4 v1, `useSubagentResyncCadence.ts`, `CADENCE_BASE_MS = +2000`, backoff to 30 s). + +Four of its manifest scenarios shipped unverified (F1, P4, P5, X1 — see +`proposal.md`). All four have the SAME root blocker: **the harness has no +subagent that stays alive long enough to be watched.** + +### Why the harness has no watchable subagent + +Already root-caused, in the harness fixtures themselves +(`qa/fixtures/faux-agent-ticks.ext.ts`): + +> A nested faux subagent cannot sustain a ≥ 10 s tick stream in the harness — +> its inner `createAgentSession` resolves a DIFFERENT faux core with an empty +> response queue, so a faux subagent dies after ~2 no-op turns. +> (`reduce-bridge-tick-bandwidth`, measurement.md — "what still blocks the L3 +> rows", item 2; cited as "Bug 2" in that change's tasks/test-plan) + +The scripted `sleep`s in `subagent-slow-inner` never execute because the inner +session dies before reaching them. The predecessor change already removed the +`subagent-*-long` fixtures for this reason and routed around it with a +**synthetic `Agent`-tick producer** (staged under `PI_SYNTH_AGENT_TICKS=1`, +first-registration-wins over the real subagents `Agent` tool). + +So the proposal's "fix or route around" question is already answered: **route +around, on the existing synthetic substrate.** This change extends that +substrate from a tick emitter into a full pull-path substrate. + +### What the synthetic producer does NOT yet drive + +| Path | Today | Needed | +|---|---|---| +| `tool_execution_update` with `toolName: "Agent"` + `details.agentId` | ✅ emitted at a scripted cadence | — | +| `details.entries` | ❌ always `[]` | must GROW 5 → 30 (F1) | +| `subagents:created/started/completed` bus frames | ❌ never emitted | required — the strip, the buffer, and the RESYNC SOURCE all live on that channel (`subagent-frame-buffer.ts` `SUBAGENT_CHANNELS`) | + +Without the bus frames there is no fat snapshot retained, so +`subagent_resync_request` is answered with a no-op and the pull path is not +exercised at all. This is the single load-bearing gap. + +The bus is shared and `registerEventBusForwarding` subscribes with `on()`, which +"observes every emitter" (`flow-event-wiring.ts`) — so frames emitted by a +fixture extension traverse the exact production strip → buffer → resync path, +with no bridge change. + +### Instrumentation inventory — what is and is NOT readable from an E2E spec + +| Signal | Where | E2E-readable? | +|---|---|---| +| Per-frame wire bytes off the browser `/ws` socket | `collectAgentTicks` (`tests/e2e/helpers/index.ts`) | ✅ `tool_execution_update` only today | +| **Resync-reply discriminator** `__resyncRequestId` | `subagent-forward-sites.ts:67`; routed requester-scoped by `server/src/pairing/subagent-resync-routing.ts` | ✅ **on the wire** — the ONLY way to tell a reply from a pushed `subagent_started` | +| Inspector-open share | `__piSubagentInspectorTelemetry()` | ✅ page-global aggregate; `resetInspectorTelemetry` is NOT exposed on `globalThis` | +| `SubagentFrameStats.resyncRequests / resyncServed / resyncCadence` | `subagent-frame-buffer.ts` | ❌ **bridge-process-internal.** The heartbeat carries `subagentTickThrottle.stats` only; `/api/health` carries server-side `storeTrim`. No spec can read these, and exposing them would be production code this change is not allowed to write. | +| Strip rollback switch | `PI_DASHBOARD_SUBAGENT_STRIP=0` (`subagent-frame-strip.ts`), read PER CALL | ⚠️ but it is a CONTAINER env var, fixed for the life of the harness → one harness start per arm | +| Outgoing `subagent_resync_request` frames, incl. their `reason` (`"open"` \| `"cadence"`) | browser `/ws`, observable via Playwright `ws.on("framesent")` | ✅ the only way to tell WHICH trigger produced a reply | +| Mid-run process kill | `force_kill` browser-socket message | ⚠️ closes the bridge WS BEFORE the signal (`session-action-handler.ts`) | + +Three facts below are load-bearing and were verified in source, not assumed: + +- **Terminal frames are never stripped** (`subagent-frame-strip.ts` terminal + guard; `tool_execution_end` stays fat) and the reducer applies their `entries` + (`event-reducer.ts` `subagent_completed` / `tool_execution_end` backfill). A + terminal frame therefore converges a rendered count all by itself. +- **The open-time resync DOES fire in F1, and there are THREE request sites, not + one.** `requestResyncIfStale` requires `emptyTimeline` — and under the strip + the client timeline IS empty at mount (thin pushes omit `entries`; the + reducer's D3 guard never overwrites with an empty array). So expand/popout + (`AgentToolRenderer.tsx:264-268`) fires it, and `App.tsx:1030-1050` fires one + more on session subscribe for every running empty-timeline subagent. A design + that assumes "the cadence is the only pull trigger" is wrong: the + producer-side entry count (5) is NOT the rendered count (0). +- **Resync replies are PERSISTED, fat.** Every `event_forward` is stored + (`event-wiring.ts` → `insertEvent`), and the reply is never stripped. So a + single resync during a run puts a full timeline in the durable store — which + is precisely what X1 must NOT have (V4). +- **The bus carrier is coalesced to 250 ms by the real producer** + (`bridge.ts:384-387`); only `tool_execution_update` is throttled at the + bridge. A fixture emitting bus frames per 50 ms tick is a ~5× inflated push + stream that exists nowhere in production. + +## Goals / Non-Goals + +**Goals:** + +- A harness substrate on which a subagent timeline demonstrably GROWS while an + inspector watches it, with the push frames provably thin and the pull replies + provably fat and identifiable. +- F1 asserted on RENDERED entry count, with no close/reopen, and with the + delivering carrier positively identified — not inferred. +- X1 asserted on a real mid-run process kill followed by a real replay. +- P4 measured as a like-for-like byte comparison over BOTH carriers at ONE + observation point, with its fixture-cadence dependence made explicit. +- P5 recorded with its watch pattern stated, so the number cannot be read as + more than it is. +- All numbers land in this change's own `heap-evidence.md`, cross-referencing + the parent's archived evidence. + +**Non-Goals:** + +- Any production behaviour change — including any new counter, config knob, or + test hook in `packages/`. Everything this change adds lives in + `qa/fixtures/`, `tests/e2e/`, `docker/`, and the evidence file. **Any + assertion that requires a production surface to exist is out of scope by + construction and must be redesigned, not enabled.** +- Making a nested faux subagent scriptable. Abandoned upstream; out of scope. +- Field-representative measurement. A scripted harness cannot produce one (V6). +- Re-verifying anything the parent already gated (P1 growth bound, terminal + fidelity, collapse). + +## Decisions + +### V1 — Extend the synthetic producer into a full pull-path substrate + +`faux-agent-ticks.ext.ts` gains, **entirely behind new prompt sentinels** — the +bus emission AND the entry growth are both sentinel-gated, so every existing +throttle row on the shared `PI_SYNTH_AGENT_TICKS=1` arm is byte-identical: + +- `[[entries:..]]` — the timeline grows from `` to `` + across the tick plan; each frame carries a FULL snapshot of the current + timeline (latest-supersedes, per the pipeline invariant — never a delta). + Entry payloads are fixed-size and small, so the whole serialized snapshot + stays far below the store's ~256 KiB `maxEventDataSize` budget (see V3). +- `[[bus:]]` — emit `subagents:created` once, `subagents:started` + **coalesced at ``, DEFAULT 250 ms to match the real producer** + (`bridge.ts:384-387`), and `subagents:completed` at the end, via + `pi.events.emit`. The bus cadence is deliberately DECOUPLED from the tick + interval: coupling them would fabricate a 5× inflated push arm in P4. + +**Frame-shape contract (load-bearing, must be asserted at L1):** + +- The bus frame's TOP-LEVEL `data.id` MUST equal `details.agentId`. + `SubagentFrameBuffer.agentIdOf` keys snapshots on `data.id` and `resync()` + looks up by it; a fixture that only sets `details.agentId` makes every resync + a silent no-op and every row below unfalsifiable-in-the-wrong-direction. +- `subagents:created` MUST carry NO entries. Its status (`"created"`) is not in + the strip allowlist (`NON_TERMINAL_STATUSES` = `queued` | `running`), so a + fat `created` frame would forward unstripped and break V2 guard (1). + +**Run-shape contract (load-bearing for F1):** the timeline MUST plateau at the +end count and the agent MUST keep emitting RUNNING frames for at least 3 cadence +intervals (≥ ~6 s at `CADENCE_BASE_MS = 2000`) before the terminal frame. If the +count reaches 30 on the last tick and `completed` follows immediately, the DOM is +still below 30 when the non-terminal assertion window closes (the first cadence +reply is 2 s out) — F1 then fails by construction and the terminal frame +"converges" it vacuously. The fixture pins tick interval, growth span, and +plateau hold as explicit numbers, not as "across the tick plan". + +*Rationale:* the bridge keys on channel name + `data.id` + status, never on +producer identity. A fixture emitting the same frames on the same bus is a +faithful substrate for every hop this change verifies, and it is fully +deterministic. + +*Fidelity boundary, stated once and repeated in the fixture header:* this proves +the bridge → server → client pull path. It does NOT prove that +`@blackbelt-technology/pi-dashboard-subagents` emits this shape; that stays +covered by the existing `subagent-spawn` scenario. + +*Alternatives rejected:* fixing the nested faux subagent (root-caused as +unfixable-in-harness upstream); a real model-backed subagent (non-deterministic, +needs credentials); asserting at L1 on a hand-built frame sequence (that is +`useSubagentResyncCadence.test.tsx`, which the proposal already calls +insufficient). + +### V2 — Prove the PULL path is what runs, using the wire discriminator + +The trap this change exists to avoid: if the pushed frames still carry +`entries`, F1 converges on PUSH traffic while looking green. + +`subagent_started` arrives BOTH as a pushed per-interval frame and as a resync +reply, so **eventType cannot classify them.** The discriminator is +`__resyncRequestId`, echoed on the reply frame only. Every classification in +this change keys on it. + +Guard rows, all read off the browser `/ws` socket: + +1. Every observed `tool_execution_update` / `subagent_*` frame for the watched + `agentId` **without** `__resyncRequestId`, while the agent is non-terminal, + carries no (or empty) `entries`. +2. At least one frame **with** `__resyncRequestId` arrives during the run and + carries a NON-empty `entries`. +3. Anti-vacuity arm: on a separate harness start with + `PI_DASHBOARD_SUBAGENT_STRIP=0`, guard (1) INVERTS — pushed frames are fat. + If it does not invert, the arm switch is not wired and every measurement + below is measuring one arm twice. + +No bridge counter is read anywhere. `resyncServed` / `resyncCadence` are +unreadable from a spec, and exposing them is forbidden by the Non-Goals. + +### V3 — F1 asserts RENDERED entries, and identifies the delivering carrier + +Mount the inspector for the running subagent BEFORE the timeline passes the +start count, hold it open for the whole run, and assert the DOM entry count +converges to the end count with no close/reopen and no reload. + +**Anti-vacuity is by carrier exhaustion, not by a disable-run.** FOUR carriers +could deliver 30 entries to the DOM: + +| Carrier | Ruled out how | +|---|---| +| Terminal frame (`subagents:completed` / `tool_execution_end`, never stripped, always fat) | The assertion window **closes while the agent is still non-terminal** — before any terminal frame for that `agentId` is observed on the socket. Asserted, not assumed; the V1 run-shape contract is what makes it reachable. | +| Fat push frames | V2 guard (1) | +| Open-time resync (`AgentToolRenderer` expand/popout, `App.tsx` subscribe) — **fires in F1**, because the rendered timeline IS empty at mount | Its reply carries the buffered snapshot as of BRIDGE-HANDLING time (`serveSubagentResync`), i.e. up to one bus interval + RTT after the request left the browser. The row asserts the DOM passes a count STRICTLY GREATER than the timeline could hold at that moment — requests read off `ws.on("framesent")`, with the bus interval added as slack. Growth spans seconds, so the margin is comfortable. | +| Cadence resync | What remains | + +To make the cadence positive rather than residual, the row ALSO asserts: + +- at least one outgoing `subagent_resync_request` with `reason: "cadence"` was + sent while the inspector stayed mounted (never closed/reopened), and +- a reply whose `__resyncRequestId` **equals that captured cadence request's + `requestId`** carried ≥ the converged count, and arrived before the DOM + reached it. Token equality, not mere ordering — otherwise a reconnect-driven + `reason: "open"` reply would satisfy the clause. + +*A cadence-disable falsifiability run is explicitly NOT used:* no such knob +exists (`CADENCE_BASE_MS` is a module constant; the only "switch" is the hook's +`key` going undefined with the view unmounted), and adding one is production +code the Non-Goals forbid. Carrier exhaustion is the stronger check anyway — it +proves what DID deliver, not merely that something broke when a knob moved. + +*Count choice:* the head-tail ceiling is a **BYTE** budget +(`exceedsSerializedSize(data, maxEventDataSize)`, ~256 KiB, K_HEAD=1/K_TAIL=4), +not an entry count, so "30" is safe only because V1 fixes the per-entry payload +small. This is stated because a count-based intuition here is a category error: +if the budget ever fired, the reply would render 1 + sentinel + 4 and F1 would +fail undiagnosably. 30 is chosen as comfortably observable, well inside the byte +budget. It is NOT chosen to probe the historical `> 20` generic-clobber +boundary: D5a landed (`locateSubagentTimeline` matches `subagent_*`), so that +clobber is unreachable for this carrier and the count tests no boundary at all. + +### V4 — X1 kills the process, and proves absence of a terminal frame in the STORE + +Drive the substrate, let the timeline accumulate, then `force_kill` the session. + +**X1 MUST be an UNWATCHED run, and that is a precondition, not a detail.** A +resync reply is stored FAT (`event-wiring.ts` → `insertEvent`), so ONE resync +during the run puts a full timeline in the durable store and the replay then +shows a mid-run timeline — failing the requirement for a reason that has nothing +to do with the regression. Both open-time request sites must therefore stay +unfired: no inspector mounted, and **the session not selected in the client +during the run** (`App.tsx` fires a subscribe-resync for every running +empty-timeline subagent). Drive the prompt via the API, keep the browser off +that session, and ASSERT zero outgoing `subagent_resync_request` for that +`agentId` before the kill. + +`force_kill` closes the bridge WebSocket BEFORE the signal lands, so "no +terminal frame observed on the socket" proves nothing — it cannot distinguish +"none emitted" from "emitted, not observed". The assertion is therefore made +where it is decidable: + +- The run had not reached its scripted end. The tick index is read POST-KILL + from the replay stream (stored ticks carry their `(running… i)` content), NOT + live: observing ticks live requires subscribing to the session, and + subscribing fires the `App.tsx` open-resync this row must avoid. +- On the post-kill **replay stream** — the stored events the server re-sends on + subscribe — there is no `subagent_completed` / `subagent_failed` / + `tool_execution_end` for that `agentId`, and no stored frame carrying + `entries` for it. There is no list-events endpoint + (`GET /api/events/:sessionId/:seq` returns ONE event by exact seq), so the + replay socket is the observation surface, not a REST sweep. + +Then assert the replayed render: scalar state present, NO mid-run timeline, and +the card neither blank nor error-rendered. Note the replay-time subscribe-resync +is a no-op only because the killed bridge holds no snapshots — pin that +explicitly rather than relying on it silently. + +*Caveat pinned as-observed:* the client's stuck-card supersede-heal may finalize +or badge a killed Agent call. The exact rendered outcome is DETERMINED +EMPIRICALLY in this step and then pinned; the row's job is to stop that +behaviour from drifting silently, not to assert a preferred outcome invented in +advance. Whatever is observed is recorded in `heap-evidence.md` as the pinned +baseline. + +*Fallback:* if `force_kill` turns out to flush a terminal frame into the store, +escalate to a container-level `SIGKILL` of the pi PID and record which mechanism +was used. + +### V5 — P4 is one observation point, both carriers, one variable + +Both halves are read off the SAME browser `/ws` socket, in bytes, over the same +workload and window, with `subagentTickThrottleMs` **identical in both arms** +(the throttle and the strip are different mechanisms; letting both move makes +the number meaningless): + +| Arm | Env | Quantity (per subscriber, for the watched `agentId`) | +|---|---|---| +| pull | strip ON (default) | bytes/s of frames bearing `__resyncRequestId` | +| push | `PI_DASHBOARD_SUBAGENT_STRIP=0` | bytes/s of subagent-carrying frames across **BOTH** carriers (`subagent_*` AND `tool_execution_update`), **EXCLUDING** `__resyncRequestId` frames (the client still pulls in this arm), minus the pull arm's push-side bytes — i.e. the bytes the strip actually removes | + +`PI_DASHBOARD_SUBAGENT_STRIP` is read per call, but it is a container env var +fixed for the life of the harness — so each arm is a SEPARATE harness start, not +a per-test toggle. + +**Byte semantics must be defined before they are compared.** `collectAgentTicks` +today records `bytes: payload.length` for EVERY event inside a batched frame, +over-counting batches N×. The generalized helper attributes bytes per FRAME +(splitting a batch by its own serialized event sizes), and the definition is +recorded alongside the numbers. + +**Pass condition:** pull bytes/s ≤ push bytes/s removed. Exceeding it triggers +the D4 v2 escalation — filed as a follow-up change — NOT a cadence tweak here. + +**The verdict is conditional on the fixture's bus cadence, and the report says +so.** At V1's production-matched 250 ms the comparison is meaningful; at the +50 ms tick interval it would be rigged in the pull arm's favour by 5×. So the +deliverable is a small **sensitivity table** — the ratio at 250 ms (the headline +verdict) plus at least one faster and one slower bus interval — not a single +pass/fail number. A verdict that flips across that table is itself the finding. + +Because the push quantity is a DIFFERENCE across two separate harness starts, it +carries run-to-run batching and timing noise. Each arm runs ≥ 3 times; the table +reports median and spread, and a verdict that falls inside the spread is +recorded as INCONCLUSIVE rather than a pass. **INCONCLUSIVE is a shippable +outcome** — the measurement infrastructure is this change's deliverable; it does +not escalate to D4 v2 (that needs a verdict, not an absence of one) and it does +not block. + +N inspectors = N browser contexts against the same session; the reply is +requester-scoped (`subagent-resync-routing.ts`), so per-subscriber is the honest +unit. N comes from a measured harness ceiling, not an assumption. + +### V6 — P5 reports a SHARE WITH ITS WATCH PATTERN, in a fresh context per arm + +A scripted harness cannot produce a field-representative inspector-open share: +the share is whatever the script's watch pattern makes it. Holding an inspector +open for the whole run yields 100 % by construction — as meaningless as the +parent's 0.0 %. + +Deliverable: **four readings**, each with its pattern and run length recorded: + +| arm | pattern | expected ≈ | +|---|---|---| +| unwatched bound | never opened | 0 % | +| glance | open at 25 % of runtime, hold 25 % | ≈ 25 % | +| threshold | open at 25 % of runtime, hold 50 % | ≈ 50 % — sits ON the C4 boundary, so it tests the boundary rather than a random point | +| watched bound | open before the first entry, never closed | 100 % | + +`__piSubagentInspectorTelemetry()` is a **page-global cumulative aggregate** and +`resetInspectorTelemetry` is not exposed on `globalThis`, so the three arms MUST +run in separate browser contexts. Sharing one context blends them into a single +meaningless share. + +**C4 reporting rule:** a share above 50 % is reported as the kill-switch +condition ONLY together with its pattern. The honest conclusion available here +is "the signal is readable and behaves as expected across the watch spectrum"; +the field number still comes from the production counter, and +`heap-evidence.md` says so in those words. + +### V7 — Evidence lives in THIS change + +`openspec/changes/verify-subagent-pull-under-load/heap-evidence.md` records the +substrate proof (V2), the F1 carrier-exhaustion verdict, the X1 pinned baseline, +the P4 sensitivity table, and the P5 curve. It cross-references +`openspec/changes/archive/2026-08-15-reduce-subagent-details-payload/heap-evidence.md` +§3/§4 by path. The archived file is NOT edited — an archived change is a record, +not a working artifact. + +## Risks / Trade-offs + +- **F1 converges on a carrier other than the cadence reply** (terminal frame, + fat pushes, or open-time resync) → V3's carrier-exhaustion table, with the + non-terminal assertion window as the load-bearing part. This is the failure + mode both review passes flagged as fatal in the first draft. +- **A push/reply mix-up in classification** (`subagent_started` is both) → + everything keys on `__resyncRequestId`, never on eventType. +- **An open-time reply is mistaken for a cadence reply** → outgoing requests are + read off `framesent` and discriminated by `reason`; F1 additionally requires + the converging count to exceed what existed at the last `open` request. +- **A stray resync fattens the X1 store and fails the row for the wrong reason** + → X1 is an explicitly unwatched run with a zero-resync-requests assertion; the + `App.tsx` subscribe-resync is the specific hazard. +- **An assertion needs a production surface that does not exist** (bridge resync + counters, a cadence-disable knob) → both were removed from the design rather + than enabled; the Non-Goals now state this as a hard rule so it cannot creep + back during implementation. +- **P4 becomes arithmetic on fixture constants** → the bus cadence defaults to + the production 250 ms and the verdict ships as a sensitivity table, so a + cadence-driven flip is visible instead of hidden. +- **The strip arm switch is silently unwired**, measuring one arm twice → V2 + guard (3)'s inversion, on a separate harness start. +- **The synthetic substrate is not the real producer** → fidelity boundary + stated in V1, in the fixture header, and in `heap-evidence.md`. +- **X1's replayed render is not what anyone predicted** (supersede-heal + finalizes or badges the card) → V4 pins what is OBSERVED rather than asserting + a guessed outcome; a surprise is recorded, not swallowed. +- **N browser contexts saturate the harness**, so P4 measures the harness → + measure the ceiling first, report per-subscriber, record N with its evidence. +- **The measurement finds the cadence costs more than the push.** A legitimate + outcome, not a failure of this change → routed to a D4 v2 follow-up. + +## Migration Plan + +Strictly ordered — each step is unusable before the previous one is green: + +1. Substrate (V1) + its L1 frame-shape test + the guard rows (V2), including the + strip-OFF inversion and the env passthrough. +2. F1 (V3) with the carrier-exhaustion assertions. +3. X1 (V4); determine and pin the replayed render. +4. P4 (V5) — harness ceiling, then the A/B, then the cadence sensitivity table. +5. P5 (V6) — the watch-pattern arms, one browser context each. +6. `heap-evidence.md` (V7); file the D4 v2 / C4 follow-up change if either + trigger fired. + +Rollback: everything added is test-side (one fixture, harness env passthrough, +E2E specs, one helper generalization). Reverting removes coverage and nothing +else — there is no production surface to roll back. + +## Open Questions + +- Does `force_kill` leave the store free of a terminal frame for the killed + agent, or does a shutdown path flush one? Determined empirically in step 3; + the fallback is a container-level kill. +- What N does the harness sustain for concurrent browser contexts before the + measurement is dominated by contention? +- Should the substrate fixture also cover `subagents:failed` (a terminal-fail + arm), or is `completed` sufficient for the four scenarios in scope? diff --git a/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/heap-evidence.md b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/heap-evidence.md new file mode 100644 index 000000000..334b31020 --- /dev/null +++ b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/heap-evidence.md @@ -0,0 +1,233 @@ +# heap-evidence — verify-subagent-pull-under-load + +Recorded numbers for the four scenarios `reduce-subagent-details-payload` +shipped UNVERIFIED (F1, P4, P5, X1). Raw readings: +`openspec/changes/verify-subagent-pull-under-load/measurements.json`, written by +the specs themselves — this file transcribes, it does not restate from memory. + +Cross-reference, NOT edited: the parent's archived evidence at +`openspec/changes/archive/2026-08-15-reduce-subagent-details-payload/heap-evidence.md` +§3 (inspector-open share, measured 0.0 % on an unwatched harness) and §4 (what +was not measured, and why). This change closes both gaps. + +Environment: `docker/test-up.sh` all-in-one harness built from this worktree, +derived `dashboardPort` from `.pi-test-harness.json` (never `:18000`). Arm +`PI_E2E_SEED=1 PI_TEST_PEERS=both PI_SYNTH_AGENT_TICKS=1`; the push arm adds +`PI_DASHBOARD_SUBAGENT_STRIP=0` and is a SEPARATE harness start. + +--- + +## 0. Substrate, and why it is not the real producer + +`qa/fixtures/faux-agent-ticks.ext.ts` gained two additive, sentinel-gated +behaviours (`[[entries:..@]]`, `[[bus:]]`): a timeline that grows +5 → 30 over ~3 s then PLATEAUS for ~9 s, and real `subagents:created/started/ +completed` EventBus frames coalesced at 250 ms — the real producer's +`PROGRESS_THROTTLE_MS`. + +The bus frames are the load-bearing part: the bridge's strip, its +`SubagentFrameBuffer`, and therefore the entire RESYNC path live on that +channel. A tick-only producer exercises none of it. + +**Fidelity boundary.** This proves the bridge → server → client pull path. It +does NOT prove that `@blackbelt-technology/pi-dashboard-subagents` emits this +frame shape; that stays covered by the `subagent-spawn` faux scenario. The +boundary is stated in the fixture header too. + +**Why not a real faux subagent:** a nested faux subagent dies after ~2 no-op +turns (its inner `createAgentSession` resolves a different faux core with an +empty response queue). Root-caused and written off upstream in +`reduce-bridge-tick-bandwidth`; the synthetic producer is that change's +established route-around. + +--- + +## 1. F1 — a mounted inspector converges via the CADENCE + +`tests/e2e/subagent-pull-under-load.spec.ts`. Inspector mounted before any entry +renders, never closed, no reload. + +| observation | value | +|---|---| +| rendered count reached 30 **while the agent was still non-terminal** | yes | +| non-terminal PUSH frames observed | 102 | +| of those, carrying a timeline | **0** | +| resync REPLIES observed | 3 | +| outgoing `reason: "cadence"` requests | 2 | +| outgoing `reason: "open"` requests | 1 | +| max entries carried by ANY open-time reply | **5** (the timeline at mount) | +| max rendered entries at any sample | 30 (never exceeded) | +| terminal frame entry count | 30 | + +**Anti-vacuity is by carrier exhaustion, not by a disable-run.** Four carriers +could have delivered 30 entries; three are excluded by measurement: + +| carrier | excluded by | +|---|---| +| terminal frame (never stripped, always fat) | the assertion window closes while the agent is non-terminal — asserted, not assumed | +| fat push frames | 0 of the 102 non-terminal pushes carried a timeline (§1 table) | +| open-time resync (expand/popout + `App.tsx` subscribe — all three DO fire, since the RENDERED timeline is empty at mount) | excluded by CONTENT, not ordering: the open-time reply carried **5** entries — the timeline as it stood at mount — so it cannot explain a converged count of 30 | +| **cadence resync** | what remains — and asserted POSITIVELY: the converging reply's `__resyncRequestId` EQUALS a captured `reason: "cadence"` request id | + +Token equality matters: mere ordering would be satisfied by a reconnect-driven +`open` reply. + +A cadence-DISABLE falsifiability run was deliberately not used — no such knob +exists (`CADENCE_BASE_MS` is a module constant) and adding one is production +code this change is not allowed to write. + +### F1 — observed, pinned, out of scope +On completion the finished Agent row is re-grouped into a tool-burst-group +header with **no reachable `Details` pill**, so the inline timeline unmounts. +That is grouping behaviour, not a timeline regression; F7 therefore asserts the +claim where it is decidable (terminal frame carries exactly 30; the rendered +count never exceeded 30) rather than asserting the grouping. + +--- + +## 2. F4 — the anti-vacuity inversion (the arm that makes §1 mean anything) + +Same workload, separate harness start, `PI_DASHBOARD_SUBAGENT_STRIP=0`: +non-terminal PUSH frames carry a timeline — **the §1 assertion inverts**. + +Without this, "0 fat pushes" could equally mean "the fixture never produced a +timeline" or "the env switch is unwired and both arms are the same arm". + +--- + +## 3. P4 — the cadence costs LESS than the push it replaced + +All bytes read off the SAME browser `/ws` socket, per subscriber, classified by +`__resyncRequestId` (never by eventType — a reply and a pushed frame are both +`subagent_started`). N = 3 subscribers, 6 s window, `subagentTickThrottleMs` +identical in both arms. + +| arm | quantity | median | spread | +|---|---|---|---| +| pull (strip ON) | resync-reply bytes/s | **1 059.8** | 9.7 | +| pull (strip ON) | its own stripped push bytes/s | 15 574.8 | 98.5 | +| push (strip OFF) | subagent-carrying push bytes/s, replies excluded | **53 229.2** | 425.5 | + +**Bytes the strip actually removes** = 53 229 − 15 575 ≈ **37 654 B/s**. +**Pull cost** = **1 060 B/s**. + +> **VERDICT: PASS — the pull path costs ~1/36th of the push traffic it removed.** +> Not inconclusive: the gap is ~36×, while the run-to-run spread is ≤ 425 B/s. +> The D4 v2 escalation is NOT triggered. + +### 3a. Bus-cadence sensitivity — the verdict does not flip + +The verdict is a function of how fast the PUSH carrier runs, so a single cadence +would report arithmetic on one fixture constant as if it were a property of the +pipeline. 250 ms is production-matched; 100/1000 ms are the flanks. + +| bus interval | push removed (B/s) | pull (B/s) | ratio | verdict | +|---|---|---|---|---| +| 100 ms | 65 535 − 18 315 = 47 220 | 1 070 | 44× | pass | +| **250 ms (headline)** | 53 229 − 15 575 = 37 654 | 1 060 | **36×** | pass | +| 1000 ms | 47 337 − 14 372 = 32 965 | 1 050 | 31× | pass | + +Pull cost is FLAT across the table (1 050–1 070 B/s) — as designed: the cadence is +one timer per subagent, not per frame. Reply rate held at 0.50/s in every arm. + +### 3b. Harness ceiling (P1) — measured, not assumed + +| subscribers | frames in a 6 s window | +|---|---| +| 1 | 145 | +| 3 | 143 | +| 5 | 143 | + +Lossless to N = 5; the byte measurement ran at N = 3, comfortably inside it. A +saturated harness therefore cannot masquerade as a byte-rate result. + +--- + +## 4. P5 — inspector-open share, WITH its watch pattern + +The parent measured 0.0 % on a workload that never opened an inspector — the +unwatched arm by construction. A scripted harness cannot produce a +FIELD-representative number either: the share is whatever the pattern makes it. +So four patterns were run, each in its OWN browser context +(`__piSubagentInspectorTelemetry()` is a page-global cumulative aggregate and +`resetInspectorTelemetry` is not exposed on `globalThis`). + +| arm | pattern | open / runtime | **realized share** | +|---|---|---|---| +| unwatched | never opened | 0 / 11 175 ms | **0.0 %** | +| glance | open at 25 % of runtime, hold 25 % | 3 055 / 17 292 ms | **17.7 %** | +| threshold | open at 25 % of runtime, hold 50 % | 6 071 / 20 311 ms | **29.9 %** | +| watched | open before the first entry, never closed | 12 003 / 12 051 ms | **99.6 %** | + +Realized ≠ nominal for the two middle arms: the telemetry's denominator is the +subagent's WHOLE runtime, which the mount interaction itself extends (17.3 s and +20.3 s vs the fixture's 12 s). The realized figure is reported; the nominal one +would have been a fiction. + +**C4 kill switch: NOT triggered.** No realistic watch pattern exceeded 50 %; only +the by-construction always-open arm did, at 99.6 %. What this run establishes is +that the signal is readable end-to-end and behaves monotonically across the watch +spectrum. **The field number still comes from the production counter** — this is +a harness reading, not a claim about real sessions. + +--- + +## 5. X1 — killed mid-run with no terminal frame, then replayed + +The parent's documented REGRESSION, previously unexercised anywhere. + +Setup validity is asserted, not assumed: a resync reply is stored FAT (every +`event_forward` is persisted; replies are never stripped), so ONE resync during +the run would put a full timeline in the store and fail X1 for a reason unrelated +to the regression. The run is therefore UNWATCHED — no inspector, session never +selected (`App.tsx` resyncs every running empty-timeline subagent on subscribe). + +| observation | value | +|---|---| +| outgoing resync requests before the kill | **0** | +| terminal frame observed before the kill | none | +| kill mechanism | `force_kill` (the container-level `SIGKILL` fallback was NOT needed) | +| replayed subagent frames | 8 | +| stored terminal frames for that agent | **0** | +| stored frames carrying a timeline | **0** | +| rendered timeline entries after replay | **0** | +| render blank or `subagent not found` | no | + +`force_kill` closes the bridge WS BEFORE the signal, so socket silence proves +nothing; the assertions above are made on the STORED transcript the server +re-sends on subscribe (there is no list-events endpoint — +`GET /api/events/:sessionId/:seq` returns one event by exact seq). + +**Pinned as observed:** the replayed card shows scalar state, no mid-run +timeline, and is neither blank nor error-rendered. This row pins CURRENT accepted +behaviour so it cannot drift silently; it does not claim the behaviour is +desirable. + +*Known weak spot:* `lastObservedTickIndex` recorded −1 — the `(running… i)` +content is carried by the `tool_execution_update` carrier, which the replay +stream did not surface for this agent, so the tick index could not be recovered +post-kill. "The run had not reached its scripted end" therefore rests on the +pre-kill assertion that no terminal frame was seen plus the 0 stored terminal +frames, not on the index. Recorded rather than papered over. + +--- + +## 6. What this change did NOT establish + +- **Field-representative inspector-open share.** Structurally impossible from a + scripted harness (§4). The production counter remains the only source. +- **Real-producer frame fidelity.** §0's boundary. +- **A pre-change baseline.** Not applicable — this change adds no production + behaviour; both P4 arms are the same binary with one env var moved. + +--- + +## 7. Transcription provenance + +Every number above is transcribed from +`openspec/changes/verify-subagent-pull-under-load/measurements.json`, written by +the specs themselves. Medians and the push−pull subtraction are computed FROM +those rows, not restated from a console reading — an earlier revision of this +file was transcribed from a first run and then went stale when the specs were +re-run after review fixes, which a round-2 review caught. If the specs are re-run, +re-derive this file from the JSON rather than editing numbers in place. diff --git a/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/measurements.json b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/measurements.json new file mode 100644 index 000000000..2e03559e3 --- /dev/null +++ b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/measurements.json @@ -0,0 +1,232 @@ +{ + "X1": { + "killMechanism": "force_kill", + "resyncRequestsDuringRun": 0, + "replayedFrames": 8, + "storedTerminalFrames": 0, + "storedTimelineFrames": 0, + "lastObservedTickIndex": -1, + "renderedEntryCount": 0 + }, + "F1": { + "convergedBeforeTerminal": true, + "pushFramesObserved": 102, + "fatPushFrames": 0, + "repliesObserved": 3, + "cadenceRequests": 2, + "openRequests": 1, + "maxEntriesInAnyOpenTimeReply": 5, + "maxRenderedDuringRun": 30, + "terminalFrameEntryCount": 30, + "observed": "on completion the Agent row is re-grouped into a tool-burst-group header with no reachable Details pill, so the inline timeline unmounts. Pinned as observed; out of scope for this change." + }, + "P1-harness-ceiling": { + "windowMs": 6000, + "samples": [ + { + "n": 1, + "frames": 145 + }, + { + "n": 3, + "frames": 143 + }, + { + "n": 5, + "frames": 143 + } + ], + "baseline": 145, + "ceiling": 5 + }, + "F4": { + "stripOffPushesAreFat": true + }, + "P2-P3-push-arm": { + "windowMs": 6000, + "N": 3, + "rows": [ + { + "bus": 250, + "run": 0, + "N": 3, + "replyBytesPerSec": 379, + "pushBytesPerSec": 53635, + "repliesPerSec": 0.16666666666666666, + "frames": 142, + "toolCarrierFrames": 117, + "busCarrierFrames": 25 + }, + { + "bus": 250, + "run": 1, + "N": 3, + "replyBytesPerSec": 379, + "pushBytesPerSec": 53229.166666666664, + "repliesPerSec": 0.16666666666666666, + "frames": 142, + "toolCarrierFrames": 118, + "busCarrierFrames": 24 + }, + { + "bus": 250, + "run": 2, + "N": 3, + "replyBytesPerSec": 379, + "pushBytesPerSec": 53209.5, + "repliesPerSec": 0.16666666666666666, + "frames": 142, + "toolCarrierFrames": 118, + "busCarrierFrames": 24 + }, + { + "bus": 100, + "run": 0, + "N": 3, + "replyBytesPerSec": 379, + "pushBytesPerSec": 65534.833333333336, + "repliesPerSec": 0.16666666666666666, + "frames": 178, + "toolCarrierFrames": 118, + "busCarrierFrames": 60 + }, + { + "bus": 1000, + "run": 0, + "N": 3, + "replyBytesPerSec": 379, + "pushBytesPerSec": 47336.83333333333, + "repliesPerSec": 0.16666666666666666, + "frames": 124, + "toolCarrierFrames": 117, + "busCarrierFrames": 7 + } + ], + "headline": { + "busIntervalMs": 250, + "pushBytesPerSecMedian": 53229.166666666664, + "pushBytesPerSecSpread": 425.5 + }, + "note": "pushBytesPerSec EXCLUDES __resyncRequestId frames: the client still pulls in this arm, so counting replies here would inflate 'the bytes the strip removes'." + }, + "P2-P3-pull-arm": { + "windowMs": 6000, + "N": 3, + "rows": [ + { + "bus": 250, + "run": 0, + "N": 3, + "replyBytesPerSec": 1050.1666666666667, + "pushBytesPerSec": 15574.833333333334, + "repliesPerSec": 0.5, + "frames": 142, + "toolCarrierFrames": 116, + "busCarrierFrames": 26 + }, + { + "bus": 250, + "run": 1, + "N": 3, + "replyBytesPerSec": 1059.8333333333335, + "pushBytesPerSec": 15554.5, + "repliesPerSec": 0.5, + "frames": 142, + "toolCarrierFrames": 116, + "busCarrierFrames": 26 + }, + { + "bus": 250, + "run": 2, + "N": 3, + "replyBytesPerSec": 1059.8333333333335, + "pushBytesPerSec": 15653, + "repliesPerSec": 0.5, + "frames": 143, + "toolCarrierFrames": 116, + "busCarrierFrames": 27 + }, + { + "bus": 100, + "run": 0, + "N": 3, + "replyBytesPerSec": 1069.5, + "pushBytesPerSec": 18315.333333333336, + "repliesPerSec": 0.5, + "frames": 177, + "toolCarrierFrames": 116, + "busCarrierFrames": 61 + }, + { + "bus": 1000, + "run": 0, + "N": 3, + "replyBytesPerSec": 1050.1666666666667, + "pushBytesPerSec": 14371.666666666666, + "repliesPerSec": 0.5, + "frames": 126, + "toolCarrierFrames": 117, + "busCarrierFrames": 9 + } + ], + "headline": { + "busIntervalMs": 250, + "replyBytesPerSecMedian": 1059.8333333333335, + "replyBytesPerSecSpread": 9.666666666666742 + }, + "note": "Pull = bytes/s of frames bearing __resyncRequestId, per subscriber. The bytes the strip REMOVES are (push_stripOff - push_stripOn), the push arm coming from the separate PI_DASHBOARD_SUBAGENT_STRIP=0 harness start. A verdict inside the run-to-run spread is INCONCLUSIVE — a shippable outcome, since the measurement infrastructure is the deliverable." + }, + "P4-inspector-open-share": { + "note": "A scripted harness cannot yield a FIELD-representative share — the number is whatever the watch pattern makes it. Each reading is reported WITH its pattern. The 50% arm sits ON the C4 boundary by construction. The field number still comes from the production counter.", + "readings": [ + { + "id": "unwatched", + "pattern": "never opened", + "openAt": -1, + "holdFor": 0, + "runtimeMs": 12000, + "telemetry": { + "runtimeMs": 11175, + "inspectorOpenMs": 0, + "share": 0 + } + }, + { + "id": "glance", + "pattern": "open at 25% of runtime, hold 25%", + "openAt": 0.25, + "holdFor": 0.25, + "runtimeMs": 12000, + "telemetry": { + "runtimeMs": 17292, + "inspectorOpenMs": 3055, + "share": 0.17667129308350682 + } + }, + { + "id": "threshold", + "pattern": "open at 25% of runtime, hold 50%", + "openAt": 0.25, + "holdFor": 0.5, + "runtimeMs": 12000, + "telemetry": { + "runtimeMs": 20311, + "inspectorOpenMs": 6071, + "share": 0.2989020727684506 + } + }, + { + "id": "watched", + "pattern": "open before the first entry, never closed", + "openAt": 0, + "holdFor": 1, + "runtimeMs": 12000, + "telemetry": { + "runtimeMs": 12051, + "inspectorOpenMs": 12003, + "share": 0.996016928055763 + } + } + ] + } +} diff --git a/openspec/changes/verify-subagent-pull-under-load/proposal.md b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/proposal.md similarity index 100% rename from openspec/changes/verify-subagent-pull-under-load/proposal.md rename to openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/proposal.md diff --git a/openspec/changes/verify-subagent-pull-under-load/specs/subagent-details-payload/spec.md b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/specs/subagent-details-payload/spec.md similarity index 100% rename from openspec/changes/verify-subagent-pull-under-load/specs/subagent-details-payload/spec.md rename to openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/specs/subagent-details-payload/spec.md diff --git a/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/tasks.md b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/tasks.md new file mode 100644 index 000000000..a6f41e053 --- /dev/null +++ b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/tasks.md @@ -0,0 +1,157 @@ +## 1. Substrate — a watchable, growing subagent timeline (design V1) + +- [x] 1.1 Add an `[[entries:..]]` sentinel to + `qa/fixtures/faux-agent-ticks.ext.ts` growing `details.entries` across the + tick plan, each frame a FULL snapshot. Entry payloads fixed-size and small + so the serialized snapshot stays far under the store's ~256 KiB + `maxEventDataSize`. +- [x] 1.2 Add a `[[bus:]]` sentinel emitting `subagents:created` + once, `subagents:started` coalesced at the interval (DEFAULT 250 ms, to + match the real producer's `PROGRESS_THROTTLE_MS`), and + `subagents:completed` at the end, via `pi.events.emit`. The bus cadence is + decoupled from the tick interval. +- [x] 1.3 Pin the run shape for F1: the timeline plateaus at the end count and + the agent keeps emitting RUNNING frames for ≥ 3 cadence intervals + (≥ ~6 s at `CADENCE_BASE_MS = 2000`) before the terminal frame. Tick + interval, growth span and plateau hold are explicit fixture numbers. +- [x] 1.4 Add the `subagent-watched-growth` faux scenario to + `qa/fixtures/faux-scenarios.ts` driving the extended producer. +- [x] 1.5 Add a `PI_DASHBOARD_SUBAGENT_STRIP` passthrough to + `docker/compose.test.yml`; update `docker/AGENTS.md`. (No + `test-entrypoint.sh` change needed — `subagent-frame-strip.ts` reads + `process.env` directly, so the compose env reaches the bridge through + normal inheritance. Verified by the arm difference: 53 229 B/s unstripped + vs 15 575 B/s stripped.) +- [x] 1.6 Generalize `collectAgentTicks` (`tests/e2e/helpers/index.ts`): + classify `subagent_*` frames, capture `__resyncRequestId`, capture + OUTGOING frames via `ws.on("framesent")` (with their `reason` + + `requestId`), and attribute bytes PER FRAME rather than per event inside a + batch (today's `payload.length` per event over-counts batches N×). Keep + the existing `TickSample` / `agentRate` contract intact for + `subagent-tick-throttle.spec.ts`. + +## 2. L1 — fixture contract (test-plan E1–E6) + +Exemplar for all of §2: `packages/server/src/__tests__/faux-agent-ticks.unit.test.ts`. + +- [x] 2.1 E1: no `[[entries:]]` sentinel · parse the growth plan · plan absent + and every frame carries `entries: []`, byte-identical to today + (test-plan #E1). +- [x] 2.2 E2: `[[entries:5..30]]`, `0..0`, `30..5`, malformed, absurdly large · + parse · 5..30 accepted, 0..0 always-empty, start>end and malformed fall + back to no-growth, count clamps with no unbounded loop (test-plan #E2). +- [x] 2.3 E3: each emitted bus frame (`created`/`started`/`completed`) · frame + construction · top-level `data.id === details.agentId` on all three (the + key `SubagentFrameBuffer.agentIdOf` reads) (test-plan #E3). +- [x] 2.4 E4: the `subagents:created` frame · frame construction · + `details.entries` empty, because `"created"` is not in + `NON_TERMINAL_STATUSES` and a fat `created` frame would forward unstripped + (test-plan #E4). +- [x] 2.5 E5: the full frame sequence for `[[entries:5..30]]` · replay it · every + `entries` is a FULL snapshot and a prefix-superset of its predecessor — + no delta encoding leaked in (test-plan #E5). +- [x] 2.6 E6: `[[bus:]]` absent / `250` / `0` / out-of-range · parse · default + 250 ms, clamped bounds, interval independent of the tick interval + (test-plan #E6). + +## 3. L3 — substrate guard, the pull path is what runs (test-plan E7, F2–F4) + +Exemplar for all of §3–§6: `tests/e2e/subagent-tick-throttle.spec.ts` (same +`PI_SYNTH_AGENT_TICKS=1` gated arm, same harness bring-up + derived port). + +- [x] 3.1 New `tests/e2e/subagent-pull-under-load.spec.ts`, self-skipping unless + `PI_SYNTH_AGENT_TICKS=1`, with the harness bring-up documented in the + header. +- [x] 3.2 E7: the existing `synthetic-agent-ticks` scenarios (no new sentinels) · + run `subagent-tick-throttle.spec.ts` unchanged · every row still passes and + no `subagents:*` bus traffic is observed — the emission is fully + sentinel-gated (test-plan #E7). +- [x] 3.3 F2: the watched run · every `/ws` frame for the watched `agentId` + WITHOUT `__resyncRequestId` while non-terminal · carries no or empty + `entries` (test-plan #F2). +- [x] 3.4 F3: the watched run · at least one `/ws` frame WITH + `__resyncRequestId` · carries a NON-empty `entries` (test-plan #F3). +- [x] 3.5 F4: the same run on a SEPARATE harness start with + `PI_DASHBOARD_SUBAGENT_STRIP=0` · same observation as F2 · the assertion + INVERTS (pushes fat). Do not record any §4–§6 measurement until this + inverts (test-plan #F4). + +## 4. L3 — F1 convergence with carrier attribution (test-plan F1, F5–F7) + +- [x] 4.1 F1: `[[entries:5..30]]` with the inspector mounted before any entry + renders and never closed · timeline grows to 30 then plateaus · the + RENDERED count reaches 30 while the agent is still non-terminal, no + close/reopen, no reload (test-plan #F1). +- [x] 4.2 F5: outgoing frames via `framesent` · the cadence fires while mounted · + ≥ 1 request with `reason: "cadence"`, and the reply carrying ≥ 30 entries + has `__resyncRequestId` EQUAL to that request's `requestId` — token + equality, not ordering (test-plan #F5). +- [x] 4.3 F6: the `reason: "open"` requests (expand/popout + `App.tsx` + subscribe — all three DO fire, the rendered timeline is empty at mount) · + the last `open` request on `framesent` · the DOM passes a count strictly + greater than the timeline could hold at that request's bridge-handling + time, with one bus interval + RTT slack (test-plan #F6). +- [x] 4.4 F7: the same run allowed to complete · terminal `subagents:completed` + after the plateau · the rendered count stays 30, no regression or + duplication (test-plan #F7). + +## 5. L3 — X1, killed mid-run and replayed (test-plan X1–X4) + +- [x] 5.1 X2: setup validity · the run is driven via the API with the session + NEVER selected in the client · ZERO outgoing `subagent_resync_request` for + that `agentId` before the kill (a stored FAT reply would fail X1 for the + wrong reason) (test-plan #X2). +- [x] 5.2 X1: pi killed mid-run · `force_kill` while the tick plan is + unexhausted, then replay · scalar state renders, NO mid-run timeline, card + neither blank nor error-rendered; the observed badge/finalize state is + recorded on the first run and pinned as the baseline (test-plan #X1). +- [x] 5.3 X3: the post-kill replay stream · subscribe after the kill · no + `subagent_completed` / `subagent_failed` / `tool_execution_end` and no + stored `entries`-bearing frame for that `agentId`; the tick index is read + from the replayed `(running… i)` content, not live (test-plan #X3). +- [x] 5.4 X4: `force_kill` flushes a terminal frame · X3 fails · escalate to a + container-level `SIGKILL` of the pi PID and record the mechanism used + (test-plan #X4). + +## 6. L3 — measurement (test-plan P1–P4) + +- [x] 6.1 P1: N browser contexts on one session, N increasing · added contexts · + the largest N at which frame delivery does not degrade — the harness + ceiling, measured not assumed (test-plan #P1). +- [x] 6.2 P2: `[[entries:5..30]][[bus:250]]` with N inspectors open, 2 separate + harness starts (strip ON / OFF), ≥ 3 runs each · measure · per-subscriber + `__resyncRequestId` bytes/s ≤ subagent-carrying bytes/s across BOTH + carriers excluding `__resyncRequestId` frames, median of ≥ 3 with spread + (test-plan #P2). +- [x] 6.3 P3: P2 repeated at a faster and a slower `[[bus:]]` interval · measure · + the pull/push ratio as a function of bus cadence; a verdict that flips + across the table is itself the finding (test-plan #P3). +- [x] 6.4 P4: four arms in four SEPARATE browser contexts (never-opened; + open@25 % hold 25 %; open@25 % hold 50 %; open-before-first-entry + never-closed) · read `__piSubagentInspectorTelemetry()` · shares ≈ 0 / 25 / + 50 / 100 %, each recorded with its pattern and run length (test-plan #P4). +- [x] 6.5 Apply the verdict rules: P2 exceeding its bound → file the D4 v2 + escalation as a FOLLOW-UP change (never a cadence tweak here); P2 inside + the spread → record INCONCLUSIVE and ship; P4 > 50 % → report as the C4 + kill-switch condition together with its pattern. + +## 7. Evidence + close-out + +- [x] 7.1 Write `openspec/changes/verify-subagent-pull-under-load/heap-evidence.md`: + substrate proof, F1 carrier-exhaustion verdict, X1 pinned baseline, P2/P3 + sensitivity table with medians and spread, P4 four-arm curve — cross- + referencing the archived parent's `heap-evidence.md` §3/§4 by path. Do NOT + edit the archived file. +- [ ] 7.2 (test-plan: manual-only, #X5) Manual verification of the replayed X1 + card beyond the automated checks — a human confirms it is not visually + corrupt. +- [ ] 7.3 (test-plan: manual-only, #M1) Manual review of the finished + `heap-evidence.md` — every number reported with its watch pattern / bus + cadence / arm, the synthetic substrate's fidelity boundary stated, the + archived evidence referenced and unedited. (Numeric transcription is + already verified MECHANICALLY: 24/24 values match measurements.json.) +- [x] 7.4 Update the per-file rows in `qa/fixtures/AGENTS.md`, + `tests/e2e/AGENTS.md`, `docker/AGENTS.md`, and the row for every other + file touched; delegate any `docs/` prose to DocScribe. +- [x] 7.5 Full test pass (`npm test`) plus the gated E2E arm; then run + `review-code` on the diff before shipping. diff --git a/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/test-plan.md b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/test-plan.md new file mode 100644 index 000000000..316e38f05 --- /dev/null +++ b/openspec/changes/archive/2026-08-24-verify-subagent-pull-under-load/test-plan.md @@ -0,0 +1,100 @@ +# Test Plan — verify-subagent-pull-under-load + +Stage: design Generated: 2026-08-24 + +No open clarifications — the three decision-forcing gaps (P5 watch pattern, X1 +expected render, P4 inconclusive handling) were answered and folded into +`design.md` (V4, V5, V6). + +Requirement refs: + +- **R1** — the open-inspector pull path SHALL be verified against a watched, + growing timeline (spec scenarios F1 / P4 / P5). +- **R2** — the crash-without-terminal-frame regression SHALL be pinned (spec + scenario X1). + +The harness arm for every L3 row is +`PI_E2E_SEED=1 PI_TEST_PEERS=both PI_SYNTH_AGENT_TICKS=1 ./docker/test-up.sh -d`, +read against the derived `dashboardPort` in `.pi-test-harness.json` — never +`:18000`. + +--- + +## Scenarios + +### Edge-case + +| id | requirement | technique | level | disposition | input | trigger | expected observable | +|----|-------------|-----------|-------|-------------|-------|---------|---------------------| +| E1 | R1 (substrate) | BVA | L1 | automated | prompt with no `[[entries:]]` sentinel | `parseTickPlan`-equivalent parse of the growth plan | growth plan absent; every frame carries `entries: []` — byte-identical to today's fixture | +| E2 | R1 (substrate) | BVA | L1 | automated | `[[entries:5..30]]`, `[[entries:0..0]]`, `[[entries:30..5]]` (start>end), `[[entries:x..y]]` (malformed), `[[entries:1..99999999]]` | parse | 5..30 accepted; 0..0 yields an always-empty timeline; start>end and malformed fall back to the no-growth default; the count clamps at the fixture's `MAX_TICKS`-style bound (no unbounded loop) | +| E3 | R1 (substrate) | decision-table | L1 | automated | each emitted bus frame (`created`, `started`, `completed`) | frame construction | top-level `data.id === details.agentId` on ALL three — the key `SubagentFrameBuffer.agentIdOf` reads; a frame keyed only inside `details` is rejected by the test | +| E4 | R1 (substrate) | decision-table | L1 | automated | the `subagents:created` frame | frame construction | `details.entries` is empty. `"created"` is NOT in `NON_TERMINAL_STATUSES` (`queued`/`running`), so a fat `created` frame would forward unstripped and silently defeat F2 | +| E5 | pipeline invariant | state-transition | L1 | automated | the full frame sequence for one run with `[[entries:5..30]]` | replay the sequence | every frame's `entries` is a FULL snapshot and a prefix-superset of its predecessor (`entries[i]` never mutates, length is monotone non-decreasing) — no delta encoding leaked in | +| E6 | R1 (substrate) | BVA | L1 | automated | `[[bus:]]` absent / `[[bus:250]]` / `[[bus:0]]` / `[[bus:999999]]` | parse | default is 250 ms (matches the real producer's `PROGRESS_THROTTLE_MS`); 0 and out-of-range clamp; the bus interval is INDEPENDENT of the tick interval | +| E7 | no-regression | state-transition | L3 | automated | the existing `synthetic-agent-ticks` / `-quiet` scenarios (no new sentinels) | run `subagent-tick-throttle.spec.ts` unchanged on the same arm | every existing throttle row still passes and observes no `subagents:*` bus traffic — the new emission is fully sentinel-gated | + +### Frontend-quirk + +| id | requirement | technique | level | disposition | input | trigger | expected observable (invariant) | +|----|-------------|-----------|-------|-------------|-------|---------|---------------------------------| +| F1 | R1 / spec F1 | state-convergence | L3 | automated | `[[entries:5..30]]` run, inspector mounted before the DOM shows any entry, never closed | timeline grows to 30 and then PLATEAUS while the agent keeps emitting running frames for ≥ 3 cadence intervals (≥ ~6 s) | the RENDERED entry count reaches 30 **while the agent is still non-terminal** (no `subagents:completed` / `tool_execution_end` for that `agentId` observed yet), with no close/reopen and no reload | +| F2 | R1 (V2 guard 1) | invariant | L3 | automated | same run | every `/ws` frame for the watched `agentId` WITHOUT `__resyncRequestId`, while non-terminal | carries no `entries` key, or an empty one — the push path is provably thin | +| F3 | R1 (V2 guard 2) | invariant | L3 | automated | same run | at least one `/ws` frame WITH `__resyncRequestId` | carries a NON-empty `entries` — the pull path is provably fat and identifiable | +| F4 | R1 (anti-vacuity) | invariant | L3 | automated | same run on a SEPARATE harness start with `PI_DASHBOARD_SUBAGENT_STRIP=0` | same observation as F2 | F2's assertion INVERTS (pushes are fat). A non-inverting arm means the env switch is unwired and every measurement below is one arm measured twice | +| F5 | R1 / spec F1 (carrier attribution) | state-transition | L3 | automated | outgoing frames captured via `ws.on("framesent")` | the cadence fires while the inspector stays mounted | ≥ 1 outgoing `subagent_resync_request` with `reason: "cadence"`; the reply that carried ≥ 30 entries has `__resyncRequestId` EQUAL to that request's `requestId` — token equality, not ordering | +| F6 | R1 / spec F1 (open-time exclusion) | state-transition | L3 | automated | the `reason: "open"` requests (expand/popout + `App.tsx` subscribe — all three DO fire, because the rendered timeline is empty at mount) | last `open` request observed on `framesent` | the DOM passes a count STRICTLY GREATER than the timeline could hold at that request's bridge-handling time (request time + one bus interval + RTT slack) — an open-time reply cannot deliver a count that did not exist yet | +| F7 | R1 (illegal edge) | state-transition | L3 | automated | the same run, allowed to complete | terminal `subagents:completed` arrives after the plateau | the rendered count stays 30 — the terminal frame does not regress or duplicate the timeline | + +### Performance + +| id | requirement | technique | level | disposition | workload | metric + threshold | window | +|----|-------------|-----------|-------|-------------|----------|--------------------|--------| +| P1 | R1 / spec P4 (prereq) | threshold | L3 | automated | N browser contexts subscribed to one session, N increasing | the largest N at which added contexts do not degrade observed frame delivery — the harness ceiling | per-N run until the rate flattens | +| P2 | R1 / spec P4 | A/B byte-rate | L3 | automated | `[[entries:5..30]][[bus:250]]`, N inspectors held open (N from P1), 2 separate harness starts (strip ON / `PI_DASHBOARD_SUBAGENT_STRIP=0`), each arm ≥ 3 runs | per-subscriber `__resyncRequestId` bytes/s (pull) ≤ subagent-carrying bytes/s across BOTH carriers excluding `__resyncRequestId` frames (push), median of ≥ 3 | the plateau window of each run | +| P3 | R1 / spec P4 (validity) | sensitivity | L3 | automated | P2 repeated at a faster and a slower `[[bus:]]` interval | the pull/push ratio as a function of bus cadence | same window | +| P4 | R1 / spec P5 | measurement | L3 | automated | four arms, each in its OWN browser context (`__piSubagentInspectorTelemetry()` is a page-global cumulative aggregate and `resetInspectorTelemetry` is not on `globalThis`): never-opened; open@25 % hold 25 %; open@25 % hold 50 %; open-before-first-entry never-closed | inspector-open share ≈ 0 % / 25 % / 50 % / 100 % respectively, each recorded WITH its pattern and run length | one full subagent run per arm | + +Verdict handling (from `design.md` V5/V6): + +- P2 exceeding its bound → file the D4 v2 escalation as a follow-up change; do + NOT tune the cadence here. +- P2 falling INSIDE the run-to-run spread → record **INCONCLUSIVE**; this is a + shippable outcome, since the measurement infrastructure is the deliverable. +- P4's 50 % arm sits ON the C4 boundary by construction, so a > 50 % reading is + reported as the C4 kill-switch condition **only together with its pattern**. + +### Error-handling + +| id | requirement | technique | level | disposition | fault | trigger | expected observable | +|----|-------------|-----------|-------|-------------|-------|---------|---------------------| +| X1 | R2 / spec X1 | fault-injection (abort) | L3 | automated | pi process killed mid-run | `force_kill` on the browser socket while the tick plan is unexhausted, then the session is replayed | the subagent renders its SCALAR state, NO mid-run timeline is shown, and the card is neither blank nor error-rendered. Whatever badge/finalize state the stuck-card supersede-heal produces is RECORDED on the first run and pinned as the baseline | +| X2 | R2 (X1 precondition) | invariant | L3 | automated | none — setup validity | the run is driven via the API with the session NEVER selected in the client | ZERO outgoing `subagent_resync_request` for that `agentId` before the kill. A single resync would store a FAT reply (`event_forward` → `insertEvent`, replies are never stripped) and fail X1 for a reason unrelated to the regression | +| X3 | R2 (X1 decidability) | invariant | L3 | automated | the post-kill replay stream (the stored events the server re-sends on subscribe — there is no list-events endpoint; `GET /api/events/:sessionId/:seq` returns one event by exact seq) | subscribe after the kill | no `subagent_completed` / `subagent_failed` / `tool_execution_end` for that `agentId`, and no stored frame carrying `entries` for it. The tick index is read from the replayed `(running… i)` content — NOT live, since live observation requires subscribing, which fires the `App.tsx` open-resync X2 forbids | +| X4 | R2 (fallback) | fault-injection (abort) | L3 | automated | `force_kill` turns out to flush a terminal frame into the store | X3 fails | escalate to a container-level `SIGKILL` of the pi PID; the mechanism actually used is recorded in `heap-evidence.md` | +| X5 | R2 / spec X1 ("neither corrupt") | visual/subjective | — | manual-only | the replayed card | a human looks at it | [judgment: "not corrupt" beyond the automatable checks in X1 — no automatable observable] | + +### Reporting + +| id | requirement | technique | level | disposition | surface | trigger | expected observable | +|----|-------------|-----------|-------|-------------|---------|---------|---------------------| +| M1 | R1 / spec P5 + P4 | review | — | manual-only | `heap-evidence.md` | a human reads the finished evidence file | [judgment: every number is reported WITH its watch pattern / bus cadence / arm, the fidelity boundary of the synthetic substrate is stated, and the archived parent evidence is cross-referenced by path and NOT edited] | + +--- + +## Coverage summary + +- Requirements covered: 2/2 (R1 via E1–E7, F1–F7, P1–P4, M1; R2 via X1–X5) +- Scenarios by class: edge 7 · perf 4 · frontend 7 · error 5 · reporting 1 +- Scenarios by level: L1 6 · L2 0 · L3 16 · — 2 +- Scenarios by disposition: automated 22 · manual-only 2 + +## New infra needed + +- None architecturally new. Extensions to existing infra only: the + `[[entries:]]` / `[[bus:]]` sentinels on `qa/fixtures/faux-agent-ticks.ext.ts`, + a `PI_DASHBOARD_SUBAGENT_STRIP` passthrough in `docker/compose.test.yml` + + `docker/test-entrypoint.sh`, and a `collectAgentTicks` generalization + (classify `subagent_*` frames, capture `__resyncRequestId`, capture outgoing + frames via `framesent`, and attribute bytes PER FRAME rather than per event + inside a batch — the current helper over-counts batches N×). diff --git a/openspec/specs/subagent-details-payload/spec.md b/openspec/specs/subagent-details-payload/spec.md index f1756de1c..69675dfdf 100644 --- a/openspec/specs/subagent-details-payload/spec.md +++ b/openspec/specs/subagent-details-payload/spec.md @@ -90,3 +90,48 @@ preserving latest-supersedes semantics at every hop. - **AND** no producer change, capability flag, or version negotiation SHALL be required, because the reduction happens downstream of the producer +### Requirement: The open-inspector pull path SHALL be verified against a watched, growing timeline + +The parent change's cadence is asserted only at the hook level. A mounted +inspector watching a RUNNING subagent SHALL be shown to converge on the rendered +timeline, and the cost of that pull SHALL be measured against the push traffic it +replaced. + +#### Scenario: A mounted inspector converges on a growing timeline + +- **GIVEN** a subagent that stays running long enough to be observed +- **AND** its inspector is mounted before the timeline grows +- **WHEN** the timeline grows from 5 to 30 entries +- **THEN** the RENDERED entry count converges to 30 +- **AND** no close/reopen of the inspector is required + +#### Scenario: The cadence costs no more than the push it replaced + +- **GIVEN** the same workload with N inspectors held open +- **WHEN** resync replies/s and per-subscriber reply bytes/s are measured +- **THEN** the reply byte rate SHALL NOT exceed the push byte rate the strip removed +- **AND** exceeding it SHALL trigger the D4 v2 escalation rather than a cadence tweak + +#### Scenario: The inspector-open share is measured representatively + +- **GIVEN** a workload in which inspectors ARE opened during subagent runtime +- **WHEN** the inspector-open share of subagent runtime is read +- **THEN** the recorded number SHALL come from that watched workload, not from an unwatched run +- **AND** a share above 50 % SHALL be reported as the C4 kill-switch condition + +### Requirement: The crash-without-terminal-frame regression SHALL be pinned + +A run that terminates without emitting a terminal frame SHALL replay as scalar +state with no mid-run timeline, and the render SHALL be neither corrupt nor +blank. The parent change documents this regression but leaves it unexercised, +which is how a documented regression quietly becomes an undocumented one. + +#### Scenario: pi killed mid-run, session replayed + +- **GIVEN** a subagent running with an accumulated timeline +- **WHEN** the pi process is killed before any terminal frame is emitted +- **AND** the session is replayed afterwards +- **THEN** the subagent renders its scalar state +- **AND** no mid-run timeline is shown +- **AND** the render is neither corrupt nor blank + diff --git a/packages/server/src/__tests__/faux-agent-ticks.unit.test.ts b/packages/server/src/__tests__/faux-agent-ticks.unit.test.ts index 30555b1d4..24af2c2c4 100644 --- a/packages/server/src/__tests__/faux-agent-ticks.unit.test.ts +++ b/packages/server/src/__tests__/faux-agent-ticks.unit.test.ts @@ -10,7 +10,39 @@ */ import { describe, expect, it } from "vitest"; -import { parseTickPlan } from "../../../../qa/fixtures/faux-agent-ticks.ext.js"; +import fauxAgentTicks, { + buildEntries, + entryCountAt, + parseBusIntervalMs, + parseEntriesPlan, + parseTickPlan, +} from "../../../../qa/fixtures/faux-agent-ticks.ext.js"; + +/** Drive the fixture's `Agent` tool with a captured EventBus + update stream. */ +async function runFixture(prompt: string): Promise<{ + bus: Array<{ channel: string; data: any }>; + updates: any[]; + result: any; +}> { + const bus: Array<{ channel: string; data: any }> = []; + const updates: any[] = []; + let tool: any; + const pi = { + registerTool: (t: unknown) => { + tool = t; + }, + events: { emit: (channel: string, data: unknown) => bus.push({ channel, data }) }, + }; + fauxAgentTicks(pi as never); + const result = await tool.execute( + "call-1", + { subagent_type: "Explore", description: "d", prompt }, + undefined, + (p: unknown) => updates.push(p), + undefined, + ); + return { bus, updates, result }; +} describe("parseTickPlan", () => { it("parses count + intervalMs from the sentinel", () => { @@ -74,3 +106,182 @@ describe("parseTickPlan", () => { }); }); }); + +/** + * Pull-path substrate contract (change: verify-subagent-pull-under-load). + * + * These are not parser trivia. Each row below is a way the L3 pull-path spec + * could go GREEN WITHOUT EXERCISING THE PULL PATH, which is the specific + * failure this change exists to prevent: + * - growth silently off -> nothing to converge on + * - `data.id` missing -> every resync is a silent no-op + * - fat `created` frame -> forwarded unstripped, thin-push guard defeated + * - a delta encoding -> breaks the latest-supersedes pipeline invariant + */ +describe("parseEntriesPlan (E1/E2)", () => { + it("E1: no sentinel => no growth plan at all (fixture stays byte-identical)", () => { + expect(parseEntriesPlan("[[ticks:240@50]] go", 240)).toBeUndefined(); + expect(parseEntriesPlan(undefined, 240)).toBeUndefined(); + // ...and with no plan every tick carries an EMPTY timeline. + expect(entryCountAt(undefined, 0)).toBe(0); + expect(entryCountAt(undefined, 999)).toBe(0); + }); + + it("E2: parses start..end and defaults the growth span to half the plan", () => { + expect(parseEntriesPlan("[[entries:5..30]] go", 240)).toEqual({ + start: 5, + end: 30, + growTicks: 120, + }); + }); + + it("E2: honours an explicit growth span, bounded by the tick plan", () => { + expect(parseEntriesPlan("[[entries:5..30@60]] go", 240)).toEqual({ + start: 5, + end: 30, + growTicks: 60, + }); + // A span longer than the run cannot outlive the run. + expect(parseEntriesPlan("[[entries:5..30@900]] go", 240)?.growTicks).toBe(240); + }); + + it("E2: 0..0 yields an always-empty timeline (a valid degenerate arm)", () => { + const plan = parseEntriesPlan("[[entries:0..0]] go", 10); + expect(plan).toEqual({ start: 0, end: 0, growTicks: 5 }); + expect(entryCountAt(plan, 0)).toBe(0); + expect(entryCountAt(plan, 10)).toBe(0); + }); + + it("E2: inverted and malformed sentinels fall back to NO growth, never a guess", () => { + expect(parseEntriesPlan("[[entries:30..5]] go", 240)).toBeUndefined(); + expect(parseEntriesPlan("[[entries:x..y]] go", 240)).toBeUndefined(); + expect(parseEntriesPlan("[[entries:5..]] go", 240)).toBeUndefined(); + expect(parseEntriesPlan("[[entries:5-30]] go", 240)).toBeUndefined(); + }); + + it("E2: an absurd end count is capped, so a snapshot cannot blow the byte budget", () => { + expect(parseEntriesPlan("[[entries:0..99999999]] go", 240)?.end).toBe(5_000); + const huge = "9".repeat(400); // Number(huge) === Infinity + expect(parseEntriesPlan(`[[entries:0..${huge}]] go`, 240)?.end).toBe(0); + }); +}); + +describe("parseBusIntervalMs (E6)", () => { + it("E6: absent => undefined, i.e. NO subagents:* traffic at all", () => { + expect(parseBusIntervalMs("[[ticks:240@50]] go")).toBeUndefined(); + expect(parseBusIntervalMs(undefined)).toBeUndefined(); + }); + + it("E6: 250 ms is the value that matches the real producer's throttle", () => { + expect(parseBusIntervalMs("[[bus:250]] go")).toBe(250); + }); + + it("E6: clamps out-of-range values instead of spinning or stalling", () => { + expect(parseBusIntervalMs("[[bus:0]] go")).toBe(10); // MIN_BUS_MS + expect(parseBusIntervalMs("[[bus:999999]] go")).toBe(60_000); // MAX_BUS_MS + const huge = "9".repeat(400); + expect(parseBusIntervalMs(`[[bus:${huge}]] go`)).toBe(250); // non-finite -> default + }); + + it("E6: the bus interval is INDEPENDENT of the tick interval", () => { + // A per-tick bus stream would fabricate a ~5x inflated push arm in the P4 + // A/B, so the two cadences must not be coupled. + const prompt = "[[ticks:240@50]][[bus:250]] go"; + expect(parseTickPlan(prompt).intervalMs).toBe(50); + expect(parseBusIntervalMs(prompt)).toBe(250); + }); +}); + +describe("entryCountAt + buildEntries (E5)", () => { + const plan = parseEntriesPlan("[[entries:5..30@60]] go", 240)!; + + it("starts at `start`, reaches `end` at the growth span, then PLATEAUS", () => { + expect(entryCountAt(plan, 0)).toBe(5); + expect(entryCountAt(plan, 30)).toBe(18); // 5 + round(25 * 30/60) + expect(entryCountAt(plan, 60)).toBe(30); + // The plateau is load-bearing: it is what lets a cadence resync converge the + // rendered count WHILE THE AGENT IS STILL RUNNING. + expect(entryCountAt(plan, 120)).toBe(30); + expect(entryCountAt(plan, 240)).toBe(30); + }); + + it("never decreases across the run", () => { + let prev = -1; + for (let i = 0; i <= 240; i++) { + const n = entryCountAt(plan, i); + expect(n).toBeGreaterThanOrEqual(prev); + prev = n; + } + }); + + it("E5: every snapshot is FULL and a prefix-superset of its predecessor", () => { + // The pipeline is latest-supersedes: the producer throttle, the bridge frame + // buffer, the server collapse predicate and WS back-pressure ALL assume a + // frame is an idempotent full snapshot. A delta breaks all four. + const startedAt = 1_000; + let prev = buildEntries(entryCountAt(plan, 0), startedAt); + for (let i = 1; i <= 240; i++) { + const cur = buildEntries(entryCountAt(plan, i), startedAt); + expect(cur.length).toBeGreaterThanOrEqual(prev.length); + // Entry k is byte-identical across frames => append-only, no rewrite. + expect(cur.slice(0, prev.length)).toEqual(prev); + prev = cur; + } + expect(prev).toHaveLength(30); + }); + + it("entries are small and fixed-shape, so the snapshot stays far under 256 KiB", () => { + const bytes = JSON.stringify(buildEntries(30, 1_000)).length; + expect(bytes).toBeLessThan(4_000); // vs the store's ~262144 B budget + }); +}); + +describe("emitted frame shape (E3/E4/E7)", () => { + it("E7: without the bus sentinel the fixture emits NO subagents:* frames", async () => { + const { bus, updates } = await runFixture("[[ticks:4@1]] go"); + expect(bus).toHaveLength(0); + // ...and the tick carrier is unchanged: still an empty timeline. + expect(updates).toHaveLength(4); + expect(updates[0].details.entries).toEqual([]); + }); + + it("E3: every bus frame carries a TOP-LEVEL id equal to details.agentId", async () => { + const { bus } = await runFixture("[[ticks:4@1]][[entries:1..3@2]][[bus:10]] go"); + expect(bus.length).toBeGreaterThan(0); + const channels = bus.map((f) => f.channel); + expect(channels[0]).toBe("subagents:created"); + expect(channels).toContain("subagents:started"); + expect(channels[channels.length - 1]).toBe("subagents:completed"); + for (const frame of bus) { + // `SubagentFrameBuffer.agentIdOf` keys the retained snapshot on `data.id` + // and `resync()` looks it up by that key. An id living only inside + // `details` makes every resync a silent no-op. + expect(typeof frame.data.id).toBe("string"); + expect(frame.data.id).not.toBe(""); + expect(frame.data.id).toBe(frame.data.details.agentId); + } + // One agent per run: every frame shares the id. + expect(new Set(bus.map((f) => f.data.id)).size).toBe(1); + }); + + it("E4: the subagents:created frame carries NO entries", async () => { + const { bus } = await runFixture("[[ticks:4@1]][[entries:5..8@2]][[bus:10]] go"); + const created = bus.find((f) => f.channel === "subagents:created")!; + // `"created"` is not in the bridge's strip allowlist (queued/running), so a + // fat created frame would forward UNSTRIPPED and defeat the thin-push guard. + expect(created.data.details.status).toBe("created"); + expect(created.data.details.entries).toEqual([]); + }); + + it("the terminal frame is fat, and the tick carrier grows alongside the bus", async () => { + const { bus, updates, result } = await runFixture( + "[[ticks:6@1]][[entries:2..5@3]][[bus:10]] go", + ); + const completed = bus.find((f) => f.channel === "subagents:completed")!; + expect(completed.data.details.status).toBe("completed"); + expect(completed.data.details.entries).toHaveLength(5); + expect(result.details.entries).toHaveLength(5); + expect(updates[0].details.entries).toHaveLength(2); + expect(updates[updates.length - 1].details.entries).toHaveLength(5); + }); +}); diff --git a/qa/fixtures/AGENTS.md b/qa/fixtures/AGENTS.md index e84e059f0..d67583128 100644 --- a/qa/fixtures/AGENTS.md +++ b/qa/fixtures/AGENTS.md @@ -4,9 +4,9 @@ Files in this directory. One row per file. Non-source area. See change: migrate- | File | Purpose | |------|---------| -| `e2e-notify.ext.ts` | pi extension fixture: the only L3 lever on `ctx.ui.notify`. Captures `ctx` at `session_start` and registers tool `e2e_notify({message, level?})` whose `execute(_toolCallId, params)` calls `ctx.ui.notify`. Seeded by `docker/test-entrypoint.sh` to `~/.pi/agent/extensions/e2e-notify/`; driven by the `notify-probe` faux scenario. See change: split-notify-from-prompt-request. | -| `faux-agent-ticks.ext.ts` | pi extension fixture (change: reduce-bridge-tick-bandwidth). Registers an `Agent` tool that streams `tool_execution_update` frames carrying `partialResult.details.agentId` at a DETERMINISTIC cadence — the SYNTHETIC substrate for the subagent-tick-throttle L3 cadence rows, since a nested faux subagent cannot sustain a ≥10 s stream (empty inner faux queue; see change measurement.md Bug 2). Cadence from a `[[ticks:@(+gap@)]]` prompt sentinel (default 240@50ms≈12 s/20 fps). Exports `parseTickPlan` (L1-tested in `packages/server/src/__tests__/faux-agent-ticks.unit.test.ts`). SHADOWS the real subagents `Agent` tool (first-registration-wins), so staged ONLY under `PI_SYNTH_AGENT_TICKS=1` by `docker/test-entrypoint.sh` (which skips the subagents producer). Proven: OFF 19.6 fps → ON(500) 2.00 fps on `/ws`. | -| `faux-provider.ext.ts` | pi extension fixture. Registers faux provider via pi-ai `registerFauxProvider({api:"faux"})`. Surfaces `faux/faux-1` in CLI catalog via `pi.registerProvider("faux",{api:"faux",streamSimple})`. `streamSimple` grabbed via `getApiProvider("faux")` so faux stream survives rpc `rebindSession()`. Reads `FAUX_SCRIPT` (scenario id) + `FAUX_TPS`. Unknown scenario → loud "faux: no scenario". Imports `@earendil-works/pi-ai` unpinned (resolves pi's bundled copy). Exports `resolveActiveStep(context)`: walks messages backward to last user msg matching `/\[\[faux:(id)\]\]/`, `stepIndex` = assistant turns after it; no sentinel → `FAUX_SCRIPT` fallback anchored at start. Self-perpetuating router factory replaces static `setResponses`: re-appends itself, resolves `SCENARIOS[id]`, replays `script[stepIndex]`, calls factory steps. `FAUX_SCRIPT` kept as fallback. See change: add-faux-model-integration-tests. See change: add-e2e-faux-model-roundtrip. | -| `faux-scenarios.ts` | Shared scenario catalog `SCENARIOS: Record`. `script` = `FauxResponseStep[]` from `fauxText`/`fauxThinking`/`fauxToolCall`/`fauxAssistantMessage` + factories. Imported by server + client test layers. Covers plain-text, slow-stream, model-error, isolation-a/b, ask-select-roundtrip, `tool-*` renderer matrix, `ask-*` interactive matrix, text-realfile, echo-system-context, mermaid-colorize. `mermaid-colorize` streams a fenced flowchart (A/C default, `style B fill:#ff0000`) for MermaidBlock default-node colorization e2e. See change: colorize-mermaid-default-nodes. Adds `text-realfile` scenario streaming `./hello.txt` (explicit `./` prefix linkifies; resolves to real `/fixtures/sample-git/hello.txt`); used by file-preview-survives-churn e2e for live-content overlay. Adds `echo-system-context` scenario: factory step echoes `context.systemPrompt` dashboard fragment back as assistant text. Exports `extractDashboardFragment`, `DASHBOARD_CONTEXT_DELIMITER`, `NO_DASHBOARD_CONTEXT_MARKER`. `FauxContext` gains `systemPrompt?`. Used by e2e session-context-injection spec. See change: add-faux-model-integration-tests. See change: fix-file-preview-survives-message-churn. See change: inject-session-context-into-agent. Reworked `tool-bash-large` to `seq 1 500` (>200 lines) for Strategy B replay-truncation E2E. See change: reduce-session-replay-traffic. Adds `burst-heterogeneous`: 3 distinct bash calls (echo/echo/`sleep 2`+echo) then text, for temporal burst-grouping E2E. See change: group-tool-call-bursts. Adds `poll-narrated` (4 identical `echo checking` each preceded by narration prose → folds to `×4` with narration absorbed) + `burst-split-by-reply` (heterogeneous run split by a mid-turn reply → two bursts) for the composition-flip E2E. See change: collapse-tool-calls-across-narration. Adds `grp-single` (1 bash + text → single-member framed group with own summary) + `grp-reasoning` (3 distinct bash then a final `thinking`+text → TRAILING reasoning folded inside the group as a ThinkingBlock; trailing-thinking pattern replays reliably) for the universal-grouping E2E. See change: enhance-tool-call-grouping. Adds `long-transcript` (via `buildLongTranscript()` = ~120 turns of thinking+text+distinct bash → spans several viewports) + exported `LONG_TRANSCRIPT_TAIL` marker; drives the chat-transcript-virtualization E2E (windowing + scroll-lock). See change: virtualize-chat-transcript-tanstack. Adds `stuck-tool-superseded` (1 bash call then a plain-text reply = a LATER assistant `message_start` proof) + exported `SUPERSEDE_HEAL_MARKER`; drives the supersede-heal E2E (drop the tool's `tool_execution_end` WS frame + 404 the reconcile → card heals + `recovered` badge). See change: fix-stuck-tool-card-superseded-heal. Adds `oversized-turn` (bash `seq 1 8000` → ~90 KB tool-result event through the real broadcast `JSON.stringify` path) + exported `OVERSIZED_TURN_MARKER`; drives the oversized-event server-liveness E2E (per-event size ceiling; server survives + stays responsive). See change: bound-subagent-event-serialization. Adds `scroll-top-heavy` (via `buildScrollTopHeavy()`: turn 0 = 16k-char thinking + 9k-char text + 24k-char bash toolResult, turn 1 = inline image, then ~40 small turns) + exported `SCROLL_TOP_HEAVY_TAIL`; the biggest rows sit near the TOP to drive the scroll-to-top convergence E2E (content-aware estimate + scroll-up lands on index 0). See change: fix-chat-scroll-to-top-estimate-drift. Adds `copy-surfaces` (assistant message with a GFM table + fenced `js` code block) driving the copy-buttons-resolve-at-click E2E (`table-copy.spec.ts`): table md/TSV, code-block, message plain-text all copy real content on a single memoized render. See change: fix-table-copy-empty-clipboard. Adds the FLOW/SUBAGENT family + `FLOW_AGENT_MARKER`/`flowAgentName(systemPrompt)`: `flow-agent-branch` (factory reads the agent's `[[flow-agent:]]` system-prompt marker, emits `finish({ note })` so a real pi-flows agent step completes per-agent) + `subagent-spawn` (emits an `Agent` tool call whose prompt carries `[[faux:plain-text]]` so a REAL subagent resolves plain-text + completes, firing subagents:* lifecycle). Drives the L3 flow-roundtrip + subagent-inspector specs. See change: add-flow-plugin-e2e-tests. Adds `tool-list-models` (step 1 emits a real `list_models` tool call → REAL bridge tool executes against the faux-populated registry; step 2 factory reads the result via `lastToolResultText` and echoes the readiness discriminator) + exported `LIST_MODELS_MARKER_PREFIX` + `summarizeListModelsResult(context)`; `faux/faux-1` guarantees a hydrated non-empty catalogue so the marker reports `registryReady=true count=N hasFaux=true`. Drives the L3 `list-models-registry-ready` spec (live proof of the `registryReady:true` populated path). See change: fix-list-models-empty-on-unhydrated-registry. Adds `tool-bash-artifact`: a `bash` tool call `echo generated-by-tool > tool-artifact.md` writes a NEW file into the session cwd (git repo) so session-diff's git-status detector + Bash attributor surface it as an `origin:"tool"` row with `producedBy`; drives the L3 `tool-created-files` spec (U1 badge + U3 other-changes group/toggle). See change: detect-tool-created-files. Adds `tool-write-out-of-cwd` (Write to `/tmp/e2e-out-of-cwd/index.html`) + `tool-write-out-of-cwd-large` (>4 KB content to `big.html`): out-of-cwd Writes the server carries payload-only (`previewable:false`, abs key); drive the L3 `out-of-cwd-session-diffs` spec (F1/F2/F5 in-stream + API-level F3). See change: opt-in-out-of-cwd-session-diffs. Adds `text-tildelink`: assistant text `see ~/.pi/agent/settings.json for the config` — the tokenizer emits ONE `~/…` FileLink; drives the L3 `file-mention-resolve` spec (server tilde resolution opens the seeded home file, not a `/`-rooted 404). See change: server-side-file-mention-resolution. Adds scenario `notify-probe` (calls the `e2e_notify` fixture tool) + exported `NOTIFY_PROBE_MESSAGE`. See change: split-notify-from-prompt-request. Adds `notify-levels` scenario: four `e2e_notify` calls, one per `NotifyLevel` (`info`/`success`/`warning`/`error`), each with an independently greppable message from the exported `NOTIFY_LEVEL_MESSAGES` map, so an e2e can assert WHICH subset survives a given `notifyMinLevel` floor. See change: gate-notify-rows-by-level. Adds `subagent-slow-inner` (three sleeping bash calls, ~6 s) and `subagent-sustained` (parent `Agent` call whose prompt embeds the former) so a subagent stays alive long enough to produce a sustained `tool_execution_update` tick stream. The docker harness bakes `pi-dashboard-subagents` (the `onProgress` producer) so these yield a REAL tick stream; without it the L3 collapse scenarios pass vacuously. Measured 36 -> 2 retained updates per buffer. See change: collapse-superseded-tool-execution-updates. Adds `synthetic-agent-ticks` (parent `Agent` call with a `[[ticks:240@50]]` prompt → drives the synthetic `faux-agent-ticks.ext.ts` producer for a deterministic ≥10 s / 20 fps Agent-tick stream) + `synthetic-agent-ticks-quiet` (a `[[ticks:120@50+gap2500@30]]` variant with a >2 s quiet gap for F5); both complete with "synthetic ticks scenario complete". Drives the subagent-tick-throttle L3 cadence rows on the `PI_SYNTH_AGENT_TICKS=1` arm. See change: reduce-bridge-tick-bandwidth. | -| `faux-roles.json` | Faux role-preset (change: add-flow-plugin-e2e-tests). Maps every built-in role (planning/coding/fast/research/compact/vision/architect) -> faux/faux-1 so flow agents using `model: @role` resolve to the key-free faux model + exercise `model:resolve`. Delivery: IMAGE-BAKED — docker/test-entrypoint.sh strips `_comment` and copies to `~/.pi/agent/providers.json` under PI_E2E_SEED. | -| `README.md` | Documents faux fixtures: purpose, `registerFauxProvider`+`pi.registerProvider` recipe, `streamSimple` survives rpc rebind, `faux/faux-1` token form, `FAUX_SCRIPT`/`FAUX_TPS` env contract, scenario-catalog shape, how to add scenario. See change: add-faux-model-integration-tests. | +| `e2e-notify.ext.ts` | pi extension fixture: the only L3 lever on `ctx.ui.notify`. → see `e2e-notify.ext.ts.AGENTS.md` | +| `faux-agent-ticks.ext.ts` | pi extension fixture (change: reduce-bridge-tick-bandwidth). → see `faux-agent-ticks.ext.ts.AGENTS.md` | +| `faux-provider.ext.ts` | pi extension fixture. Registers faux provider via pi-ai `registerFauxProvider({api:"faux"})`. → see `faux-provider.ext.ts.AGENTS.md` | +| `faux-scenarios.ts` | Shared scenario catalog `SCENARIOS: Record`. → see `faux-scenarios.ts.AGENTS.md` | +| `faux-roles.json` | Faux role-preset (change: add-flow-plugin-e2e-tests). Maps every built-in role… → see `faux-roles.json.AGENTS.md` | +| `README.md` | Documents faux fixtures: purpose, `registerFauxProvider`+`pi.registerProvider` recipe, `streamSimple`… → see `README.md.AGENTS.md` | diff --git a/qa/fixtures/README.md.AGENTS.md b/qa/fixtures/README.md.AGENTS.md new file mode 100644 index 000000000..edf1735b9 --- /dev/null +++ b/qa/fixtures/README.md.AGENTS.md @@ -0,0 +1,3 @@ +# README.md — index + +Documents faux fixtures: purpose, `registerFauxProvider`+`pi.registerProvider` recipe, `streamSimple` survives rpc rebind, `faux/faux-1` token form, `FAUX_SCRIPT`/`FAUX_TPS` env contract, scenario-catalog shape, how to add scenario. See change: add-faux-model-integration-tests. diff --git a/qa/fixtures/e2e-notify.ext.ts.AGENTS.md b/qa/fixtures/e2e-notify.ext.ts.AGENTS.md new file mode 100644 index 000000000..35f94351e --- /dev/null +++ b/qa/fixtures/e2e-notify.ext.ts.AGENTS.md @@ -0,0 +1,3 @@ +# e2e-notify.ext.ts — index + +pi extension fixture: the only L3 lever on `ctx.ui.notify`. Captures `ctx` at `session_start` and registers tool `e2e_notify({message, level?})` whose `execute(_toolCallId, params)` calls `ctx.ui.notify`. Seeded by `docker/test-entrypoint.sh` to `~/.pi/agent/extensions/e2e-notify/`; driven by the `notify-probe` faux scenario. See change: split-notify-from-prompt-request. diff --git a/qa/fixtures/faux-agent-ticks.ext.ts b/qa/fixtures/faux-agent-ticks.ext.ts index 26a62b6f3..065b0e0d7 100644 --- a/qa/fixtures/faux-agent-ticks.ext.ts +++ b/qa/fixtures/faux-agent-ticks.ext.ts @@ -28,6 +28,35 @@ * `[[ticks:@+gap@]]` — insert one quiet gap * before tick index (F5) * Defaults (no sentinel): 240 ticks @ 50 ms ≈ 12 s of 20 fps. + * + * PULL-PATH SUBSTRATE (change: verify-subagent-pull-under-load, V1). Two more + * sentinels, both OFF unless present, so every pre-existing throttle row is + * byte-identical: + * `[[entries:..]]` — grow `details.entries` from to + * `[[entries:..@]]` over the first ticks (default + * half the plan), then PLATEAU for the + * rest of the run. Each frame carries a + * FULL snapshot — never a delta (the + * pipeline is latest-supersedes). + * `[[bus:]]` — also emit the `subagents:*` EventBus + * frames (`created` once, `started` + * coalesced at , `completed` + * at the end). DEFAULT 250 ms, matching + * the real producer's + * PROGRESS_THROTTLE_MS; deliberately + * DECOUPLED from the tick interval, since + * a per-tick bus stream would fabricate + * an inflated push arm in the P4 A/B. + * + * Why the bus frames matter: the bridge's strip, its `SubagentFrameBuffer`, and + * therefore the whole RESYNC (pull) path live on the `subagents:*` channel. A + * tick-only producer exercises none of it. The bus is shared and the bridge + * subscribes with `on()` ("observes every emitter"), so frames emitted here + * traverse the exact production path with no bridge change. + * + * FIDELITY BOUNDARY: this proves the bridge → server → client pull path. It does + * NOT prove that `@blackbelt-technology/pi-dashboard-subagents` emits this + * shape; that stays covered by the `subagent-spawn` faux scenario. */ import { randomUUID } from "node:crypto"; @@ -47,6 +76,8 @@ const AGENT_TICK_PARAMS = { } as const; const SENTINEL = /\[\[ticks:(\d+)@(\d+)(?:\+gap(\d+)@(\d+))?\]\]/; +const ENTRIES_SENTINEL = /\[\[entries:(\d+)\.\.(\d+)(?:@(\d+))?\]\]/; +const BUS_SENTINEL = /\[\[bus:(\d+)\]\]/; interface TickPlan { count: number; @@ -55,12 +86,28 @@ interface TickPlan { gapAt: number; } +/** Timeline-growth plan. `undefined` = no growth (entries stay `[]`). */ +export interface EntriesPlan { + start: number; + end: number; + /** Ticks over which the growth is spread; the run PLATEAUS afterwards. */ + growTicks: number; +} + // Hard bounds so a hostile/typo'd sentinel cannot wedge the loop. `\d+` with // enough digits makes `Number()` return `Infinity` (unbounded loop), and a `0` // interval spins the event loop; clamp both. const MAX_TICKS = 100_000; const MAX_INTERVAL_MS = 60_000; const MAX_GAP_MS = 600_000; +/** Entry-count ceiling. Entries are small + fixed-size, so the serialized + * snapshot stays far under the store's ~256 KiB `maxEventDataSize` budget — + * head-tail truncation is a BYTE budget, not a count, and a fired ceiling + * would rewrite the timeline to head + sentinel + tail. */ +const MAX_ENTRIES = 5_000; +const DEFAULT_BUS_MS = 250; +const MIN_BUS_MS = 10; +const MAX_BUS_MS = 60_000; function clampInt(raw: string | undefined, fallback: number, min: number, max: number): number { if (raw === undefined) return fallback; @@ -81,6 +128,56 @@ export function parseTickPlan(prompt: string | undefined): TickPlan { }; } +/** + * Parse the timeline-growth plan. Returns `undefined` (no growth) when the + * sentinel is absent, malformed, or inverted (`start > end`) — inventing a + * growth curve from a typo'd sentinel would silently fake the substrate. + */ +export function parseEntriesPlan( + prompt: string | undefined, + tickCount: number, +): EntriesPlan | undefined { + const m = prompt ? ENTRIES_SENTINEL.exec(prompt) : null; + if (!m) return undefined; + const start = clampInt(m[1], 0, 0, MAX_ENTRIES); + const end = clampInt(m[2], 0, 0, MAX_ENTRIES); + if (start > end) return undefined; + const defaultGrow = Math.max(1, Math.ceil(tickCount / 2)); + const growTicks = m[3] + ? Math.min(Math.max(1, clampInt(m[3], defaultGrow, 1, MAX_TICKS)), Math.max(1, tickCount)) + : defaultGrow; + return { start, end, growTicks }; +} + +/** Parse the bus-emission interval. `undefined` = emit no `subagents:*` frames. */ +export function parseBusIntervalMs(prompt: string | undefined): number | undefined { + const m = prompt ? BUS_SENTINEL.exec(prompt) : null; + if (!m) return undefined; + return clampInt(m[1], DEFAULT_BUS_MS, MIN_BUS_MS, MAX_BUS_MS); +} + +/** Entry count at tick `i`: linear over `growTicks`, then flat at `end`. */ +export function entryCountAt(plan: EntriesPlan | undefined, i: number): number { + if (!plan) return 0; + if (i >= plan.growTicks) return plan.end; + const span = plan.end - plan.start; + return plan.start + Math.round((span * i) / plan.growTicks); +} + +/** + * Build the FULL timeline snapshot of length `count`. Deterministic and + * append-only: entry `i` is byte-identical across every frame of a run, so each + * snapshot is a prefix-superset of its predecessor (latest-supersedes holds and + * no delta encoding can leak in). + */ +export function buildEntries(count: number, startedAt: number): Array> { + const out: Array> = []; + for (let i = 0; i < count; i++) { + out.push({ kind: "text", text: `faux-entry ${i}`, ts: startedAt + i }); + } + return out; +} + function sleep(ms: number, signal?: AbortSignal): Promise { return new Promise((resolve) => { if (signal?.aborted) return resolve(); @@ -90,14 +187,21 @@ function sleep(ms: number, signal?: AbortSignal): Promise { } /** Build the full AgentDetails snapshot the client hydration + throttle read. */ -function details(agentId: string, subagentType: string, status: string, turn: number, elapsedMs: number) { +function details( + agentId: string, + subagentType: string, + status: string, + turn: number, + elapsedMs: number, + entries: Array> = [], +) { return { agentId, displayName: subagentType, description: "synthetic tick producer", subagentType, status, - entries: [], + entries, toolUses: 0, tokens: "0", tokensUsage: { input: 0, output: 0, total: 0 }, @@ -123,10 +227,32 @@ export default function fauxAgentTicks(pi: ExtensionAPI): void { _ctx: unknown, ) { const plan = parseTickPlan(params.prompt); + const entriesPlan = parseEntriesPlan(params.prompt, plan.count); + const busIntervalMs = parseBusIntervalMs(params.prompt); const subagentType = params.subagent_type ?? "Explore"; const agentId = randomUUID(); const started = Date.now(); + const description = "synthetic tick producer"; + // TOP-LEVEL `id` is load-bearing: `SubagentFrameBuffer.agentIdOf` keys the + // retained snapshot on `data.id`, and `resync()` looks it up by that key. + // A frame carrying the id only inside `details` makes every resync a + // silent no-op — i.e. the pull path would look green while never running. + const busEmit = (channel: string, extra: Record, snapshot: unknown): void => { + if (busIntervalMs === undefined) return; + pi.events?.emit(channel, { id: agentId, ...extra, details: snapshot }); + }; + + busEmit( + "subagents:created", + { type: subagentType, description }, + // `"created"` is NOT in the bridge's strip allowlist (queued/running), so + // a fat `created` frame would forward UNSTRIPPED and defeat the + // thin-push guard. It carries no entries by construction. + details(agentId, subagentType, "created", 0, 0), + ); + + let lastBusAt = 0; for (let i = 0; i < plan.count; i++) { if (signal?.aborted) break; if (plan.gapAt === i && plan.gapMs > 0) { @@ -135,16 +261,45 @@ export default function fauxAgentTicks(pi: ExtensionAPI): void { // gap does not emit one extra tick. if (signal?.aborted) break; } + const entries = buildEntries(entryCountAt(entriesPlan, i), started); + const snapshot = details(agentId, subagentType, "running", i, Date.now() - started, entries); onUpdate?.({ content: [{ type: "text", text: `(running… ${i})` }], - details: details(agentId, subagentType, "running", i, Date.now() - started), + details: snapshot, }); + // Coalesce the bus carrier on its OWN interval, independent of the tick + // interval — mirroring the real producer's 250 ms progress throttle. + const now = Date.now(); + if (busIntervalMs !== undefined && now - lastBusAt >= busIntervalMs) { + lastBusAt = now; + busEmit("subagents:started", { type: subagentType, description }, snapshot); + } await sleep(plan.intervalMs, signal); } + const finalEntries = buildEntries(entryCountAt(entriesPlan, plan.count), started); + const finalDetails = details( + agentId, + subagentType, + "completed", + plan.count, + Date.now() - started, + finalEntries, + ); + busEmit( + "subagents:completed", + { + result: "synthetic ticks complete", + durationMs: Date.now() - started, + tokens: { input: 0, output: 0, total: 0 }, + toolUses: 0, + }, + finalDetails, + ); + return { content: [{ type: "text", text: "synthetic ticks complete" }], - details: details(agentId, subagentType, "completed", plan.count, Date.now() - started), + details: finalDetails, }; }, }); diff --git a/qa/fixtures/faux-agent-ticks.ext.ts.AGENTS.md b/qa/fixtures/faux-agent-ticks.ext.ts.AGENTS.md new file mode 100644 index 000000000..07d4809e8 --- /dev/null +++ b/qa/fixtures/faux-agent-ticks.ext.ts.AGENTS.md @@ -0,0 +1,3 @@ +# faux-agent-ticks.ext.ts — index + +pi extension fixture (change: reduce-bridge-tick-bandwidth). Registers an `Agent` tool that streams `tool_execution_update` frames carrying `partialResult.details.agentId` at a DETERMINISTIC cadence — the SYNTHETIC substrate for the subagent-tick-throttle L3 cadence rows, since a nested faux subagent cannot sustain a ≥10 s stream (empty inner faux queue; see change measurement.md Bug 2). Cadence from a `[[ticks:@(+gap@)]]` prompt sentinel (default 240@50ms≈12 s/20 fps). Exports `parseTickPlan` (L1-tested in `packages/server/src/__tests__/faux-agent-ticks.unit.test.ts`). SHADOWS the real subagents `Agent` tool (first-registration-wins), so staged ONLY under `PI_SYNTH_AGENT_TICKS=1` by `docker/test-entrypoint.sh` (which skips the subagents producer). Proven: OFF 19.6 fps → ON(500) 2.00 fps on `/ws`. PULL-PATH SUBSTRATE (change: verify-subagent-pull-under-load): two additive, sentinel-gated behaviours, both OFF unless present, so pre-existing throttle rows stay byte-identical. `[[entries:..@]]` grows `details.entries` → over the first ticks (default half the plan) then PLATEAUS — each frame a FULL snapshot, never a delta. `[[bus:]]` also emits `subagents:created/started/completed` via `pi.events.emit`, coalesced at (default 250, matching the real producer's PROGRESS_THROTTLE_MS) and DECOUPLED from the tick interval (a per-tick bus stream would fabricate an inflated push arm in the P4 A/B). Bus frames carry top-level `data.id` === `details.agentId` — `SubagentFrameBuffer.agentIdOf` keys on `data.id`, so a details-only id makes every resync a silent no-op. `subagents:created` carries NO entries (`"created"` is not in the strip allowlist, so a fat one forwards unstripped). Exports `parseEntriesPlan`/`parseBusIntervalMs`/`entryCountAt`/`buildEntries` for L1. Fidelity boundary: proves the bridge→server→client pull path, NOT the real producer's frame shape. diff --git a/qa/fixtures/faux-provider.ext.ts.AGENTS.md b/qa/fixtures/faux-provider.ext.ts.AGENTS.md new file mode 100644 index 000000000..499ce9347 --- /dev/null +++ b/qa/fixtures/faux-provider.ext.ts.AGENTS.md @@ -0,0 +1,3 @@ +# faux-provider.ext.ts — index + +pi extension fixture. Registers faux provider via pi-ai `registerFauxProvider({api:"faux"})`. Surfaces `faux/faux-1` in CLI catalog via `pi.registerProvider("faux",{api:"faux",streamSimple})`. `streamSimple` grabbed via `getApiProvider("faux")` so faux stream survives rpc `rebindSession()`. Reads `FAUX_SCRIPT` (scenario id) + `FAUX_TPS`. Unknown scenario → loud "faux: no scenario". Imports `@earendil-works/pi-ai` unpinned (resolves pi's bundled copy). Exports `resolveActiveStep(context)`: walks messages backward to last user msg matching `/\[\[faux:(id)\]\]/`, `stepIndex` = assistant turns after it; no sentinel → `FAUX_SCRIPT` fallback anchored at start. Self-perpetuating router factory replaces static `setResponses`: re-appends itself, resolves `SCENARIOS[id]`, replays `script[stepIndex]`, calls factory steps. `FAUX_SCRIPT` kept as fallback. See change: add-faux-model-integration-tests. See change: add-e2e-faux-model-roundtrip. diff --git a/qa/fixtures/faux-roles.json.AGENTS.md b/qa/fixtures/faux-roles.json.AGENTS.md new file mode 100644 index 000000000..ebcc836eb --- /dev/null +++ b/qa/fixtures/faux-roles.json.AGENTS.md @@ -0,0 +1,3 @@ +# faux-roles.json — index + +Faux role-preset (change: add-flow-plugin-e2e-tests). Maps every built-in role (planning/coding/fast/research/compact/vision/architect) -> faux/faux-1 so flow agents using `model: @role` resolve to the key-free faux model + exercise `model:resolve`. Delivery: IMAGE-BAKED — docker/test-entrypoint.sh strips `_comment` and copies to `~/.pi/agent/providers.json` under PI_E2E_SEED. diff --git a/qa/fixtures/faux-scenarios.ts b/qa/fixtures/faux-scenarios.ts index 74052a3d8..ce33e13d6 100644 --- a/qa/fixtures/faux-scenarios.ts +++ b/qa/fixtures/faux-scenarios.ts @@ -1064,6 +1064,76 @@ export const SCENARIOS: Record = { expect: { text: "synthetic ticks scenario complete" }, }, + // Watched-growth substrate for the PULL path (change: + // verify-subagent-pull-under-load, V1). Same synthetic Agent-tick producer, + // with the two additive sentinels switched on: + // [[ticks:240@50]] — 240 ticks @ 50 ms ≈ 12 s of run + // [[entries:5..30@60]] — timeline grows 5 → 30 over the first 60 ticks + // (≈ 3 s), then PLATEAUS for the remaining ≈ 9 s + // [[bus:250]] — `subagents:*` frames coalesced at the real + // producer's 250 ms, so the strip → frame-buffer → + // RESYNC path actually runs + // The ≈ 9 s plateau is load-bearing, not padding: it is ≥ 3 cadence intervals + // (CADENCE_BASE_MS = 2000), so the rendered count can reach 30 via a cadence + // resync reply WHILE THE AGENT IS STILL RUNNING. Without it the count would + // only converge on the never-stripped terminal frame and F1 would be vacuous. + "subagent-watched-growth": { + script: [ + fauxAssistantMessage( + [ + fauxToolCall("Agent", { + subagent_type: "Explore", + description: "watched growing subagent", + prompt: "[[ticks:240@50]][[entries:5..30@60]][[bus:250]] stream a watched growing timeline", + }), + ], + { stopReason: "toolUse" }, + ), + fauxAssistantMessage([fauxText("watched growth scenario complete")]), + ], + expect: { text: "watched growth scenario complete" }, + }, + + // Bus-cadence variants of `subagent-watched-growth`, for the P4 sensitivity + // table. The pull-vs-push byte verdict is a function of how fast the PUSH + // carrier runs, so a single cadence would report arithmetic on one fixture + // constant as if it were a property of the pipeline. 250 ms is the + // production-matched headline; these are the faster/slower flanks. + // See change: verify-subagent-pull-under-load (V5). + "subagent-watched-growth-bus100": { + script: [ + fauxAssistantMessage( + [ + fauxToolCall("Agent", { + subagent_type: "Explore", + description: "watched growing subagent", + prompt: "[[ticks:240@50]][[entries:5..30@60]][[bus:100]] stream a watched growing timeline", + }), + ], + { stopReason: "toolUse" }, + ), + fauxAssistantMessage([fauxText("watched growth scenario complete")]), + ], + expect: { text: "watched growth scenario complete" }, + }, + + "subagent-watched-growth-bus1000": { + script: [ + fauxAssistantMessage( + [ + fauxToolCall("Agent", { + subagent_type: "Explore", + description: "watched growing subagent", + prompt: "[[ticks:240@50]][[entries:5..30@60]][[bus:1000]] stream a watched growing timeline", + }), + ], + { stopReason: "toolUse" }, + ), + fauxAssistantMessage([fauxText("watched growth scenario complete")]), + ], + expect: { text: "watched growth scenario complete" }, + }, + // ── OpenSpec auto-attach locality gate (change: // scope-openspec-auto-attach-to-session-cwd) ─────────────────────────────── // The verbatim incident shape: an openspec CLI invocation prefixed with a diff --git a/qa/fixtures/faux-scenarios.ts.AGENTS.md b/qa/fixtures/faux-scenarios.ts.AGENTS.md new file mode 100644 index 000000000..fcbddb042 --- /dev/null +++ b/qa/fixtures/faux-scenarios.ts.AGENTS.md @@ -0,0 +1,3 @@ +# faux-scenarios.ts — index + +Shared scenario catalog `SCENARIOS: Record`. `script` = `FauxResponseStep[]` from `fauxText`/`fauxThinking`/`fauxToolCall`/`fauxAssistantMessage` + factories. Imported by server + client test layers. Covers plain-text, slow-stream, model-error, isolation-a/b, ask-select-roundtrip, `tool-*` renderer matrix, `ask-*` interactive matrix, text-realfile, echo-system-context, mermaid-colorize. `mermaid-colorize` streams a fenced flowchart (A/C default, `style B fill:#ff0000`) for MermaidBlock default-node colorization e2e. See change: colorize-mermaid-default-nodes. Adds `text-realfile` scenario streaming `./hello.txt` (explicit `./` prefix linkifies; resolves to real `/fixtures/sample-git/hello.txt`); used by file-preview-survives-churn e2e for live-content overlay. Adds `echo-system-context` scenario: factory step echoes `context.systemPrompt` dashboard fragment back as assistant text. Exports `extractDashboardFragment`, `DASHBOARD_CONTEXT_DELIMITER`, `NO_DASHBOARD_CONTEXT_MARKER`. `FauxContext` gains `systemPrompt?`. Used by e2e session-context-injection spec. See change: add-faux-model-integration-tests. See change: fix-file-preview-survives-message-churn. See change: inject-session-context-into-agent. Reworked `tool-bash-large` to `seq 1 500` (>200 lines) for Strategy B replay-truncation E2E. See change: reduce-session-replay-traffic. Adds `burst-heterogeneous`: 3 distinct bash calls (echo/echo/`sleep 2`+echo) then text, for temporal burst-grouping E2E. See change: group-tool-call-bursts. Adds `poll-narrated` (4 identical `echo checking` each preceded by narration prose → folds to `×4` with narration absorbed) + `burst-split-by-reply` (heterogeneous run split by a mid-turn reply → two bursts) for the composition-flip E2E. See change: collapse-tool-calls-across-narration. Adds `grp-single` (1 bash + text → single-member framed group with own summary) + `grp-reasoning` (3 distinct bash then a final `thinking`+text → TRAILING reasoning folded inside the group as a ThinkingBlock; trailing-thinking pattern replays reliably) for the universal-grouping E2E. See change: enhance-tool-call-grouping. Adds `long-transcript` (via `buildLongTranscript()` = ~120 turns of thinking+text+distinct bash → spans several viewports) + exported `LONG_TRANSCRIPT_TAIL` marker; drives the chat-transcript-virtualization E2E (windowing + scroll-lock). See change: virtualize-chat-transcript-tanstack. Adds `stuck-tool-superseded` (1 bash call then a plain-text reply = a LATER assistant `message_start` proof) + exported `SUPERSEDE_HEAL_MARKER`; drives the supersede-heal E2E (drop the tool's `tool_execution_end` WS frame + 404 the reconcile → card heals + `recovered` badge). See change: fix-stuck-tool-card-superseded-heal. Adds `oversized-turn` (bash `seq 1 8000` → ~90 KB tool-result event through the real broadcast `JSON.stringify` path) + exported `OVERSIZED_TURN_MARKER`; drives the oversized-event server-liveness E2E (per-event size ceiling; server survives + stays responsive). See change: bound-subagent-event-serialization. Adds `scroll-top-heavy` (via `buildScrollTopHeavy()`: turn 0 = 16k-char thinking + 9k-char text + 24k-char bash toolResult, turn 1 = inline image, then ~40 small turns) + exported `SCROLL_TOP_HEAVY_TAIL`; the biggest rows sit near the TOP to drive the scroll-to-top convergence E2E (content-aware estimate + scroll-up lands on index 0). See change: fix-chat-scroll-to-top-estimate-drift. Adds `copy-surfaces` (assistant message with a GFM table + fenced `js` code block) driving the copy-buttons-resolve-at-click E2E (`table-copy.spec.ts`): table md/TSV, code-block, message plain-text all copy real content on a single memoized render. See change: fix-table-copy-empty-clipboard. Adds the FLOW/SUBAGENT family + `FLOW_AGENT_MARKER`/`flowAgentName(systemPrompt)`: `flow-agent-branch` (factory reads the agent's `[[flow-agent:]]` system-prompt marker, emits `finish({ note })` so a real pi-flows agent step completes per-agent) + `subagent-spawn` (emits an `Agent` tool call whose prompt carries `[[faux:plain-text]]` so a REAL subagent resolves plain-text + completes, firing subagents:* lifecycle). Drives the L3 flow-roundtrip + subagent-inspector specs. See change: add-flow-plugin-e2e-tests. Adds `tool-list-models` (step 1 emits a real `list_models` tool call → REAL bridge tool executes against the faux-populated registry; step 2 factory reads the result via `lastToolResultText` and echoes the readiness discriminator) + exported `LIST_MODELS_MARKER_PREFIX` + `summarizeListModelsResult(context)`; `faux/faux-1` guarantees a hydrated non-empty catalogue so the marker reports `registryReady=true count=N hasFaux=true`. Drives the L3 `list-models-registry-ready` spec (live proof of the `registryReady:true` populated path). See change: fix-list-models-empty-on-unhydrated-registry. Adds `tool-bash-artifact`: a `bash` tool call `echo generated-by-tool > tool-artifact.md` writes a NEW file into the session cwd (git repo) so session-diff's git-status detector + Bash attributor surface it as an `origin:"tool"` row with `producedBy`; drives the L3 `tool-created-files` spec (U1 badge + U3 other-changes group/toggle). See change: detect-tool-created-files. Adds `tool-write-out-of-cwd` (Write to `/tmp/e2e-out-of-cwd/index.html`) + `tool-write-out-of-cwd-large` (>4 KB content to `big.html`): out-of-cwd Writes the server carries payload-only (`previewable:false`, abs key); drive the L3 `out-of-cwd-session-diffs` spec (F1/F2/F5 in-stream + API-level F3). See change: opt-in-out-of-cwd-session-diffs. Adds `text-tildelink`: assistant text `see ~/.pi/agent/settings.json for the config` — the tokenizer emits ONE `~/…` FileLink; drives the L3 `file-mention-resolve` spec (server tilde resolution opens the seeded home file, not a `/`-rooted 404). See change: server-side-file-mention-resolution. Adds scenario `notify-probe` (calls the `e2e_notify` fixture tool) + exported `NOTIFY_PROBE_MESSAGE`. See change: split-notify-from-prompt-request. Adds `notify-levels` scenario: four `e2e_notify` calls, one per `NotifyLevel` (`info`/`success`/`warning`/`error`), each with an independently greppable message from the exported `NOTIFY_LEVEL_MESSAGES` map, so an e2e can assert WHICH subset survives a given `notifyMinLevel` floor. See change: gate-notify-rows-by-level. Adds `subagent-slow-inner` (three sleeping bash calls, ~6 s) and `subagent-sustained` (parent `Agent` call whose prompt embeds the former) so a subagent stays alive long enough to produce a sustained `tool_execution_update` tick stream. The docker harness bakes `pi-dashboard-subagents` (the `onProgress` producer) so these yield a REAL tick stream; without it the L3 collapse scenarios pass vacuously. Measured 36 -> 2 retained updates per buffer. See change: collapse-superseded-tool-execution-updates. Adds `synthetic-agent-ticks` (parent `Agent` call with a `[[ticks:240@50]]` prompt → drives the synthetic `faux-agent-ticks.ext.ts` producer for a deterministic ≥10 s / 20 fps Agent-tick stream) + `synthetic-agent-ticks-quiet` (a `[[ticks:120@50+gap2500@30]]` variant with a >2 s quiet gap for F5); both complete with "synthetic ticks scenario complete". Drives the subagent-tick-throttle L3 cadence rows on the `PI_SYNTH_AGENT_TICKS=1` arm. See change: reduce-bridge-tick-bandwidth. Adds `subagent-watched-growth` (`[[ticks:240@50]][[entries:5..30@60]][[bus:250]]` — 12 s run, timeline 5→30 over ~3 s then a ~9 s plateau ≥ 3 cadence intervals so a cadence resync can converge the render while the agent is STILL RUNNING; without the plateau the count would only converge on the never-stripped terminal frame and F1 would be vacuous) plus `-bus100`/`-bus1000` cadence flanks for the P4 sensitivity table. See change: verify-subagent-pull-under-load. diff --git a/tests/e2e/AGENTS.md b/tests/e2e/AGENTS.md index cbe17e833..405b788e0 100644 --- a/tests/e2e/AGENTS.md +++ b/tests/e2e/AGENTS.md @@ -31,9 +31,9 @@ Files in this directory. One row per file. Non-source area (migrated from `docs/ | `file-preview-survives-churn.spec.ts` | Playwright spec. Rendered-DOM regression for hoisted file-preview overlay. → see `file-preview-survives-churn.spec.ts.AGENTS.md` | | `fixtures.ts` | The suite's `test`/`expect` entry point — EVERY spec imports from here, not `@playwright/test` (guard:… → see `fixtures.ts.AGENTS.md` | | `flow-roundtrip.spec.ts` | L3 spec (change: add-flow-plugin-e2e-tests). Real pi-flows engine + faux agents. → see `flow-roundtrip.spec.ts.AGENTS.md` | -| `folder-actions-menu.spec.ts` | Playwright spec. Folder actions menu after the slot-pill controls moved in (test-plan #E1, #E2, #F7, #F8, #F9). Trigger neither navigates nor collapses; only one panel open across the sidebar; the rendered card's four slot sections expose exactly ONE interactive node each (their pill root) and none of the ten moved test ids; `refresh-folder` + `directory-settings` reachable from the menu. Own `test.use({viewport:{width:390}})` describe for the mobile sheet: `data-menu-form=sheet`, full viewport width, Escape returns focus to the trigger. Registry mechanics stay L1 in `dashboard-plugin-runtime/src/__tests__/folder-menu-contributions.test.tsx`. See change: move-slot-actions-to-menu. | +| `folder-actions-menu.spec.ts` | Playwright spec. Folder actions menu after the slot-pill controls moved in (test-plan #E1, #E2, #F7, #F8,… → see `folder-actions-menu.spec.ts.AGENTS.md` | | `folder-membership-drag.spec.ts` | L3 for `drag-folders-across-workspaces` (test-plan #F11-#F13, #F16, #F18, #F20, #X6) — the scenarios needing… → see `folder-membership-drag.spec.ts.AGENTS.md` | -| `followup-image-queue.spec.ts` | L3 for `fix-bridge-followup-image-drop` (test-plan #F1, #F2, #F3, #F5, #F6, #F7, #F8). Drives `send_prompt → bridge buffer → queue_update → chip` against the docker harness. Image-bearing follow-ups posted via same-origin `POST /api/session/:id/prompt` (accepts `images`, defaults to `followUp` delivery) because a browser paste is not scriptable — same bridge handler either way. Asserts the attachment COUNT indicator (`queue-followup-attachments`, never a thumbnail), its survival across an inline edit, absence for text-only, a `routeWebSocket`-downgraded legacy `string[]` still rendering readable text, a cap refusal surfacing in chat, a dropped `image/svg+xml` reported, and a drained over-ceiling image-bearing message still rendering a chat row. Needs `PI_E2E_SEED=1`. | +| `followup-image-queue.spec.ts` | L3 for `fix-bridge-followup-image-drop` (test-plan #F1, #F2, #F3, #F5, #F6, #F7, #F8). → see `followup-image-queue.spec.ts.AGENTS.md` | | `gateway-board-mobile.spec.ts` | 6.8/6.9 (D11) — the readiness board at 375×667: one 52px line per row, no horizontal overflow, ≥44px targets,… → see `gateway-board-mobile.spec.ts.AGENTS.md` | | `gateway-origin-surfaces.spec.ts` | L3 for the transport/identity CLIENT surfaces (test-plan #F1–#F5 → tasks 12.47–12.51). → see `gateway-origin-surfaces.spec.ts.AGENTS.md` | | `gateway-primary-offer.spec.ts` | F7/F8 of add-zrok-custom-reserved-name — confirm-gated primary switch and the registration offer. → see `gateway-primary-offer.spec.ts.AGENTS.md` | @@ -105,6 +105,8 @@ Files in this directory. One row per file. Non-source area (migrated from `docs/ | `split-composer-overflow.spec.ts` | Browser E2E gate for `fix-split-composer-overflow`. Opens `split-toggle` at viewport 1280 (≥ md); asserts… → see `split-composer-overflow.spec.ts.AGENTS.md` | | `subagent-detail-dialog.spec.ts` | Playwright spec (change: fix-subagent-live-detail-reliability D4). → see `subagent-detail-dialog.spec.ts.AGENTS.md` | | `subagent-inspector.spec.ts` | L3 spec (change: add-flow-plugin-e2e-tests). Drives `[[faux:subagent-spawn]]` — parent emits an `Agent` tool…… → see `subagent-inspector.spec.ts.AGENTS.md` | +| `subagent-pull-measurements.spec.ts` | RECORDED-EVIDENCE measurement rows for the subagent pull path (change: verify-subagent-pull-under-load): P1… → see `subagent-pull-measurements.spec.ts.AGENTS.md` | +| `subagent-pull-under-load.spec.ts` | L3 behavioural rows for the open-inspector PULL path (change: verify-subagent-pull-under-load). → see `subagent-pull-under-load.spec.ts.AGENTS.md` | | `subagent-thin-tick-liveness.spec.ts` | L3 for the subagent push/pull split: terminal fidelity after refresh, `/api/health`… → see `subagent-thin-tick-liveness.spec.ts.AGENTS.md` | | `subagent-tick-throttle.spec.ts` | L3 cadence rows for the subagent-tick throttle (change: reduce-bridge-tick-bandwidth): F1/P1 (floor ≥5 in 10… → see `subagent-tick-throttle.spec.ts.AGENTS.md` | | `superseded-heal.spec.ts` | Playwright spec (task 7.1, change: fix-stuck-tool-card-superseded-heal). → see `superseded-heal.spec.ts.AGENTS.md` | diff --git a/tests/e2e/folder-actions-menu.spec.ts.AGENTS.md b/tests/e2e/folder-actions-menu.spec.ts.AGENTS.md new file mode 100644 index 000000000..90c1950e3 --- /dev/null +++ b/tests/e2e/folder-actions-menu.spec.ts.AGENTS.md @@ -0,0 +1,3 @@ +# folder-actions-menu.spec.ts — index + +Playwright spec. Folder actions menu after the slot-pill controls moved in (test-plan #E1, #E2, #F7, #F8, #F9). Trigger neither navigates nor collapses; only one panel open across the sidebar; the rendered card's four slot sections expose exactly ONE interactive node each (their pill root) and none of the ten moved test ids; `refresh-folder` + `directory-settings` reachable from the menu. Own `test.use({viewport:{width:390}})` describe for the mobile sheet: `data-menu-form=sheet`, full viewport width, Escape returns focus to the trigger. Registry mechanics stay L1 in `dashboard-plugin-runtime/src/__tests__/folder-menu-contributions.test.tsx`. See change: move-slot-actions-to-menu. diff --git a/tests/e2e/followup-image-queue.spec.ts.AGENTS.md b/tests/e2e/followup-image-queue.spec.ts.AGENTS.md new file mode 100644 index 000000000..d82202135 --- /dev/null +++ b/tests/e2e/followup-image-queue.spec.ts.AGENTS.md @@ -0,0 +1,3 @@ +# followup-image-queue.spec.ts — index + +L3 for `fix-bridge-followup-image-drop` (test-plan #F1, #F2, #F3, #F5, #F6, #F7, #F8). Drives `send_prompt → bridge buffer → queue_update → chip` against the docker harness. Image-bearing follow-ups posted via same-origin `POST /api/session/:id/prompt` (accepts `images`, defaults to `followUp` delivery) because a browser paste is not scriptable — same bridge handler either way. Asserts the attachment COUNT indicator (`queue-followup-attachments`, never a thumbnail), its survival across an inline edit, absence for text-only, a `routeWebSocket`-downgraded legacy `string[]` still rendering readable text, a cap refusal surfacing in chat, a dropped `image/svg+xml` reported, and a drained over-ceiling image-bearing message still rendering a chat row. Needs `PI_E2E_SEED=1`. diff --git a/tests/e2e/helpers/index.ts b/tests/e2e/helpers/index.ts index fb1afb157..d5f4996eb 100644 --- a/tests/e2e/helpers/index.ts +++ b/tests/e2e/helpers/index.ts @@ -396,6 +396,188 @@ export function collectAgentTicks(page: Page): TickCollector { }; } +/** One observed subagent-carrying frame, on either carrier. */ +export interface SubagentFrameSample { + at: number; + /** Owning session id — the shared harness runs one container, so a previous + * spec's producer can still be streaming into this window. */ + sessionId: string; + /** `tool_execution_update` | `subagent_created` | `subagent_started` | … */ + eventType: string; + /** The watched agent, read from whichever slot the carrier uses. */ + agentId: string; + status: string; + /** Timeline length carried by THIS frame (0 when stripped). */ + entryCount: number; + /** + * Set ONLY on a resync REPLY (`subagent-forward-sites.ts` echoes the + * requester's token). This is the ONLY way to tell a reply from a pushed + * `subagent_started` — they share an eventType, so eventType cannot classify + * them. See change: verify-subagent-pull-under-load (V2). + */ + resyncRequestId?: string; + /** + * Bytes attributed to THIS frame. A batched replay message carries many + * events; `collectAgentTicks` charges each of them the WHOLE payload length + * (an N× over-count). Here each event is charged its own serialized size. + */ + bytes: number; +} + +/** + * One `subagent_resync_request` the CLIENT sent (outgoing direction). + * Not exported: reached only through `SubagentWireCollector`, so exporting it + * would be a dead export (knip `types` class). + */ +interface ResyncRequestSample { + at: number; + agentId: string; + requestId: string; + /** `"open"` (expand / popout / subscribe) or `"cadence"` (the D4 v1 timer). */ + reason: string; +} + +export interface SubagentWireCollector { + /** Inbound subagent-carrying frames, in receive order. */ + frames: SubagentFrameSample[]; + /** Outbound resync requests, in send order. */ + requests: ResyncRequestSample[]; + /** Inbound frames for one agent. */ + forAgent: (agentId: string) => SubagentFrameSample[]; + /** Resync REPLIES for one agent (frames bearing a requester token). */ + repliesFor: (agentId: string) => SubagentFrameSample[]; + /** PUSHED frames for one agent (everything without a requester token). */ + pushesFor: (agentId: string) => SubagentFrameSample[]; + /** Outbound requests for one agent, optionally filtered by reason. */ + requestsFor: (agentId: string, reason?: string) => ResyncRequestSample[]; +} + +const SUBAGENT_STATUS_OF = (ev: any): string => + String( + ev?.data?.details?.status ?? + ev?.data?.partialResult?.details?.status ?? + // `tool_execution_end` carries the terminal snapshot under `result`, the + // same slot ENTRY_COUNT_OF reads. Omitting it made every status filter + // silently miss the terminal carrier. + ev?.data?.result?.details?.status ?? + "", + ); + +const AGENT_ID_OF = (ev: any): string => + String( + ev?.data?.id ?? + ev?.data?.details?.agentId ?? + ev?.data?.partialResult?.details?.agentId ?? + "", + ); + +const ENTRY_COUNT_OF = (ev: any): number => { + const e = + ev?.data?.details?.entries ?? + ev?.data?.partialResult?.details?.entries ?? + // `tool_execution_end` carries the final snapshot under `result`. + ev?.data?.result?.details?.entries; + return Array.isArray(e) ? e.length : 0; +}; + +/** + * Collect BOTH subagent carriers off the browser `/ws` socket, in both + * directions. + * + * Why this exists next to `collectAgentTicks` rather than replacing it: the + * throttle rows measure `tool_execution_update` FRAME RATE filtered by + * `toolName`, while the pull-path rows measure BYTES per carrier and need the + * `__resyncRequestId` discriminator plus the outgoing requests. Same socket, + * different questions. + * + * Attach BEFORE the run starts — `page.on("websocket")` only sees sockets opened + * after it is registered. + * + * See change: verify-subagent-pull-under-load (V2/V5). + */ +export function collectSubagentWire(page: Page): SubagentWireCollector { + const frames: SubagentFrameSample[] = []; + const requests: ResyncRequestSample[] = []; + + const ingest = (payload: string): void => { + if ( + !payload.includes("subagent") && + !payload.includes("tool_execution_update") && + !payload.includes("tool_execution_end") + ) { + return; + } + let parsed: any; + try { + parsed = JSON.parse(payload); + } catch { + return; + } + const events: any[] = parsed?.event + ? [parsed.event] + : Array.isArray(parsed?.events) + ? parsed.events.map((e: any) => e?.event).filter(Boolean) + : []; + for (const ev of events) { + const eventType = String(ev?.eventType ?? ""); + // `tool_execution_end` is admitted deliberately: it is a TERMINAL carrier + // for an Agent run, and a caller asking "did any terminal frame arrive for + // this agent?" would otherwise get a vacuous `false` because the event + // never entered the collector at all. + const isToolCarrier = + (eventType === "tool_execution_update" || eventType === "tool_execution_end") && + String(ev?.data?.toolName ?? "") === "Agent"; + if (!eventType.startsWith("subagent_") && !isToolCarrier) continue; + const token = ev?.data?.__resyncRequestId; + frames.push({ + at: Date.now(), + sessionId: String(parsed?.sessionId ?? ""), + eventType, + agentId: AGENT_ID_OF(ev), + status: SUBAGENT_STATUS_OF(ev), + entryCount: ENTRY_COUNT_OF(ev), + resyncRequestId: typeof token === "string" && token ? token : undefined, + // Per-frame attribution: charge this event its OWN serialized size, so a + // batched replay message is not counted once per contained event. + bytes: JSON.stringify(ev).length, + }); + } + }; + + page.on("websocket", (ws) => { + ws.on("framereceived", (frame) => { + ingest(typeof frame.payload === "string" ? frame.payload : ""); + }); + ws.on("framesent", (frame) => { + const payload = typeof frame.payload === "string" ? frame.payload : ""; + if (!payload.includes("subagent_resync_request")) return; + try { + const msg = JSON.parse(payload); + if (msg?.type !== "subagent_resync_request") return; + requests.push({ + at: Date.now(), + agentId: String(msg.agentId ?? ""), + requestId: String(msg.requestId ?? ""), + reason: String(msg.reason ?? ""), + }); + } catch { + /* non-JSON frame: not ours */ + } + }); + }); + + const forAgent = (agentId: string) => frames.filter((f) => f.agentId === agentId); + return { + frames, + requests, + forAgent, + repliesFor: (agentId) => forAgent(agentId).filter((f) => f.resyncRequestId !== undefined), + pushesFor: (agentId) => forAgent(agentId).filter((f) => f.resyncRequestId === undefined), + requestsFor: (agentId, reason) => + requests.filter((r) => r.agentId === agentId && (reason === undefined || r.reason === reason)), + }; +} + /** * Write `subagentTickThrottleMs` into the CONTAINER's dashboard config file. * diff --git a/tests/e2e/helpers/index.ts.AGENTS.md b/tests/e2e/helpers/index.ts.AGENTS.md index 8836f7d55..05af5fe82 100644 --- a/tests/e2e/helpers/index.ts.AGENTS.md +++ b/tests/e2e/helpers/index.ts.AGENTS.md @@ -3,3 +3,5 @@ E2E helpers. `gotoDashboard(page)` navigates `/`, waits for `header-app-bar`, and arms a one-per-page Playwright `addLocatorHandler` that auto-clicks the scoped `first-launch-display-backdrop` “Skip” whenever that modal appears (it renders ASYNC after /ws display-prefs load and its backdrop intercepts clicks; a one-shot check races it — needed for isolated specs on a wiped/fresh harness). See change: fix-stuck-tool-card-on-dropped-event. `TESTIDS` testid→locator map. Rides existing app data-testids; no new app testids. Adds TESTIDS keys `sessionCardDesktop`=session-card-desktop, `sessionSearchInput`=session-search-input, `onboardingStep2Cta`=onboarding-step-2-cta, `onboardingStep3Cta`=onboarding-step-3-cta, `addFoldersDialog`=add-folders-dialog, `addFoldersCommit`=add-folders-commit. Exports `FIXTURE_GIT`=/fixtures/sample-git. `pinDirectory(page,absPath)` opens the add-folders dialog via `onboarding-step-2-cta` or sidebar `dashboard-add-folder-btn`, fills the PathPicker textbox with the FULL path (the picker lists that path's parent filtered by its leaf), ticks `path-picker-check-`, clicks `add-folders-commit`. `pinDirectoryDialog` is RETIRED: the "Add folder" CTAs open `AddFoldersDialog`, not `PinDirectoryDialog` (which still exists under Settings > Packages). `PinDirectoryDialog` still exists and still renders `pin-directory-dialog`, but only under Settings ▸ Packages — the map kept pointing at it, so every pinning spec hung to its 180 s cap. Pinning is IMPLICIT: adding a folder IS pinning it, so the dialog offers no pin control. `byTestId(scope,key)` accepts a Page OR a Locator so a lookup can be dialog-scoped; `pinDirectory` asserts `add-folders-commit` ENABLED before clicking, proving the basket took the path rather than committing empty. See change: project-scope-disable-global-resources. `ensureGitSession(page)` idempotent; bounded-wait reuse of `session-card-desktop` else `pinDirectory`+spawn via `onboarding-step-3-cta` or `folder-spawn-session-btn`, returns card locator. New TESTIDS keys `gitBranchBtn`, `dashboardAddFolderBtn`, `folderSpawnSessionBtn`, `settingsContent`, `openspecBoard`, `archiveBrowser`, `specsBrowser`. Adds `sendPrompt(page,text)`: fills composer (placeholder `/message/i`), clicks `send-button` testid. Adds `sendButton` testid. `spawnFreshGitSession(page)` spawns isolated new session, resolves by unseen `data-session-id` (avoids ask_user pending-state blocking shared session). TESTIDS gains `pendingPromptCard`=pending-prompt-card + `queueChipFollowup`=queue-chip-followup. See change: add-playwright-e2e. See change: add-e2e-spawn-scenarios. See change: add-e2e-faux-model-roundtrip. See change: optimistic-prompt-progress. TESTIDS gains `chatScrollContainer`=chat-scroll-container, `scrollToBottom`=scroll-to-bottom, `turnBar`=turn-bar (windowed-transcript scroll handle + jump affordance; deliberate exception to no-new-testids). See change: virtualize-chat-transcript-tanstack. TESTIDS gains `flowLaunchRun`=flow-launch-run (existing app testid on FlowLaunchDialog; no new app testid). See change: add-flow-plugin-e2e-tests. TESTIDS gains `gitDirtyPill`, `gitDirtyCount`, `gitAhead`, `gitBehind`, `groupCommitBtn`, `commitDialog`, `commitFileList`, `commitSelectAll`, `commitSubject`, `commitBody`, `commitAiDraft`, `commitDraftUnavailable`, `commitSubmit`, `commitCancel`, `commitError` (existing app testids). Adds git working-tree helpers driving same-origin REST from the page context (cookie auth like the app): `readGitStatus(page,cwd)`, `readChangedFiles(page,cwd)`, `dirtyMarkdown(page,cwd,relPath,marker)` (md-read→md-write append; markdown-only), `cleanupCommit(page,cwd)` (commit all changed → clean, for spec isolation). Envelope is `{success,data}` (ApiResponse), NOT `{ok}`. See change: add-session-uncommitted-indicator-and-commit. `TESTIDS.refreshChat` → app testid `refresh-chat` (SessionHeader refresh control). Added because `replay-delta-on-reload.spec.ts` passed the RAW testid as the `byTestId` KEY; `TESTIDS[key]` was then `undefined` and `getByTestId(undefined)` threw `Cannot read properties of undefined (reading 'unicode')` at runtime. `tsc -p tests/tsconfig.json` catches this class now. See change: purge-replay-cache-on-reset-paths. + +`collectSubagentWire(page)` collects BOTH subagent carriers off the browser `/ws` socket, in BOTH directions. Sibling of `collectAgentTicks` rather than a replacement: the throttle rows measure `tool_execution_update` FRAME RATE filtered by `toolName`, these rows measure BYTES per carrier. Returns `{frames, requests, forAgent, repliesFor, pushesFor, requestsFor}`. Inbound `SubagentFrameSample` carries `sessionId` (shared harness → isolate your own run), `eventType`, `agentId` (read from `data.id` OR `details.agentId` OR `partialResult.details.agentId`), `status`, `entryCount`, `resyncRequestId`, `bytes`. `resyncRequestId` is the ONLY discriminator between a resync REPLY and a pushed frame — both are `subagent_started`, so eventType cannot classify them. Outbound via `ws.on("framesent")`: `subagent_resync_request` with its `requestId` + `reason` (`"open"` | `"cadence"`) — the only way to attribute a reply to the trigger that caused it. `bytes` is attributed PER FRAME (`JSON.stringify(ev).length`), unlike `collectAgentTicks` which charges every event in a batched message the whole payload length (an N× over-count). See change: verify-subagent-pull-under-load. diff --git a/tests/e2e/subagent-pull-measurements.spec.ts b/tests/e2e/subagent-pull-measurements.spec.ts new file mode 100644 index 000000000..107e7a87a --- /dev/null +++ b/tests/e2e/subagent-pull-measurements.spec.ts @@ -0,0 +1,386 @@ +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; +import type { BrowserContext, Locator, Page } from "@playwright/test"; +import { expect, test } from "./fixtures.js"; +import { + collectSubagentWire, + type SubagentWireCollector, + sendPrompt, + spawnFreshGitSession, +} from "./helpers/index.js"; + +/** + * MEASUREMENT rows for the subagent pull path (change: + * verify-subagent-pull-under-load — scenarios P1/P2/P3/P4). + * + * These are RECORDED EVIDENCE, not CI gates: the deliverable is a number written + * to `measurements.json` and transcribed into `heap-evidence.md`. They are split + * out of `subagent-pull-under-load.spec.ts` and gated on `PI_E2E_MEASURE=1` + * because each row spawns several sessions and holds several browser contexts — + * on the shared single-container harness that is minutes of wall clock, and + * bundling them with the behavioural rows blew the global teardown budget. + * + * Every row spawns a FRESH session per measured run. Re-prompting one session is + * cheaper but does NOT reliably produce a second subagent in this harness, and a + * silent zero would be transcribed into the evidence file as if it were data — + * so each row asserts that every arm actually produced frames. + * + * Two harness starts are required: + * pull arm: PI_E2E_SEED=1 PI_TEST_PEERS=both PI_SYNTH_AGENT_TICKS=1 ./docker/test-up.sh -d + * push arm: ... plus PI_DASHBOARD_SUBAGENT_STRIP=0 + * then, against the derived `.pi-test-harness.json` port: + * PW_E2E_USE_RUNNING=1 PW_E2E_PORT=$PORT PI_SYNTH_AGENT_TICKS=1 PI_E2E_MEASURE=1 \ + * PW_CHANNEL=chrome npx playwright test subagent-pull-measurements \ + * --global-timeout=2400000 + * + * `--global-timeout` is REQUIRED: the config default is 15 min, and these rows + * spawn a session per measured run. Without it the run aborts mid-suite and + * reports "did not run" rather than a real failure. + */ + +const WATCHED_BY_BUS: Record = { + 100: "[[faux:subagent-watched-growth-bus100]] go", + 250: "[[faux:subagent-watched-growth]] go", + 1000: "[[faux:subagent-watched-growth-bus1000]] go", +}; +const EVIDENCE_PATH = "openspec/changes/verify-subagent-pull-under-load/measurements.json"; +const STRIP_OFF = process.env.PI_DASHBOARD_SUBAGENT_STRIP === "0"; +/** Fixture runtime: 240 ticks @ 50 ms. */ +const RUNTIME_MS = 12_000; +const WINDOW_MS = 6_000; + +function record(key: string, value: unknown): void { + mkdirSync(dirname(EVIDENCE_PATH), { recursive: true }); + let current: Record = {}; + try { + current = JSON.parse(readFileSync(EVIDENCE_PATH, "utf8")) as Record; + } catch { + /* first write */ + } + current[key] = value; + writeFileSync(EVIDENCE_PATH, `${JSON.stringify(current, null, 2)}\n`); +} + +async function startWatchedRun( + page: Page, + prompt: string, +): Promise<{ wire: SubagentWireCollector; sessionId: string }> { + await page.goto("/"); + const wire = collectSubagentWire(page); + const card = await spawnFreshGitSession(page); + const sessionId = (await card.getAttribute("data-session-id")) ?? ""; + await card.click(); + await page.keyboard.press("Escape").catch(() => {}); + await sendPrompt(page, prompt); + return { wire, sessionId }; +} + +async function resolveAgentId( + wire: SubagentWireCollector, + sessionId: string, + timeoutMs = 90_000, +): Promise { + const mine = () => wire.frames.filter((f) => f.sessionId === sessionId && f.agentId); + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (mine().length > 0) return mine()[0]!.agentId; + await new Promise((r) => setTimeout(r, 250)); + } + return null; +} + +/** + * Mount the inline inspector and return a collapse handle. + * + * The control's LABEL flips `Details` -> `Collapse` once expanded + * (`AgentToolRenderer` CardControls), so the locator used to open it can never + * close it — the collapse handle has to match the expanded label. + */ +async function mountInspector(page: Page): Promise<() => Promise> { + const details = page.getByRole("button", { name: /^Details$/ }).first(); + const toggles = page.getByRole("button", { name: /watched growing subagent/i }); + const deadline = Date.now() + 60_000; + while (Date.now() < deadline) { + if (await details.isVisible().catch(() => false)) break; + const count = await toggles.count(); + for (let i = 0; i < count; i++) { + if (await details.isVisible().catch(() => false)) break; + await toggles.nth(i).click().catch(() => {}); + } + await page.waitForTimeout(250); + } + await expect(details).toBeVisible({ timeout: 30_000 }); + await details.click(); + return async () => { + const collapse = page.getByRole("button", { name: /^Collapse$/ }).first(); + await expect(collapse).toBeVisible({ timeout: 15_000 }); + await collapse.click(); + }; +} + +interface Rates { + replyBytesPerSec: number; + pushBytesPerSec: number; + repliesPerSec: number; + frames: number; + toolCarrierFrames: number; + busCarrierFrames: number; +} + +function byteRates( + wire: SubagentWireCollector, + agentId: string, + t0: number, + windowMs: number, +): Rates { + const inWindow = wire.forAgent(agentId).filter((f) => f.at >= t0 && f.at <= t0 + windowMs); + // Classify by the `__resyncRequestId` discriminator, NEVER by eventType: a + // reply and a pushed frame are both `subagent_started`. + const replies = inWindow.filter((f) => f.resyncRequestId !== undefined); + const pushes = inWindow.filter((f) => f.resyncRequestId === undefined); + const per = (n: number) => (n / windowMs) * 1000; + return { + replyBytesPerSec: per(replies.reduce((a, f) => a + f.bytes, 0)), + pushBytesPerSec: per(pushes.reduce((a, f) => a + f.bytes, 0)), + repliesPerSec: per(replies.length), + frames: inWindow.length, + toolCarrierFrames: inWindow.filter((f) => f.eventType === "tool_execution_update").length, + busCarrierFrames: inWindow.filter((f) => f.eventType.startsWith("subagent_")).length, + }; +} + +/** One measured run: fresh session, inspector mounted, N-1 extra subscribers. */ +async function measureOnce( + page: Page, + context: BrowserContext, + prompt: string, + subscribers: number, + attempt = 0, + /** + * Pull arms must also observe a NON-ZERO reply rate. `frames > 0` alone is + * satisfied by pushes, so a silently-broken pull path would write + * `replyBytesPerSec: 0` into the evidence file and the verdict + * `0 <= push_removed` would "pass" trivially. + */ + requireReplies = false, +): Promise { + const { wire, sessionId } = await startWatchedRun(page, prompt); + const agentId = await resolveAgentId(wire, sessionId); + if (agentId === null) { + // A spawn that never registers is HOST contention (this harness shares the + // machine with other test containers), not a product signal. Retry once, + // then fail loudly rather than recording a silent zero. + expect(attempt, "a fresh session produced no subagent twice in a row").toBeLessThan(1); + // `requireReplies` MUST survive the retry: dropping it would let a pull arm + // silently record `replyBytesPerSec: 0` on its second attempt. + return measureOnce(page, context, prompt, subscribers, attempt + 1, requireReplies); + } + await mountInspector(page); + const extras: Page[] = []; + for (let i = 1; i < subscribers; i++) { + const p = await context.newPage(); + await p.goto(`/session/${sessionId}`); + extras.push(p); + } + const t0 = Date.now(); + await page.waitForTimeout(WINDOW_MS); + const rates = byteRates(wire, agentId, t0, WINDOW_MS); + for (const p of extras) await p.close(); + // Non-vacuity: a silent zero must never reach the evidence file as data. + expect(rates.frames, "the measured window observed subagent frames").toBeGreaterThan(0); + if (requireReplies) { + expect( + rates.repliesPerSec, + "the pull arm observed resync REPLIES (a zero here would make the verdict vacuous)", + ).toBeGreaterThan(0); + } + return rates; +} + +const median = (xs: number[]): number => { + const s = [...xs].sort((a, b) => a - b); + return s.length % 2 ? s[(s.length - 1) / 2]! : (s[s.length / 2 - 1]! + s[s.length / 2]!) / 2; +}; +const spread = (xs: number[]): number => Math.max(...xs) - Math.min(...xs); + +test.describe("subagent pull path — recorded measurements", () => { + test.skip( + process.env.PI_SYNTH_AGENT_TICKS !== "1", + "requires the PI_SYNTH_AGENT_TICKS=1 harness arm", + ); + test.skip( + process.env.PI_E2E_MEASURE !== "1", + "measurement rows are evidence collection — opt in with PI_E2E_MEASURE=1", + ); + + test("P1: harness ceiling for concurrent subscribers", async ({ page, context }) => { + test.skip(STRIP_OFF, "the ceiling is measured once, on the default arm"); + test.setTimeout(600_000); + + // Measured, not assumed: without this, a saturated harness would masquerade + // as a byte-rate result in P2. + const samples: Array<{ n: number; frames: number }> = []; + for (const n of [1, 3, 5]) { + const rates = await measureOnce(page, context, WATCHED_BY_BUS[250]!, n); + samples.push({ n, frames: rates.frames }); + } + const baseline = samples[0]!.frames; + const ceiling = Math.max(...samples.filter((s) => s.frames >= baseline * 0.8).map((s) => s.n)); + record("P1-harness-ceiling", { windowMs: WINDOW_MS, samples, baseline, ceiling }); + + // `measureOnce` already fails an arm that produced nothing, so a ceiling + // computed over a silently-empty arm cannot happen. + expect(baseline).toBeGreaterThan(0); + expect(ceiling).toBeGreaterThanOrEqual(1); + }); + + test("P2/P3 pull arm: reply byte rate + bus-cadence sensitivity", async ({ page, context }) => { + test.skip(STRIP_OFF, "the pull arm requires the strip ON (default harness)"); + test.setTimeout(900_000); + + const N = 3; + const rows: Array> = []; + // 250 ms is the production-matched HEADLINE (3 runs -> median + spread); + // 100/1000 are the sensitivity flanks (1 run each). A verdict that flips + // across the flanks is itself the finding. + const matrix: Array<[number, number]> = [ + [250, 3], + [100, 1], + [1000, 1], + ]; + for (const [bus, runs] of matrix) { + for (let run = 0; run < runs; run++) { + rows.push({ + bus, + run, + N, + ...(await measureOnce(page, context, WATCHED_BY_BUS[bus]!, N, 0, true)), + }); + } + } + + const headline = rows.filter((r) => r.bus === 250).map((r) => r.replyBytesPerSec as number); + record("P2-P3-pull-arm", { + windowMs: WINDOW_MS, + N, + rows, + headline: { + busIntervalMs: 250, + replyBytesPerSecMedian: median(headline), + replyBytesPerSecSpread: spread(headline), + }, + note: + "Pull = bytes/s of frames bearing __resyncRequestId, per subscriber. The bytes the strip " + + "REMOVES are (push_stripOff - push_stripOn), the push arm coming from the separate " + + "PI_DASHBOARD_SUBAGENT_STRIP=0 harness start. A verdict inside the run-to-run spread is " + + "INCONCLUSIVE — a shippable outcome, since the measurement infrastructure is the deliverable.", + }); + + expect(headline.length, "the headline cadence was measured 3x").toBe(3); + }); + + test("P2/P3 push arm: subagent-carrying byte rate with the strip OFF", async ({ + page, + context, + }) => { + test.skip(!STRIP_OFF, "requires the PI_DASHBOARD_SUBAGENT_STRIP=0 harness start"); + test.setTimeout(900_000); + + const N = 3; + const rows: Array> = []; + const matrix: Array<[number, number]> = [ + [250, 3], + [100, 1], + [1000, 1], + ]; + for (const [bus, runs] of matrix) { + for (let run = 0; run < runs; run++) { + rows.push({ bus, run, N, ...(await measureOnce(page, context, WATCHED_BY_BUS[bus]!, N)) }); + } + } + + const headline = rows.filter((r) => r.bus === 250).map((r) => r.pushBytesPerSec as number); + record("P2-P3-push-arm", { + windowMs: WINDOW_MS, + N, + rows, + headline: { + busIntervalMs: 250, + pushBytesPerSecMedian: median(headline), + pushBytesPerSecSpread: spread(headline), + }, + note: + "pushBytesPerSec EXCLUDES __resyncRequestId frames: the client still pulls in this arm, so " + + "counting replies here would inflate 'the bytes the strip removes'. The push/pull subtraction " + + "is only valid against a pull arm whose reply rate was non-zero (asserted there).", + }); + + expect(headline.length).toBe(3); + }); + + test("P4: inspector-open share across four watch patterns", async ({ page, context }) => { + test.skip(STRIP_OFF, "the share is a client-side reading, measured on the default arm"); + // MUST stay below the config's `globalTimeout` (15 min). At 900_000 the two + // budgets coincide, so the RUN aborted before the test could report and the + // failure surfaced as "1 did not run" with no diagnostic at all. Four arms + // need more than 15 min of wall clock on a loaded host, so this row is run + // with an explicit `--global-timeout` (see the header). + test.setTimeout(600_000); + void page; + + // `__piSubagentInspectorTelemetry()` is a page-global CUMULATIVE aggregate + // and `resetInspectorTelemetry` is NOT exposed on globalThis, so each arm + // needs its OWN PAGE (a fresh JS realm resets the module-level state). + // Sharing one page blends all four arms into a single meaningless number. + const arms = [ + { id: "unwatched", pattern: "never opened", openAt: -1, holdFor: 0 }, + { id: "glance", pattern: "open at 25% of runtime, hold 25%", openAt: 0.25, holdFor: 0.25 }, + { id: "threshold", pattern: "open at 25% of runtime, hold 50%", openAt: 0.25, holdFor: 0.5 }, + { id: "watched", pattern: "open before the first entry, never closed", openAt: 0, holdFor: 1 }, + ]; + const readings: Array> = []; + + for (const arm of arms) { + // eslint-disable-next-line no-console -- a 4-arm row is long; without a + // progress line a stall is indistinguishable from a hang. + console.log(`[P4] arm=${arm.id} (${arm.pattern})`); + const p = await context.newPage(); + await startWatchedRun(p, WATCHED_BY_BUS[250]!); + if (arm.openAt >= 0) { + if (arm.openAt > 0) await p.waitForTimeout(RUNTIME_MS * arm.openAt); + const collapse = await mountInspector(p); + if (arm.holdFor < 1) { + await p.waitForTimeout(RUNTIME_MS * arm.holdFor); + await collapse(); // the inspector is no longer mounted + } + } + await p.waitForTimeout(RUNTIME_MS); + const telemetry = await p.evaluate(() => { + const fn = (globalThis as Record).__piSubagentInspectorTelemetry as + | (() => unknown) + | undefined; + return fn ? fn() : null; + }); + readings.push({ ...arm, runtimeMs: RUNTIME_MS, telemetry }); + await p.close(); + } + + record("P4-inspector-open-share", { + note: + "A scripted harness cannot yield a FIELD-representative share — the number is whatever the " + + "watch pattern makes it. Each reading is reported WITH its pattern. The 50% arm sits ON the " + + "C4 boundary by construction. The field number still comes from the production counter.", + readings, + }); + + // Non-vacuity: the signal must MOVE across the spectrum, else a constant + // would be transcribed into the evidence file as a measurement. + const share = (r: Record): number => + Number((r.telemetry as { share?: number } | null)?.share ?? -1); + expect(share(readings[0]!), "the unwatched arm reads ~0").toBeLessThan(0.05); + expect(share(readings[3]!), "the fully-watched arm reads high").toBeGreaterThan(0.5); + expect(share(readings[3]!), "the share tracks how long the inspector was held").toBeGreaterThan( + share(readings[1]!), + ); + }); +}); diff --git a/tests/e2e/subagent-pull-measurements.spec.ts.AGENTS.md b/tests/e2e/subagent-pull-measurements.spec.ts.AGENTS.md new file mode 100644 index 000000000..1c6479d81 --- /dev/null +++ b/tests/e2e/subagent-pull-measurements.spec.ts.AGENTS.md @@ -0,0 +1,3 @@ +# subagent-pull-measurements.spec.ts — index + +RECORDED-EVIDENCE measurement rows for the subagent pull path (change: verify-subagent-pull-under-load): P1 harness ceiling, P2/P3 pull+push byte rates across a bus-cadence sensitivity table, P4 inspector-open share across four watch patterns. Gated on BOTH `PI_SYNTH_AGENT_TICKS=1` AND `PI_E2E_MEASURE=1`; push rows need a second harness start with `PI_DASHBOARD_SUBAGENT_STRIP=0`. Writes `openspec/changes/verify-subagent-pull-under-load/measurements.json`. MUST run with `--global-timeout` above the 15-min config default — at parity the run aborts and reports "did not run" with no diagnostic. Every arm asserts frames>0, and PULL arms additionally assert repliesPerSec>0 — `frames>0` alone is satisfied by pushes, so a silently-broken pull path would write `replyBytesPerSec: 0` and the `0 <= push_removed` verdict would pass trivially. diff --git a/tests/e2e/subagent-pull-under-load.spec.ts b/tests/e2e/subagent-pull-under-load.spec.ts new file mode 100644 index 000000000..1f3bbe540 --- /dev/null +++ b/tests/e2e/subagent-pull-under-load.spec.ts @@ -0,0 +1,450 @@ +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; +import type { Locator, Page } from "@playwright/test"; +import { expect, test } from "./fixtures.js"; +import { + collectSubagentWire, + type SubagentFrameSample, + type SubagentWireCollector, + sendPrompt, + spawnFreshGitSession, +} from "./helpers/index.js"; + +/** + * L3 rows for the open-inspector PULL path (change: + * verify-subagent-pull-under-load). Verifies the four scenarios + * `reduce-subagent-details-payload` shipped UNVERIFIED: F1 (a mounted inspector + * converges on a growing timeline), P4 (the cadence costs no more than the push + * it replaced), P5 (a representative inspector-open share) and X1 (a run killed + * with no terminal frame). + * + * SUBSTRATE — the synthetic Agent-tick producer with its two pull-path + * sentinels (`qa/fixtures/faux-agent-ticks.ext.ts`, scenario + * `subagent-watched-growth`): a 12 s run whose timeline grows 5 -> 30 over the + * first ~3 s and then PLATEAUS for ~9 s, emitting `subagents:*` bus frames + * coalesced at the real producer's 250 ms. A nested faux subagent cannot be + * used — it dies after ~2 no-op turns (see reduce-bridge-tick-bandwidth + * measurement.md), which is precisely why the parent's four scenarios were + * unverifiable. + * + * WHY THE PLATEAU IS LOAD-BEARING: terminal frames are NEVER stripped, so if + * the timeline only reached 30 at the last tick, the rendered count would + * converge on `subagents:completed` and F1 would pass with the pull path never + * running. The ~9 s plateau (>= 3 cadence intervals at CADENCE_BASE_MS = 2000) + * is what lets the count converge WHILE THE AGENT IS STILL RUNNING. + * + * ANTI-VACUITY — every row keys on `__resyncRequestId`, not on eventType: a + * resync reply and a pushed frame are BOTH `subagent_started`, so eventType + * cannot tell them apart. + * + * HARNESS ARM — this producer SHADOWS the real subagents `Agent` tool + * (first-registration-wins), so it is staged ONLY under `PI_SYNTH_AGENT_TICKS=1`. + * This spec self-skips otherwise. Run it on its own arm: + * + * PI_E2E_SEED=1 PI_TEST_PEERS=both PI_SYNTH_AGENT_TICKS=1 ./docker/test-up.sh -d + * PORT=$(jq -r '.dashboardPort' .pi-test-harness.json) + * PW_E2E_USE_RUNNING=1 PW_E2E_PORT=$PORT PI_SYNTH_AGENT_TICKS=1 PW_CHANNEL=chrome \ + * npx playwright test subagent-pull-under-load + * ./docker/test-down.sh + * + * The PUSH arm of P4 (and the F4 anti-vacuity inversion) needs a SECOND harness + * start with `PI_DASHBOARD_SUBAGENT_STRIP=0` — the bridge reads that env per + * call, but a container env var is fixed for the life of the harness. Those + * rows self-skip on the arm they do not belong to. + */ + +const WATCHED = "[[faux:subagent-watched-growth]] go"; +/** Bus-cadence flanks for the P4 sensitivity table (250 ms = production-matched). */ +const WATCHED_BY_BUS: Record = { + 100: "[[faux:subagent-watched-growth-bus100]] go", + 250: WATCHED, + 1000: "[[faux:subagent-watched-growth-bus1000]] go", +}; +const START_ENTRIES = 5; +const END_ENTRIES = 30; +/** Matches the fixture's entry text, so a DOM count needs no production hook. */ +const ENTRY_TEXT = /^faux-entry \d+$/; +const CADENCE_BASE_MS = 2_000; +const EVIDENCE_PATH = "openspec/changes/verify-subagent-pull-under-load/measurements.json"; + +/** True on the harness start that disables the bridge strip (the push arm). */ +const STRIP_OFF = process.env.PI_DASHBOARD_SUBAGENT_STRIP === "0"; + +/** Append one recorded measurement so heap-evidence.md is transcribed, not invented. */ +function record(key: string, value: unknown): void { + mkdirSync(dirname(EVIDENCE_PATH), { recursive: true }); + let current: Record = {}; + try { + current = JSON.parse(readFileSync(EVIDENCE_PATH, "utf8")) as Record; + } catch { + /* first write */ + } + current[key] = value; + writeFileSync(EVIDENCE_PATH, `${JSON.stringify(current, null, 2)}\n`); +} + +/** Start a watched run and return its session id + wire collector. */ +async function startWatchedRun( + page: Page, + prompt: string = WATCHED, +): Promise<{ wire: SubagentWireCollector; sessionId: string }> { + await page.goto("/"); + const wire = collectSubagentWire(page); + const card = await spawnFreshGitSession(page); + const sessionId = (await card.getAttribute("data-session-id")) ?? ""; + await card.click(); + await page.keyboard.press("Escape").catch(() => {}); + await sendPrompt(page, prompt); + return { wire, sessionId }; +} + +/** Sum bytes/s over a window, split by the resync-reply discriminator. */ +function byteRates( + wire: SubagentWireCollector, + agentId: string, + t0: number, + windowMs: number, +): { replyBytesPerSec: number; pushBytesPerSec: number; repliesPerSec: number; frames: number } { + const inWindow = wire.forAgent(agentId).filter((f) => f.at >= t0 && f.at <= t0 + windowMs); + const replies = inWindow.filter((f) => f.resyncRequestId !== undefined); + const pushes = inWindow.filter((f) => f.resyncRequestId === undefined); + const per = (n: number) => (n / windowMs) * 1000; + return { + replyBytesPerSec: per(replies.reduce((a, f) => a + f.bytes, 0)), + pushBytesPerSec: per(pushes.reduce((a, f) => a + f.bytes, 0)), + repliesPerSec: per(replies.length), + frames: inWindow.length, + }; +} + +/** Read the page-global inspector-open telemetry (cumulative, no exposed reset). */ +async function readInspectorShare(page: Page): Promise { + return page.evaluate(() => { + const fn = (globalThis as Record).__piSubagentInspectorTelemetry as + | (() => unknown) + | undefined; + return fn ? fn() : null; + }); +} + +/** + * The agentId of THIS run, resolved off the wire (never guessed). + * + * `seen` excludes agents from EARLIER runs on the same session: the measurement + * rows re-prompt one session instead of spawning a fresh one per run (nine + * session spawns saturate the shared single-container harness, and a saturated + * harness would masquerade as a byte-rate result). + */ +async function resolveAgentId( + wire: SubagentWireCollector, + sessionId: string, + seen: Set = new Set(), +): Promise { + const fresh = () => + wire.frames.filter((f) => f.sessionId === sessionId && f.agentId && !seen.has(f.agentId)); + await expect.poll(() => fresh().length, { timeout: 90_000, intervals: [250] }).toBeGreaterThan(0); + const id = fresh()[0]!.agentId; + seen.add(id); + return id; +} + +/** Mount the inline inspector (the "Details" pill) for the watched subagent. */ +async function mountInspector(page: Page): Promise { + const details = page.getByRole("button", { name: /^Details$/ }).first(); + const toggles = page.getByRole("button", { name: /watched growing subagent/i }); + const deadline = Date.now() + 60_000; + while (Date.now() < deadline) { + if (await details.isVisible().catch(() => false)) break; + const count = await toggles.count(); + for (let i = 0; i < count; i++) { + if (await details.isVisible().catch(() => false)) break; + await toggles.nth(i).click().catch(() => {}); + } + await page.waitForTimeout(250); + } + await expect(details).toBeVisible({ timeout: 30_000 }); + await details.click(); + return details; +} + +/** Rendered timeline entries currently in the DOM. */ +const renderedEntries = (page: Page) => page.getByText(ENTRY_TEXT); + +/** A frame is TERMINAL for the watched agent. */ +const isTerminal = (f: SubagentFrameSample): boolean => + f.eventType === "subagent_completed" || + f.eventType === "subagent_failed" || + f.eventType === "tool_execution_end" || + f.status === "completed" || + f.status === "failed"; + +test.describe("subagent pull path under load (synthetic watched-growth substrate)", () => { + test.skip( + process.env.PI_SYNTH_AGENT_TICKS !== "1", + "requires the PI_SYNTH_AGENT_TICKS=1 harness arm (synthetic Agent tool shadows the real subagents Agent tool)", + ); + + test("F1/F2/F3/F5/F6/F7: a mounted inspector converges via the CADENCE, not via a push or a terminal frame", async ({ + page, + }) => { + test.skip(STRIP_OFF, "the pull arm requires the strip ON (default harness)"); + test.setTimeout(240_000); + + const { wire, sessionId } = await startWatchedRun(page); + const agentId = await resolveAgentId(wire, sessionId); + + // Mount BEFORE the timeline renders anything: the rendered count starts at + // 0 (pushes are thin), so the inspector is watching a growing timeline. + await mountInspector(page); + + // --- F1: the rendered count converges while the agent is NON-TERMINAL ---- + // Poll both together: the moment a terminal frame lands the row is over, + // because a terminal frame is never stripped and would converge the count + // by itself. + let convergedAt = 0; + let maxRendered = 0; + await expect + .poll( + async () => { + if (wire.forAgent(agentId).some(isTerminal)) return "terminal"; + const n = await renderedEntries(page).count(); + maxRendered = Math.max(maxRendered, n); + if (n >= END_ENTRIES) convergedAt = Date.now(); + return n >= END_ENTRIES ? "converged" : `n=${n}`; + }, + { timeout: 60_000, intervals: [250] }, + ) + .toBe("converged"); + expect(convergedAt, "converged before any terminal frame").toBeGreaterThan(0); + + const beforeConvergence = (f: SubagentFrameSample) => f.at <= convergedAt; + const mine = wire.forAgent(agentId).filter(beforeConvergence); + + // --- F2: every PUSHED frame was thin ----------------------------------- + const fatPushes = mine.filter( + (f) => f.resyncRequestId === undefined && !isTerminal(f) && f.entryCount > 0, + ); + expect( + fatPushes.map((f) => `${f.eventType}:${f.entryCount}`), + "no non-terminal PUSH frame may carry a timeline (the strip is what makes the pull path load-bearing)", + ).toEqual([]); + + // --- F3: at least one REPLY was fat ------------------------------------- + const replies = mine.filter((f) => f.resyncRequestId !== undefined); + expect(replies.length, "at least one resync reply arrived").toBeGreaterThan(0); + expect( + replies.some((f) => f.entryCount > 0), + "a resync reply carried a real timeline", + ).toBe(true); + + // --- F5: the converging reply belongs to a CADENCE request -------------- + // Token equality, not ordering: a reconnect-driven `reason:"open"` reply + // would satisfy mere ordering and prove nothing about the cadence. + const cadenceIds = new Set( + wire.requestsFor(agentId, "cadence").filter((r) => r.at <= convergedAt).map((r) => r.requestId), + ); + expect(cadenceIds.size, "the mounted inspector fired at least one cadence resync").toBeGreaterThan(0); + const convergingCadenceReply = replies.find( + (f) => f.entryCount >= END_ENTRIES && cadenceIds.has(f.resyncRequestId!), + ); + expect( + convergingCadenceReply, + "the reply that carried the full timeline was answering a CADENCE request", + ).toBeTruthy(); + + // --- F6: an open-time reply cannot explain the convergence -------------- + // The open-time trigger DOES fire here (the rendered timeline is empty at + // mount, so `emptyTimeline` holds, and App.tsx also resyncs on subscribe). + // Ruled out by CONTENT, not by ordering: every reply answering an + // open-time request is inspected, and none of them carried enough of the + // timeline to explain the converged count. A wall-clock "the cadence reply + // came later" check would be satisfied trivially and prove nothing. + const opens = wire.requestsFor(agentId, "open").filter((r) => r.at <= convergedAt); + const openIds = new Set(opens.map((r) => r.requestId)); + const openReplyMax = Math.max( + 0, + ...replies.filter((f) => openIds.has(f.resyncRequestId!)).map((f) => f.entryCount), + ); + expect( + openReplyMax, + "no open-time reply carried the full timeline, so it cannot explain the convergence", + ).toBeLessThan(END_ENTRIES); + // Stronger than the count bound alone (which an open reply carrying 29 + // would satisfy): the reply that ACTUALLY carried the converged timeline is + // identified by token, and that token is not one of the open-time requests. + expect( + openIds.has(convergingCadenceReply!.resyncRequestId!), + "the reply that carried the converged timeline answered a CADENCE request, not an open-time one", + ).toBe(false); + + // --- F7: the terminal frame neither loses nor duplicates the timeline --- + // Asserted where it is decidable. OBSERVED: on completion the finished Agent + // row is re-grouped into a tool-burst-group header and the inline body is no + // longer reachable at all (no Details pill in the completed row) — so a + // post-terminal DOM count would assert that GROUPING behaviour, not this + // change's claim, and would hang waiting for a control that never appears. + // The claim itself is asserted on the terminal frame's own payload plus the + // no-duplication bound the DOM did honour while it was mounted. + await expect + .poll(() => wire.forAgent(agentId).some(isTerminal), { timeout: 90_000, intervals: [500] }) + .toBe(true); + const terminal = wire.forAgent(agentId).filter(isTerminal).filter((f) => f.entryCount > 0); + expect(terminal.length, "the terminal frame is fat (never stripped)").toBeGreaterThan(0); + for (const f of terminal) { + expect(f.entryCount, `terminal ${f.eventType} carries exactly the final timeline`).toBe( + END_ENTRIES, + ); + } + expect(maxRendered, "the rendered timeline never exceeded the produced one").toBeLessThanOrEqual( + END_ENTRIES, + ); + + record("F1", { + convergedBeforeTerminal: true, + pushFramesObserved: mine.filter((f) => f.resyncRequestId === undefined).length, + fatPushFrames: fatPushes.length, + repliesObserved: replies.length, + cadenceRequests: cadenceIds.size, + openRequests: opens.length, + maxEntriesInAnyOpenTimeReply: openReplyMax, + maxRenderedDuringRun: maxRendered, + terminalFrameEntryCount: END_ENTRIES, + observed: + "on completion the Agent row is re-grouped into a tool-burst-group header with no reachable " + + "Details pill, so the inline timeline unmounts. Pinned as observed; out of scope for this change.", + }); + }); + + test("F4: anti-vacuity — with the strip OFF the pushed frames are FAT", async ({ page }) => { + test.skip( + !STRIP_OFF, + "requires the PI_DASHBOARD_SUBAGENT_STRIP=0 harness start (the push arm)", + ); + test.setTimeout(240_000); + + const { wire, sessionId } = await startWatchedRun(page); + const agentId = await resolveAgentId(wire, sessionId); + + // The SAME observation as F2 must INVERT. If it does not, the env switch is + // unwired and every measurement in this spec is one arm measured twice. + await expect + .poll( + () => + wire + .forAgent(agentId) + .filter((f) => f.resyncRequestId === undefined && !isTerminal(f) && f.entryCount > 0) + .length, + { timeout: 90_000, intervals: [250] }, + ) + .toBeGreaterThan(0); + + record("F4", { stripOffPushesAreFat: true }); + }); + + test("X1/X2/X3: killed mid-run with no terminal frame, then replayed", async ({ page }) => { + test.skip(STRIP_OFF, "the regression is a property of the strip-ON pull model"); + test.setTimeout(240_000); + + // X2 — UNWATCHED by construction. A single resync stores a FAT reply + // (every event_forward is persisted and replies are never stripped), which + // would show a mid-run timeline on replay and fail X1 for a reason that has + // nothing to do with the regression. So: no inspector, and the session is + // never SELECTED (App.tsx resyncs every running empty-timeline subagent on + // subscribe). + await page.goto("/"); + const wire = collectSubagentWire(page); + const card = await spawnFreshGitSession(page); + const sessionId = (await card.getAttribute("data-session-id")) ?? ""; + await card.click(); + await page.keyboard.press("Escape").catch(() => {}); + await sendPrompt(page, WATCHED); + const agentId = await resolveAgentId(wire, sessionId); + + // Let a timeline accumulate on the producer side (thin on the wire). + await expect + .poll(() => wire.forAgent(agentId).length, { timeout: 90_000, intervals: [250] }) + .toBeGreaterThan(10); + + // Navigate AWAY so no inspector is mounted and no further open-resync can + // fire while the run is killed. + await page.goto("/"); + await page.waitForTimeout(500); + + expect( + wire.requestsFor(agentId).map((r) => `${r.reason}:${r.requestId}`), + "X2: an unwatched run must issue ZERO resync requests — a stored fat reply would fail X1 spuriously", + ).toEqual([]); + expect( + wire.forAgent(agentId).some(isTerminal), + "the run must still be mid-flight at kill time", + ).toBe(false); + + // --- the kill ---------------------------------------------------------- + await page.evaluate((id) => { + const ws = new WebSocket(`${location.origin.replace(/^http/, "ws")}/ws`); + return new Promise((resolve) => { + ws.onopen = () => { + ws.send(JSON.stringify({ type: "force_kill", sessionId: id })); + setTimeout(() => { + ws.close(); + resolve(); + }, 500); + }; + ws.onerror = () => resolve(); + }); + }, sessionId); + + // --- the replay -------------------------------------------------------- + // `force_kill` closes the bridge WS BEFORE the signal, so socket silence is + // not proof of anything. The decidable surface is the STORED transcript the + // server re-sends on subscribe. + // Address the ended session by ROUTE. An ended card is hidden behind its + // folder's ended group in the list, and hunting that grouping would make the + // row assert list behaviour instead of replay behaviour. + const replay = collectSubagentWire(page); + await page.goto(`/session/${sessionId}`); + await page.waitForTimeout(8_000); + + const replayed = replay.forAgent(agentId); + expect(replayed.length, "the killed run replayed some subagent state").toBeGreaterThan(0); + + // X3 — no terminal frame and no stored timeline for that agent. + expect( + replayed.filter(isTerminal).map((f) => f.eventType), + "X3: a run killed mid-flight has no terminal frame in the store", + ).toEqual([]); + expect( + replayed.filter((f) => f.entryCount > 0).map((f) => `${f.eventType}:${f.entryCount}`), + "X3: no stored frame carries a timeline for the killed agent", + ).toEqual([]); + + // The tick index is read from the REPLAY, not live: reading it live would + // require subscribing during the run, which fires the open-resync X2 forbids. + const tickIndexOf = (f: SubagentFrameSample): number => { + const m = /\(running… (\d+)\)/.exec(JSON.stringify(f)); + return m ? Number(m[1]) : -1; + }; + const lastTick = Math.max(-1, ...replayed.map(tickIndexOf)); + + // X1 — scalar state renders, no mid-run timeline, neither blank nor corrupt. + await expect(page.getByText(/watched growing subagent/i).first()).toBeVisible({ + timeout: 30_000, + }); + expect(await renderedEntries(page).count(), "X1: no mid-run timeline is shown").toBe(0); + const bodyText = (await page.locator("body").innerText()).trim(); + expect(bodyText.length, "X1: the render is not blank").toBeGreaterThan(0); + expect(bodyText).not.toMatch(/subagent not found/i); + + record("X1", { + killMechanism: "force_kill", + resyncRequestsDuringRun: 0, + replayedFrames: replayed.length, + storedTerminalFrames: 0, + storedTimelineFrames: 0, + lastObservedTickIndex: lastTick, + // Pinned as OBSERVED (V4): whatever the stuck-card supersede-heal does to + // a killed Agent call is the baseline this row protects from drift. + renderedEntryCount: 0, + }); + }); +}); diff --git a/tests/e2e/subagent-pull-under-load.spec.ts.AGENTS.md b/tests/e2e/subagent-pull-under-load.spec.ts.AGENTS.md new file mode 100644 index 000000000..adad8751f --- /dev/null +++ b/tests/e2e/subagent-pull-under-load.spec.ts.AGENTS.md @@ -0,0 +1,3 @@ +# subagent-pull-under-load.spec.ts — index + +L3 behavioural rows for the open-inspector PULL path (change: verify-subagent-pull-under-load). F1/F2/F3/F5/F6/F7: a mounted inspector converges 5→30 rendered entries while the agent is NON-TERMINAL, with the delivering carrier identified positively (the converging reply's `__resyncRequestId` equals a captured `reason:"cadence"` request id). Classification keys on `__resyncRequestId`, never eventType — a reply and a pushed frame are both `subagent_started`. F4: anti-vacuity inversion, requires the `PI_DASHBOARD_SUBAGENT_STRIP=0` harness start. X1/X2/X3: `force_kill` mid-run on an explicitly UNWATCHED run (a stored fat resync reply would fail X1 spuriously), asserted on the post-kill replay stream. Gated on `PI_SYNTH_AGENT_TICKS=1`.