docs: make the playbook answer the question that decides the PoC - #50
docs: make the playbook answer the question that decides the PoC#50yuting0624 wants to merge 1 commit into
Conversation
Two independent efforts -- ours on Harbor, a colleague's on SWE-bench and SWE-bench Pro -- spent days measuring the regime where delegation cannot win, and both landed within ~10% of solo. Neither had a reason to ask first. That is a defect in this playbook, not in either effort. New principle 0.1: does the conductor have to keep the raw material in context to be accountable for the result? If yes, expect parity and do not build a cost story on it -- Claude re-reads handed-over context rather than trusting it, which is the same rule that makes the hybrid safe. You cannot have 'the conductor doesn't re-read' and 'the conductor owns correctness' at once. Section 5 rewritten from 'record these fields' to conditions that are requirements, each one having cost us a rerun: every trial cold (our +46% headline was a cache draw), identical caps on both arms, executor usage to a file because tail eats the stderr line, and delegation count -- a trial with 0 delegations is the baseline arm wearing a hat, which is what the plugin does when merely installed. Headline metric moved to cost-of-pass. Cost per trial is gameable by failing cheaply and it got us: the cheapest hybrid trial was the one that timed out, and including it read 14.7% where excluding it read 42.1%. Added the two accounting systems side by side. cache_read is an inner subset on the Claude side and a separate counter on agy's; merging them is the easiest way to be badly wrong in either direction. New section 8 for the likely outcome. The playbook only had a happy path, which is how someone spends a week proving nothing -- parity is a result, and there are four things to present instead, ordered by how hard they are to argue with.
Quorum review
Beyond the diff, the models made 35 read-only lookup(s) into the checkout at Refuted by the other model (3)
No new issues found in this diff. Usage
Reviewed |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| Measured: a 62k-token corpus offloaded to a digest leaves the conductor carrying **4.4k | ||
| instead of 62k**, i.e. **$0.002 vs $0.031 per subsequent turn** — $0.53 vs $2.05 across | ||
| 50 turns. | ||
|
|
There was a problem hiding this comment.
This "Measured:" claim (62k→4.4k tokens, $0.002 vs $0.031/turn, $0.53 vs $2.05 across 50 turns) has no supporting data anywhere in this diff or the repo — not in docs/AB-RESULTS.md, not in CHANGELOG.md. That's inconsistent with this PR's own stated principle ("this repo doesn't quote numbers it hasn't measured") and with how the rest of this same PR behaves: the "~5.7 delegations" figure later in the diff correctly traces back to the 0.22.0 changelog entry, but this one, and several others added here (lines 132, 135–136, 143–144, 232: the "+46%→indistinguishable" story, "0 delegations in 122 turns / 6 trials", "1.7–3.1× within an arm", "14.7% / 42.1%", "1.3–1.9× latency"), don't cite a source, version, or config the way the existing benchmark entries in CHANGELOG.md/SKILL.md do (e.g. "Opus 5 conductor · Gemini 3.6 Flash High executor · agy 1.1.8 · n=3/arm, cold cache"). If these came from the "colleague's SWE-bench" effort mentioned in the PR description, say so explicitly and note it wasn't run against this repo's tooling; if they're from an unpublished run of this repo's own harness, they should get the same citation treatment as the other numbers in this file.
|
Reviewed — this is docs-only ( Structural changes check out: the new principle 1 renumbers 1→5 correctly (the one downstream cross-reference at line 91, "per principle 5," was updated to match), the new §8 anchor and the One finding worth a look (left inline on lines 33–36): several new numbers in this diff are presented as "Measured:" facts but aren't backed by anything in the repo — not Everything else — the two-accounting-systems table, the report template, §8's framing of parity as a reportable result — reads consistently with the existing playbook's voice and doesn't contradict anything else in the repo. |
1 participant
Two independent efforts — ours on Harbor, Len's
on SWE-bench and SWE-bench Pro — spent days measuring the regime where delegation cannot
win, and both landed within ~10% of solo. Neither had a reason to ask first. That's a
defect in this playbook, not in either effort.
New principle 0.1 — pick the regime before measuring
Yes → expect parity; don't build a cost story on it. Claude re-reads handed-over context
rather than trusting it — the same rule that makes the hybrid safe (agy has been caught
patching its own environment to force a green test). You cannot have "the conductor doesn't
re-read" and "the conductor owns correctness" at once.
No → the digest is the deliverable, and that's where the saving lives: 62k-token corpus
offloaded leaves the conductor carrying 4.4k instead of 62k — $0.002 vs $0.031 per
subsequent turn.
This is also confirmed by prior art, which names our design as the failure mode: "if every
sub-agent bounces results back to the orchestrator for validation, you pay frontier model
costs at every step."
§5 rewritten: conditions as requirements, not advice
Each of these cost us a rerun. Getting one wrong doesn't add noise — it produces a
confident wrong number.
AGY_USAGE_LOG, not stderr —2>&1 | tail -Neats exactly that linethat's what the plugin does when merely installed (0 in 122 turns; 0 in 6 trials even with
break-even guidance every turn)
Headline metric → cost-of-pass
Cost per trial is gameable by failing cheaply, and it got us: our cheapest hybrid trial was
the one that hit the wall-clock cap and failed. Including it read 14.7%; excluding it,
42.1%. Neither number was wrong — the metric was.
Also added the two accounting systems side by side.
cache_readis an inner subset on theClaude side and a separate counter on agy's; merging them is the easiest way to be badly
wrong in either direction.
New §8 — when the answer is parity
The playbook only had a happy path, which is how someone spends a week proving nothing.
Parity is a result. Four things to present instead, ordered by how hard they are to argue
with: work the frontier model can't do at all, the context ceiling, recall via two
independent scans (a reviewer can't raise recall — it only sees what was reported), and
wall-clock if you fan out — noting it moves the wrong way for a single delegation.
Docs only — no plugin code, no version bump. 191 tests, validate passes.
Still to come: the GAIA run this playbook is written for. Deliberately not written up yet —
this repo doesn't quote numbers it hasn't measured.