GAUGE

A dataset of real per-task agent-coordination outcomes, and a surprising result.

_{Coordination shape clearly matters in aggregate. Yet which shape wins on a given task is unpredictable from standard features, so the lever is not prediction but calibrated abstention. Measured on real outcomes, reproducible from frozen data.}

GAUGE releases (task → per-shape outcome) labels nobody else has: all five coordination shapes (solve single-agent, fan out and vote, decompose, chain, orchestrator-workers) run on 159 hard reasoning and coding tasks across 3 model families, recording per task which shape actually won. The result that comes out of them is counterintuitive. Shape moves aggregate accuracy a lot (best-of-N voting beats single-agent by twelve points on code). But per instance, the winning shape is not recoverable: no standard task-difficulty feature, no strong text embedding, and no capable LLM reading the task beats the always-single majority floor. What survives is knowing when not to trust the guess: a calibrated, abstaining router lifts accuracy where the predictor itself is at chance.

This repository is the dataset plus the paper's reproducible artifact: every headline number falls out of data/frozen/ via one script. It is not a product and has no demo: the finding is a measured negative, and the honest thing is to ship the data and the evidence, not a dashboard.

The dataset

The core artifact is real (task → per-shape outcome) labels that, as far as a live same-venue search could find, don't exist elsewhere:

• 159 hard tasks, competition mathematics (AIME, integer-checked) and competitive programming (CodeContests, checked in a sandbox), each with a deterministic oracle, so a per-instance win-label ("did this shape solve it") is well-defined.
• × 3 model families × 5 coordination shapes = 2385 outcome cells (how each shape did on each task with each model: accuracy, latency, tokens, cost), from which 477 win-labels are derived, for every (task, model), which shape won by a speed objective (fastest solver) and a reliability objective. Not an aggregate leaderboard.
• Plus the long-context endurance probes behind the chunking case study, per-task difficulty features, and a zero-shot LLM's shape predictions.

// one outcome cell                                   // one derived win-label
{ "task_id":"aime0", "shape":"single", "model":"haiku",   { "speed_win":"single",
  "accuracy":1, "latency_proxy":788, "cost_usd":0.002 }     "reliability_win":"AMBIGUOUS" }  // shape, or TIE/AMBIGUOUS

So a row tells you how each shape performed on a task; the win-labels tell you which one to have picked. That per-instance granularity is the point: it's what lets you ask "is the winner predictable?", which an aggregate dataset can't answer.

It lives in data/frozen/ (small; the reproduction backbone) and is mirrored to HuggingFace Datasets for discovery and citation. Third-party task prompts (AIME, CodeContests) are not rehosted, they refetch from public HuggingFace datasets; the win-labels derived from them are frozen here. The map from each file to the number it backs is REPRODUCE.md. The paper itself is archived on Zenodo (DOI above).

▶ Reproduce every number in the paper, one command

uv run --extra experiments --with scikit-learn --with sentence-transformers \
  python scripts/selfcheck.py

Runs offline from data/frozen/ (no database, no API key) and prints each of the 11 load-bearing numbers next to its Expected (paper) value. That self-check is the demo: the claims prove themselves from the bytes. (The prompt-text rows fetch the public AIME/CodeContests sets from HuggingFace, free and keyless; offline they print a notice and skip.)

What the data says (measured, not claimed)

Every row reproduces from data/frozen/ via scripts/selfcheck.py; n and the held-out protocol are stated in the paper.

Question	Result	Reading
Shape matters in aggregate?	best-of-N voting beats single-agent by +12 points on code	yes, the regime is built to favor routing
...so can you predict the winner per task from features?	per-feature ROC-AUC 0.46–0.53	no, every feature sits on the chance line
...from a trained predictor, embeddings, or an LLM reading the task?	0.652 vs 0.646 floor; embeddings 0.646 / 0.635; zero-shot LLM predicts single for all 353 cells	no, nothing clears the always-single majority floor
Does calibrated abstention extract value anyway?	0.76 accuracy at a quarter coverage, 95% CI clear of the random-abstention null	yes, ranking which decisions to distrust is easier than naming the winner
Does that confidence transfer across model families?	ECE 0.0 → 0.13 under family shift	no, usable within a family, not across (the honest fragility)
Where does shape win decisively?	context-chunking holds long-context accuracy: single 0.18 vs chunk 0.82 at 16k	by design, not selection, and only on decomposable tasks
What bounds capturing that win?	oracle-deployable gap 0.92 vs 0.75 (widens to 22 pts on the weaker family)	the same calibration problem one level up: the model is over-confident about when its own lever applies

The through-line: across selection, design, and deployment, calibration is the binding constraint.

Honest caveats (the scope is the point)

• The negative is scoped. It is about these representations (hand-scored difficulty axes, strong text embeddings, a capable model reading the task) at these tiers (cheap-to-mid) on these tasks (verifiable reasoning and coding). We do not test the role, cost, and multi-round signals learned routers also use; a router that exploits those is the natural rejoinder.
• The abstention lift is modest, certified around a quarter of decisions, and does not transfer across model families. Deployable inside a family today; cross-family calibration is left open.
• The chunking study is one worked example, not a general claim about chunking; the gap's magnitude is model-specific.
• Labels are finite (159 tasks; n=30/cell for the chunking router). Every number ships with its n.

No demo, no product, no flashy number, because the result is a careful negative, and inflating it would contradict the data. If the honest result is less impressive, this ships the honest result and says so.

The method, and what is reused (accounted honestly)

GAUGE is a recombination, not an invention, and says so. It imports the per-instance algorithm-selection framing (Rice; SATzilla; MASIF) and the calibrated selective-prediction layer (Geifman & El-Yaniv; calibrated selective classification; UCCI's calibrated-abstaining cascade router), and retargets them from model-routing onto the agent-shape decision, then measures the per-instance signal instead of assuming it. src/gauge/calibration is reused machinery (Platt / isotonic / debiased-ECE / selective-abstain); the calibrated-abstaining-routing mechanism is UCCI's, for model cascades. GAUGE claims only the retargeting onto agent shape, the purpose-cut taxonomy, the dataset, and the measured result. Model-routing is not shape-routing, and this repo keeps that line.

Repository layout

src/gauge/            # pure-stdlib calibration core (Platt / isotonic / debiased-ECE / selective-abstain) + the router scaffold
experiments/path_b/   # the (task → per-shape outcome) harness: shapes, providers, the endurance/chunking probes, the eval
scripts/
  selfcheck.py        # reproduces all 11 headline numbers from data/frozen, each vs its Expected (paper) value
  make_paper_figures.py
data/frozen/           # the frozen per-shape outcome labels + endurance probes (the dataset)
REPRODUCE.md           # the number → script → frozen-input → expected map (the paper is on Zenodo)
tests/                 # 54 tests (calibration round-trips + the router smoke suite)

Built by

Benaja Soren Obounou Lekogo Nguia, AI Systems Engineer.

Cite

If you use the dataset or the method, please cite the Zenodo record, which archives the paper: DOI 10.5281/zenodo.21030368.

License

MIT. See LICENSE. The reused gauge.calibration package is attributed above; third-party benchmark prompts (AIME, CodeContests) are not rehosted.