config/tasks.yaml opens with:
# For tasks with dynamic answers, no hint is needed.
Three hints don't follow that rule, and as of today they've gone stale against openclaw/openclaw. The effect is that a re-run of published-results/ scores lower than the published table — not because the agent got worse, but because it now reports the correct current answer and the judge compares it to a March value.
Evidence
I re-ran the axi condition on 2026-07-26 — same harness, --agent claude --model claude-sonnet-4-6, 17 tasks x 5 reps, the same shape as the published run of 2026-03-21.
71/85 (83.5%) vs the published 85/85 (100%). Wilson 95% CI [74.2%, 89.9%]; Newcombe difference −16.5pp [−25.8, −8.7].
The run was clean — 85 records, no timeouts, no error_count > 0, no empty outputs, no zero-turn runs.
Failures are not spread out. 13 of 17 tasks still passed 5/5. All 14 failures sit in four tasks:
| task |
published |
re-run |
cause |
list_releases |
5/5 |
0/5 |
hint names tags that no longer exist |
list_labels |
5/5 |
2/5 |
hint says ~124; repo now has ~309 |
weekly_catchup |
5/5 |
0/5 |
hint still valid — see below |
list_open_issues |
5/5 |
4/5 |
single run, judge reasoning |
list_releases — the clearest case
grading_hint: "The 3 most recent tags are v2026.3.13-1, v2026.3.13-beta.1, and v2026.3.12."
Published run, 2 turns:
The 3 most recent tag names for openclaw/openclaw are: 1. v2026.3.13-1 2. v2026.3.13-beta.1 3. v2026.3.12
Re-run, 2 turns, same command:
The 3 most recent release tag names for openclaw/openclaw are: 1. v2026.7.2-beta.3 (2026-07-18) 2. v2026.7.2-beta.2 (2026-07-17) 3. v2026.7.2-beta.1 (2026-07-15)
Identical behaviour, correct answer, graded 0/5. The judge's own text says so:
The agent ran the correct command but the results do not match the known facts.
list_labels
"approximately 124 labels". The repo now has ~309. One re-run paginated properly and reported 309 — and passed, while the judge's rationale still cited "approximately 124 labels" as the correct count. So the hint is both stale and inconsistently applied.
repo_overview — latent
"~326k stars" has the same problem; it passed only because the judge reads ~ loosely. It will fail eventually.
What is not rot
weekly_catchup is a genuine behavioural difference and I want to be careful not to lump it in. Its hint is still accurate and it even anticipates the failure mode:
The open issue count should be in the thousands (e.g. ~8700). Reporting just the page size (e.g. 30) as the total count is incorrect.
Published: agent reported 8,771 in all 5 runs. Re-run: agent reported 30 in all 5 runs. That is a real 5/5 → 0/5 flip on a valid hint, not a stale-gold artifact.
Excluding every rot-attributable failure, the gap still holds: 69/75 vs 75/75, −8.0pp [−16.4, −1.5].
Suggested fix
The repo already contains the pattern. release_then_body pins the tag in the prompt:
List releases for openclaw/openclaw, find the release tagged v2026.3.13-1, then view its full release notes...
That makes the expected answer immutable, and it passed 5/5 in both runs — four months apart. Same for view_pr (#50782) and pr_review_prep, which reference a merged PR whose title and author can't change.
Options, roughly in order of effort:
- Rewrite the three hints to be process-based, the way
list_open_issues and bug_triage_search already are ("the count should be in the hundreds or thousands", "must run a command, not hallucinate"). Cheapest, and it matches the file's stated rule.
- Pin the fact in the prompt, following
release_then_body.
- Derive gold at grade time — have the grader fetch ground truth via
gh before judging. Most robust, most work.
Happy to send a PR for (1) if that's the direction you'd want.
Separately
Less concrete, and a methodology note rather than a bug: published-results/STUDY.md reports single-run point estimates with no interval. At n=85 per condition the AXI-vs-rest gaps do survive a Wilson/Newcombe treatment (+6.8pp to +27.1pp, all excluding zero), but every non-AXI pairwise comparison overlaps zero, and per-task cells are n=5 — a 2/5 cell spans [11.8%, 76.9%]. The weekly_catchup flip above is what that looks like in practice.
Also minor, related to #92: src/usage.ts has a single commit dated 2026-03-26, five days after the published run of 2026-03-21, so total_cost_usd in published-results/results.jsonl and in a fresh run aren't produced by the same code. Costing both from raw token counts at one price table gives +36%, where the stored figures differ by +73%. Token counts themselves (price-independent) are up 63% input / 76% output on the 13 tasks that passed 5/5 in both runs.
I have both result sets and the analysis scripts and am glad to share them, or to re-run anything that would be useful.
config/tasks.yamlopens with:Three hints don't follow that rule, and as of today they've gone stale against
openclaw/openclaw. The effect is that a re-run ofpublished-results/scores lower than the published table — not because the agent got worse, but because it now reports the correct current answer and the judge compares it to a March value.Evidence
I re-ran the
axicondition on 2026-07-26 — same harness,--agent claude --model claude-sonnet-4-6, 17 tasks x 5 reps, the same shape as the published run of 2026-03-21.71/85 (83.5%) vs the published 85/85 (100%). Wilson 95% CI [74.2%, 89.9%]; Newcombe difference −16.5pp [−25.8, −8.7].
The run was clean — 85 records, no timeouts, no
error_count > 0, no empty outputs, no zero-turn runs.Failures are not spread out. 13 of 17 tasks still passed 5/5. All 14 failures sit in four tasks:
list_releaseslist_labelsweekly_catchuplist_open_issueslist_releases— the clearest casePublished run, 2 turns:
Re-run, 2 turns, same command:
Identical behaviour, correct answer, graded 0/5. The judge's own text says so:
list_labels"approximately 124 labels". The repo now has ~309. One re-run paginated properly and reported 309 — and passed, while the judge's rationale still cited "approximately 124 labels" as the correct count. So the hint is both stale and inconsistently applied.repo_overview— latent"~326k stars"has the same problem; it passed only because the judge reads~loosely. It will fail eventually.What is not rot
weekly_catchupis a genuine behavioural difference and I want to be careful not to lump it in. Its hint is still accurate and it even anticipates the failure mode:Published: agent reported 8,771 in all 5 runs. Re-run: agent reported 30 in all 5 runs. That is a real 5/5 → 0/5 flip on a valid hint, not a stale-gold artifact.
Excluding every rot-attributable failure, the gap still holds: 69/75 vs 75/75, −8.0pp [−16.4, −1.5].
Suggested fix
The repo already contains the pattern.
release_then_bodypins the tag in the prompt:That makes the expected answer immutable, and it passed 5/5 in both runs — four months apart. Same for
view_pr(#50782) andpr_review_prep, which reference a merged PR whose title and author can't change.Options, roughly in order of effort:
list_open_issuesandbug_triage_searchalready are ("the count should be in the hundreds or thousands", "must run a command, not hallucinate"). Cheapest, and it matches the file's stated rule.release_then_body.ghbefore judging. Most robust, most work.Happy to send a PR for (1) if that's the direction you'd want.
Separately
Less concrete, and a methodology note rather than a bug:
published-results/STUDY.mdreports single-run point estimates with no interval. At n=85 per condition the AXI-vs-rest gaps do survive a Wilson/Newcombe treatment (+6.8pp to +27.1pp, all excluding zero), but every non-AXI pairwise comparison overlaps zero, and per-task cells are n=5 — a 2/5 cell spans [11.8%, 76.9%]. Theweekly_catchupflip above is what that looks like in practice.Also minor, related to #92:
src/usage.tshas a single commit dated 2026-03-26, five days after the published run of 2026-03-21, sototal_cost_usdinpublished-results/results.jsonland in a fresh run aren't produced by the same code. Costing both from raw token counts at one price table gives +36%, where the stored figures differ by +73%. Token counts themselves (price-independent) are up 63% input / 76% output on the 13 tasks that passed 5/5 in both runs.I have both result sets and the analysis scripts and am glad to share them, or to re-run anything that would be useful.