delegation-and-review §1: a pinned model string does not pin behavior - #54
Merged
F-e-u-e-r merged 2 commits intoJul 22, 2026
Merged
Conversation
Two independent private incidents in one week, two vendors: a pre-registered one-day re-measurement of a CLI's unstated-edge guard rate (same --model flag, same prompt battery) flipped enough to force retraction of the prior day's published claim; a second vendor's endpoint inverted a reproduced infinite-loop failure into a fully guarded pass over twelve days behind unchanged model strings. §1 already says lineups are volatile; this says the MEASUREMENTS are too — a behavioral number about a hosted endpoint is a measurement of a day, not of the string. Date-stamp such numbers, re-run the probe when a routing/safety decision depends on one, treat undated claims as expired. Ships unprobed per the README covenant (nothing can fixture scheduled endpoint drift); Provenance records longitudinal re-measurement as the honest test.
…view PR F-e-u-e-r#54 round-1 gate (grok-4.5 high: FIX 1; gpt-5.6-sol max: FIX 1-6). All reproduced and addressed: - decision-time re-run made canonical; DAY framing replaced by not-a-durable-attribute ontology; fresh results INFORM routing and never replace S2's edge spec + proof gate; no measurement pins the next request (grok F1 / codex F1) - 'number' -> 'measurement' covering categorical properties (grok n2 / codex F2) - terminal re-verification line now includes recorded hosted-endpoint behavioral claims (codex F3) - Done defined (fresh probe cited w/ timestamp+config, or the unknown-property fallback: route as if unguarded + spec the edge); probe-unavailable disposition named; pos example added; neg example generalized to any prior measurement incl. this morning's (grok F1 / codex F4) - provenance: rates-not-restated claim qualified (intervals are contributor-reported shape); in-body interval phrasing loosened to match (codex F5) - probe debt re-characterized as the behavioral executor trap (stale dated result + changed same-slug probe), distinct from premise re-verification (codex F6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds one bullet to
delegation-and-review§1, directly extending the existing "Model lineups are volatile facts" sentence: a pinned model string does not pin behavior — a measured behavioral property of a hosted endpoint (an edge-guard rate, a failure signature, a latency class) is a measurement of a day, not of the string. Date-stamp such numbers where recorded; when a routing or safety decision depends on one, re-run the probe instead of reusing it; an undated behavioral claim about a hosted endpoint is expired on arrival.Why
§1 already covers lineup volatility (which models exist) and cross-model-review §1 covers identity discovery — neither covers the measurements themselves drifting behind an unchanged slug. Two independent private incidents in one week, two vendors: a pre-registered one-day re-measurement of a CLI's unstated-edge guard rate (same
--modelflag, same prompt battery) flipped enough to force retraction of the prior day's published claim; a second vendor's endpoint inverted a reproduced infinite-loop failure into a fully guarded pass over twelve days, model strings unchanged. The failure the rule targets is reusing a stale benchmark number to route edge-risky work.Covenant status
Both incidents contributor-reported (private benchmark harnesses, verifiable by the contributor, not linkable); cited as shape, numbers not restated. In-body
unprobedmarker; Provenance notes that scheduled endpoint drift cannot be fixtured — longitudinal re-measurement is the honest test, recorded as debt.Checks
python3 .github/checks.py: all checks passed🤖 Generated with Claude Code