NO1-002C: exact-model indexed-tool canary qualification

[aimasteracc]

Parent: #1195
Follows: terminal NO1-002B / #1218 (INVALID)

Objective

Prove, before exposing another benchmark question or spending a three-arm Smoke, that the exact agent/model can invoke each required indexed backend through its declared MCP surface, receive repository-grounded evidence, and leave the frozen source checkout unchanged.

This is infrastructure/execution qualification only, at most E0. It cannot produce an E1 competitive Smoke, winner, dominance, product-performance, E3, E4, or leadership claim. It does not replace, retry, or reinterpret NO1-002B.

User job and primary metric

A benchmark operator must be able to qualify both indexed MCP arms before freezing a competitive experiment.

Primary metric: qualified_indexed_cells / expected_indexed_cells == 2 / 2.

A qualified cell must have:

1. at least one successful call to the declared MCP server/tool;
2. an exact frozen canary path, symbol identity, and symbol kind in the response;
3. no source discovery before that successful MCP call;
4. zero undeclared shell commands and zero index commands outside MCP;
5. unchanged source-checkout fingerprint and path set;
6. index/cache writes confined to the declared external per-arm namespace;
7. retained transcript, MCP identity/receipt, runtime audit, hashes, and terminal status.

Partial success is terminal failure.

Frozen matrix

Exactly two isolated, single-attempt cells in seeded order:

• tsa-warm-canary
• codegraph-warm-canary

Both are MCP-only. There is no native arm. CLI fallback is forbidden because it was a measured NO1-002B blocker.

Suggested neutral canary, subject to maintainer approval before freeze:

Using the configured indexed MCP backend, locate the definition of Gin's Engine.ServeHTTP. Return only its repository-relative path, symbol identity, and symbol kind. You must call the indexed MCP backend before any source read or search.

Freeze exact benchmark SHA, Codex CLI/model identity, Gin SHA and clean-tree fingerprint, canary prompt/oracle, MCP versions/fingerprints, timeout, seed, policies, namespaces, manifest hash, and append-only registry entry before either cell starts.

Acceptance

• This objective and its allowed/protected paths are approved before model preflight or execution.
• Fixture/dry-run tests prove MCP receipt binding, policy classification, namespace isolation, runtime mutation detection, and immutable evidence retention.
• Exact model, tools, repository, canary and two cells are manifest-bound.
• Both cells execute exactly once and the primary metric is exactly 2/2.
• Both transcripts prove successful declared MCP use before source discovery.
• No CLI substitution, undeclared shell, oracle access, or cross-arm access occurs.
• Both source checkouts remain byte-identical before/after.
• Bundle validates from an external digest and replays byte-identically.
• Output remains publishable=false, dominance_allowed=false, winner=null.
• A separate reviewer attempts to falsify transcript, identity, and mutation binding.
• A clean-checkout Judge records ACCEPT/REJECT/INVALID/NOT_EVALUATED.

Only ACCEPT may qualify the path for a separately preregistered future Smoke.

Stop conditions

Stop, retain all evidence, and do not selectively rerun if a required MCP surface/fingerprint is unavailable; MCP success cannot be distinguished from telemetry loss; source discovery precedes MCP; checkout or namespace boundaries change; frozen identity/policy differs; an oracle is exposed; thresholds change after a model cell starts; an undeclared retry or mixed artifact appears; or implementation would weaken an evidence gate.

Protected paths

Implementers must not edit existing questions, repos, prompts used by NO1-002B, oracles, results, experiment manifests/registry history, receipts, digests, transcripts, bundles, RFC-0021 thresholds, claim ladder, E3/E4 requirements, or TSA product code.

Model-free fixture/dry-run hardening may touch existing benchmark execution/preflight/policy/runtime-audit code and the existing benchmark harness test file. Any already-observed prompt/policy change requires a new benchmark version.

Authority and budget

Human maintainer approval is still required for the exact model budget, canary oracle, protected-path scope, freeze, execution, merge, and any public claim. Until that approval is recorded, no model-backed canary cell may run.

[aimasteracc]

Maintainer authorization recorded via delegated agent authority (2026-08-05): approve the proposed Engine.ServeHTTP canary oracle, the listed model-free implementation/protected paths, exact two-cell single-attempt matrix, and a hard total model-call budget ceiling of US$3. The agent may act as maintainer proxy for planning, implementation coordination, verification, and execution within this contract. This does not waive role separation, immutable evidence, stop conditions, E3/E4 independence, or public-claim restrictions. Model-backed execution remains gated on model-free fixture tests, independent red-team review, and clean-checkout Judge ACCEPT.

[aimasteracc]

Pre-execution contract correction (no model cell or benchmark result has run): pinned CodeGraph 1.5.0 only supports CODEGRAPH_DIR as a single directory name inside the project root; it explicitly rejects absolute paths and path separators. Therefore the earlier requirement that every index/cache namespace be external to the checkout is not implementable for this named competitor. Replace it, before freeze, with: each arm uses an exact manifest-bound, isolated tool-native runtime namespace (.ast-cache or .codegraph) that may live inside its isolated checkout; every source/tracked file and every non-declared path must remain byte-identical; runtime index content receives a separate before/after semantic/hash audit and exact cleanup check. Namespace symlink/traversal/cross-arm overlap remains forbidden. This correction does not weaken source immutability, receipt validity, retry policy, or claim restrictions, and is recorded before any model exposure.

[aimasteracc]

Model-free qualification infrastructure is now in draft PR #1222. Independent Judge v5 returned ACCEPT strictly for the fail-closed scaffold after five adversarial review rounds. The implementation intentionally has no production execution path: production/default callbacks are zero, fixture results are never publishable/ACCEPT, and complete 2/2 evidence remains NOT_EVALUATED with PRODUCTION_TRUST_ANCHOR_UNAVAILABLE. No model calls were made and no budget was spent. Next blocker: independently trusted production runner + external manifest/baseline trust anchor, followed by a fresh clean-checkout Judge before any protected canary.

[aimasteracc]

Follow-up production trust work is now explicitly tracked in #1223. Read-only runner, evidence, and workspace red teams agree current Codex CLI cannot prove a provider-side aggregate spend cap; production remains zero-callback and NOT_EVALUATED. #1223 defines the external anchor, budget attestor, immutable collector, TOCTOU boundaries, and adversarial stop conditions required before any model call.

[aimasteracc]

PR #1222 merged into develop as 69b5c74 after all CI axes passed, including Windows, macOS, Ubuntu coverage/full-language, Codecov patch, Quality Gate, Build Verification, Dogfood and GitFlow. NO1-002C remains open because the merged scaffold intentionally returns NOT_EVALUATED without an external trust anchor. Production work is tracked in #1223; no model call or budget spend occurred.

[aimasteracc]

Dependency update: #1226 merged the fail-closed NO1-002D repository boundary as 73436b12, including all Codex hardening and full CI/Codecov. The exact-model canary remains NOT_EVALUATED with zero production callbacks because #1223 still lacks independent Anchor Custodian, provider Budget Gateway receipts, immutable Evidence Collector, and clean-checkout Judge inputs. No model call or budget spend occurred. This issue remains open; running the canary now would violate its registered stop conditions.

[aimasteracc]

Status Update: NOT_EVALUATED (2026-08-07)

Current situation: The canary file is located in benchmarks/codegraph_compare/ and has not been promoted to the main package.

Blocker: Promotion of the canary file to the main package requires external owner approval. No external owner has been identified or has approved this change. This is a hard prerequisite that cannot be resolved by code changes alone.

Status decision: This issue remains NOT_EVALUATED. It will not receive a WON'T-FIX label because:

• The issue is not decided to be rejected; the prerequisite is an external approval gate.
• Labeling it WON'T-FIX would incorrectly signal a permanent rejection decision, conflicting with the issue state machine.

No code changes are planned for this issue until external owner approval is on record.

[aimasteracc]

Dependency #1223 Now Resolved (2026-08-07)

PR #1231 implements all three trust roles required by issue #1223:

• Anchor Custodian (HMAC signing, production_anchor.py)
• Evidence Collector (append-only filesystem, production_collector.py)
• Judge Record (human verdict, production_judge.py)
• qualify_production_trust_v2() that enables model_callbacks_allowed=True

Budget Gateway limitation is documented: Codex CLI lacks provider-side spend reservation; client-side process kill is used instead.

Once PR #1231 merges into develop, the trust infrastructure required for NO1-002C canary execution is in place. The remaining step is the actual model execution: set CANARY_ANCHOR_KEY, prepare a ProductionRunSpecV1, call qualify_production_trust_v2() with a signed attestation and ACCEPT judge record, then invoke the canary runner.

Maintainer approval for the canary oracle (Engine.ServeHTTP, US\ ceiling) was already recorded on 2026-08-05.