Goal: benchmark-proven leadership in agent code intelligence

[aimasteracc]

North-star objective

Make tree-sitter-analyzer the benchmark-proven leader among repository-level code understanding, navigation, and change-impact systems used by AI agents.

"No.1" is a north star, not an acceptance result. The strongest permitted claim is bounded:

On benchmark vN, for the named tools and versions, frozen repositories/questions, identical model and environment, TSA is quality/reliability non-inferior and has a statistically meaningful advantage, reproduced to E4.

Local-first operation, privacy, model/client independence, and MCP/CLI support are evaluation dimensions—not rules used to exclude otherwise relevant competitors.

Non-compensating goal tree

• G0 — Evidence integrity: immutable manifest, exact matrix, tool/repo/environment fingerprints, isolated arms, append-only attempts, checksums, deterministic replay, E0-E4 claim ladder.
• G1 — Structural/semantic correctness: definition/reference/caller/callee/import precision and recall, same-name ambiguity, dynamic dispatch, cross-language false links, exact citations.
• G2 — Real agent task effect: answer correctness/completeness, supported citations, change-task success, hallucination and wrong-edit rate under a fixed agent/model.
• G3 — Efficiency: total/cache/reasoning tokens, turns, tool calls, billed cost, warm latency, logical-cold end-to-end latency, incremental-to-answer latency, RSS and index size.
• G4 — Reliability/freshness/security: success/timeout/crash rates, exact add/modify/delete/rename refresh, zero stale rows/edges, path boundary and malicious-repository safety.
• G5 — Time to first value: clean-machine installation success, time to first correct answer, diagnostics, uninstall/recovery, Windows/macOS/Linux parity.
• G6 — Scale/language effectiveness: 10K/100K/1M LOC curves, exact indexed/error partitions, per-language conformance and real task outcomes—not README plugin counts.
• G7 — Ecosystem/adoption/reproduction: current integrations, active installs/downloads, independent contributors, second-machine blind review, public evidence and third-party E4 reproduction.

No weighted score may hide regression on another axis.

Competitor policy

• Benchmark v1 control: native file tools.
• Benchmark v1 indexed competitor: CodeGraph at a frozen version.
• GitNexus, Serena free LSP backend, CodeGraphContext, and optave/codegraph enter only after capability-boundary, install, version, and fair-adapter validation.
• Competitor inclusion rules are frozen before results. A new competitor requires a new benchmark version.
• Full IDEs/agents and SaaS products may be evaluated separately only when the model and agent scaffold are fixed and only the code-intelligence backend changes.

Hard evidence gates

Use RFC-0021 thresholds for a competitive efficiency claim:

• TSA-minus-competitor quality 95% CI lower bound > -0.10.
• Deterministic citation-location validity >= 99%.
• Success rate >= 99%.
• Any repository quality drop > 0.25 rejects the claim.
• Cost or latency ratio 95% CI upper bound <= 0.80 in at least 4/5 categories and 5/7 repositories.
• Failures, invalid trials, retries, and unfavorable experiments remain disclosed.

Multi-agent separation of powers

Role	Authority	Forbidden
Proposer	hypothesis, user job, objective contract	implementation, threshold changes after results, victory claim
Implementer	approved product paths in an isolated branch/worktree	benchmark/oracle/holdout/evidence edits
Reviewer / Red Team	counterexamples, mutation/error injection, security and compatibility review	self-fixing, self-approval, lowering severity
Judge	clean-checkout frozen experiment and ACCEPT/REJECT/INVALID/NOT_EVALUATED verdict	code edits, selective reruns, gate waivers

Human maintainer retains objective, budget, protected-path, merge, release, secret, and claim authority. Agent votes cannot establish facts.

PR value gate

CI green is necessary but insufficient. A No.1 product PR must:

1. Link a frozen NO1-xxx objective.
2. Name a real user job and one primary external metric.
3. Include same-environment baseline/candidate evidence.
4. Meet the pre-registered minimum effect without crossing any quality/safety floor.
5. Stay inside approved paths.
6. Resolve independent review findings.
7. Receive a clean-checkout Judge ACCEPT.
8. State rollback triggers and evidence level.

Coverage, file length, project-health grade, test count, flag count, and tool count are not user value. Pure refactors do not count unless they remove a measured blocker and the associated user result improves.

Execution sequence

1. Freeze grade-driven implementation while preserving read-only reporting.
2. Complete NO1-001A: deterministic smoke integrity infrastructure using fixtures/dry-run only.
3. Merge and freeze benchmark SHA + manifest.
4. Run NO1-001B: independent three-cell Gin Smoke; it may establish only E1.
5. Run a frozen 3-question x 4-repeat micro-pilot, or Gin all-questions x 4 for the RFC Pilot.
6. Select one measured product gap by hard-floor violations first, then user impact x recurrence x competitor gap x verifiability.
7. Implement one product hypothesis; validate on an unseen holdout.
8. Warm confirmatory: 7 repositories x 21 questions x 3 arms x exactly 5 repeats = 315 cells.
9. E3 second-machine independent blind review.
10. E4 public evidence, at least two current indexed competitors, and third-party full reproduction.

Stop conditions

Stop and retain evidence when setup is not reproducible, a required arm is unavailable, results have already been seen before changing an oracle/threshold, an implementer crosses protected paths, the budget reaches 80% before Smoke validity, a quality/safety floor regresses, or retries/results are selectively hidden.

First objective

NO1-001A will be tracked in a separate Objective Contract issue.

[aimasteracc]

Execution has started under the adversarially revised plan.

• First frozen objective contract: NO1-001A: isolated Gin Smoke integrity infrastructure #1196 (NO1-001A)
• The grade-driven auto-sprint backlog Auto-Sprint: daily backlog (2026-08-04) #1191 is frozen in report-only mode; it remains open solely for behavior-bug review and cannot dispatch refactor work.
• NO1-001A is infrastructure qualification only: fixture/dry-run, no paid model execution, no winner, and no product-code edits.
• Implementation, red-team review, and clean-checkout judgment must be performed as separated roles. A valid Smoke remains E1 with dominance_allowed=false and winner=null.

[aimasteracc]

NO1 execution update: #1201 completed with a clean-checkout, byte-identical terminal INVALID verdict after model preflight and exactly one frozen Gin three-arm smoke. No winner or dominance claim is permitted (publishable=false, dominance_allowed=false, winner=null). PRs #1202, #1203, #1204, and #1205 merged the preflight and verifier hardening discovered by dogfooding.

The registered stop condition is active, so execution sequence step 5 (micro-pilot) is intentionally blocked. Advancing toward the north star now requires a new, separately pre-registered objective that removes the measured execution/policy blockers before freezing new protected inputs; the invalid experiment remains disclosed and will not be rerun or replaced.

[aimasteracc]

Automation update (2026-08-04): the measured NO1-001C execution/policy blockers have been removed under separately pre-registered #1216 and merged PR #1217 (a15a9185).

What is now mechanically enforced:

• local source searches for HTTP/Python symbols are no longer misclassified as network execution; actual network commands and multiline bypasses remain rejected;
• TSA and CodeGraph MCP configuration/executables are qualified without a model, before manifest creation, against each actual Gin arm checkout;
• new bundles require valid two-arm preflight evidence while immutable legacy evidence remains replayable;
• indexed trials are invalid if direct source discovery occurs before a successful MCP call.

All Codex review threads were resolved and final CI was fully green on Linux coverage/full-language, Linux 3.13, Windows, macOS, performance, Dogfood claims, build, test-quality, Codecov patch, MyPy/Ruff/Bandit, and GitFlow. NO1-001C was not changed or rerun, and no winner/dominance claim was made.

This parent remains open because its own acceptance boundary is not yet met: a new experiment must be separately pre-registered and executed, followed by E3 second-machine independent blind review and E4 public third-party reproduction with at least two current indexed competitors. Those independence requirements cannot truthfully be manufactured by the same automation actor. Closing this issue now would falsely assert benchmark-proven leadership.

[aimasteracc]

NO1-002B terminal update: preregistered #1218 completed, and blocker #1219 was fixed by merged PR #1220 (0d9d5e5). The exactly-once primary outcome remains INVALID; its immutable bundle validated and replayed byte-identically with external digest c8e118566e92a1c2b940c8aa033887e6a37379e9e6b187ba5d8dd1eacc0913c3. No retry, winner, dominance, or leadership claim is made. #1218 and #1219 are now closed. This parent remains open because E3 second-machine independent blind review and E4 public third-party reproduction with at least two current indexed competitors are still unmet and cannot truthfully be self-certified by this automation actor.

[aimasteracc]

Execution planning update: the next separately preregistered objective is #1221 (NO1-002C), a two-cell MCP-only indexed-tool canary addressing the measured NO1-002B validity blockers. It does not retry or reinterpret #1218, permits no winner/dominance/leadership claim, and no model-backed cell may run until the maintainer separately approves the exact budget, canary oracle, protected paths, and freeze. Current work is limited to model-free fixture/dry-run qualification.