Skip to content

verify.md

Latest commit

 

History

History
106 lines (88 loc) · 7.19 KB

verify.md

File metadata and controls

106 lines (88 loc) · 7.19 KB
Raw

The verification layer — confident-wrong detection

The live tool the experiment produced (its origin story is in history.md). The rift-verify / ue-verify MCP servers catch confident-wrong claims: statements that are in-domain but false. It is the same null-driven discipline as the rest of Aporia, pointed at code — and it abstains rather than bluff.

checker.py — the detector (deterministic + judge pipeline)

A modular, swappable-checker pipeline:

  • DeterministicChecker — parses the codebase into an API index (return types, existence, authority/purity specifiers, delegate arity) and checks claims against ground truth. 100%, zero false positives, no model, instant. Abstains on anything it can't parse.
  • JudgeChecker — fallback for prose/behavioural claims: retrieves the exact docstring + .cpp implementation and asks a local open LLM (qwen2.5:7b, Apache-2.0) — CONTRADICTED / CONSISTENT / UNSURE (abstain, don't bluff).
  • Pipeline runs checkers in order; an abstain routes to the next. Add/remove/swap = one line.
.venv\Scripts\python.exe checker.py "URiftInventoryComponent::AddItem returns bool"   # -> FLAG
.venv\Scripts\python.exe checker.py --eval          # deterministic only, instant
.venv\Scripts\python.exe checker.py --eval-full     # + judge on prose claims
# switch domains: $env:CARTRIDGE_DOMAIN="ue"; $env:CARTRIDGE_FILTER="UFUNCTION"; $env:CARTRIDGE_MAXFILES="40"

runtime_verify.py — the runtime loop (prose in, per-fact verdicts out)

qwen converts prose into typed facts (EXISTS / RETURN_TYPE / AUTHORITY / PURITY / DELEGATE_ARITY, each anchored Class::Member; BEHAVIOR for everything else). Structural facts are checked as a pure index lookup (no phrasing regex — the runtime smoke test proved the deterministic rules were phrasing-locked to the eval's own templates). BEHAVIOR facts route to the judge carrying their anchor, so symbol-precise evidence always attaches. Every anchor gets an implicit EXISTS check, so invented methods are caught regardless of how extraction typed them. Typed decomposition + anchoring cracked the cross-method polarity fake ("bind in ShutdownViewModel") that survived prompt-level fixes.

learn_null.py — the learning loop

Abstains are a work-list of identified gaps. The loop: detect (abstained BEHAVIOR facts in field_log.jsonl) → acquire (relaxed-owner evidence search: every doc/impl for the member under any owner) → verify (judge, trinary) → write back (learned_facts.jsonl), served by the judge as evidence from then on. Facts the evidence can't support stay parked — learning never forces a verdict. --reverify re-checks every active learned fact against the current corpus; changed verdicts are RETIRED (kept as history, never served stale).

2026-07-19 hardenings — freshness, confidence, call-graph

Three additions, patterns adopted from studying Graphify after the fact-553 staleness incident (the rift index silently aged out against osprey-AIE — worse, the corpus roots still pointed at the abandoned MagicReborn\Plugins tree):

  • Freshness (index_cache.py): header parses and .cpp implementation offsets are cached per file, content-keyed ((mtime, size) fast path, sha1 fallback) — rebuild cost scales with the diff, not the corpus. corpus.py now globs the canonical RiftSuite58AIE\Plugins\*-AIE clones (repos-as-source-of-truth, 2026-07-18). install_hooks.py drops post-commit / post-checkout hooks into every bird clone: they touch .corpus_dirty_rift and spawn a detached prewarm_cache.py; verify_server.py stats the marker per call and hot-refreshes — a commit lands in the verifier without a restart. Both servers also expose a refresh_index tool for uncommitted edits (and the UE cart, which has no git to hook).
  • Confidence (checker.py rubric): every verdict carries confidence + basis — extracted 1.0 (parsed-index fact), inferred <1.0 (corpus substring, call-graph hop, judge by evidence tier: impl 0.9 / doc 0.75 / retrieval-only 0.55), ambiguous 0.0 (abstain). A judge inference no longer reads with the same weight as a parsed signature.
  • Call-graph (CALLS: typed fact): implementation bodies are brace-match parsed (fixing a latent gap — the old 45-line window's ");" heuristic silently dropped any body whose first statement was a call) into a who-calls-whom + who-Broadcasts-what index. "X calls Y" / "X fires OnZ" verdicts: direct = pass extracted 0.9; one hop = pass inferred 0.7; absent = flag.

Verdict laws — paid for empirically, do not relitigate without new evidence

  • Ownership / location / usage claims are inherently null ("lives on X", "attached to", "owned by"): a spawnable component can live on ANY actor, so these are design decisions, not source facts. ABSTAIN always — never flag (it would false-flag legitimate design choices), never pass on a bare substring (the fabricated-anchor false-green, fixed 2026-07-01). Documented design intent is not a code constraint.
  • Shape the claims, don't pressure the judge. The quote-gate experiment (forcing reasoning before the verdict) regressed and was reverted — a local reproduction of the AbstentionBench finding. Typed decomposition + anchoring is what cracked polarity, not prompt pressure. (This same law is why the New Brain extractor enforces discourse structure in code, not the prompt.)

Results

  • Full pipeline 100% / 100% on 374 RiftSuite claims (5 structural types + prose fires_event).
  • Generalizes to the UE engine for structural claims (280 claims, 100%) — usable for Unreal now.
  • Honest caveats: the structural 100% is partly by-construction (checker + labels share the parser); the prose judge test is small (~14) — this codebase is prose-sparse.

Files

file role
checker.py the confident-wrong detector (deterministic + judge pipeline)
runtime_verify.py the runtime loop: prose → typed facts → index lookup / anchored judge
verify_server.py the live MCP server (rift-verify): verify_statement / learn_from_log / reverify_learned / describe
ue_verify_server.py the UE-domain variant (ue-verify)
learn_null.py the learning loop: abstains → acquire → verify → write back / retire
test_verify_loop.py MCP round-trip plumbing check (verify + learning tools)
field_log.jsonl, learned_facts.jsonl field-prose dataset (auto-accumulates) / learned-fact store
generate_claims.py claim generator: parses headers → typed real/fake claim pairs (+ domain switch)
claims.py original hand-written confident-wrong test set (validated by Aaron)
extract_claims.py LLM-based claim extraction (preliminary, generalization path)
embed.py, corpus.py Ollama batch embedder + on-disk cache / corpus ingest + domain switch
index_cache.py content-keyed parse cache: headers, impl bodies, call-graph (the freshness spine)
install_hooks.py, prewarm_cache.py bird-clone git hooks (dirty marker + detached prewarm)
freshness_calls_test.py test suite for cache freshness, the confidence rubric, and CALLS