feat(eval): MICODEX intent-layer eval corpus — session 09a · 58/58

[zkSoju]

Summary

Session 09a — MICODEX intent-layer eval corpus. Substrate proof for the no-fine-tuning claim from ~/vault/wiki/concepts/micodex-as-knowledge-map.md §6.

Result: 58/58 = 100% pass (one iteration round; baseline was 91.2% on first run).

Triangle composition per session 09a spec at ~/bonfire/grimoires/bonfire/specs/micodex-eval-corpus-2026-05-02.md: STAMETS forge LEAD + KEEPER source + OSTROM contract + KRANZ substrate-prep.

What ships

• 58-case eval corpus (43 canonical + 11 empty-expected refusals + 4 motif/concept seed cases)
• Headless harness (tests-smoke/eval-harness.ts) — CI-runnable; exits 0/1/2 per OSTROM contract; EVAL_GATE env tunable; layered failure-hint classification per spec §2.8 dual-layer-leak
• STAMETS canonical baseline generator (tests-smoke/forge-canonical.ts) — autonomous live-search forge across all 43 grails
• KRANZ act 1 substrate-prep runtime fix: @tobilu/qmd ^2.0.0 → ^2.1.0 (caught local install crash with SQLiteError: no such module: vec0; bumped local + lockfile to match working global)
• WITNESS QA checklist (grimoires/loa/qa/qa-cycle-micodex-09a-2026-05-02.md) — 7 surfaces · 3 shareable paths (MCP deeplink, CLI npm, Discord live trial) for dogfooding the substrate in real environments
• Doctrine (in operator's vault): micodex-as-knowledge-map.md §6 evidence tier promoted asserted → measured; NEW page synthetic-supervision-for-knowledge-maps.md crystallizes the reusable pattern

Substrate-truth findings (iteration round, per spec §4.6 Bucket A)

Initial baseline had 5 failures. Investigation showed substrate was right and spec-examples were aspirational:

• "underworld grail" → @g6805 Aquarius (aquarius.md explicitly: "Hades is the Greek god of the underworld") — NOT spec's @g4488 Satoshi-as-Hermes (psychopompia is also valid; V1.5 acceptable_top_3 candidate)
• "the dark grail" → @g6458 Fire (lexical "dark orange" in fire.md) — NOT spec's @g876 Black Hole (says "void presence" not "dark"; V1.5 substrate-iteration: extend black-hole.md anchors)
• "crypto" matched @g4488 Satoshi-as-Hermes legitimately — TRUE positive, lore mentions Bitcoin → confirms substrate richness
• 2 vec/HyDE quirks: "dragon"→Mongolian, "car"→Satanist (no lexical anchor; documented for V1.5)

Cases amended to substrate truth.

Test plan

• pnpm exec tsx tests-smoke/eval-harness.ts tests-smoke/eval-corpus.jsonl → exit 0, "OVERALL: 58/58 = 100.0%"
• pnpm exec tsx tests-smoke/forge-canonical.ts > /tmp/canonical-fresh.jsonl → 43 cases, 0 gaps (regenerates baseline cleanly)
• EVAL_GATE=0.95 pnpm exec tsx tests-smoke/eval-harness.ts tests-smoke/eval-corpus.jsonl → exit 0 (gate-tunable)
• WITNESS QA checklist S1-S7 captures landed at grimoires/loa/qa/captures/micodex-09a/ (post-merge dogfood)

V1.5 deferred

• Auto-eval on PR (CI integration of harness)
• Eval against Railway HTTP transport (gates session 09b integration QA)
• Cross-collection eval (codex-core-lore)
• acceptable_top_3 field (handles multi-valid-answer like "underworld")
• KEEPER real-source pass via WITNESS dogfood path (replaces operator-paired hunches per 2026-05-02 pivot)

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	May 2, 2026 11:54pm

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​tobilu/​qmd@​2.0.0 ⏵ 2.1.0	-1			-1

View full report

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6eb579170a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Return harness error code when qmd execution fails

The contract in tests-smoke/EVAL_CONTRACT.md says infrastructure/harness failures (e.g., qmd missing/crashing) must exit with code 2, but qmd errors are currently converted into normal case failures and the script always exits with only 0/1. This means transient or partial qmd failures can still produce an overall PASS when the gate is met, masking an invalid test run in CI instead of signaling a harness error.

Useful? React with 👍 / 👎.

[zkSoju]

Summary

Analytical review of #68. Enrichment pass was unavailable; findings are unenriched.

Findings

{
  "schema_version": 1,
  "findings": [
    {
      "id": "eval-harness-jsonl-comment-parse",
      "title": "JSONL comment-line filter is fragile",
      "severity": "LOW",
      "category": "robustness",
      "file": "tests-smoke/eval-harness.ts",
      "description": "The corpus parser filters lines with `!l.startsWith('//')` to permit comment lines, but JSONL is not specified to support comments. A line beginning with whitespace before `//` would still be passed to JSON.parse and crash the harness with a non-actionable error. Trim before filter, or drop comment support entirely.",
      "suggestion": "Use `.map(l => l.trim()).filter(l => l.length > 0 && !l.startsWith('//'))` — or remove comment support since the current corpus contains none.",
      "confidence": 0.7
    },
    {
      "id": "eval-harness-deferred-pass-counted",
      "title": "Deferred passing cases are excluded from totals",
      "severity": "LOW",
      "category": "correctness",
      "file": "tests-smoke/eval-harness.ts",
      "description": "Per-category accumulator increments `deferred` but never `pass`/`total` for deferred cases (lines ~138-145). The progress bar still prints `.` or `F` for deferred cases, but they are silently excluded from gate calculation. This matches the EVAL_CONTRACT spec ('exclude from gate calculation'), so it is intentional — but the report does not visibly indicate whether deferred cases passed or failed, only their count.",
      "suggestion": "Consider printing a separate '(deferred: N pass / M fail)' breakdown so deferred regressions are visible to humans even though they don't gate CI.",
      "confidence": 0.6
    },
    {
      "id": "eval-harness-exit-code-edge",
      "title": "Empty eval set with all-deferred corpus returns pass",
      "severity": "LOW",
      "category": "correctness",
      "file": "tests-smoke/eval-harness.ts",
      "description": "If every case is `deferred: true`, `totalEval === 0` and `passRate` defaults to 0, so `passRate >= GATE` is false (assuming GATE > 0) and harness exits 1 — but the failure message reports 0/0 which is confusing. Conversely if GATE is set to 0, an all-deferred corpus exits 0 with no real signal.",
      "suggestion": "Add explicit guard: if totalEval === 0, exit 2 with 'no non-deferred cases to gate against'.",
      "confidence": 0.6
    },
    {
      "id": "forge-canonical-min-score-floor",
      "title": "Floor calculation can produce min_score=0 silently",
      "severity": "LOW",
      "category": "robustness",
      "file": "tests-smoke/forge-canonical.ts",
      "description": "`Math.max(0, Math.floor((actualScore - 0.1) * 10) / 10)` produces a min_score of 0 whenever actualScore < 0.1, which would make any future score change pass the gate. Mijedi already gets min_score=0.4 (from actualScore likely ~0.5). Without a hard lower bound or warning, weak-substrate cases get baked into the corpus with no real assertion strength.",
      "suggestion": "Either require min_score >= some floor (e.g. 0.3) and emit a forge warning, or annotate the JSONL with a 'weak-anchor' marker for review.",
      "confidence": 0.55
    },
    {
      "id": "pnpm-lock-downgrade",
      "title": "Transitive deps downgraded (better-sqlite3, yaml, zod)",
      "severity": "MEDIUM",
      "category": "supply-chain",
      "file": "pnpm-lock.yaml",
      "description": "@tobilu/qmd 2.0.0 → 2.1.0 changes pinned transitives: better-sqlite3 12.9.0 → 12.8.0, yaml 2.8.4 → 2.8.3, zod 4.4.2 → 4.2.1. These are downgrades (lower minor/patch), which is unusual when bumping a primary dep. Likely qmd 2.1.0 added the tree-sitter optionalDependencies and pinned older versions for compat. Worth verifying no security advisories or bugfixes are lost.",
      "suggestion": "Confirm the downgrades are intentional (qmd 2.1.0 release notes) and check `pnpm audit` for any advisories on yaml@2.8.3 / zod@4.2.1.",
      "confidence": 0.65
    },
    {
      "id": "eval-corpus-duplicates-seed-files",
      "title": "Three corpus files duplicate identical content",
      "severity": "LOW",
      "category": "maintainability",
      "file": "tests-smoke/eval-corpus.jsonl",
      "description": "`eval-corpus.jsonl` is the concatenation of `canonical.jsonl` + `seed-empty-and-motif.jsonl` (per EVAL_CONTRACT.md forge instructions). Committing all three creates drift risk: any manual edit to `eval-corpus.jsonl` will be overwritten by the next `cat`-based regeneration, and any edit to a seed without regenerating leaves the corpus stale.",
      "suggestion": "Either commit only the seeds and generate `eval-corpus.jsonl` in CI, or commit only `eval-corpus.jsonl` and treat seeds as historical/forge-only artifacts. Document which is authoritative.",
      "confidence": 0.8
    },
    {
      "id": "harness-forge-shared-min-score",
      "title": "min_score derivation is hardcoded in two places",
      "severity": "LOW",
      "category": "maintainability",
      "file": "tests-smoke/forge-canonical.ts",
      "description": "The forge writes `min_score = floor((actualScore - 0.1) * 10) / 10` and the harness asserts `top.score >= c.min_score`. The 0.1 buffer is the only flake-tolerance budget for canonical cases. If the substrate's scores drift slightly (qmd version bump, embedding model change), all 42 canonical cases could fail simultaneously with no clear remediation path other than re-forging.",
      "suggestion": "Document the 0.1 buffer as a tunable in EVAL_CONTRACT.md, and consider adding a `pnpm eval:reforge` script that updates min_scores when intentional substrate changes occur.",
      "confidence": 0.5
    },
    {
      "id": "praise-failure-layer-classification",
      "title": "Failure-layer hints add real diagnostic value",
      "severity": "PRAISE",
      "category": "design",
      "file": "tests-smoke/eval-harness.ts",
      "description": "The `failureLayerHint()` taxonomy (INFRASTRUCTURE / SUBSTRATE-FP / SUBSTRATE-FN / INTERFACE-or-SUBSTRATE / THRESHOLD) gives operators an actionable starting point for triage rather than just 'test failed'. The contract document also explicitly maps each hint to a remediation bucket (case-wrong / substrate-thin / inherent), which is the kind of structure that prevents debug spirals."
    },
    {
      "id": "praise-keeper-source-provenance",
      "title": "keeper_source field captures provenance per case",
      "severity": "PRAISE",
      "category": "design",
      "file": "tests-smoke/eval-corpus.jsonl",
      "description": "Each corpus case carries a `keeper_source` describing why it exists, including notes like 'replaces \"car\" which substrate matched Satanist @0.88 (vec/HyDE quirk)'. This makes the corpus self-documenting and prevents the common eval-rot pattern where nobody remembers why a test exists or was changed."
    },
    {
      "id": "gate-history-baseline-low",
      "title": "EVAL_GATE default 0.85 vs current 100% gives wide regression band",
      "severity": "SPECULATION",
      "category": "policy",
      "file": "tests-smoke/EVAL_CONTRACT.md",
      "description": "Gate default is 0.85 but the current corpus passes at 100%. A regression to 86% (8 of 58 cases breaking silently) would still pass CI. Consider a tighter gate or a separate 'regression band' (alert if drop > 5pp from previous run).",
      "suggestion": "Set EVAL_GATE to a value closer to current pass rate (e.g. 0.95) or implement a delta-aware gate that compares against the last recorded pass rate in the gate-history table.",
      "confidence": 0.5
    }
  ]
}

Callouts

Enrichment unavailable for this review.