test(parity): gate complete-corpus strict capability and engine output - #681
Conversation
There was a problem hiding this comment.
Thank you for putting this together. The strict-mode audit and backend-parity direction are valuable + the careful evidence and the prerequisite hook fixes included here are appreciated. Ran both new targets locally: the happy-path gate passes all 10 Tier-1 fixtures, and the audit reproduces the reported 10/6/0 classification.
I found three issues to tighten up before merging, mainly so the gate provides the enforcement and strict-AOT guarantees it promises. I’ve left the details inline.
dc925e1 to
bf760a2
Compare
bf760a2 to
d502302
Compare
Implements the three-tier classification model for AOT-testable fixtures:
- Tier 1: both backends pass *ir-compile-strict* and agree (counts toward parity)
- Tier 2: bytecode fails strict (would trampoline; excluded from coverage)
- Tier 3: divergent (goes to per-backend allowlist as known miscompile)
Audit Results (test/gold-aot/ fixtures):
- 10 Tier-1 fixtures: arith, closure, destructure, quot, seq, setbang,
setvar_load_stability, setvar_load_straightline, tryfinally, var
- 6 Tier-2 fixtures: binding, closure_capture, ir_pipeline, math_float,
pipeline_dump_ir, typed_cross_fn (excluded from parity coverage)
- 0 Tier-3 fixtures (no current divergences)
Implementation:
- TestStrictModeAudit: classifies all 16 fixtures under strict mode
- TestParityGatePhase1: runs Tier-1 fixtures under both backends with
per-backend allowlists (test/parity-xfail-{bytecode,gogen}.txt)
- Shrink-only ratchet: NEW divergence fails regression test; allowlisted
fixture that now agrees fails as stale (improvement must be explicit)
- Make targets: parity-gate-phase1 (run gate), strict-audit (reclassify)
Verification: Green drill executed (stale allowlist detection proven);
gate passes with all 10 Tier-1 fixtures agreeing on both backends.
The pre-push hook environment (GIT_DIR, GIT_WORK_TREE, GIT_INDEX_FILE, GIT_PREFIX) leaks into git subprocess calls in the compat-suite overlay helpers (syncJankSubmodule, applyPatchOverlay, revertPatchOverlay). When these vars are set, git honors them over the -C repo-discovery path, causing submodule gitlink resolution to fail in worktree contexts. The lg-char-unicode-scalar overlay silently no-ops and the gate reds on TestJankSuiteCoversLetGoUnicodeScalar and TestClojureTestSuite/char tests during push, but only in worktrees. Filter GIT_DIR, GIT_WORK_TREE, GIT_INDEX_FILE, and GIT_PREFIX from the environment for these git calls, restoring path-based -C discovery. No behavior change outside hook context (Fixes issue nooga#679).
2be976d to
32abf99
Compare
nnunley
changed the title
mparrett
left a comment
mparrett
left a comment
There was a problem hiding this comment.
Thanks for the substantial follow-up here. I rechecked the current implementation and reran make gogen-diff: both mandatory legs pass, strict failures are now classified independently from allowlisted capability gaps, and the documented rederive path is connected and reproduces the committed ledger and baselines byte-for-byte. All three requested changes are addressed.
Approving with one non-blocking whitespace nit inline.
| # The target state is empty. Rederive only from a complete valid census. | ||
| # Partial and divergence are independent facts; a fixture may appear in both. | ||
| # LETGO_PARITY_REDERIVE=1 make engine-parity-gate | ||
|
|
There was a problem hiding this comment.
Non-blocking nit: git diff --check reports this as a new blank line at EOF. The rederive path reproduces it, so the durable cleanup would be in baselineBytes (avoid the second trailing newline when the generated baseline has no entries), rather than editing this generated file alone.
32abf99 to
4a374bc
Compare
2 participants
Backend miscompiles can pass human review when a differential run is optional, and a successful
gogen_irrun does not by itself show that strict IR lowering succeeded. This PR turns engine-output parity and strict-capability coverage into a complete-corpus, measured, committed contract.What changed
make engine-parity-gatediscovers everytest/gold-aot/*.lgfixture instead of maintaining hand-curated tiers.gogen_ircontrol, bytecode strict, andgogen_irstrict.ir-compile-strict:stays fatal.Reporting: what is and isn't working
The gate now prints a coverage report and writes it to
LETGO_PARITY_REPORTwhen set. Non-full fixtures are measured a second time with strict off, so the report names the exactdefnand the raw lowering reason behind every gap, plus a bucket census of distinct reasons and their sites. Full-parity fixtures need no extra run: strict success already proves zero fallback.Committed case ledger
test/parity-ledger.txtrecords one line per case, so a commit diff shows exactly what changed:ADDED,REMOVED, orSTATUS-CHANGEDcases;TOMBSTONEentry, and anUNEXPLAINEDtombstone fails;ADDED, so the tombstone must be removed deliberately;LETGO_PARITY_LEDGER_CLEANUP=1clears tombstones inherited from an earlier revision.Two shrink-only baselines still encode direction (partial capability, unattributed divergence). The ledger, both baselines, and any rederive are written as one staged, validated, rollback-safe set.
Measured result
pipeline_dump_ir.lg,defn type-display:ir/lower: junk-below mismatch for block 15 (0 vs 2); unsupported shapeThat single gap is one instance of the largest structural class on the bytecode lowering path, so the ratchet has a real target rather than an aspiration.
Scope
This proves strict capability plus normalized full-output agreement across the bytecode and
gogen_irengines. It does not claim that fixture source executed through generated Go or a native entry frame. Native-entry proof usinglg-compile --entry-frameand the scoped Go AST oracle is the immediate follow-up.Validation
GOFLAGS=-shortfalsifiermake gogen-diffap
Related: #660, #678
This branch still contains the identical hook-environment scrub from #680; it resolves as overlap whichever lands first. The earlier #676 overlap is no longer part of this PR diff.