Skip to content

conformance(chant): answer the whole-build fixtures in isolated mode (chant#2442, chant#2438) - #115

Merged
lex00 merged 2 commits into
mainfrom
chant-2442-isolated
Sep 12, 2026
Merged

lex00 merged 2 commits into
mainfrom
chant-2442-isolated

Conversation

@lex00

@lex00 lex00 commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Closes the last gap in chant's adapter. isolated is answerable, the three fixtures it was blocking now agree with the reference, and the pin moves to the release that made that possible.

Also satisfies chant#2438's done-when — "no skipped fixture except the round-trip family, and compareAdapters covers findings and hosted builds" — so that issue closes with this too, not just #2442.

Why isolated was unavailable

chant has no mode that means isolated. Its nearest setting, --sandbox (chant#1093), is a different rule: isolated refuses to invoke, so F-Call step 5 fires and step 6 never imports, while chant's sandbox refuses to import project code at all. Wiring them together traded one disagreement for another — the four negatives agreed and good.ts and named.ts broke, reference fold against chant run — so the adapter skipped visibly instead.

That turned out to be chant#2442 rather than a real difference between the two rules. A host's Composite was not recognised as the registration form, so nothing was interpreted and every fixture fell through to the invocation arm the sandbox refuses. chant-v0.72.1 recognises it, and returns the host's members as the factory built them instead of wrapping them in chant's own Composite, whose member validation asks for chant's Declarable and refused an entity carrying a different marker.

What changed

foldOnDisk passes sandbox: true under isolated, behind a probe. The probe is a behaviour test rather than a version comparison, matching acceptsHostPackages beside it: it folds a host-registered composite under sandbox, which answers only if the form is recognised and the host's own entity survives validation. A chant without the fix reports run, so an older pin skips visibly rather than reporting wrong verdicts. Verified by running the suite against 0.72.0, where it correctly goes unavailable.

Three fixtures move from skipped to answered, all agreeing on every verdict and every export:

F-IsolatedRefusal/a-project-factory-is-refused-not-invoked
S-FactoryBody/consts-then-a-final-return
S-FactoryParams/one-plainly-bound-parameter

Measured 3 → 0 skips across 56 project fixtures, taken both ways — the before figure came from checking out the previous adapter and re-running, not from assuming.

Two things beyond the plain restore

The skip allowance is retired. It let any fixture naming a host skip, first under chant#2408 and then chant#2442. Both are fixed, so it had nothing left to excuse, and the test now asserts zero skips. That closes a real hole: compareAdapters reports nothing when one side is "unavailable", so a silent return to unavailable read as agreement — the false signal that cost a day earlier in this work.

The hold-out list is split by reason. F-Call/a-host-factory-is-invoked was sitting inside foldsAtDepth, a list named for chant#2441's depth bug, so four entries sat under two unrelated reasons with a guard test that said three. Now foldsAtDepth (chant#2441) and openSpecQuestion (#110) are separate, heldOut is their union, and the guard runs per list. Each empties on its own event — a chant fix retires one, a J1 ruling the other — and neither can go on excusing the other's fixtures.

Pins and the corpus

Both pins move together, the devDependency and corpus.yml's CHANT_REF, because a corpus number is only reproducible if the engine under test and the corpus come from the same tag.

Rerun at chant-v0.72.1 (75c05827) in a worktree seeded per the usual procedure. The numbers do not move, which is the point of recording them:

Files Comparable Agreed Both fold No host Host call in value position
441 372 372 258 1 68

data-host: 441 files, 441 agreed, 142 both fold.

isolated becoming answerable adds three fixtures to the agreement suite and takes nothing from the corpus, so movement here would have been a regression rather than the fix landing.

Gates

tsc --noEmit clean; 37 passed / 10 skipped.

…(chant#2442)

`isolated` returned "unavailable" because chant had no honest way to
answer it. Its nearest setting, `--sandbox` (chant#1093), is not the same
rule: isolated refuses to INVOKE, so F-Call step 5 fires and step 6 never
imports, while chant's sandbox refuses to import project code at all.
Wiring them together traded one disagreement for another — the four
negatives agreed and `good.ts` and `named.ts` broke, reference `fold`
against chant `run` — so the adapter skipped visibly instead.

That was chant#2442 rather than a difference between the two rules. A
host's `Composite` was not recognised as the registration form, so nothing
was interpreted and every fixture fell to the invocation arm the sandbox
refuses. chant-v0.72.1 recognises it, and returns the host's members as the
factory built them rather than wrapping them in chant's own `Composite`,
whose member validation asks for chant's `Declarable` and refused an entity
carrying a different marker.

So `foldOnDisk` now passes `sandbox: true` under isolated, behind a probe.
The probe is a behaviour test rather than a version comparison, matching
`acceptsHostPackages` next to it: it folds a host-registered composite
under sandbox, which answers only if the form is recognised AND the host's
own entity survives validation. A chant without the fix reports `run` and
the mode stays unavailable, so an older pin skips visibly instead of
reporting wrong verdicts.

Three fixtures move from skipped to answered, and all three agree with the
reference on every verdict and every export:

  F-IsolatedRefusal/a-project-factory-is-refused-not-invoked
  S-FactoryBody/consts-then-a-final-return
  S-FactoryParams/one-plainly-bound-parameter

With nothing left unanswered, the skip allowance in the agreement test is
retired. It let any fixture naming a host skip, first for chant#2408 and
then for chant#2442; both are fixed, so the test now asserts zero skips
across all 56 project fixtures. A silent return to "unavailable" fails
there rather than reading as agreement, which is the failure that cost a
day when `compareAdapters` reported nothing for an unavailable side.
The release carrying chant#2442. Both pins move together — the
devDependency and corpus.yml's CHANT_REF — because a corpus number is only
reproducible if the engine under test and the corpus come from the same
tag, and moving one without the other measures a mixture of two.

The numbers do not move, which is the point of recording them:

  files 441, comparable 372, agreed 372, both fold 258
  no host 1, host call in value position 68
  data-host: 441 files, 441 agreed, 142 both fold

`isolated` becoming answerable adds three fixtures to the agreement suite
and takes nothing away from the corpus, so a change in these would have
been a regression rather than the fix landing.
@lex00
lex00 merged commit 22bdaf0 into main Sep 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant