Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docs/src/content/docs/conformance/chant-cross-check.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ sidebar:
order: 4
---

import figures from "../../../data/figures.json";

`packages/conformance/src/chant-agreement.test.ts` runs the fixture set against
chant, and then runs both implementations against each other.

The pin is a real dependency. `@intentius/chant` is in this repository's
`devDependencies` at version `0.70.1`, and the adapter in
`devDependencies` at version <code>{figures.chantPin}</code>, and the adapter in
`packages/conformance/src/adapters/chant.ts` reaches it through its public
entry only: `fold`, `collectConsts`, `FoldError`, and, since `chant-v0.64.0`,
`findSubsetViolation` for the shape half. No source is vendored and no internal
Expand Down Expand Up @@ -51,9 +53,10 @@ is asserted rather than assumed.

| Pin | Fixtures | Rules with a fixture | Shape and fold agreement |
|---|---|---|---|
| chant 0.70.1 | 68, of which 30 are whole-build | 100 of 127 | all, on the 52 the pin can answer |
| chant <code>{figures.chantPin}</code> | {figures.fixtures}, of which {figures.wholeBuildFixtures} are whole-build | {figures.rulesWithFixture} of {figures.rulesTotal} | all, on every fixture the pin can answer |

Numbers taken from `paper/measurements.md`.
The counts are read from the fixture tree at build time; the agreement figure
is `paper/measurements.md`'s.

The same file reports one disagreement that existed between the two, on an
envelope inside a template span. It was triaged the way `spec/README.md`
Expand Down
44 changes: 28 additions & 16 deletions docs/src/content/docs/conformance/corpus.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
---
title: The corpus cross-check
description: Running chant's whole example corpus through both implementations, the four named limits, and what the numbers mean.
description: Running chant's whole example corpus through both implementations, the two named limits, and what the numbers mean.
sidebar:
order: 5
---

import figures from "../../../data/figures.json";

`packages/conformance/src/corpus.ts` runs chant's example corpus through both
implementations and compares verdicts file by file, and export namespaces where
both folded. It is issue #25.
Expand Down Expand Up @@ -53,19 +55,22 @@ gives the reason, and it is the same reason the port was removed from the
reference: a transcription would make the cross-check a test of the
transcription.

## The four limits
## The two limits

A file is comparable when nothing disarmed either implementation before the
comparison started. Four things do, and each is counted apart from the
comparison started. Two things do, and each is counted apart from the
agreement figure. `corpus.ts` defines them as a closed union and records which
side each one disarms, because a limit can only make its own side refuse more.

| Limit | Disarms | What it is |
|---|---|---|
| `composite` | the reference | The reference implements no composite factory form. |
| `host` | the reference | The reference has no bindings for a package it cannot load. |
| `lexicon-list` | chant | chant's whole-build entry takes no lexicon list, so `F-Host-Trust` arm 1 is off on its side, exactly as inventory row L9.4 requires for a build with no package list. |
| `build-params` | chant | chant's whole-build entry takes no build parameters, so a file reading one cannot fold on its side. |

Both are the reference's. chant is given each entry's lexicons and build
parameters through `foldProject`'s options, the inputs a real build has, since
chant#2422 (#96); before that two more limits stood for what its entry point
could not be told.

A limit spreads. `corpus.ts` propagates each one along two edges: `F-Import`,
because a file that imports an unfoldable module cannot resolve what it
Expand All @@ -88,21 +93,21 @@ From `paper/measurements.md`:

| Corpus | Files | Comparable | Agreed | Both fold, namespaces identical |
|---|---|---|---|---|
| chant-v0.70.1 at `730e7f7e`, 108 entries | 409 | 65 | 65 | 50 |
| chant <code>{figures.corpus.corpusVersion}</code> at <code>{figures.corpus.revision}</code>, {figures.corpus.entries} entries | {figures.corpus.files} | {figures.corpus.comparable} | {figures.corpus.agreed} | {figures.corpus.bothFold} |

And the limits, from the same file: 272 files reach a composite factory call, a
form the reference does not implement. 52 read a host data export as a value.
19 sit in entries that declare build parameters. 1 imports a package the host
cannot load.
And the limits, from the same file: {figures.corpus.noComposite} files reach a
composite factory call, a form the reference does not implement, and
{figures.corpus.noHost} imports a package the host cannot load.

`paper/measurements.md` states what this establishes: on 65 files nobody wrote
for the purpose, the two implementations agree on every verdict, and on the 50
`paper/measurements.md` states what this establishes: on
{figures.corpus.comparable} files nobody wrote for the purpose, the two
implementations agree on every verdict, and on the {figures.corpus.bothFold}
that fold on both sides the export namespaces are structurally identical,
entity class and properties included. It is the first agreement between the two
that was observed rather than designed.

It also states the limit, and the limit is the more useful half. 65 of 409 is
the comparable set, not the corpus, and every limit is an over-approximation,
It also states the limit, and the limit is the more useful half.
{figures.corpus.comparable} of {figures.corpus.files} is the comparable set, not the corpus, and every limit is an over-approximation,
so a file under one may also be hiding a disagreement. The composite limit
alone removes two thirds of the corpus, and the most common form in real
projects is therefore the one the cross-check cannot yet speak to.
Expand Down Expand Up @@ -148,8 +153,15 @@ without a host because an unrevived entity is a plain object. The defect is
issue #68: `F-Eval-Ident` step 1 read an instance as "J2 pre-built" that no
rule of J2 built, so the reference refused 22 files chant folds, and
`F-Prebuild` now states the rule. The artifact was 21 files on which the
harness had asked the two implementations different questions, now counted
under the lexicon-list limit.
harness had asked the two implementations different questions, counted under
a lexicon-list limit until chant#2422 retired it.

The second run had that limit and the build-parameters one gone; it exposed 79
more files and 10 of them disagreed. All ten were the harness or the
reference: build parameters bound to the module's live export, a `../`
specifier never joined to the importer's directory, and a capture decided over
the namespace alone where `F-Import` records it at the import.
`paper/measurements.md` lists them, and every one agrees now.

## In CI

Expand Down
25 changes: 15 additions & 10 deletions docs/src/content/docs/conformance/coverage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,19 @@ sidebar:
order: 3
---

import figures from "../../../data/figures.json";

One of the process rules in `spec/README.md` is that every identifier is
exercised by at least one fixture, and every fixture cites a real identifier,
in both directions, in CI. `spec/fixtures.test.ts` is that gate.

## The three assertions
## The four assertions

The test collects every `S-` and `F-` identifier defined across `grammar.md`,
`judgments.md`, `values.md`, `divergence.md` and `hosts.md`. It also collects
`judgments.md`, `values.md`, `divergence.md`, `hosts.md` and `rules.md`. It also
collects
every identifier cited by a fixture's `rules` array, and reads
`spec/fixtures/UNCOVERED.md` as an allowlist. Then it asserts three things.
`spec/fixtures/UNCOVERED.md` as an allowlist. Then it asserts four things.

Every rule a fixture cites is defined. A fixture citing an identifier that no
spec file introduces is a citation of nothing.
Expand All @@ -27,6 +30,9 @@ now has a fixture, or that names a rule that no longer exists, fails the gate.
That is what makes the list shrink-only: you cannot leave a stale exemption
behind.

The header's count is the one the gate computes, so a rule added without
touching the file is noticed there rather than on a downstream page.

The identifier collection is keyed on the identifiers themselves and never on
document structure. `spec/coverage.test.ts` carries the reason in a comment,
and it is the lesson of chant#2306, where a gate keyed on headings could not
Expand All @@ -50,14 +56,13 @@ rule identifier in backticks, an em dash, and a reason. `fixtures.test.ts`
parses exactly that shape with a regular expression, so an entry written any
other way is not an exemption and the gate will still fail on the rule.

The file's own header states the count. At the time of writing it says 100 of
127 rules have fixtures, and `paper/measurements.md` reports the same figure
for the chant `0.70.1` pin.
The file's own header states the count, {figures.rulesWithFixture} of
{figures.rulesTotal} rules with a fixture.

`paper/measurements.md` also groups the 27 rules that are still uncovered: five
wait on a form the reference implementation does not implement, the composite
factory and isolation mode; one waits on a specification decision, issue #71,
the call-depth bound; and the rest wait on the corpus growing, issue #24.
`paper/measurements.md` sorts the rest by what each waits on. Five need a
form the reference implementation lacks, the composite factory and isolation
mode. The six of `rules.md` need the harness half of the rules contract
(#101), and the remainder are waiting for the corpus to grow (#24).

A few entries are worth reading as a group, because they say something about
where the limits of a conformance suite are rather than where the gaps in the
Expand Down
4 changes: 2 additions & 2 deletions docs/src/content/docs/evidence.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ No figure on this site was typed by hand. Each one below is read from an artifac

| Figure | Value | Artifact | Limit |
|---|---|---|---|
| Specification version | {figures.specVersion} | `spec/VERSION`, tag `spec-{figures.specVersion}` | A version names a rule set, not a chant release |
| Specification version | {figures.specVersion} | `spec/VERSION`, tag <code>spec-{figures.specVersion}</code> | A version names a rule set, not a chant release |
| chant pin | {figures.chantPin} | root `package.json` | The engine under test, not necessarily the corpus's revision |
| Reference implementation | {figures.referenceVersion} | `packages/reference/package.json` | Same author as chant; not an independent implementation |
| Conformance suite | {figures.conformanceVersion} | `packages/conformance/package.json` | Carries the fixtures at the declared spec version |
| Rules with a fixture | {figures.rulesWithFixture} of {figures.rulesTotal} | `spec/fixtures/UNCOVERED.md` | The rest are listed with a reason each |
| Fixtures | {figures.fixtures}, of which {figures.wholeBuildFixtures} whole-build | `spec/fixtures/` | Cases somebody chose |
| Corpus | {figures.corpus.entries} projects, {figures.corpus.files} files | chant `{figures.corpus.corpusVersion}` at `{figures.corpus.revision}` | chant's own examples, not a sample of real-world source |
| Corpus | {figures.corpus.entries} projects, {figures.corpus.files} files | chant <code>{figures.corpus.corpusVersion}</code> at <code>{figures.corpus.revision}</code> | chant's own examples, not a sample of real-world source |
| Comparable files | {figures.corpus.comparable}, all agreed | `packages/conformance/corpus-report.md` | {figures.corpus.noComposite} need a form the reference lacks, {figures.corpus.noHost} loads no package |
| Both fold, identical namespaces | {figures.corpus.bothFold} | same | Entities compared as class plus properties, on one shared host |

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/for-your-platform/chant.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ chant {figures.chantPin} passes every fixture it can answer and agrees with the

| Over chant's example corpus | Count |
|---|---|
| Projects and files at `{figures.corpus.revision}` | {figures.corpus.entries} projects, {figures.corpus.files} files |
| Projects and files at <code>{figures.corpus.revision}</code> | {figures.corpus.entries} projects, {figures.corpus.files} files |
| Comparable, all agreed | {figures.corpus.comparable} |
| Both fold, namespaces identical | {figures.corpus.bothFold} |

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/for-your-platform/forgejo-warden.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ sidebar:

## Author

A `governance.ts` is typed by the package's own `GovernanceConfig`, with `satisfies` on `export const policy`; a default export does not fold ([#94](https://github.com/INTENTIUS/typescript-as-data/issues/94)), and a shared helper lives in a sibling file ([#95](https://github.com/INTENTIUS/typescript-as-data/issues/95)). The schema is 195 lines of hand-written interfaces covering eight cycles: org settings, membership and teams; repo settings, branch protection and repo baseline; secrets and variables, and webhooks. Everything the YAML parser could not do, flow style and multi-line scalars and anchors, is ordinary TypeScript: a template literal, a `const`, a spread.
A `governance.ts` is typed by the package's own `GovernanceConfig`, with `satisfies` on its `export default`, and its helper is an arrow in the same file; both forms fold under `data-host` since spec `1.2` ([#94](https://github.com/INTENTIUS/typescript-as-data/issues/94), [#95](https://github.com/INTENTIUS/typescript-as-data/issues/95)). The schema is 195 lines of hand-written interfaces covering eight cycles: org settings, membership and teams; repo settings, branch protection and repo baseline; secrets and variables, and webhooks. Everything the YAML parser could not do, flow style and multi-line scalars and anchors, is ordinary TypeScript: a template literal, a `const`, a spread.

## Check

Expand Down
6 changes: 3 additions & 3 deletions paper/measurements.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ The chant side's `false` is also an unsampled invariant: the run fails unless ev

| Pin | Fixtures | Rules with a fixture | Shape and fold agreement |
|---|---|---|---|
| chant `0.71.0` | 90, of which 36 are whole-build | 107 of 132 | all, on the 73 the pin can answer |
| chant `0.71.0` | 90, of which 36 are whole-build | 107 of 138 | all, on the 73 the pin can answer |

The 25 rules without a fixture are listed in `spec/fixtures/UNCOVERED.md` with a reason each; five wait on a form the reference does not implement (the composite factory and isolation mode) and the rest on the corpus growing (#24). One disagreement existed between the reference and chant, on an envelope inside a template span. The spec recorded the recommendation, chant-v0.68.0 implemented it, and a fixture now pins it (chant#2349).
The 31 rules without a fixture are listed in `spec/fixtures/UNCOVERED.md` with a reason each; five wait on a form the reference does not implement (the composite factory and isolation mode), the six of `rules.md` wait on the harness half of the rules contract (#101) and the rest on the corpus growing (#24). One disagreement existed between the reference and chant, on an envelope inside a template span. The spec recorded the recommendation, chant-v0.68.0 implemented it, and a fixture now pins it (chant#2349).

**Two profiles.** Spec `1.1` names `full` and `data-host` (F-Profile, F-Profile-DataHost); the second is the specification for an evaluator with no JavaScript runtime, and it is what a platform in another language implements. Fixtures carry profile tags, and 59 of the 90 belong to `data-host`, and the reference with an empty host passes every one of them. That is a JavaScript implementation passing a profile defined by the absence of JavaScript, so it establishes that the profile is consistent, not that it is implementable without an engine; the latter is #86's to establish.

Expand Down Expand Up @@ -135,7 +135,7 @@ The corpus is chant's own examples, and chant's documentation says the number is
## Limits

- Twelve mixed entries and one adversarial build are a small sample, all from one project.
- Fixture coverage is 107 of 132 rules. The 25 without one are listed with a reason, and the list may only shrink.
- Fixture coverage is 107 of 138 rules. The 31 without one are listed with a reason, and the list may only shrink.
- J3's whole-build fixtures reach both implementations where no host is involved, and one where a host is. Comparing verdicts alone would not be enough, since a seed and a taint casualty are both `run`; the tentative verdict and the taint edge are compared too.
- Revival is implemented for five of the six envelopes; `{__compositeStep}` needs a composite factory form the reference does not have (`packages/reference/CAVEATS.md`).
- The independent rewrite found two specification gaps. Two is a small sample, and it is the sample a single author working alone can produce.
7 changes: 6 additions & 1 deletion scripts/lint-docs-sentences.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,11 @@ function docFiles(dir) {
}

/** The frontmatter block replaced by spaces, so every offset after it is unchanged. */
// An MDX expression such as `{figures.corpus.files}` is a value, not prose; the
// dots inside it would end sentences. It is read as the number it renders.
function blankExpressions(text) {
return text.replace(/\{[^{}\n]*\}/g, "0");
}
function blankFrontmatter(text) {
const m = /^---\r?\n[\s\S]*?\r?\n---[ \t]*(\r?\n|$)/.exec(text);
return m ? m[0].replace(/[^\n]/g, " ") + text.slice(m[0].length) : text;
Expand All @@ -85,7 +90,7 @@ for (const file of files) {
// tables and inline code but leaves the `---` block, whose delimiters read
// as em dashes and whose `key: value` lines read as colon nameplates; every
// page was paying for its own frontmatter. Blank it, offsets preserved.
const prose = extractProse(blankFrontmatter(text));
const prose = extractProse(blankExpressions(blankFrontmatter(text)));
const doc = buildDocAnalysis(prose);
const { findings, errors } = runRules(RULES, doc);
for (const e of errors) detail.push(`${rel}: rule ${e.ruleId} errored: ${e.message}`);
Expand Down
5 changes: 5 additions & 0 deletions spec/fixtures.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ describe("fixture coverage gate (#8)", () => {
const bad = [...cited].filter((c) => !defined.has(c));
expect(bad, `fixtures cite undefined rules: ${bad.join(", ")}`).toEqual([]);
});
test("the header states the count the gate computes", () => {
const header = /^(\d+) of (\d+) rules have fixtures\./m.exec(readFileSync(join(specDir, "fixtures", "UNCOVERED.md"), "utf8"));
const covered = [...defined].filter((d) => cited.has(d)).length;
expect(header && `${header[1]} of ${header[2]}`, "UNCOVERED.md's header count").toBe(`${covered} of ${defined.size}`);
});
test("every defined rule is exercised by a fixture or deliberately listed as uncovered", () => {
const missing = [...defined].filter((d) => !cited.has(d) && !allow.has(d)).sort();
expect(missing, `rules with no fixture and no UNCOVERED entry:\n${missing.join("\n")}`).toEqual([]);
Expand Down
2 changes: 1 addition & 1 deletion spec/fixtures/UNCOVERED.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Deliberate, and this list may only shrink (#8). Each entry names the rule and why it is still uncovered. Remove an entry when its fixture lands; the gate fails if an entry is stale.

107 of 132 rules have fixtures. Most of the rules below are uncovered because the corpus is still growing (#24); where a rule needs something the reference implementation does not have, the entry says so.
107 of 138 rules have fixtures. Most of the rules below are uncovered because the corpus is still growing (#24); where a rule needs something the reference implementation does not have, the entry says so.

- `F-Rule-Input` — the harness has no rule hook and no findings-as-data fixture kind yet (#101)
- `F-Rule-Phase` — the harness has no rule hook and no findings-as-data fixture kind yet (#101)
Expand Down