[FEATURE]: differential conformance — run every implementation of a component over one corpus

[diegolopezrm]

Summary

A2UI has four implementations of the expression parser and four of DataModel — Dart, TypeScript, Python and Swift — each a close port of the others. Every one has its own unit tests, and every suite passes. None of them compares one implementation against another, so a port that drifts from its siblings does so silently, and stays that way until somebody notices by hand.

This proposes making that comparison a first-class tool in the repository: run every implementation of a component over one shared corpus, group them by what they answered, and treat the grouping itself as the signal.

Why: what the manual version already found

I ran this by hand over the last few days, on two components. It is not hypothetical:

run	corpus	disagreements
expression parser, Dart vs TypeScript	2,592 templates	408
DataModel, Dart vs TypeScript	1,392 operation programs	221
expression parser, all four implementations	the same 2,592 templates	141

Issues and PRs that came out of it: #2490, #2491, #2492 (an unreachable depth guard, in two languages); #2496 (four classes of parser divergence); #2498, #2499 (DataModel losing writes silently, and a falsy root being discarded); #2500 (non-ASCII keys unreachable from templates in three of four); #2497 (the first conformance suite the client implementations share).

The most useful property showed up as soon as the third and fourth implementations joined the run. The shape of the disagreement classifies it:

• dart+py+swift | ts — one implementation alone against three. ${1.2.3} returned NaN in TypeScript, an error in the rest. That is a bug, and the outlier names itself.
• dart+ts | py+swift — an even split. The empty template produces [""] in two and [] in two. That is nobody's bug; it is an unstated decision.
• dart | py+swift+ts — a lone implementation on a question the specification itself is ambiguous about (the null keyword, [BUG]: Dart and web_core expression parsers disagree on 408 of 2,592 templates #2496). Evidence for the discussion, not a verdict.

And the counter-example that keeps the tool honest: on non-ASCII identifiers (#2500) the vote is 3-1, and the majority is wrong — three implementations carry the same hand-ported ASCII range, Swift uses its language's Unicode-aware predicate and is the one that behaves correctly. The grouping says where to look. It never says who is right.

What it would be

A tools/ command with one adapter per language. The adapter contract is deliberately tiny — read cases from a file, write one normalised JSON result per line — so adding an implementation is a small program, not an integration. I have working versions of all four for the parser; the Swift one is 30 lines against the existing BasicCatalog product.

Three layers on top:

1. Corpora per component, generated from a seeded grammar plus hand-written edge cases, versioned in the repository so a run is reproducible.
2. The agreement matrix, grouping implementations per case and reporting by pattern rather than by pair. With N implementations there are N(N-1)/2 pairs; the grouping is what makes the report readable at four and still readable at six.
3. Export into conformance/. Cases where every implementation agrees become conformance cases automatically — the agreed behaviour, written down. Disagreements become a triage queue. This is the part that pays off long-term: the suites stop being hand-written and start being derived from what the implementations already agree on, plus explicit decisions on the rest.

Where it fits

conformance/ already exists, with suites consumed by the Python and Kotlin agent SDKs, and #2497 adds the first one run by the client implementations. This is the machine that fills it. It also gives the skill for implementing an SDK in a new language something concrete to check against on day one: a new port either matches the others or produces a list of the exact inputs where it does not.

What I am not proposing

• Not a CI gate, at least not at first. A divergence is not automatically a failure — half of the ones above are open questions. A nightly run producing a report is enough to start, and it can tighten once the corpora settle.
• Not a reference implementation. Deriving a fifth "correct" parser from the spec sounds appealing and would mostly encode whoever wrote it.
• Not the reactive layer yet. The binder is where the remaining complexity lives, but comparing signal graphs means fixing notification order and async behaviour first, and the noise would swamp the signal.

The honest risk

The bottleneck is triage, not code. Four implementations over a generous corpus produce hundreds of raw disagreements, and most are variations of the same handful of causes. Without the grouping — and without someone willing to read the report — this generates noise and gets ignored. That is the main reason the classification by pattern matters, and why I would start it as a report rather than a gate.

Suggested next step

I have the parser side working end to end across the four languages, so the smallest useful step is to bring the adapters and the corpus generator into tools/ behind a documented interface, on the parser alone, and see whether the report reads usefully to people who did not write it. If it does, the DataModel corpus is already written and Kotlin is the obvious fifth adapter.

Happy to send that as a PR, or to write it up as a design doc first if that suits the process better — glad to take direction on the shape before writing the code.

[Varun-S10]

Hi @diegolopezrm, thank you for your feedback. We appreciate you taking the time to share your thoughts with us.

Hi @gspencergoog, I would like to escalate this issue regarding the proposal for differential conformance testing across implementations. Could you please take a look?

[gspencergoog]

This is a good idea, and we're slowly building the conformance suite that will help with this. In the future, all of the implementations will be based on the corresponding "core" library for the language they are written in, and the core libraries will all pass the conformance suite. We plan core libraries for each of Python, TypeScript, Dart, Kotlin, and Swift. Some of this is already in progress as we implement the 1.0 spec for Python and TypeScript, which will then be ported to the others.

I'm not sure the tools/ directory is the right place for this, it seems more like part of the conformance suite, but that's an implementation detail.

[diegolopezrm]

Thanks — and you are right about the location. Inside the conformance suite is where it belongs: the corpus and the adapters are how you decide what a suite should contain, not a separate tool that happens to live nearby. I will put it under conformance/ rather than tools/, and I would rather you name the exact directory than pick one myself.

Your plan is also what makes the timing worth a comment. If every core library is going to pass one suite, the interesting question is what happens before a case exists for something. A suite catches an implementation that breaks a rule already written down. Running the implementations against each other catches the rules nobody wrote down yet — and porting the 1.0 spec from Python and TypeScript to Dart, Kotlin and Swift is precisely when those appear, because a port inherits its siblings' assumptions without inheriting their tests.

That is not hypothetical here. Running the four existing expression parsers over one corpus turned up, in a single pass:

• [BUG]: non-ASCII data model keys are unreachable from templates in Dart, web_core and Python #2500 — ${señor}, ${café/precio} and ${日本} parse in Swift and are errors in Dart, TypeScript and Python. Three implementations carry the same hand-ported ASCII range; Swift used its language's Unicode-aware predicate. Any data model key outside English ASCII is unreachable from a template on three of four platforms.
• The null keyword: Dart alone against the other three ([BUG]: Dart and web_core expression parsers disagree on 408 of 2,592 templates #2496), on a question the spec answers inconsistently — the implementation guide lists null as a literal, DynamicValue has no null branch.
• ${1.2.3}: TypeScript alone, returning NaN where the rest raise. Fixed in test(conformance): add a cross-language conformance suite for the expression parser #2497.

None of those would have been caught by a suite, because none of them had a case. The shape of the disagreement is what classifies it: alone against three usually means a bug in the one; an even split means a decision nobody has taken. It is not an oracle — on the Unicode row the majority is the side that is wrong — but it tells you where to look, and that is most of the work.

Concretely, and small enough to review: the corpus generator plus one adapter per language for the expression parser, under whatever path in conformance/ you prefer, on top of #2497. The adapter contract is deliberately trivial — read cases from a file, write one normalised JSON result per line — so a new core library becomes a short program rather than an integration. I have working adapters for the Dart, TypeScript, Python and Swift parsers already; Kotlin would be the fifth once its core exists.

Happy to send that, or to hold it until the 1.0 ports settle if you would rather not have a moving target underneath it.

[gspencergoog]

Yes, at the same time we are expanding the conformance suite. Probably better to wait until at least the Python and TypeScript 1.0 ports land in main.