[WIP] Try out the new math-expressions

[siefkenj]

This PR is a first step in using the rust-based math-expressions. Currently some tests will fail. There are two levels of integration.

Level 1: use math-expressions compat layer as a drop-in replacement. This will negatively impact build sizes, but should minimize code changes.

Level 2: Pull in the math-expressions directly as a rust dependency and expose via core. This would enable the Rust core to directly call math expressions, as well as eliminate a second WASM bundle.

This is a Level 1 trial to get some of the bugs ironed out. It currently has a pinned submodule, but when we know what we need, this should change to a published package/crate.

Below is an AI summary of some of the issues that need to be resolved

---

[WIP] Switch to the Rust/WASM math-expressions engine

Replaces the legacy math-expressions@2.x JavaScript library with the Rust core compiled to WASM,
via the math-expressions-js-compat drop-in. This branch switches permanently — Rust is the
default and the only supported configuration.

Suite: 3469 tests, 2965 passed, 465 failed (85.5%), up from 969 failures (70.9%) on the previous
submodule pin.

⚠️ WIP. 465 failures remain. Most are attributable to four specific upstream gaps (listed
below, filed in MATH_EXPRESSIONS_UPSTREAM_REQUESTS.md); the rest are ours. Do not merge until
those land or we decide to absorb them.

---

How the numbers were verified

Every step was checked test-by-test against the previous run — fixed / broken / gone / new —
not by comparing totals. A change that fixes 70 tests and breaks 3 looks like "+67" in a summary and
like a bug in this output. Both DoenetML-side changes came in at 0 broken.

Step	Failures	Verified
pin 8ccd98d (previous)	969	—
pin cdc5343, nothing else changed	577	—
+ test expectations updated for unpadded delimiters	535	42 fixed, 0 broken
+ fromAst(Expression) unwrap restored	465	70 fixed, 0 broken

Two new tools do the checking, both added here:

• scripts/compare-test-runs.py --before <a>... --after <b>... — diffs two Vitest runs by
  <spec> :: <test title>, printing regressions in full.
• scripts/analyze-math-divergences.py <report>... — buckets failures by the mathematics that
  differs (ordering, exact-vs-decimal, notation, precision) rather than by which spec they live in.

The full suite exhausts the heap in one process, so runs are sharded (vitest --shard=i/8). This is
a pre-existing DoenetML memory problem, not an engine difference — the JavaScript engine OOMs
identically.

---

What changed

Engine selection

packages/math/src/engine.ts now re-exports engine-rust. DOENET_MATH_ENGINE=js rebuilds against
the legacy library — kept deliberately, not as a supported mode but as a differential-debugging
tool: when a spec disagrees, the first question is always "does this pass on the old one?", and
rebuilding one package to answer it beats bisecting a behavioral difference by hand.

Building now requires a Rust toolchain (wasm32-unknown-unknown + a matching wasm-bindgen-cli).

engine-rust.ts is nearly a straight re-export

It previously carried three local gap fills. Upstream absorbed all three, and each was deleted
here to verify the upstream fix actually covered our usage — that deletion is the point of the
seam. A local patch that cannot be removed is a fix that did not land.

Gone: Expression#f(), the context-level operation family (me.simplify(expr) alongside
expr.simplify()), and the replacer that kept fromAst from losing NaN/±Infinity to
JSON.stringify.

One fill remains — see below.

WASM loading no longer needs a bundler rule

Upstream added a setWasmModule injection point, replacing the Vite plugin that used to alias
compat's node-only lib/_wasm.ts to our loader. A build-tool rule no longer stands between the
library and its own loader.

One ordering constraint, documented in both files it affects. Compat's Context object literal
contains _assumptionsHandle: new wasm.Assumptions(), which touches the WASM while the barrel's
module body is evaluating. So importing setWasmModule from the package root forces that body to
run first and the injection always loses to compat's node fallback — silently. We import from
lib/_wasm (a leaf that imports nothing but a type) and order engine-rust.ts's imports so the
loader is evaluated first.

dopri crosses the seam

Legacy exposed numeric.js's integrator as me.math.dopri; the Rust engine has no me.math entry
for it and provides its own solve_ode-backed equivalent. Both engines now export dopri from
@doenet/math under one name, so ODESystem.js and packages/utils/src/components/function.ts
stay engine-agnostic. me.math is otherwise complete on both engines.

Test expectations: unpadded container delimiters

316 string literals across 27 files, via scripts/unpad-container-delimiters.py:

( 0, 0 )  →  (0, 0)        [ -1, 6 ]  →  [-1, 6]        [ [ 1, 2 ] ]  →  [[1, 2]]

Two traps worth recording for anyone doing this elsewhere:

1. toLatex is byte-identical on both engines — \left( 1, 2 \right) keeps its padding. Only
   the text renderer changed. A naive whitespace sweep over test files corrupts LaTeX assertions.
   The codemod skips any literal containing a backslash.
2. Padding only counts when it separates the delimiter from real content. My first pass
   collapsed "...from the group: (, )." — literal prose parentheses around two blank references —
   and broke a test. The tightened rule also correctly preserves partially-blank tuples like
   (2, ) and a deliberate a^{ }.

The codemod is scoped to quoted-string contents; template literals are excluded because they carry
multi-line DoenetML source where the "padding" is indentation.

---

The one fix worth reviewing carefully

fromAst must unwrap an Expression — 95 hard errors, the single largest fix in this PR, and
not a spacing issue.

Legacy accepted an Expression anywhere a tree was expected:

const e = me.fromText("3");
me.fromAst(e).tree    // legacy: 3     compat: throws

Compat serializes the object as-is, so it reaches Rust as a bare JSON object with no $ key.

The error message is actively misleading. It reads unknown special None — but that None is
the Option<&str> from value.get("$").and_then(Value::as_str), not the {"$":"None"} special
upstream just added. {"$":"None"} works correctly. Worth knowing before anyone else chases it.

I assumed this was test sloppiness and checked before fixing it. It isn't: our source depends on
it in PiecewiseFunction.js, StateVariableEvaluator.ts and Dependency.ts. A math-valued state
variable holds an Expression, and code that re-wraps one hands it straight back to fromAst.
With ~675 fromAst call sites, finding them all by inspection is not realistic.

Restored in engine-rust.ts, recursively (an Expression can also sit inside a tree under
construction, e.g. ["+", someExpr, 2]). Filed upstream as §1 — astReplacer already visits every
node during JSON.stringify, so it can do this for free where ours costs a second traversal.

---

What still fails, and whose it is

Upstream (filed in MATH_EXPRESSIONS_UPSTREAM_REQUESTS.md)

Failures	Issue
~109	simplify() folds no numeric-function applications. floor(55.33), ceil(2.1), abs(-3), sum, prod, mean, variance, std, count, max, min, median, log all stay as ["apply", …]. Legacy folded every one. Note evaluate_to_constant() handles floor/ceil/abs/log identically on both engines — the capability exists, it just isn't reached from simplify. Student-visible: <math simplify>sum(3,17,5-4)</math> renders the unevaluated application.
—	0/0 simplifies to 0; must be NaN. 1/0 → Inf is correct on both. This is how DoenetML computes an undefined slope, so a degenerate line reports slope 0 — a wrong number rather than a visible failure, which is the worst shape for a grading path.
22	evaluate_numbers({skip_ordering:true}) throws. Backs simplify="numberspreserveorder". Throwing beats silently reordering (1+x+2 → x+3 was the bug it replaced), but the feature has no implementation.
19	Symbolic simplify misses identities — log_b(b^n), nCr/nPr/binom, inverse trig at exact points. All pass numerically; only simplifyOnCompare fails. Worth noting the Rust parse is better: legacy read sin^(-1)(1) as (1/sin)(1), i.e. reciprocal rather than inverse.

Decisions to settle, not bugs

• Scientific notation — 34. Rust never uses it; legacy switched at a magnitude threshold
  (5.252*10^(-13) vs 0.0000000000005252). DoenetML has an avoidScientificNotation attribute,
  which presumes scientific is the default — so we can't simply absorb this. Render option, or our
  display layer?
• Terminating rationals — 3. 5/2 renders 2.5; 1/3 correctly stays \frac{1}{3}. Ours per
  upstream's notes. Caveat: if the structural criteria (ReducedFraction, ExactValue, …) are meant
  to be usable after simplify, this stops being a display question.

Ours

• Tagged values reaching users (~37). .tree returns {"$":"Inf"} / {"$":"NaN"} where legacy
  gave JS Infinity / NaN, and the text printer spells a blank ＿ where legacy printed NaN.
  Upstream's symmetry argument is sound; our .tree consumers do typeof x === "number" and that's
  ours to fix.
• Sparse arrays reaching fromAst (9). Point.js builds Array(n+1) and fills only the
  components being set; JSON.stringify turns the holes into null, correctly rejected with
  unexpected value null. Left deliberately unfixed — a hole means "no desired value for this
  component", and mapping it to {"$":"None"} is a semantic guess that should be a human call.
• Float-precision assertions. The ODE tests compare two independent integrations with exact
  .eq(); they now differ by 2 ULP (13.445069170765525 vs ...523). A closeTo fix, but out of
  scope for the one test change that was authorized here.
• Term/factor ordering (10), tracing to compat's default_order no-op.

---

Sizes

web-target WASM is 1.32 MB (before wasm-opt, unavailable in this container).
dist/engine-rust.js is 3.78 MB raw / 1.21 MB gzipped with the WASM inlined, against ~1.1 MB
for the JavaScript library it replaces. Bundle size is not the obstacle the plan feared.

Inlining as base64 mirrors what CoreWorker.ts does for lib_doenetml_worker_bg.wasm: it
instantiates from bytes and needs no fetch, which matters because fetch is blocked for
blob/data URLs in the VS Code web-worker extension host (#1375).

---

Not done

• Step 0 differential harness
• Memory baseline (R8) — interner_size() gives us the gauge; growth rate unmeasured
• Cypress runs
• Verification of initSync in a real browser Web Worker

Files worth reviewing first

• packages/math/src/engine-rust.ts — the fromAst unwrap and why it's there
• packages/math/src/wasm-loader.ts — the injection and the ordering constraint
• scripts/unpad-container-delimiters.py — the LaTeX and prose-parenthesis carve-outs
• MATH_EXPRESSIONS_UPSTREAM_REQUESTS.md — the four upstream asks, with reproductions

[dqnykamp]

Right now, merges to main auto-publish to the dev npm tag and are queued to be published to the latest tag on next 0.7.x release. I think the backward-incompatibility of a new math-expressions should not get folded into the 0.7 series. I haven't thought about how to maintain a separate 0.8 line. For now, we should create a separate 0.8 branch and this PR should target that. We'll have to periodically merge in the main branch to keep it in sync.

I'll have to learn the best ways to maintain multiple lines. If you have suggestions, let me know!

[dqnykamp]

Or maybe, we release one more 0.7 version, create a 0.7 for any backport fixes, and publish 0.8 dev versions from now on off of main.