Skip to content

feat(fields): fast KoalaBear quartic extension and coset interpolation - #399

Draft
mitschabaude-bot wants to merge 139 commits into
Verified-zkEVM:mainfrom
mitschabaude-bot:koalabear-fast-ext4
Draft

mitschabaude-bot wants to merge 139 commits into
Verified-zkEVM:mainfrom
mitschabaude-bot:koalabear-fast-ext4

Conversation

@mitschabaude-bot

@mitschabaude-bot mitschabaude-bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Depends on #395, which depends on #390. This branch starts at #395's tip. GitHub cannot use a fork branch as an upstream base, so this draft targets main and its diff includes both earlier PRs. Review only the new changes.

A fast, fully proved degree-4 extension of KoalaBear, KoalaBear[X] / (X^4 - 3): the same field as the spec KoalaBear.Ext4 and Plonky3's BinomialExtensionField<KoalaBear, 4>. On top of it, a proved coset interpolation at an extension point, benchmarked against Plonky3's stock interpolate_coset.

  • KoalaBear.Fast.Ext4 holds four Montgomery coefficients. toSpec maps it into the spec field; toSpec_add, toSpec_mul, toSpec_inv, toSpec_div and toSpec_ofBase prove agreement operation by operation, and specEquiv is the ring isomorphism.
  • Multiplication folds 3 into the left factors, so each output coefficient is a sum of four products with a single Montgomery reduction. The new Montgomery.Native32.mulAdd4 does this for every 32-bit Montgomery field: four products stay below 2 · p · 2^32, so one conditional subtraction brings them into reduce's range.
  • Inversion uses the norm to the base field: with σ : X ↦ -X, a · σ(a) = g₀ + g₁ X² and a⁻¹ = σ(a) (g₀ - g₁ X²) / (g₀² - 3 g₁²), one base-field inversion. The norm of a nonzero element is nonzero because σ is an injective ring endomorphism of the quotient field and 3 is a quadratic non-residue mod p.
  • Codec: the four coefficient encodings, identical to the spec field's encoding (toBytes_toSpec), with round trip through ByteCodec.

Results

The existing fields-extension-koalabear-ext4-mul and -inv benchmark groups now time both types on the same operands; the harness checks that both produce the same checksum. Per call, including the harness's operand lookup and the result allocation:

Operation Spec KoalaBear.Ext4 Fast KoalaBear.Fast.Ext4
Multiplication 7.33 µs 28.1 ns
Inversion 913 µs (Fermat) 161 ns (norm)

The tests check known answers computed with Plonky3 (p3-field 0.4.2) for a product and two inverses, field laws, agreement with the spec field, and the codec.

Coset interpolation

KoalaBear.Fast.interpolateCoset is Plonky3's interpolate_coset for KoalaBear and its quartic extension: given a row-major 2^logN × width matrix whose columns hold the values of polynomials on the coset s · ⟨ω⟩, it evaluates every column's polynomial at a point z of the extension, by (zⁿ - sⁿ) / (n sⁿ) · ∑ᵢ xᵢ / (z - xᵢ) · p(xᵢ).

  • NTT.Domain.aeval_eq_cosetBarycentric proves that formula for any field extension E / F: the coset's nodal polynomial is Xⁿ - sⁿ, so each barycentric weight is xᵢ / (n sⁿ).
  • List.batchInv is Montgomery's batch inversion, proved correct for inputs without zeros and transferable along maps preserving *, ⁻¹ and 1, so it runs on the fast extension, which has no Field instance.
  • toSpec_interpolateCoset proves every output equal to aeval z pⱼ for the column polynomials pⱼ.
  • --suite interpolate runs it against p3_interpolation::interpolate_coset 0.4.2 on the same fixtures; full output digests agree before timing, from single rows to 2²⁰ × 16.

KoalaBear.Fast.interpolateCosetPacked is the fast version. It takes the matrix as Montgomery words in a ByteArray, the layout of Plonky3's RowMajorMatrix, and interpolateCosetPacked_eq proves it equal to the reference for every point off the coset.

  • Base-field weights. For t = z₀ - x, (z - x)⁻¹ = (b₀ + t b₁ + t² b₂ + t³) / N(t), with extension constants bₖ that depend on z only and N(t) the norm of z - x. So each row's weight is four base-field values x tᵏ / N, only base-field norms need inverting, and a column sum is ∑ₖ Sₖ bₖ for four base-field sums. That is about twelve base-field multiplications per row.
  • Blocks of 512 rows. A forward pass stores each row's node, t², norm and prefix product, one inversion per block inverts the product, and a backward pass overwrites each row with its weights.
  • Lazy sums. Column sums accumulate unreduced 64-bit products with one conditional subtraction each (Montgomery.Native32.LazyAcc) and one Montgomery reduction per block. Four columns share each row's weights.
  • Tasks. Eight row ranges run in parallel, with the caller taking the lowest.

It reads and writes words with the packed NTT's backported ByteArray word accessors (lean4#14053) and adds no extern of its own. The benchmark suite also checks its full outputs against the reference and Plonky3 before timing.

Five rounds of --suite interpolate on eight workers at 130c36a, milliseconds per call:

Rows Columns Lean packed Lean reference Plonky3
2^12 1 0.08 0.68 0.20
2^12 16 0.09 1.76 0.15
2^16 1 0.30 11.92 0.88
2^16 16 0.47 27.97 0.64
2^20 1 3.26 248.35 8.08
2^20 16 4.84 523.12 8.80

graikos and others added 30 commits September 29, 2026 07:33
…ocks

Fold the 64x64 product's middle terms without carry bookkeeping so clang emits
one widening multiply for squarings too; `mul x x` inlined into `square` used
to compile to a four-imul chain. Run the Fermat inverse and the pow ladder on
unreduced words (`mulLazy`, `squareNLazy`, `invLazy`, `powLazy`) with a single
canonicalization at the end, and make `squareN` tail recursive. Correctness
rests on word-level cast lemmas (`foldUInt128Lazy_cast`, `mulLazy_cast`,
`squareNLazy_cast`, `invLazy_cast`, `powLazy_cast`).

A/B against master on the same core: inv 311 -> 167 ns, pow 99 -> 74 ns,
mul and add unchanged.
`conditionalSubtract` was a plain def, so every KoalaBear and BabyBear `add`
paid an out-of-line call. A/B: add latency 844 -> 606 ps, throughput
876 -> 166 ps.
…nreduced words

`univariate-dense-goldilocks` now runs the native-word carrier beside ZMod like
the other fast fields, with rows named in the shared `-fast` scheme. Add tests
that unreduced `ofUInt64` words compare and multiply as their residues.
Compute the canonical sum as `x - (p - y)`: the subtraction borrows exactly
when `x + y < p`, so one borrow-selected correction by `2^32 - 1` yields the
reduced result, three dependent instructions instead of a carry check plus a
separate compare against `p`. `addRaw_toNat` carries the bound and the cast
lemma; the carry-based reducer and its lemmas are removed.

A/B against master on the same core: add latency 1021 -> 616 ps, throughput
381 -> 230 ps; mul, inv and pow unchanged.
# Conflicts:
#	bench/CompPolyBench/Fields/Arith.lean
… the Goldilocks fold

Two changes to the 128-bit fold behind every fast Goldilocks multiply. The borrow
of the low word against the top limb has probability about 2^-32, but a Lean `if`
compiled to a select on the critical path; moving that arm into a `@[noinline]`
function forces a predicted branch. The middle term `hi_lo * (2^32 - 1)` compiled
to a three-cycle `imul` inside loops (clang holds the constant in a register); it
is now `(hi <<< 32) - hi_lo`, two one-cycle operations. `shiftLeft32_sub_low` and
`foldTail_cast` carry the proofs; the fold lemma cases on the borrow.

A/B against master on the same core: mul latency 3077 -> 2485 ps, throughput
815 -> 739 ps, inv 314 -> 136 ns, pow 101 -> 64 ns; add unchanged.
One-sentence docstrings, `2 ^ k` literals instead of decimals, a `_toNat`
bridge for the product's high word, implicit bounds on the `addRaw` lemmas,
camelCase limb names, and README rows for the Goldilocks files.
An odd-size leaf now runs its radix-two layer first, with a sixteen-lane
kernel (`pair16`), then radix-four passes down to the fused final
layers and reversal (`lastRev`) that even leaves use.
`runStagesDIFRadix4WithTwiddles_eq_top` proves the two stage orders
equal. Leaves below 2^8 words run all passes and the scalar `decode`.

With every leaf on one path, the sequential `stages` pipeline, the
`leaf16` kernel, the separate `leafRev` pass and their refinement
files are deleted. The native library root list now names
`Packed.Columns` instead of the deleted `SliceTree`.

Forward, ms, before → after: CPU 0 2^17 2.93 → 0.878, 2^19 13.2 → 3.92;
CPUs 0-7 2^17 0.749 → 0.353, 2^19 2.96 → 1.18. Even sizes unchanged
within noise.
Both plans' column transforms are now built from three task
combinators in `CompPoly/Data/Tasks.lean`: `spawnAll` runs independent
tasks, `spawnAfter` runs tasks once a set of tasks is done, and `join`
collects results. `map_get_spawnAll` and `map_get_spawnAfter` state
the results as plain maps, which replaces the task bookkeeping in each
transform's proof.

The fused kernels spill AVX registers with unaligned moves, and their
speed depended on the stack alignment inherited from callers: builds
with byte-identical kernels differed by 20% on one worker, and so did
one binary under different environments. `CompPolyPackedNative` now
compiles with `-mstackrealign`, which aligns the spill slots; timings
are stable within 2% of the fast case.
The NTT kernels' native library now realigns the stack, which costs the
packed interpolation loops about 2.5% without benefit: they don't spill
AVX registers. They get their own library with -march=native only.
The packed interpolation loops do spill a few AVX registers; what
justifies their separate library is that realigning measured 2.5%
slower for them.
The sixteen-lane kernels and their field-array models are now expanded
by the macros of `Packed/Unrolled.lean` from one description per
family: `step16`, `pair16`, `lastHalf`, `transposeStep` and `line` on
packed words, `step16Field`, `leaf16Field` and `laneLine` on field
arrays. The expansions are the terms that were written out before, so
the refinement proofs are unchanged and the generated C is identical.
About 1,700 lines of unrolled source are gone.
The worker count only chose between the column transform and one leaf,
so both plans now take `parallel : Bool := true`; the runner passes it
when it has more than one worker.
The packed kernels compile with -march=native, and the restored .lake/build may
come from a runner with another CPU: the NTT bench died with SIGILL on objects
Lake did not need to rebuild.
# Conflicts:
#	CompPoly/Fields/Secp256k1.lean
lean.h drops lean_copy_byte_array; the backported word stores take the array
exclusive with lean_sarray_ensure_exclusive, as core's byte stores now do. A
rewrite in DifSpec closes its forward case without the trailing rfl.
The coefficient-list index in toBytes_toSpec names its bound, which v4.35 no
longer infers from the length. The interpolation proofs split off the shared
factor with congrArg, where congr 1 now recurses trying to close the sum.
Montgomery.Native32.Ext4 p is FastField p [X] / (X^4 - W) for any
Mont32Field p with a Binomial4 p instance, which supplies W, a fast
multiplication by W, and the irreducibility of X^4 - W over FastField p.
Its spec field Ext4.Spec p is Ext over FastField p itself, so consumers
never meet ZMod; specEquiv, the toSpec_* agreement lemmas and the byte
codec carry over. The norm inverse gets its non-square W from the
irreducibility.

KoalaBear.Fast.Ext4 becomes the X^4 - 3 instance (mulW x = x + x + x),
and BabyBear.Fast.Ext4 the X^4 - 11 instance, with Plonky3 known answers.
Both transfer their ZMod certificates along ringEquiv with the new
Extension.Map (ExtensionParams.map, irreducible_map_ringEquiv).
FastField gets Fintype and card lemmas. Coset interpolation is respecced
to the new spec field. Extension benchmarks and packed interpolation
time the same as before.
The packed kernels, plan and refinement proofs work on FastField p for any
Mont32Field p. The raw word operations take the modulus and
montgomeryNegInv from the instance, ofWord checks against p, and every
kernel takes p and is @[specialize]d; the unrolled generators emit add p.

The plan calls the class PackedNTT p, whose run is Native.run p
(run_eq), so a field's kernels are specialized and compiled once, in the
CPU-targeted native library: Packed/Instances.lean provides KoalaBear and
BabyBear, and their hot kernels disassemble to the instructions of the
KoalaBear-only kernels. CompPolyNativeSmoke runs both instances against
the DFT specification and the parallel path against the sequential one.
interpolateCoset, interpolateCosetPacked and their correctness proofs move
to Fields/Montgomery and work over FastField p with any Binomial4 quartic
extension; the weights' constants use Binomial4.mulW, and stored words are
checked against the instance's modulus.

The packed loops are @[specialize]d, and interpolateCosetPacked calls the
class PackedInterpolation p, whose instances in KoalaBear/InterpolateCoset
and BabyBear/InterpolateCoset (the roots of CompPolyInterpolateNative)
compile them at the field's modulus. CompPolyNativeSmoke checks both
instances against the reference.
…r radix-four pass

`lastHalf` checks its output once per call, writes the output's last word back unchanged
(`touchLast`), and then stores its sixteen half lines with proved bounds (`store8`): the
exclusivity check of the first store no longer holds the 128 lane values across a copy, so they
stay in registers. `radix4Blocks` checks the whole pass once for quarters of whole sixteen-cell
batches and runs the blocks in a machine-word loop (`blocks16Loop`), without the per-block
natural-number checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants