Repository navigation
feat(fields): fast KoalaBear quartic extension and coset interpolation - #399
Draft
mitschabaude-bot wants to merge 139 commits into
Draft
mitschabaude-bot wants to merge 139 commits into
mitschabaude-bot wants to merge 139 commits into
Conversation
…ocks Fold the 64x64 product's middle terms without carry bookkeeping so clang emits one widening multiply for squarings too; `mul x x` inlined into `square` used to compile to a four-imul chain. Run the Fermat inverse and the pow ladder on unreduced words (`mulLazy`, `squareNLazy`, `invLazy`, `powLazy`) with a single canonicalization at the end, and make `squareN` tail recursive. Correctness rests on word-level cast lemmas (`foldUInt128Lazy_cast`, `mulLazy_cast`, `squareNLazy_cast`, `invLazy_cast`, `powLazy_cast`). A/B against master on the same core: inv 311 -> 167 ns, pow 99 -> 74 ns, mul and add unchanged.
`conditionalSubtract` was a plain def, so every KoalaBear and BabyBear `add` paid an out-of-line call. A/B: add latency 844 -> 606 ps, throughput 876 -> 166 ps.
…nreduced words `univariate-dense-goldilocks` now runs the native-word carrier beside ZMod like the other fast fields, with rows named in the shared `-fast` scheme. Add tests that unreduced `ofUInt64` words compare and multiply as their residues.
Compute the canonical sum as `x - (p - y)`: the subtraction borrows exactly when `x + y < p`, so one borrow-selected correction by `2^32 - 1` yields the reduced result, three dependent instructions instead of a carry check plus a separate compare against `p`. `addRaw_toNat` carries the bound and the cast lemma; the carry-based reducer and its lemmas are removed. A/B against master on the same core: add latency 1021 -> 616 ps, throughput 381 -> 230 ps; mul, inv and pow unchanged.
# Conflicts: # bench/CompPolyBench/Fields/Arith.lean
… the Goldilocks fold Two changes to the 128-bit fold behind every fast Goldilocks multiply. The borrow of the low word against the top limb has probability about 2^-32, but a Lean `if` compiled to a select on the critical path; moving that arm into a `@[noinline]` function forces a predicted branch. The middle term `hi_lo * (2^32 - 1)` compiled to a three-cycle `imul` inside loops (clang holds the constant in a register); it is now `(hi <<< 32) - hi_lo`, two one-cycle operations. `shiftLeft32_sub_low` and `foldTail_cast` carry the proofs; the fold lemma cases on the borrow. A/B against master on the same core: mul latency 3077 -> 2485 ps, throughput 815 -> 739 ps, inv 314 -> 136 ns, pow 101 -> 64 ns; add unchanged.
One-sentence docstrings, `2 ^ k` literals instead of decimals, a `_toNat` bridge for the product's high word, implicit bounds on the `addRaw` lemmas, camelCase limb names, and README rows for the Goldilocks files.
# Conflicts: # CompPoly/Fields/README.md
An odd-size leaf now runs its radix-two layer first, with a sixteen-lane kernel (`pair16`), then radix-four passes down to the fused final layers and reversal (`lastRev`) that even leaves use. `runStagesDIFRadix4WithTwiddles_eq_top` proves the two stage orders equal. Leaves below 2^8 words run all passes and the scalar `decode`. With every leaf on one path, the sequential `stages` pipeline, the `leaf16` kernel, the separate `leafRev` pass and their refinement files are deleted. The native library root list now names `Packed.Columns` instead of the deleted `SliceTree`. Forward, ms, before → after: CPU 0 2^17 2.93 → 0.878, 2^19 13.2 → 3.92; CPUs 0-7 2^17 0.749 → 0.353, 2^19 2.96 → 1.18. Even sizes unchanged within noise.
Both plans' column transforms are now built from three task combinators in `CompPoly/Data/Tasks.lean`: `spawnAll` runs independent tasks, `spawnAfter` runs tasks once a set of tasks is done, and `join` collects results. `map_get_spawnAll` and `map_get_spawnAfter` state the results as plain maps, which replaces the task bookkeeping in each transform's proof. The fused kernels spill AVX registers with unaligned moves, and their speed depended on the stack alignment inherited from callers: builds with byte-identical kernels differed by 20% on one worker, and so did one binary under different environments. `CompPolyPackedNative` now compiles with `-mstackrealign`, which aligns the spill slots; timings are stable within 2% of the fast case.
# Conflicts: # lakefile.lean
The NTT kernels' native library now realigns the stack, which costs the packed interpolation loops about 2.5% without benefit: they don't spill AVX registers. They get their own library with -march=native only.
The packed interpolation loops do spill a few AVX registers; what justifies their separate library is that realigning measured 2.5% slower for them.
The sixteen-lane kernels and their field-array models are now expanded by the macros of `Packed/Unrolled.lean` from one description per family: `step16`, `pair16`, `lastHalf`, `transposeStep` and `line` on packed words, `step16Field`, `leaf16Field` and `laneLine` on field arrays. The expansions are the terms that were written out before, so the refinement proofs are unchanged and the generated C is identical. About 1,700 lines of unrolled source are gone.
The worker count only chose between the column transform and one leaf, so both plans now take `parallel : Bool := true`; the runner passes it when it has more than one worker.
The packed kernels compile with -march=native, and the restored .lake/build may come from a runner with another CPU: the NTT bench died with SIGILL on objects Lake did not need to rebuild.
# Conflicts: # CompPoly/Fields/Secp256k1.lean
lean.h drops lean_copy_byte_array; the backported word stores take the array exclusive with lean_sarray_ensure_exclusive, as core's byte stores now do. A rewrite in DifSpec closes its forward case without the trailing rfl.
The coefficient-list index in toBytes_toSpec names its bound, which v4.35 no longer infers from the length. The interpolation proofs split off the shared factor with congrArg, where congr 1 now recurses trying to close the sum.
Montgomery.Native32.Ext4 p is FastField p [X] / (X^4 - W) for any Mont32Field p with a Binomial4 p instance, which supplies W, a fast multiplication by W, and the irreducibility of X^4 - W over FastField p. Its spec field Ext4.Spec p is Ext over FastField p itself, so consumers never meet ZMod; specEquiv, the toSpec_* agreement lemmas and the byte codec carry over. The norm inverse gets its non-square W from the irreducibility. KoalaBear.Fast.Ext4 becomes the X^4 - 3 instance (mulW x = x + x + x), and BabyBear.Fast.Ext4 the X^4 - 11 instance, with Plonky3 known answers. Both transfer their ZMod certificates along ringEquiv with the new Extension.Map (ExtensionParams.map, irreducible_map_ringEquiv). FastField gets Fintype and card lemmas. Coset interpolation is respecced to the new spec field. Extension benchmarks and packed interpolation time the same as before.
The packed kernels, plan and refinement proofs work on FastField p for any Mont32Field p. The raw word operations take the modulus and montgomeryNegInv from the instance, ofWord checks against p, and every kernel takes p and is @[specialize]d; the unrolled generators emit add p. The plan calls the class PackedNTT p, whose run is Native.run p (run_eq), so a field's kernels are specialized and compiled once, in the CPU-targeted native library: Packed/Instances.lean provides KoalaBear and BabyBear, and their hot kernels disassemble to the instructions of the KoalaBear-only kernels. CompPolyNativeSmoke runs both instances against the DFT specification and the parallel path against the sequential one.
interpolateCoset, interpolateCosetPacked and their correctness proofs move to Fields/Montgomery and work over FastField p with any Binomial4 quartic extension; the weights' constants use Binomial4.mulW, and stored words are checked against the instance's modulus. The packed loops are @[specialize]d, and interpolateCosetPacked calls the class PackedInterpolation p, whose instances in KoalaBear/InterpolateCoset and BabyBear/InterpolateCoset (the roots of CompPolyInterpolateNative) compile them at the field's modulus. CompPolyNativeSmoke checks both instances against the reference.
…r radix-four pass `lastHalf` checks its output once per call, writes the output's last word back unchanged (`touchLast`), and then stores its sixteen half lines with proved bounds (`store8`): the exclusivity check of the first store no longer holds the 128 lane values across a copy, so they stay in registers. `radix4Blocks` checks the whole pass once for quarters of whole sixteen-cell batches and runs the blocks in a machine-word loop (`blocks16Loop`), without the per-block natural-number checks.
…alabear-fast-ext4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Depends on #395, which depends on #390. This branch starts at #395's tip. GitHub cannot use a fork branch as an upstream base, so this draft targets
mainand its diff includes both earlier PRs. Review only the new changes.A fast, fully proved degree-4 extension of KoalaBear,
KoalaBear[X] / (X^4 - 3): the same field as the specKoalaBear.Ext4and Plonky3'sBinomialExtensionField<KoalaBear, 4>. On top of it, a proved coset interpolation at an extension point, benchmarked against Plonky3's stockinterpolate_coset.KoalaBear.Fast.Ext4holds four Montgomery coefficients.toSpecmaps it into the spec field;toSpec_add,toSpec_mul,toSpec_inv,toSpec_divandtoSpec_ofBaseprove agreement operation by operation, andspecEquivis the ring isomorphism.3into the left factors, so each output coefficient is a sum of four products with a single Montgomery reduction. The newMontgomery.Native32.mulAdd4does this for every 32-bit Montgomery field: four products stay below2 · p · 2^32, so one conditional subtraction brings them intoreduce's range.σ : X ↦ -X,a · σ(a) = g₀ + g₁ X²anda⁻¹ = σ(a) (g₀ - g₁ X²) / (g₀² - 3 g₁²), one base-field inversion. The norm of a nonzero element is nonzero becauseσis an injective ring endomorphism of the quotient field and3is a quadratic non-residue modp.toBytes_toSpec), with round trip throughByteCodec.Results
The existing
fields-extension-koalabear-ext4-muland-invbenchmark groups now time both types on the same operands; the harness checks that both produce the same checksum. Per call, including the harness's operand lookup and the result allocation:KoalaBear.Ext4KoalaBear.Fast.Ext4The tests check known answers computed with Plonky3 (
p3-field0.4.2) for a product and two inverses, field laws, agreement with the spec field, and the codec.Coset interpolation
KoalaBear.Fast.interpolateCosetis Plonky3'sinterpolate_cosetfor KoalaBear and its quartic extension: given a row-major2^logN × widthmatrix whose columns hold the values of polynomials on the cosets · ⟨ω⟩, it evaluates every column's polynomial at a pointzof the extension, by(zⁿ - sⁿ) / (n sⁿ) · ∑ᵢ xᵢ / (z - xᵢ) · p(xᵢ).NTT.Domain.aeval_eq_cosetBarycentricproves that formula for any field extensionE / F: the coset's nodal polynomial isXⁿ - sⁿ, so each barycentric weight isxᵢ / (n sⁿ).List.batchInvis Montgomery's batch inversion, proved correct for inputs without zeros and transferable along maps preserving*,⁻¹and1, so it runs on the fast extension, which has noFieldinstance.toSpec_interpolateCosetproves every output equal toaeval z pⱼfor the column polynomialspⱼ.--suite interpolateruns it againstp3_interpolation::interpolate_coset0.4.2 on the same fixtures; full output digests agree before timing, from single rows to 2²⁰ × 16.KoalaBear.Fast.interpolateCosetPackedis the fast version. It takes the matrix as Montgomery words in aByteArray, the layout of Plonky3'sRowMajorMatrix, andinterpolateCosetPacked_eqproves it equal to the reference for every point off the coset.t = z₀ - x,(z - x)⁻¹ = (b₀ + t b₁ + t² b₂ + t³) / N(t), with extension constantsbₖthat depend onzonly andN(t)the norm ofz - x. So each row's weight is four base-field valuesx tᵏ / N, only base-field norms need inverting, and a column sum is∑ₖ Sₖ bₖfor four base-field sums. That is about twelve base-field multiplications per row.t², norm and prefix product, one inversion per block inverts the product, and a backward pass overwrites each row with its weights.Montgomery.Native32.LazyAcc) and one Montgomery reduction per block. Four columns share each row's weights.It reads and writes words with the packed NTT's backported
ByteArrayword accessors (lean4#14053) and adds no extern of its own. The benchmark suite also checks its full outputs against the reference and Plonky3 before timing.Five rounds of
--suite interpolateon eight workers at130c36a, milliseconds per call: