Skip to content

ci: gate proofs on the base revision's policy and a kernel replay - #414

Open
parithosh wants to merge 2 commits into
Verified-zkEVM:mainfrom
parithosh:proof-gates-2
Open

parithosh wants to merge 2 commits into
Verified-zkEVM:mainfrom
parithosh:proof-gates-2

Conversation

@parithosh

Copy link
Copy Markdown
Contributor

Closes the gaps an audit of the proof tooling found, each shown with a planted mutation:

  • A PR could approve its own sorry or axiom by listing it in scripts/axiom_baseline.json. A new trusted-gates job runs the base revision's axiom sweep against the base revision's baseline, with no PR code: base checkout, toolchain and Mathlib cache, and only the PR's compiled oleans. A wider policy, or a toolchain or manifest change, needs an approving review of the head commit by a base CODEOWNERS user who did not author the PR or its commits.
  • A declaration added without kernel checking passed the sweep. lake exe kernelreplay (scripts/KernelReplay.lean) replays every library and test declaration through the kernel in import-ordered chunks. It checks that exported constants match the private ones it checks, checks every copy of a name several modules store, and with --require-modules fails if a module is outside the roots' imports.
  • sorry in examples, structure defaults or tests: lake build --wfail over the library and tests is now blocking, and the lint rejects decide +native, native := true, trustCompiler, skipKernelTC, addDeclWithoutChecking, warn.sorry and #guard_msgs (drop ...).
  • axiomsweep sweeps CompPolyTests and no longer runs initializers, so swept code cannot run inside it; --statements exports digests that scripts/statement_diff.py reports on PRs (report only).
  • build-timing rows record CPU, CPU count and cache state and are compared only when those match.
  • checkout no longer persists the token, and CODEOWNERS covers the files that decide what counts as proved.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1a66d-72db-7532-b204-db0dff8754a6

Closes the gaps an audit of the proof tooling found, each shown with a
planted mutation:

- A PR could approve its own sorry or axiom by listing it in
  scripts/axiom_baseline.json. A new trusted-gates job runs the base
  revision's axiom sweep against the base revision's baseline, with no
  PR code: base checkout, toolchain and Mathlib cache, and only the PR's
  compiled oleans. A wider policy, or a toolchain or manifest change,
  needs an approving review of the head commit by a base CODEOWNERS user
  who did not author the PR or its commits.
- A declaration added without kernel checking passed the sweep.
  lake exe kernelreplay (scripts/KernelReplay.lean) replays every
  library and test declaration through the kernel in import-ordered
  chunks. It checks that exported constants match the private ones it
  checks, checks every copy of a name several modules store, and with
  --require-modules fails if a module is outside the roots' imports.
- sorry in examples, structure defaults or tests: lake build --wfail
  over the library and tests is now blocking, and the lint rejects
  decide +native, native := true, trustCompiler, skipKernelTC,
  addDeclWithoutChecking, warn.sorry and #guard_msgs (drop ...).
- axiomsweep sweeps CompPolyTests and no longer runs initializers, so
  swept code cannot run inside it; --statements exports digests that
  scripts/statement_diff.py reports on PRs (report only).
- build-timing rows record CPU, CPU count and cache state and are
  compared only when those match.
- checkout no longer persists the token, and CODEOWNERS covers the
  files that decide what counts as proved.

Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1a66d-72db-7532-b204-db0dff8754a6
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

🤖 PR Summary

DQ


Statistics

Metric Count
📝 Files Changed 13
✅ Lines Added 1180
❌ Lines Removed 47

Lean Declarations

✏️ Added: 15 declaration(s)

scripts/AxiomSweep.lean (6)

  • abbrev DigestM
  • def buildStatements (env : Environment) (roots : Array Name) : Array StatementEntry
  • def sweptTargets (env : Environment) (roots : Array Name) : Array (Name × Name) × Nat
  • def telescopeBinderInfos : Expr → List BinderInfo
  • partial def exprDigest (env : Environment) (gray : List Name) (e : Expr) : DigestM UInt64
  • partial def fullDigest (env : Environment) (gray : List Name) (n : Name) : DigestM UInt64

scripts/KernelReplay.lean (9)

  • def defaultRoots : Array Name
  • def exportedMatches (exported priv : ConstantInfo) : Bool
  • def parseArgs : List String → Config → Except String Config
  • def renamedTheorem (fresh : Name) : ConstantInfo → Option ConstantInfo
  • def sameSignature (a b : ConstantInfo) : Bool
  • unsafe def main (args : List String) : IO UInt32
  • unsafe def modulesInOrder (roots : Array Name) :
  • unsafe def readParts (m : Name) : IO ModuleParts
  • unsafe def replayChunk (roots : Array Name) (chunkSize i : Nat) : IO UInt32
✏️ Affected: 2 declaration(s) (line number changed)
  • def buildEntries (roots : Array Name) : CoreM (Array Entry × Nat) in scripts/AxiomSweep.lean moved from L186 to L207
  • def defaultRoots : Array Name in scripts/AxiomSweep.lean moved from L57 to L63

sorry Tracking

  • No sorrys were added, removed, or affected.

📋 **Additional Analysis**

No findings. The diff is a CI/script/infrastructure change (CODEOWNERS, workflow hardening, new kernelreplay executable, statement-diff script, expanded linting, build-timing provenance) and does not touch any Lean library or test source, formalize a theorem, or modify a mathematical definition. None of the deployment-supplied review categories (style guide, tier transitions, obligation mappings, registry/roadmap drift, etc.) apply to this material.


📄 **Per-File Summaries**
  • .github/CODEOWNERS: This change adds a new .github/CODEOWNERS file to the repository. It specifies that changes to files governing the verification pipeline (the axiom sweep script and its baseline, the kernel replay script, the style linter and its exceptions, the build configuration, toolchain and dependency lock files, the CI workflow directory, and the CODEOWNERS file itself) require review from three named maintainers before they can be merged, once a branch protection rule is enabled.
  • .github/workflows/lean_action_ci.yml: The workflow now disables persisted credentials during checkout to prevent token leakage from Lean code execution; adds a build cache state recorder for accurate timing comparisons; modifies the axiom sweep step to export its results as a JSON artifact (uploaded for 90 days) and introduces a statement comparison against the base commit via a new script; adjusts the fail-on-build-warnings step to fail on warnings and also build CompPolyTests; uploads the compiled library and test oleans as artifacts for PRs; and adds a new 'trusted-gates' job that re-runs the axiom sweep and kernel replay under the base revision's environment and tools, requiring code-owner sign-off if the sweep finds new taint or if the lock files changed, ensuring the PR's code cannot tamper with trust-policy decisions.
  • AGENTS.md: Updated AGENTS.md to add two new verification commands (lake test and lake exe kernelreplay) to step 6 of the contributor workflow, which already required lake exe axiomsweep --check when filling or adding sorry declarations. This change ensures contributors now run a broader test suite and a kernel replay check alongside the existing axiom-sweep validation before refreshing the axiom baseline.
  • CONTRIBUTING.md: Two changes were made to CONTRIBUTING.md. In the CI mapping table, the command for work that is axiom-clean or involves sorry now includes && lake test before lake exe axiomsweep --check and adds && lake exe kernelreplay after it. In the benchmarking workflow instructions, the command following a successful faster run (without SUSPECT) also appends && lake exe kernelreplay to the existing lake build && lake test && lake exe axiomsweep --check. This ensures that kernel replay validation is always performed alongside the axiom sweep for axiom-critical changes and performance refinements.
  • docs/wiki/generated-files.md: Updated docs/wiki/generated-files.md to clarify the CI enforcement for scripts/axiom_baseline.json: on a PR, the base revision's axiom sweep is also run against the base revision's baseline, so any increase in the baseline requires an approving review from a code owner.
  • docs/wiki/quickstart.md: Expanded the documentation for CI gates in docs/wiki/quickstart.md: added lake test and lake exe kernelreplay to the sorry-filling workflow; widened the axiom-sweep scope to cover CompPolyTests.* modules and added --wfail build coverage; introduced the trusted-gates job that runs the base revision's axiom sweep on PRs to prevent baseline manipulation, with specific sign-off rules for toolchain/manifest changes or new taint; described the kernelreplay gate that re-checks every declaration in the Lean kernel to catch declarations added under skipKernelTC or via addDeclWithoutChecking; listed remaining blind spots (workflow definition from PR, olean-to-source trust, initializers); and updated the CI summary to include kernel replay, --wfail, and declaration-diff reporting, while adding lake exe kernelreplay to the local developer checklist.
  • lakefile.lean: The lakefile.lean adds a new lean_exe target kernelreplay in the scripts/ directory (root KernelReplay) that re-checks every declaration under the CompPoly and CompPolyTests namespaces in the kernel, ensuring any added declaration missing kernel checking causes a failure. The executable is intended to be run after lake build and lake test.
  • scripts/AxiomSweep.lean: The diff makes several independent changes to scripts/AxiomSweep.lean. It extends the set of default root modules to include CompPolyTests alongside CompPoly, updates the module-level documentation to reflect this and to clarify that CI now covers two known blind spots (source-level sorry via lake build --wfail and kernel re-checking via KernelReplay.lean), and adds a new --statements FILE command-line option. To support the latter, it introduces StatementEntry (a structure with JSON output), a DigestM state monad abbreviation, and several new functions — telescopeBinderInfos, exprDigest, fullDigest, and buildStatements — that compute a structural digest of each reportable declaration's type and (for non-theorems) value, recursively folding in digests of internal constants. The existing buildEntries function is refactored to extract the target enumeration into a pure sweptTargets helper. In main, loading now disables initializer execution (loadExts := false) and the import error message advises building both roots with lake build and lake test. The usage string and parseArgs are updated to accept --statements, and the Config structure gains a statements? field. The runtime behavior now writes a statement digest JSON file when the flag is supplied.
  • scripts/KernelReplay.lean: Added scripts/KernelReplay.lean, a new tool that re-checks every declaration of the CompPoly project in Lean's kernel, closing a gap where axiomsweep could report unchecked declarations (e.g., a “proof” of False with no axioms). It defines Config (with fields roots, chunkSize, jobs, chunk?, requireModules?, selfScript?), ModuleParts (holding imports, exported, priv), and core functions: readParts reads .olean parts; modulesInOrder computes a topologically sorted import order; sameSignature and exportedMatches verify that exported copies are consistent with private constants; renamedTheorem renames duplicate theorem copies so each proof is checked individually; replayChunk replays a contiguous chunk of modules in a child process against a base environment of outside modules; and main chunks modules, spawns child processes (up to jobs at a time), and reports failures. The tool handles duplicate constant names across modules, unsafe/partial skip, and a --require-modules flag to verify required modules are reachable.
  • scripts/README.md: This update expands the scripts/README.md documentation to cover the new KernelReplay executable and the statement_diff.py script, and updates descriptions for the existing AxiomSweep and lint-style.py tools.

Specifically:

  • KernelReplay (lake exe kernelreplay) is introduced. It re-checks every declaration in CompPoly and CompPolyTests using Lean’s kernel replay mechanism, running in import-ordered chunks to bound memory. The documentation notes it should be run after lake build and lake test.
  • statement_diff.py is documented. It compares two axiomsweep --statements exports and produces a Markdown report of removed declarations and changes to theorem statements or values. It is used in CI but never fails a run.
  • AxiomSweep’s scope is extended: it now covers CompPolyTests.* declarations in addition to CompPoly.*. The --statements FILE flag is documented (writes per-declaration digests for statement_diff.py). The invocation line now shows it should be run after lake build and lake test.
  • lint-style.py’s description is expanded to explicitly name the local TCB policy checks it enforces: native_decide, decide +native, ofReduceBool, trustCompiler, and the kernel-check bypasses skipKernelTC and addDeclWithoutChecking.
  • The AxiomSweep baseline argument now includes --chunk-size N in its chunk-size default reference.
  • scripts/build_timing_report.sh: The script now records per-run provenance: a machine field with CPU model (from /proc/cpuinfo) and core count, and a cache field set to 'clean' for clean builds or read from the BUILD_TIMING_CACHE_STATE environment variable otherwise. A new not_comparable function checks that two timing records have matching exit codes, machine info, and cache state before comparing them; the comparison table displays n/a (reason) when they differ. The printed header now lists the measured machine(s) and per-label cache states, and notes that deltas are shown only when machine and cache match. Target-level baselines also respect the same comparability check, skipping and explaining when the build log types differ or the timing records are missing or incomparable.
  • scripts/lint-style.py: Extended the forbidden_tcb_check function in scripts/lint-style.py to detect additional TCB-bypassing constructs: ofReduceBool, trustCompiler, skipKernelTC, addDeclWithoutChecking, decide + native, native := true, and patterns that hide sorry warnings (warn.sorry, #guard_msgs (drop ...), #guard_msgs (substring := ...), and #guard_msgs docstrings that mention sorry). Also updated the error message for ERR_NDEC to list all newly forbidden patterns, and the comment describing the error constant now refers broadly to 'native evaluation or kernel-check bypass' instead of only native_decide / Lean.ofReduceBool.
  • scripts/statement_diff.py: Added a new script scripts/statement_diff.py that compares two JSON exports from lake exe axiomsweep --statements and produces a Markdown summary of declaration changes for pull requests. It reports removed declarations, theorems whose statement (type and universe parameters) changed, other declarations whose type changed, and definitions whose value changed (by digest, ignoring binder names), up to a limit of 50 rows per section with a count of overflow. The script exits 0 even when files differ, and notes mismatched toolchains or roots; it does not gate the PR.

Last updated: 2026-11-18 00:40 UTC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants