window-oracle-nightly #47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: window-oracle-nightly | |
| # Deep generative window-tree parity run against a LIVE GNU Emacs. | |
| # | |
| # `test-suite.yml` runs the oracle corpus in snapshot mode (expect vs neomacs, GNU never | |
| # invoked) and shards it 20 ways to stay fast. That is the right trade for a | |
| # per-commit gate, but it means the generative property | |
| # (`oracle_prop_window_tree_survives_random_operation_sequences`) only ever runs | |
| # at ORACLE_PROP_CASES depth, which catches gross regressions and little else. | |
| # | |
| # The value of that property is in DEEP runs: it found eight window bugs at | |
| # 400-1200 sequences, each shrunk to a minimal operation list. A deep run takes | |
| # ~200s wall, too slow per-commit but trivial nightly. | |
| # | |
| # `proptest-regressions/` is gitignored in this repo, so failing seeds do NOT | |
| # survive a clean checkout. Each nightly run therefore rediscovers cases from | |
| # scratch, and a failure must be captured from the log rather than replayed from | |
| # a committed seed -- hence the artifact upload below. | |
| on: | |
| schedule: | |
| # 03:43 UTC daily. Offset from test-suite.yml (05:17 Sun) and codeql (04:23 Mon) so | |
| # the three never contend for runners. | |
| - cron: "43 3 * * *" | |
| workflow_dispatch: | |
| inputs: | |
| cases: | |
| description: "Number of random operation sequences" | |
| required: false | |
| default: "1200" | |
| permissions: | |
| contents: read | |
| concurrency: | |
| group: window-oracle-nightly | |
| cancel-in-progress: false | |
| jobs: | |
| deep-window-parity: | |
| name: generative window-tree parity vs live GNU Emacs | |
| runs-on: ubuntu-24.04 | |
| timeout-minutes: 60 | |
| env: | |
| CARGO_TERM_COLOR: always | |
| RUST_BACKTRACE: 1 | |
| # `live` runs BOTH engines and compares them directly, with no recorded | |
| # expectation involved. That is deliberate: recorded expects drift with | |
| # the GNU build (frame geometry, configure flags), whereas a direct | |
| # comparison only asserts that the two engines agree right now. | |
| NEOVM_ORACLE_MODE: live | |
| NEOVM_WINDOW_PROP_CASES: ${{ github.event.inputs.cases || '1200' }} | |
| NEOVM_FORCE_ORACLE_PATH: emacs | |
| TMPDIR: ${{ github.workspace }}/tmp | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Prepare workspace temp directory | |
| run: mkdir -p "$TMPDIR/nextest" | |
| - name: Install Linux system dependencies | |
| run: scripts/ci/setup-linux.sh oracle | |
| - name: Set up GNU Emacs oracle | |
| uses: ./.github/actions/setup-gnu-emacs | |
| - name: Record GNU Emacs oracle | |
| run: | | |
| # Record the oracle build: a divergence reported here is only | |
| # meaningful against a known GNU version, and window geometry in | |
| # particular varies between builds. | |
| emacs --version | head -n 1 | |
| - name: Set up Rust tooling | |
| uses: ./.github/actions/setup-rust | |
| with: | |
| cache-key: window-oracle-nightly-ubuntu-24.04 | |
| install-nextest: true | |
| # The oracle spawns the real neomacs binary per case, so it needs the | |
| # runtime tree (lisp/, etc.) in place, not just the test binary. | |
| - name: Build neomacs runtime | |
| run: cargo xtask fresh-build --release | |
| - name: Deep generative window-tree parity run | |
| # `shell: bash` is required, not cosmetic: the default Linux shell is | |
| # `bash -e` WITHOUT pipefail, so `cargo nextest ... | tee` would report | |
| # tee's exit status and the job would pass on a failing run. | |
| shell: bash | |
| run: | | |
| set -o pipefail | |
| cargo nextest run --release -p neovm-oracle-tests \ | |
| -E 'test(oracle_prop_window_tree_survives_random_operation_sequences)' \ | |
| --no-capture 2>&1 | tee window-oracle-nightly.log | |
| # A failure prints the minimal shrunk operation list AND the exact | |
| # generated elisp under `program:`. Hand-translating the op list is | |
| # error-prone -- lift the program verbatim from this artifact instead. | |
| - name: Upload run log | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: window-oracle-nightly-log | |
| path: window-oracle-nightly.log | |
| retention-days: 30 |