Add nightly automation and statistical regression detection#16
Open
vito1317 wants to merge 3 commits into
Open
Add nightly automation and statistical regression detection#16vito1317 wants to merge 3 commits into
vito1317 wants to merge 3 commits into
Conversation
…nnotations - load_release_data picked releases_info[-1] as mostRecentRelease, but releases.json is ordered newest-first, so the oldest release (41.0.0) was pinned and the last week / last months time filters anchored to the wrong revision. Now select the release with the latest git_revision_timestamp among those present in the loaded data, falling back to releases_info[0]. - Pin the Plotly CDN script to 2.35.2 instead of plotly-latest, so a future Plotly major release cannot break the published dashboard. - Merge nightly/out/detected_events.json (written by the new regression detector) into the chart annotations: rendered orange dash-dot with their own 'Show detected changes' toggle, deduplicated by revision with hand-curated events.json taking precedence. A missing or corrupt file degrades to no detected annotations instead of failing the report.
Closes the automation TODOs: the pipeline (build recent commits, run clickbench, regenerate the dashboard) can now run unattended every night from cron or a systemd timer, and regressions are detected statistically instead of by eyeballing charts. New tooling under nightly/ (python is stdlib-only, 3.9+): - nightly.py: orchestrator running build -> bench -> detect -> report -> publish with per-step logs, a portable ownership-checked lockfile, effect assertions (a night where binaries were built but no new revisions landed in results.csv fails loudly instead of reporting a silent green), builds/ pruning, status.json for monitoring, and --dry-run for safe inspection. Publishing is config-gated and off by default, with a branch guard before any commit/rebase/push. - detect.py: two detectors over per-query median series - (1) per-query significance thresholds learned from each query's own historical night-over-night noise (IQR fence, as used by rustc-perf), (2) E-divisive-means changepoint detection with a permutation significance test (as used by MongoDB), pure stdlib. Emits alerts.json / alerts.md and detected_events.json, which report.py renders as dashboard annotations. Exit code 2 signals regressions. Validated against the results_metal history: the collect_statistics change (2d7ae0926) is found as a changepoint on 7 queries. - nightly.sh: cron entry point - flock guard, dirty-tree-aware git pull, failure and regression-alert notifications via a configurable NIGHTLY_NOTIFY_CMD hook. - tune.sh / check_env.sh: benchmark-box tuning (cpu governor, turbo, SMT, ASLR, irqbalance) and pre-run assertions, following LLVM and rustc-perf benchmarking practice. - systemd units, crontab example, and a self-hosted GitHub Actions example (schedule-only, with security caveats documented). - tests: 100 unit/integration tests, including a real-data fixture carved from results_metal. See nightly/README.md for setup and nightly/DESIGN.md for the design.
There was a problem hiding this comment.
Pull request overview
This PR adds a new nightly/ automation pipeline to regularly build and run benchmarks, detect statistically significant performance regressions, and optionally publish updated dashboards—completing the “Automation” TODOs in the top-level README and integrating regression annotations into the report dashboard.
Changes:
- Introduces
nightly/nightly.py+nightly/nightly.shorchestration (lockfile, logging, status.json, optional publish). - Adds stdlib-only statistical regression detection (
nightly/detect.py) with alerts and detected event outputs plus extensive unit/integration tests. - Updates
report.py/ generateddocs/index.htmlto pin Plotly, fix most-recent-release selection, and render/toggle detected regression annotations.
Reviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| report.py | Pins Plotly JS, adds detected-regression annotations/toggle, and adjusts most-recent-release logic. |
| docs/index.html | Regenerated dashboard output reflecting the report changes (Plotly pin + detected toggle + release selection). |
| README.md | Updates Automation/TODO sections to point to nightly/ pipeline docs. |
| nightly/nightly.py | New stdlib-only orchestrator (build → bench → detect → report → optional publish) with locking/logging/status. |
| nightly/detect.py | New stdlib-only regression detection (threshold + e-divisive changepoints) emitting alerts + detected events. |
| nightly/nightly.sh | Cron/systemd entry wrapper with flock guard, optional pull, logging, and notification hook. |
| nightly/tune.sh | Linux benchmark-box tuning helper (governor/turbo/SMT/ASLR/irqbalance) + reset mode. |
| nightly/check_env.sh | Read-only environment assertions for tuned benchmark stability (optional pipeline gate). |
| nightly/config.json | Default nightly configuration (publish off by default). |
| nightly/README.md | Operator manual: install/schedule/configure/run/interpret alerts, plus methodology details. |
| nightly/DESIGN.md | Design rationale and interface/contract documentation for the nightly pipeline. |
| nightly/crontab.example | Example crontab entry for scheduling nightly runs. |
| nightly/systemd/datafusion-nightly.service | Example systemd service unit for running the nightly pipeline. |
| nightly/systemd/datafusion-nightly.timer | Example systemd timer unit to schedule the service daily (UTC). |
| nightly/github-workflow.example.yml | Optional self-hosted Actions workflow example with explicit security caveats. |
| nightly/tests/test_nightly.py | Extensive tests for orchestrator config merge, locking, step dependencies, publish guard, dry-run, etc. |
| nightly/tests/test_detect.py | Detector tests (synthetic + real-data fixture, parsing robustness, determinism, exit codes). |
| nightly/.gitignore | Ignores runtime artifacts for nightly runs while keeping nightly/out/ publishable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1173
to
+1177
| releases_in_data = [rev for rev in releases if rev in rev_ts] | ||
| if releases_in_data: | ||
| most_recent_release = max(releases_in_data, key=lambda rev: pd.to_datetime(rev_ts[rev], utc=True)) | ||
| else: | ||
| most_recent_release = releases_info[0]['revision'] |
Comment on lines
+1171
to
+1172
| except: | ||
| rev_ts = {} |
Comment on lines
+779
to
+780
| with open(detected_path, 'r') as f: | ||
| detected_labels = {item['revision']: item['label'] for item in json.load(f) if item['revision'] not in event_labels} |
Comment on lines
+977
to
+978
| with open(detected_path, 'r') as f: | ||
| detected_labels = {item['revision']: item['label'] for item in json.load(f) if item['revision'] not in event_labels} |
Comment on lines
+102
to
+105
| with open(path, 'r', newline='', encoding='utf-8') as fh: | ||
| reader = csv.reader(fh) | ||
| first = True | ||
| for record in reader: |
- report.py: expose mostRecentReleaseLabel alongside mostRecentRelease so the dashboard's <=30-day annotation filter matches release labels (e.g. "54.0.0") instead of revisions (e.g. "branch-54") - report.py: replace the bare except in load_release_data with Exception plus a warning so fallback reasons show up in generation logs - report.py: open detected_events.json with explicit UTF-8 encoding - nightly/detect.py: make load_results tolerate NUL bytes, undecodable bytes, and oversized fields (csv.Error) so a single corrupted results file cannot abort the nightly run - add regression tests for both corruption modes
vito1317
force-pushed
the
nightly-automation
branch
from
5147c57 to
4fa6e7c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements the two remaining TODOs in the README ("rerun the benchmarks on a regular basis (cron job?)" / "Automation") and adds the analysis layer discussed in apache/datafusion#5504 and the "Trackable over time" goal of apache/datafusion#21165.
What's included (new
nightly/directory, python is stdlib-only, 3.9+):nightly.py— orchestrator: build recent commits (via the existingensure_datafusion_cli.py) → runbenchmark.py→ detect regressions → regenerate the report → optionally commit/push. Ownership-checked lockfile, per-step logs,status.jsonfor monitoring,--dry-run, and effect assertions so a night where binaries were built but no results landed fails loudly instead of reporting green. Publishing is config-gated and off by default, with a branch guard.detect.py— two detectors over per-query median series: (1) per-query significance thresholds learned from each query's own night-over-night noise history (IQR fence, the approach used by rustc-perf), and (2) E-divisive-means changepoint detection with a permutation significance test (the approach used by MongoDB). Emitsalerts.json/alerts.mdanddetected_events.json, whichreport.pynow renders as dashboard annotations. Exit code 2 signals regressions, so cron wrappers can alert.nightly.sh(cron entry, with aNIGHTLY_NOTIFY_CMDhook that fires on failures and on regression alerts), systemd units,tune.sh/check_env.sh(governor/turbo/SMT/ASLR, following LLVM & rustc-perf benchmarking practice), and a schedule-only self-hosted GitHub Actions example with the security caveats documented.python -m unittest discover -s nightly/tests), including a real-data fixture carved fromresults_metal.Validation on real data: running the detector over the
results_metalhistory (383 per-commit revisions) finds thecollect_statisticsdefault change (2d7ae0926, already hand-curated inevents.json) as a statistically significant changepoint on 7 queries — without being told where to look.Also fixes three dashboard issues in
report.py:mostRecentReleasepicked the oldest release (releases.json is newest-first), breaking the "last week/months" filters; the Plotly CDN script is now pinned to 2.35.2 instead ofplotly-latest; detected events degrade gracefully when the JSON is missing or corrupt.See
nightly/README.mdfor setup andnightly/DESIGN.mdfor design rationale. Happy to adjust anything — especially the default detection thresholds and the alert output format.