|
| 1 | +# Per-script smoke timings are a standing dataset — one runner change, ten repos inherit |
| 2 | + |
| 3 | +PyAutoHands#265 → `d2a22f4` (closing PyAutoHands#264) + PyAutoHeart#167 → |
| 4 | +`3df42b5`, merged 2026-08-24. Phase 2 of the test-performance board arc |
| 5 | +(`complete/2026/08/test-performance-board.md`); answers item 4 of |
| 6 | +`draft/research/ci/smoke_timing_and_profiling.md` — "should the runner record |
| 7 | +per-script timings routinely?" — with **yes, shipped**. The smoke-runner |
| 8 | +delegation (#260–#263) is what made this one change instead of ten repo |
| 9 | +sweeps. |
| 10 | + |
| 11 | +## What shipped |
| 12 | + |
| 13 | +- **`smoke_timings.json`** (schema `smoke_timings/1`) emitted from |
| 14 | + `RunReport.write()` — one call site covers `run_python.py`, `run.py` and |
| 15 | + `generate.py`, so every delegated gate inherits it with zero per-repo |
| 16 | + edits. Per entry: `{entry, kind, status, seconds, cap_s, exit_code}` — |
| 17 | + `seconds` is the runner's own measured duration (the number the |
| 18 | + `[PASS] — <n>s` line prints), TIMEOUT entries carry the cap they hit, |
| 19 | + skips are `seconds: null`, never a fabricated 0. One merged file per |
| 20 | + report directory, rows keyed on entry path (script + notebook legs both |
| 21 | + survive; a re-run replaces its own rows; `legs` records contributors). |
| 22 | +- **Step summary**: with `$GITHUB_STEP_SUMMARY` set, a slowest-first timing |
| 23 | + table per leg — every gate run's timings one click away, no artifact |
| 24 | + download. |
| 25 | +- **`ScriptResult` gained `cap_seconds`/`exit_code`** at all 8 execution |
| 26 | + sites — deliberately absent from `to_dict()` so the per-run JSONs Heart's |
| 27 | + `script_timing`/`test_run` and `aggregate_results` read stay |
| 28 | + byte-compatible (pinned by a test); `aggregate_results` skips the sidecar |
| 29 | + by name so the mega-run surface stays clean. |
| 30 | +- **PyAutoHeart `smoke-tests.yml`** uploads the report dir as |
| 31 | + `smoke-timings-<python-version>` (`if: always()`, |
| 32 | + `if-no-files-found: ignore`, no `retention-days` — full default artifact |
| 33 | + retention). The path is a glob (`test-results/` + `**/smoke_timings.json`) |
| 34 | + because the reusable workflow never passes `--report-dir` — each |
| 35 | + workspace's `run_smoke.py` does. |
| 36 | +- Drive-by fix: `run.py` never passed `env_profile`, so every notebook |
| 37 | + report claimed `unknown` — threaded through, negative-tested. |
| 38 | +- Tests: PyAutoHands 363 → 382 passed (14 pre-existing environmental |
| 39 | + failures unchanged — missing `ipynb-py-convert`/`pngquant` locally, |
| 40 | + identical set on main); PyAutoHeart 573 → 576. |
| 41 | + |
| 42 | +## Key traps / findings |
| 43 | + |
| 44 | +- **The notebook report leg is `run.py`, not `run_notebook.py`** — the |
| 45 | + latter is the single-notebook kernel-cwd shim `build_util` shells out to |
| 46 | + and has no report path. Emit from `RunReport.write()` and every leg is |
| 47 | + covered. |
| 48 | +- **A fixed per-leg filename would clobber across directories in the |
| 49 | + mega-run** — one report dir hosts many `(project, directory, run_type)` |
| 50 | + invocations; hence the merged file keyed on entry path. Known limit: two |
| 51 | + workspaces sharing a relative path in one mega-run dir collapse to one |
| 52 | + row (documented in the docstring; that dir is not a consumer). |
| 53 | +- **Keep new fields out of `to_dict()` until every consumer is audited** — |
| 54 | + the per-run JSON shape is load-bearing for three downstream readers; |
| 55 | + extending the dataclass without extending the serialization is the |
| 56 | + compatible move, pinned by a byte-compat test. |
| 57 | +- `aggregate_results` globs `**/*.json` — any new sidecar in the report dir |
| 58 | + becomes a phantom run unless excluded by name. |
| 59 | + |
| 60 | +## Follow-ups |
| 61 | + |
| 62 | +- The Heart board ingesting these artifacts into per-script rows with |
| 63 | + STALL/SLOW verdicts (reuse `retime.py`'s vocabulary; bimodality |
| 64 | + first-class) once a few weeks of data exist — the deferred phase 3 named |
| 65 | + in the board record. |
| 66 | + |
| 67 | +## Original prompt |
| 68 | + |
| 69 | +# Per-script smoke timings as a standing dataset — one runner change, ten repos inherit |
| 70 | + |
| 71 | +Type: feature |
| 72 | +Target: pyautohands |
| 73 | +Repos: |
| 74 | +- @PyAutoHands |
| 75 | +- @PyAutoHeart |
| 76 | +Difficulty: medium |
| 77 | +Autonomy: supervised |
| 78 | +Priority: high |
| 79 | +Status: formalised |
| 80 | +Filed: 2026-08-24 |
| 81 | +Issued: 2026-08-24 |
| 82 | + |
| 83 | +Phase 2 of the test-performance board |
| 84 | +([`../../../docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md); |
| 85 | +phase 1 shipped 2026-08-24 as PyAutoHeart#164 + PyAutoBrain#261). Today the |
| 86 | +smoke runner's per-entry timings exist only as `[PASS] <name> — <n>s` lines in |
| 87 | +job logs, recovered by hand-scraping; the 2026-08-23 slow-vs-stall audit and |
| 88 | +the jax_grad budget work both had to rebuild their datasets that way. Since |
| 89 | +the smoke-runner delegation (PyAutoHands#260–#263) all ten workspace runners |
| 90 | +are thin shims over `autohands/run_python.py` — **so recording per-script |
| 91 | +timings routinely is now one PyAutoHands change, not ten repo sweeps.** This |
| 92 | +answers item 4 of `draft/research/ci/smoke_timing_and_profiling.md` |
| 93 | +("should the runner record per-script timings routinely?") with yes. |
| 94 | + |
| 95 | +## Task |
| 96 | + |
| 97 | +1. **PyAutoHands** — the report machinery (`result_collector.RunReport`, |
| 98 | + already mandatory in the PR gate via `--report-dir`) additionally emits a |
| 99 | + consolidated `smoke_timings.json` in the report dir: one entry per |
| 100 | + script/notebook — `{entry, kind, status, seconds, cap_s (the cap in force |
| 101 | + from build_util.timeout_for), exit_code}` — plus run metadata (project, |
| 102 | + env profile, python version). When `$GITHUB_STEP_SUMMARY` is set, append a |
| 103 | + compact per-entry timing table (slowest first) so every smoke run's |
| 104 | + timings are one click away in the Actions UI with no artifact download. |
| 105 | +2. **PyAutoHeart** — the reusable `smoke-tests.yml` uploads the report dir as |
| 106 | + a run artifact (`smoke-timings-<python-version>`, `if: always()`, |
| 107 | + `if-no-files-found: ignore`), so the dataset persists the full artifact |
| 108 | + retention window for every gate run across all ten repos at once. |
| 109 | +3. Timing must come from the runner's own measurement (the same clock the |
| 110 | + `[PASS] — <n>s` line prints), never re-derived; a TIMEOUT entry records |
| 111 | + the cap it hit. |
| 112 | + |
| 113 | +## Acceptance |
| 114 | + |
| 115 | +- A PR-gate smoke run on any workspace produces `smoke_timings.json` with one |
| 116 | + timed entry per executed script/notebook and a step-summary table, with no |
| 117 | + per-repo changes. |
| 118 | +- TIMEOUT entries carry `cap_s`; skipped entries are absent or explicitly |
| 119 | + marked, never silently timed as 0. |
| 120 | +- Existing report consumers (`run_all.py`, Heart `test_run`/`script_timing`) |
| 121 | + are unaffected. |
| 122 | + |
| 123 | +Follow-up (not this task): the Heart board ingesting these artifacts into |
| 124 | +per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s vocabulary) |
| 125 | +once a few weeks of data exist. |
0 commit comments