From adc6dd7c374e781ddc97e885e178e02410b7af42 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 02:53:59 +0000 Subject: [PATCH 1/2] =?UTF-8?q?prompt:=20file=20issues=20for=20the=20two?= =?UTF-8?q?=20test-performance=20follow-ups=20=E2=86=92=20library-dev?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit script-timing-baselines-fix (PyAutoHeart#165, phase 0: dedup/floor/ migration/provenance) and smoke-timings-dataset (PyAutoHands#264, phase 2: the delegated runner records per-script timings routinely). Both prompts active/, registered in active.md, dashboard regenerated. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EoDPz2LevKeBaDwqFKtZrU --- active.md | 29 ++++++++++ ...ng_baselines_orphaned_and_window_filled.md | 1 + active/smoke_timings_dataset.md | 57 +++++++++++++++++++ dashboard.html | 37 ++++++------ dashboard.md | 52 ++++++++++------- 5 files changed, 136 insertions(+), 40 deletions(-) rename {draft/bug/pyautoheart => active}/script_timing_baselines_orphaned_and_window_filled.md (99%) create mode 100644 active/smoke_timings_dataset.md diff --git a/active.md b/active.md index fb25a165..7cf61bb1 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,34 @@ # Active Tasks +## script-timing-baselines-fix +- issue: https://github.com/PyAutoLabs/PyAutoHeart/issues/165 +- issued: 2026-08-24 +- status: library-dev +- prompt: active/script_timing_baselines_orphaned_and_window_filled.md +- repos: + - PyAutoHeart: claude/test-performance-dashboard-y3fdy7 +- summary: | + Phase 0 of the test-performance board arc: run-identity dedup (the real + cause of the one-value-repeated-7x windows), >=3-distinct-run + classification floor, rename-aware slug migration with loud orphan + counts, per-duration run provenance. Cloud session, no local worktree. + +## smoke-timings-dataset +- issue: https://github.com/PyAutoLabs/PyAutoHands/issues/264 +- issued: 2026-08-24 +- status: library-dev +- prompt: active/smoke_timings_dataset.md +- repos: + - PyAutoHands: claude/test-performance-dashboard-y3fdy7 + - PyAutoHeart: claude/smoke-timings-artifact-y3fdy7 +- summary: | + Phase 2 of the test-performance board arc: the delegated runner emits + smoke_timings.json + a step-summary timing table for every gate run + (one change, ten repos inherit); Heart's reusable smoke-tests.yml + uploads the report dir as a run artifact. Answers item 4 of + draft/research/ci/smoke_timing_and_profiling.md with yes. Cloud + session, no local worktree. + ## transformed-message-factor-gradient-unpack - issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1501 (issued 2026-08-19) - issued: 2026-08-19 diff --git a/draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md b/active/script_timing_baselines_orphaned_and_window_filled.md similarity index 99% rename from draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md rename to active/script_timing_baselines_orphaned_and_window_filled.md index 6ad122d7..2b4a98b6 100644 --- a/draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md +++ b/active/script_timing_baselines_orphaned_and_window_filled.md @@ -9,6 +9,7 @@ Autonomy: supervised Priority: medium Status: formalised Filed: 2026-08-04 (backfilled from git) +Issued: 2026-08-24 Two independent defects in `PyAutoHeart/heart/checks/script_timing.py`, both found while diagnosing the jax_grad smoke timeouts (PyAutoHands#226). Neither is fixed by diff --git a/active/smoke_timings_dataset.md b/active/smoke_timings_dataset.md new file mode 100644 index 00000000..50bdb79c --- /dev/null +++ b/active/smoke_timings_dataset.md @@ -0,0 +1,57 @@ +# Per-script smoke timings as a standing dataset — one runner change, ten repos inherit + +Type: feature +Target: pyautohands +Repos: +- @PyAutoHands +- @PyAutoHeart +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised +Filed: 2026-08-24 +Issued: 2026-08-24 + +Phase 2 of the test-performance board +([`../../../docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md); +phase 1 shipped 2026-08-24 as PyAutoHeart#164 + PyAutoBrain#261). Today the +smoke runner's per-entry timings exist only as `[PASS] — s` lines in +job logs, recovered by hand-scraping; the 2026-08-23 slow-vs-stall audit and +the jax_grad budget work both had to rebuild their datasets that way. Since +the smoke-runner delegation (PyAutoHands#260–#263) all ten workspace runners +are thin shims over `autohands/run_python.py` — **so recording per-script +timings routinely is now one PyAutoHands change, not ten repo sweeps.** This +answers item 4 of `draft/research/ci/smoke_timing_and_profiling.md` +("should the runner record per-script timings routinely?") with yes. + +## Task + +1. **PyAutoHands** — the report machinery (`result_collector.RunReport`, + already mandatory in the PR gate via `--report-dir`) additionally emits a + consolidated `smoke_timings.json` in the report dir: one entry per + script/notebook — `{entry, kind, status, seconds, cap_s (the cap in force + from build_util.timeout_for), exit_code}` — plus run metadata (project, + env profile, python version). When `$GITHUB_STEP_SUMMARY` is set, append a + compact per-entry timing table (slowest first) so every smoke run's + timings are one click away in the Actions UI with no artifact download. +2. **PyAutoHeart** — the reusable `smoke-tests.yml` uploads the report dir as + a run artifact (`smoke-timings-`, `if: always()`, + `if-no-files-found: ignore`), so the dataset persists the full artifact + retention window for every gate run across all ten repos at once. +3. Timing must come from the runner's own measurement (the same clock the + `[PASS] — s` line prints), never re-derived; a TIMEOUT entry records + the cap it hit. + +## Acceptance + +- A PR-gate smoke run on any workspace produces `smoke_timings.json` with one + timed entry per executed script/notebook and a step-summary table, with no + per-repo changes. +- TIMEOUT entries carry `cap_s`; skipped entries are absent or explicitly + marked, never silently timed as 0. +- Existing report consumers (`run_all.py`, Heart `test_run`/`script_timing`) + are unaffected. + +Follow-up (not this task): the Heart board ingesting these artifacts into +per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s vocabulary) +once a few weeks of data exist. diff --git a/dashboard.html b/dashboard.html index 1bf29b17..65c6a3d6 100644 --- a/dashboard.html +++ b/dashboard.html @@ -123,7 +123,7 @@
📋

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-
  • 1In flight
  • 3Parked
  • 6Planned
  • 155Backlog
+
  • 3In flight
  • 3Parked
  • 6Planned
  • 154Backlog

markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

@@ -145,6 +145,8 @@

Quick wins (small enough, and safe enough to run unatte

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

+

Heart script_timing baselines are orphaned by path moves and filled… — issue #165 — issued 2026-08-24 — library-dev

+

Per-script smoke timings as a standing dataset — one runner change, ten… — issue #264 — issued 2026-08-24 — library-dev

Parked markdown version

3 task(s) @@ -163,7 +165,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

155 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

154 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 29

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

@@ -197,12 +199,11 @@

Backlog

Teach repos_sync --write to stamp organ config surfaces✨ featurepyautomindhardsupervisedlow

-bug — 32 +bug — 31

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

Fix release result/sample parameter-path regressions🐛 bughealth_fixestoo-largesupervisedhigh

jax 0.11 breaks beta/gamma message log_partition under jit…🐛 bugautofitsmallsupervisedmedium

-

Heart script_timing baselines are orphaned by path moves and filled…🐛 bugpyautoheartsmallsupervisedmedium

Numba PSF gathers derive the y/x kernel shifts from the wrong kernel…🐛 bugautoarraylowsupervisedmedium

LogGaussianPrior misreports its own support as (-inf, inf)🐛 bugautofitsmallsupervisednormal

autofit.plot functions accept **kwargs and silently discard them🐛 bugautofitsmallsupervisednormal

@@ -325,6 +326,18 @@

Backlog The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from complete/index.md, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.

+ + + + + + + + + + + + @@ -372,13 +385,13 @@

Backlog The reconstruction noise map describes a different estimator than the…

- + - + @@ -612,18 +625,6 @@

Backlog cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of…

- - - - - - - - - - - -
2026-08-24issuedPer-script smoke timings as a standing dataset — one runner change…
2026-08-24issuedHeart script_timing baselines are orphaned by path moves and filled…
2026-08-23 filed pynufft removal: unswept downstream residue (1 hard break + stale…

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index 26e04989..4c00e0bb 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,10 +8,10 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 1 | +| [In flight](#in-flight) (`active/`) | 3 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 155 | +| [Backlog](#backlog) (`draft/`) | 154 | ## Start here @@ -143,6 +143,22 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
+
📋 Heart script_timing baselines are orphaned by path moves and filled… — issue #165 — issued 2026-08-24 — library-dev + +``` +/start_dev active/script_timing_baselines_orphaned_and_window_filled.md +``` + +
+ +
📋 Per-script smoke timings as a standing dataset — one runner change, ten… — issue #264 — issued 2026-08-24 — library-dev + +``` +/start_dev active/smoke_timings_dataset.md +``` + +
+ ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -235,7 +251,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**155** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below. +**154** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
feature — 29 @@ -475,7 +491,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-bug — 32 +bug — 31
📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high @@ -509,14 +525,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-
📋 Heart script_timing baselines are orphaned by path moves and filled… — pyautoheart · small · supervised · medium - -``` -/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md -``` - -
-
📋 Numba PSF gathers derive the y/x kernel shifts from the wrong kernel… — autoarray · low · supervised · medium ``` @@ -1328,6 +1336,8 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| +| 2026-08-24 | issued | Per-script smoke timings as a standing dataset — one runner change… | +| 2026-08-24 | issued | Heart script_timing baselines are orphaned by path moves and filled… | | 2026-08-23 | filed | pynufft removal: unswept downstream residue (1 hard break + stale… | | 2026-08-23 | filed | Properly time and profile the smoke/release script surface | | 2026-08-23 | filed | Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… | @@ -1336,13 +1346,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA… | | 2026-08-22 | filed | Untrack the generated FITS test artifacts in autoarray | | 2026-08-22 | filed | The reconstruction noise map describes a different estimator than the… | -| 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | -| 2026-08-22 | filed | Point-source JSON datasets record no resolution regime |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| +| 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | +| 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… | | 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of… | | 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | @@ -1351,13 +1361,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | | 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | | 2026-08-19 | filed | status.sh --repos sources a file that no longer exists | -| 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit… | -| 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| +| 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit… | +| 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | | 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | @@ -1366,13 +1376,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | RTD organism docs currency: Nerves page, organ-count drift, hands.md… | | 2026-08-19 | filed | PyAutoConf rename leftovers in Brain functional surfaces | | 2026-08-19 | filed | Explore: dashboardify the Brain's operational surfaces with pasteable… | -| 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | -| 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| +| 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | +| 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | @@ -1381,13 +1391,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-16 | filed | LogGaussianPrior misreports its own support as (-inf, inf) | | 2026-08-14 | filed | Three jax_likelihood pins are stale by ~1.24e-4 and fail the smoke… | | 2026-08-09 | found | isothermal-ell-sph-oversampling-at-the-cusp | -| 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | -| 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| +| 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | +| 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | | 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | @@ -1396,8 +1406,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-05 | filed | Give PyAutoFit searches a seed — today no search can be made… | | 2026-08-04 | filed | dataset/imaging/jwst_lw is untracked because the gitignore was never… | | 2026-08-04 | filed | cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of… | -| 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | -| 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos |
From e31c6a6c194da7ee5d5fa2813c806680262da9fc Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 03:12:27 +0000 Subject: [PATCH 2/2] =?UTF-8?q?prompt:=20ship=20the=20two=20test-performan?= =?UTF-8?q?ce=20follow-ups=20=E2=86=92=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit script-timing-baselines-fix (PyAutoHeart#166 ee915f3, closes #165) and smoke-timings-dataset (PyAutoHands#265 d2a22f4 + PyAutoHeart#167 3df42b5, closes PyAutoHands#264). Records under complete/2026/08/, active.md released, index + dashboard regenerated. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EoDPz2LevKeBaDwqFKtZrU --- active.md | 29 ---- active/smoke_timings_dataset.md | 57 -------- .../2026/08/script-timing-baselines-fix.md | 52 ++++++++ complete/2026/08/smoke-timings-dataset.md | 125 ++++++++++++++++++ complete/index.md | 4 +- dashboard.html | 32 +++-- dashboard.md | 38 ++---- 7 files changed, 206 insertions(+), 131 deletions(-) delete mode 100644 active/smoke_timings_dataset.md rename active/script_timing_baselines_orphaned_and_window_filled.md => complete/2026/08/script-timing-baselines-fix.md (55%) create mode 100644 complete/2026/08/smoke-timings-dataset.md diff --git a/active.md b/active.md index 7cf61bb1..fb25a165 100644 --- a/active.md +++ b/active.md @@ -1,34 +1,5 @@ # Active Tasks -## script-timing-baselines-fix -- issue: https://github.com/PyAutoLabs/PyAutoHeart/issues/165 -- issued: 2026-08-24 -- status: library-dev -- prompt: active/script_timing_baselines_orphaned_and_window_filled.md -- repos: - - PyAutoHeart: claude/test-performance-dashboard-y3fdy7 -- summary: | - Phase 0 of the test-performance board arc: run-identity dedup (the real - cause of the one-value-repeated-7x windows), >=3-distinct-run - classification floor, rename-aware slug migration with loud orphan - counts, per-duration run provenance. Cloud session, no local worktree. - -## smoke-timings-dataset -- issue: https://github.com/PyAutoLabs/PyAutoHands/issues/264 -- issued: 2026-08-24 -- status: library-dev -- prompt: active/smoke_timings_dataset.md -- repos: - - PyAutoHands: claude/test-performance-dashboard-y3fdy7 - - PyAutoHeart: claude/smoke-timings-artifact-y3fdy7 -- summary: | - Phase 2 of the test-performance board arc: the delegated runner emits - smoke_timings.json + a step-summary timing table for every gate run - (one change, ten repos inherit); Heart's reusable smoke-tests.yml - uploads the report dir as a run artifact. Answers item 4 of - draft/research/ci/smoke_timing_and_profiling.md with yes. Cloud - session, no local worktree. - ## transformed-message-factor-gradient-unpack - issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1501 (issued 2026-08-19) - issued: 2026-08-19 diff --git a/active/smoke_timings_dataset.md b/active/smoke_timings_dataset.md deleted file mode 100644 index 50bdb79c..00000000 --- a/active/smoke_timings_dataset.md +++ /dev/null @@ -1,57 +0,0 @@ -# Per-script smoke timings as a standing dataset — one runner change, ten repos inherit - -Type: feature -Target: pyautohands -Repos: -- @PyAutoHands -- @PyAutoHeart -Difficulty: medium -Autonomy: supervised -Priority: high -Status: formalised -Filed: 2026-08-24 -Issued: 2026-08-24 - -Phase 2 of the test-performance board -([`../../../docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md); -phase 1 shipped 2026-08-24 as PyAutoHeart#164 + PyAutoBrain#261). Today the -smoke runner's per-entry timings exist only as `[PASS] — s` lines in -job logs, recovered by hand-scraping; the 2026-08-23 slow-vs-stall audit and -the jax_grad budget work both had to rebuild their datasets that way. Since -the smoke-runner delegation (PyAutoHands#260–#263) all ten workspace runners -are thin shims over `autohands/run_python.py` — **so recording per-script -timings routinely is now one PyAutoHands change, not ten repo sweeps.** This -answers item 4 of `draft/research/ci/smoke_timing_and_profiling.md` -("should the runner record per-script timings routinely?") with yes. - -## Task - -1. **PyAutoHands** — the report machinery (`result_collector.RunReport`, - already mandatory in the PR gate via `--report-dir`) additionally emits a - consolidated `smoke_timings.json` in the report dir: one entry per - script/notebook — `{entry, kind, status, seconds, cap_s (the cap in force - from build_util.timeout_for), exit_code}` — plus run metadata (project, - env profile, python version). When `$GITHUB_STEP_SUMMARY` is set, append a - compact per-entry timing table (slowest first) so every smoke run's - timings are one click away in the Actions UI with no artifact download. -2. **PyAutoHeart** — the reusable `smoke-tests.yml` uploads the report dir as - a run artifact (`smoke-timings-`, `if: always()`, - `if-no-files-found: ignore`), so the dataset persists the full artifact - retention window for every gate run across all ten repos at once. -3. Timing must come from the runner's own measurement (the same clock the - `[PASS] — s` line prints), never re-derived; a TIMEOUT entry records - the cap it hit. - -## Acceptance - -- A PR-gate smoke run on any workspace produces `smoke_timings.json` with one - timed entry per executed script/notebook and a step-summary table, with no - per-repo changes. -- TIMEOUT entries carry `cap_s`; skipped entries are absent or explicitly - marked, never silently timed as 0. -- Existing report consumers (`run_all.py`, Heart `test_run`/`script_timing`) - are unaffected. - -Follow-up (not this task): the Heart board ingesting these artifacts into -per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s vocabulary) -once a few weeks of data exist. diff --git a/active/script_timing_baselines_orphaned_and_window_filled.md b/complete/2026/08/script-timing-baselines-fix.md similarity index 55% rename from active/script_timing_baselines_orphaned_and_window_filled.md rename to complete/2026/08/script-timing-baselines-fix.md index 2b4a98b6..64e8036e 100644 --- a/active/script_timing_baselines_orphaned_and_window_filled.md +++ b/complete/2026/08/script-timing-baselines-fix.md @@ -1,3 +1,55 @@ +# script_timing baselines made real — run-identity dedup + rename-aware slugs + +PyAutoHeart#166 → `ee915f3`, closing PyAutoHeart#165, merged 2026-08-24 on +branch `claude/test-performance-dashboard-y3fdy7` (restarted from main after +the phase-1 board merge). Phase 0 of the test-performance board arc +(`complete/2026/08/test-performance-board.md`) — the per-script trend surface +reads what this fixes. + +## What shipped + +- **The "every history is one value repeated 7×" defect was not seeding — it + was re-ingestion.** `run()` re-read `run_logs/latest` on every tick and + re-appended the same observation until the window filled with copies of one + run. History entries now carry `{duration_s, run_id, ts}` (run_id = the + resolved timestamped run dir behind the `latest` symlink); a re-tick on the + same run replaces the newest entry, so windows only grow across distinct + runs. Legacy bare-float histories still read; a legacy window of + all-identical values collapses to the single observation it provably was. +- **Classification floor**: the yellow/red ratio fires only against ≥3 + distinct-run samples (`MIN_BASELINE_RUNS`); below that a script counts as + `building_count` instead of being judged against a fake-stable median. + `red/yellow/green_count` keep their exact meaning; `dashboard.py` untouched. +- **Rename-aware slugs, loud orphans**: history files untouched by a scan are + orphans; a new slug with no history adopts an orphan on an unambiguous + (workspace, script-name) match — the #216 restructure scenario now heals + itself (`migrated_count`). Ambiguous/unmatched orphans are reported + (`orphaned_count`), never deleted, never silent. +- Histories written via `atomic_write_json` (was a bare `write_text`). +- Tests 573 → 583; `test_script_timing.py` 6 → 16. + +## Key traps / findings + +- **Four existing tests had encoded the bug itself** — they re-ran `run()` on + the same results dir as a stand-in for repeated runs, which is exactly the + defect. When a test suite's fixture shape mirrors a bug, the tests pass for + the wrong reason; the fixtures now create one dir per run. +- **Legacy empty run_ids each count as a distinct run** in the floor — history + that predates provenance is real accumulation once the identical-window + collapse has run; treating it as one run would have zeroed every surviving + baseline. +- Orphan reporting is scan-relative: pointing Heart at a partial results dir + surfaces other workspaces' histories as "orphaned" (reporting only — + migration matching is workspace-scoped, so nothing can be misattached). + +## Follow-ups + +- `unit_test_timing` / `workspace_testmode_timing` keep their own history + mechanics — if they share the re-ingestion pattern, the same dedup applies; + not audited here (out of the prompt's scope). + +## Original prompt + # Heart script_timing baselines are orphaned by path moves and filled with one repeated value Type: bug diff --git a/complete/2026/08/smoke-timings-dataset.md b/complete/2026/08/smoke-timings-dataset.md new file mode 100644 index 00000000..9d8472af --- /dev/null +++ b/complete/2026/08/smoke-timings-dataset.md @@ -0,0 +1,125 @@ +# Per-script smoke timings are a standing dataset — one runner change, ten repos inherit + +PyAutoHands#265 → `d2a22f4` (closing PyAutoHands#264) + PyAutoHeart#167 → +`3df42b5`, merged 2026-08-24. Phase 2 of the test-performance board arc +(`complete/2026/08/test-performance-board.md`); answers item 4 of +`draft/research/ci/smoke_timing_and_profiling.md` — "should the runner record +per-script timings routinely?" — with **yes, shipped**. The smoke-runner +delegation (#260–#263) is what made this one change instead of ten repo +sweeps. + +## What shipped + +- **`smoke_timings.json`** (schema `smoke_timings/1`) emitted from + `RunReport.write()` — one call site covers `run_python.py`, `run.py` and + `generate.py`, so every delegated gate inherits it with zero per-repo + edits. Per entry: `{entry, kind, status, seconds, cap_s, exit_code}` — + `seconds` is the runner's own measured duration (the number the + `[PASS] — s` line prints), TIMEOUT entries carry the cap they hit, + skips are `seconds: null`, never a fabricated 0. One merged file per + report directory, rows keyed on entry path (script + notebook legs both + survive; a re-run replaces its own rows; `legs` records contributors). +- **Step summary**: with `$GITHUB_STEP_SUMMARY` set, a slowest-first timing + table per leg — every gate run's timings one click away, no artifact + download. +- **`ScriptResult` gained `cap_seconds`/`exit_code`** at all 8 execution + sites — deliberately absent from `to_dict()` so the per-run JSONs Heart's + `script_timing`/`test_run` and `aggregate_results` read stay + byte-compatible (pinned by a test); `aggregate_results` skips the sidecar + by name so the mega-run surface stays clean. +- **PyAutoHeart `smoke-tests.yml`** uploads the report dir as + `smoke-timings-` (`if: always()`, + `if-no-files-found: ignore`, no `retention-days` — full default artifact + retention). The path is a glob (`test-results/` + `**/smoke_timings.json`) + because the reusable workflow never passes `--report-dir` — each + workspace's `run_smoke.py` does. +- Drive-by fix: `run.py` never passed `env_profile`, so every notebook + report claimed `unknown` — threaded through, negative-tested. +- Tests: PyAutoHands 363 → 382 passed (14 pre-existing environmental + failures unchanged — missing `ipynb-py-convert`/`pngquant` locally, + identical set on main); PyAutoHeart 573 → 576. + +## Key traps / findings + +- **The notebook report leg is `run.py`, not `run_notebook.py`** — the + latter is the single-notebook kernel-cwd shim `build_util` shells out to + and has no report path. Emit from `RunReport.write()` and every leg is + covered. +- **A fixed per-leg filename would clobber across directories in the + mega-run** — one report dir hosts many `(project, directory, run_type)` + invocations; hence the merged file keyed on entry path. Known limit: two + workspaces sharing a relative path in one mega-run dir collapse to one + row (documented in the docstring; that dir is not a consumer). +- **Keep new fields out of `to_dict()` until every consumer is audited** — + the per-run JSON shape is load-bearing for three downstream readers; + extending the dataclass without extending the serialization is the + compatible move, pinned by a byte-compat test. +- `aggregate_results` globs `**/*.json` — any new sidecar in the report dir + becomes a phantom run unless excluded by name. + +## Follow-ups + +- The Heart board ingesting these artifacts into per-script rows with + STALL/SLOW verdicts (reuse `retime.py`'s vocabulary; bimodality + first-class) once a few weeks of data exist — the deferred phase 3 named + in the board record. + +## Original prompt + +# Per-script smoke timings as a standing dataset — one runner change, ten repos inherit + +Type: feature +Target: pyautohands +Repos: +- @PyAutoHands +- @PyAutoHeart +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised +Filed: 2026-08-24 +Issued: 2026-08-24 + +Phase 2 of the test-performance board +([`../../../docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md); +phase 1 shipped 2026-08-24 as PyAutoHeart#164 + PyAutoBrain#261). Today the +smoke runner's per-entry timings exist only as `[PASS] — s` lines in +job logs, recovered by hand-scraping; the 2026-08-23 slow-vs-stall audit and +the jax_grad budget work both had to rebuild their datasets that way. Since +the smoke-runner delegation (PyAutoHands#260–#263) all ten workspace runners +are thin shims over `autohands/run_python.py` — **so recording per-script +timings routinely is now one PyAutoHands change, not ten repo sweeps.** This +answers item 4 of `draft/research/ci/smoke_timing_and_profiling.md` +("should the runner record per-script timings routinely?") with yes. + +## Task + +1. **PyAutoHands** — the report machinery (`result_collector.RunReport`, + already mandatory in the PR gate via `--report-dir`) additionally emits a + consolidated `smoke_timings.json` in the report dir: one entry per + script/notebook — `{entry, kind, status, seconds, cap_s (the cap in force + from build_util.timeout_for), exit_code}` — plus run metadata (project, + env profile, python version). When `$GITHUB_STEP_SUMMARY` is set, append a + compact per-entry timing table (slowest first) so every smoke run's + timings are one click away in the Actions UI with no artifact download. +2. **PyAutoHeart** — the reusable `smoke-tests.yml` uploads the report dir as + a run artifact (`smoke-timings-`, `if: always()`, + `if-no-files-found: ignore`), so the dataset persists the full artifact + retention window for every gate run across all ten repos at once. +3. Timing must come from the runner's own measurement (the same clock the + `[PASS] — s` line prints), never re-derived; a TIMEOUT entry records + the cap it hit. + +## Acceptance + +- A PR-gate smoke run on any workspace produces `smoke_timings.json` with one + timed entry per executed script/notebook and a step-summary table, with no + per-repo changes. +- TIMEOUT entries carry `cap_s`; skipped entries are absent or explicitly + marked, never silently timed as 0. +- Existing report consumers (`run_all.py`, Heart `test_run`/`script_timing`) + are unaffected. + +Follow-up (not this task): the Heart board ingesting these artifacts into +per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s vocabulary) +once a few weeks of data exist. diff --git a/complete/index.md b/complete/index.md index e5032cee..af18ac68 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1095 records across 7 buckets. +1097 records across 7 buckets. ## Highlights @@ -187,6 +187,7 @@ _(curate hard-won records here — survives regeneration.)_ - [samplers-surface-autolens-tiers](2026/08/samplers-surface-autolens-tiers.md) — the samplers faculty's SamplerSurface now scans the findings - [save-json-numpy-scalar-typeerror](2026/08/save-json-numpy-scalar-typeerror.md) — Adds `NumpyEncoder` in `autofit/tools/util.py` (`np.ndarray` -> - [script-size-guard-git-based](2026/08/script-size-guard-git-based.md) — replaced the rotting `.script_sizes.json` snapshot with a git-diff truncation +- [script-timing-baselines-fix](2026/08/script-timing-baselines-fix.md) - [script-title-underline-off-by-one](2026/08/script-title-underline-off-by-one.md) - [searches-readme-dashboard](2026/08/searches-readme-dashboard.md) - [should-simulate-capped-branch-reuse](2026/08/should-simulate-capped-branch-reuse.md) — small, single-repo follow-up of the closed PyAutoNerves#153 @@ -196,6 +197,7 @@ _(curate hard-won records here — survives regeneration.)_ - [small-datasets-rmtree-committed-data](2026/08/small-datasets-rmtree-committed-data.md) - [smoke-runner-delegation](2026/08/smoke-runner-delegation.md) - [smoke-runner-jupyter-guard](2026/08/smoke-runner-jupyter-guard.md) +- [smoke-timings-dataset](2026/08/smoke-timings-dataset.md) - [spawn-autonomy-log-generated](2026/08/spawn-autonomy-log-generated.md) - [spawn-drift-self-heal](2026/08/spawn-drift-self-heal.md) - [spawn-empty-body-privacy-fix](2026/08/spawn-empty-body-privacy-fix.md) diff --git a/dashboard.html b/dashboard.html index 65c6a3d6..c4d18319 100644 --- a/dashboard.html +++ b/dashboard.html @@ -123,7 +123,7 @@
📋

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-
  • 3In flight
  • 3Parked
  • 6Planned
  • 154Backlog
+
  • 1In flight
  • 3Parked
  • 6Planned
  • 154Backlog

markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

@@ -145,8 +145,6 @@

Quick wins (small enough, and safe enough to run unatte

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

-

Heart script_timing baselines are orphaned by path moves and filled… — issue #165 — issued 2026-08-24 — library-dev

-

Per-script smoke timings as a standing dataset — one runner change, ten… — issue #264 — issued 2026-08-24 — library-dev

Parked markdown version

3 task(s) @@ -326,18 +324,6 @@

Backlog The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from complete/index.md, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.

- - - - - - - - - - - - @@ -385,13 +371,13 @@

Backlog The reconstruction noise map describes a different estimator than the…

- + - + @@ -625,6 +611,18 @@

Backlog cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of…

+ + + + + + + + + + + +
2026-08-24issuedPer-script smoke timings as a standing dataset — one runner change…
2026-08-24issuedHeart script_timing baselines are orphaned by path moves and filled…
2026-08-23 filed pynufft removal: unswept downstream residue (1 hard break + stale…
2026-08-22 filed Remove pynufft + legacy TransformerNUFFTPyNUFFT
2026-08-22 filed Point-source JSON datasets record no resolution regime

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index 4c00e0bb..c653c2fd 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,7 +8,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 3 | +| [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | | [Backlog](#backlog) (`draft/`) | 154 | @@ -143,22 +143,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
-
📋 Heart script_timing baselines are orphaned by path moves and filled… — issue #165 — issued 2026-08-24 — library-dev - -``` -/start_dev active/script_timing_baselines_orphaned_and_window_filled.md -``` - -
- -
📋 Per-script smoke timings as a standing dataset — one runner change, ten… — issue #264 — issued 2026-08-24 — library-dev - -``` -/start_dev active/smoke_timings_dataset.md -``` - -
- ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -1336,8 +1320,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| -| 2026-08-24 | issued | Per-script smoke timings as a standing dataset — one runner change… | -| 2026-08-24 | issued | Heart script_timing baselines are orphaned by path moves and filled… | | 2026-08-23 | filed | pynufft removal: unswept downstream residue (1 hard break + stale… | | 2026-08-23 | filed | Properly time and profile the smoke/release script surface | | 2026-08-23 | filed | Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… | @@ -1346,13 +1328,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA… | | 2026-08-22 | filed | Untrack the generated FITS test artifacts in autoarray | | 2026-08-22 | filed | The reconstruction noise map describes a different estimator than the… | +| 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | +| 2026-08-22 | filed | Point-source JSON datasets record no resolution regime |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| -| 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | -| 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… | | 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of… | | 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | @@ -1361,13 +1343,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | | 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | | 2026-08-19 | filed | status.sh --repos sources a file that no longer exists | +| 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit… | +| 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| -| 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit… | -| 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | | 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | @@ -1376,13 +1358,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | RTD organism docs currency: Nerves page, organ-count drift, hands.md… | | 2026-08-19 | filed | PyAutoConf rename leftovers in Brain functional surfaces | | 2026-08-19 | filed | Explore: dashboardify the Brain's operational surfaces with pasteable… | +| 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | +| 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| -| 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | -| 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | @@ -1391,13 +1373,13 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-16 | filed | LogGaussianPrior misreports its own support as (-inf, inf) | | 2026-08-14 | filed | Three jax_likelihood pins are stale by ~1.24e-4 and fail the smoke… | | 2026-08-09 | found | isothermal-ell-sph-oversampling-at-the-cusp | +| 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | +| 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| -| 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | -| 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | | 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | @@ -1406,6 +1388,8 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-05 | filed | Give PyAutoFit searches a seed — today no search can be made… | | 2026-08-04 | filed | dataset/imaging/jwst_lw is untracked because the gitignore was never… | | 2026-08-04 | filed | cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of… | +| 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | +| 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos |