diff --git a/active.md b/active.md index 6b188a9d..fb25a165 100644 --- a/active.md +++ b/active.md @@ -13,15 +13,3 @@ - registered: 2026-08-19 by the wake_up session — the issuing session (claude/autofit-priors-messages-audit-ylvenv) filed the prompt + issue but not this entry, tripping Lifecycle Drift on main. - repos-none-claimed: this entry claims NO repos — one line deliberately, not 2-space bullets. - -## jax-stall-shared-preloads-retime-refutation -- issue: https://github.com/PyAutoLabs/PyAutoMind/issues/328 (issued 2026-08-25) -- issued: 2026-08-25 -- prompt: active/jax_stall_shared_preloads_retime_refutation.md -- epic: jax-compile-stall (evidence-recording only; not a phase) -- status: library-dev -- worktree: ~/Code/PyAutoLabs-wt/jax-stall-shared-preloads-retime-refutation -- repos: - - PyAutoMind (branch claude/weekly-smoke-timings-naming-012fuj) -- note: web-github session — no local worktree created. Mind-only record change; parks nothing. - diff --git a/active/jax_stall_shared_preloads_retime_refutation.md b/complete/2026/08/jax-stall-shared-preloads-retime-refutation.md similarity index 56% rename from active/jax_stall_shared_preloads_retime_refutation.md rename to complete/2026/08/jax-stall-shared-preloads-retime-refutation.md index 77e759f2..eebdf68a 100644 --- a/active/jax_stall_shared_preloads_retime_refutation.md +++ b/complete/2026/08/jax-stall-shared-preloads-retime-refutation.md @@ -1,3 +1,78 @@ +# A retime refuted a stall marker; one day later the script stalled + +PyAutoMind#329 → `5eeee59` (closing PyAutoMind#328), merged 2026-08-25. +Evidence-recording task against the `jax-compile-stall` ledger. Filed as a +one-line follow-up ("add `shared_preloads.py` to the family list") from +`complete/2026/08/weekly-smoke-timings-naming.md`; the investigation found +something materially larger. + +## What shipped + +- **`draft/bug/ci/jax_vmap_jit_compile_stall.md`** gained a + `## New occurrence — 2026-08-25` section, mirroring the 2026-08-24 precedent: + run/job ids, a ten-row sibling timing table, the retime refutation, the + exclusion-list disagreement, the widened affected-set shape, and the channel + finding. +- **`epics.md`** — the `jax-compile-stall` entry carries a + `NEW EVIDENCE 2026-08-25` note so the epic's state reflects it. +- Nothing parked, disabled, or re-marked. `autolens_workspace_test` read for + evidence only. + +## The finding + +`multi_dataset/jax_likelihood/shared_preloads.py` hit `TIMEOUT (300s)` in +PyAutoHeart `Workspace Smoke` run **32902243623** (job **97978549465**, +2026-08-25) while all nine siblings passed in 8.6–49.3s — a >6x outlier with the +bimodal signature the ledger uses to separate a stall from slowness. + +**The 2026-08-24 retime had removed this entry's SLOW marker and returned it to +coverage the day before.** Verified against `autolens_workspace_test` at +`7fc497d`: absent from `config/build/no_run.yaml` (in mega-run/weekly coverage), +still commented out at `smoke_tests.txt` line 15 (out of the PR gate). + +## Key traps / findings + +- **N=5 cannot refute a bimodal failure.** The retime (5 repeats per leg, 300s + cap; runs 32741371308 + 32741386752) removed eight PyAutoHeart#74 entries on + "completed 5/5 on both legs, slowest 54.5s — marker refuted". ~10 executions + measure the fast mode and say nothing about the tail. The retime was right + that the script is not *slow*; it was not powered to see that it *stalls*. + **The other seven entries that sweep readmitted carry identical uncertainty** + and are in the weekly channel now — the standing follow-up below. +- **The two exclusion lists disagree, and different gates read different ones.** + `smoke_tests.txt` gates the PR run; `no_run.yaml` gates mega-run/weekly. A + script disabled since 2026-07-22 was therefore still able to burn a 300s cap + in CI. Resolve deliberately rather than by whichever file is edited next. +- **First occurrence via the weekly `workspace-validation` channel.** Every + prior one came from a workspace repo's own `Smoke Tests` gate or from + `release-integrate`. A different, PyAutoHeart-owned, `script_matrix.py`-driven + harness reproducing the same signature is cross-harness corroboration that the + stall is not one repo's runner configuration. +- **The common factor is wider than MGE.** `shared_preloads.py` is not an MGE + variant, so the ledger's "composite MGE vmap graph" shape no longer covers the + affected set. What the members share is heavy `multi_dataset` vmap + composition. +- **A one-line follow-up note was not the whole finding.** The note said "add it + to the family list". Checking the workspace's actual exclusion state — rather + than trusting the note — is what surfaced the refutation. Cheap to do, and it + changed the task's meaning. + +## Follow-ups + +- **Re-check the seven other entries the 2026-08-24 sweep readmitted.** One of + eight hung within a day; the rest were cleared by the same under-powered + protocol and sit in weekly coverage. Now cheaper to settle than before: the + weekly `smoke-timings-*` dataset (shipped same day, PyAutoHeart#182) makes + per-script weekly timings globbable, so a few weeks of data answers it without + a bespoke sweep. +- **Reconcile `smoke_tests.txt` vs `no_run.yaml`** for the entries where they + disagree — a decision, not code. +- `pyauto-brain bug` returns `owner: unresolved` / `fix locus: unresolved` for a + prompt whose only repo is `@PyAutoMind`, because Mind is absent from its + library-repo owner map. Cosmetic; classification was otherwise sound. + +## Original prompt + # The 2026-08-24 retime returned `shared_preloads.py` to coverage; it stalled the next day Type: bug diff --git a/complete/index.md b/complete/index.md index d6751a34..63edd867 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1139 records across 7 buckets. +1140 records across 7 buckets. ## Highlights @@ -122,6 +122,7 @@ _(curate hard-won records here — survives regeneration.)_ - [jax-grad-local-vs-ci-assertions](2026/08/jax-grad-local-vs-ci-assertions.md) - [jax-grad-smoke-timeout-budget](2026/08/jax-grad-smoke-timeout-budget.md) - [jax-likelihood-mass-sensitivity](2026/08/jax-likelihood-mass-sensitivity.md) — Audit + fix for median-literal mass blindness across all jax_likelihood/datacube scripts (phase 2 of the findi… +- [jax-stall-shared-preloads-retime-refutation](2026/08/jax-stall-shared-preloads-retime-refutation.md) - [knowledge-board](2026/08/knowledge-board.md) — auto-closed on merge - [lacosmic-cr-option-and-star-pass-decoupling](2026/08/lacosmic-cr-option-and-star-pass-decoupling.md) — both deliberately OPEN — closure is the default-flip decision, not this ship - [latex-docstrings-invalid-escape-warnings](2026/08/latex-docstrings-invalid-escape-warnings.md) diff --git a/dashboard.html b/dashboard.html index 3b6eb0d6..401dcefc 100644 --- a/dashboard.html +++ b/dashboard.html @@ -185,7 +185,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- +

Last updated 2026-08-25. This page is generated from active/, draft/ and the registry files, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to complete/ keeps rendering here as pickable backlog. Reconciling those is the refresh below.

@PyAutoFit TransformedMessage.factor_gradient crashes on first callissue #1501 — issued 2026-08-19HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

-

The 2026-08-24 retime returned shared_preloads.py to coverage; it stalled the next dayissue #328 — issued 2026-08-25library-dev

Parked markdown version

3 task(s) @@ -395,12 +394,6 @@

Backlog The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from complete/index.md, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.

- - - - - - @@ -454,7 +447,7 @@

Backlog pynufft removal: unswept downstream residue (1 hard break + stale…

- + @@ -694,6 +687,12 @@

Backlog interferometer/start_here.py OOM in nightly release-validation…

+ + + + + +
2026-08-25issuedThe 2026-08-24 retime returned shared_preloads.py to coverage; it…
2026-08-24 filed wiki-currency's --check-version gate rots on every library main merge
2026-08-23 filed Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI…

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index bf9ff131..4b9aed47 100644 --- a/dashboard.md +++ b/dashboard.md @@ -42,7 +42,7 @@ anything you could not verify. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 2 | +| [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 5 | | [Backlog](#backlog) (`draft/`) | 137 | @@ -163,14 +163,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
-
📋 The 2026-08-24 retime returned shared_preloads.py to coverage; it stalled the next dayissue #328 — issued 2026-08-25 — library-dev - -``` -/start_dev active/jax_stall_shared_preloads_retime_refutation.md -``` - -
- ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -1204,7 +1196,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| -| 2026-08-25 | issued | The 2026-08-24 retime returned shared_preloads.py to coverage; it… | | 2026-08-24 | filed | wiki-currency's --check-version gate rots on every library main merge | | 2026-08-24 | filed | interferometer/jax_grad/gradient.py: eager and jitted likelihoods… | | 2026-08-24 | filed | euclid: CRLF has reached the HPC submit scripts AGENTS.md warns about | @@ -1214,12 +1205,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-24 | filed | Induct PyAutoReduce into the PyAutoHands release machinery (date… | | 2026-08-24 | filed | Heart's local smoke runner cannot run any CTI workspace — no autocti… | | 2026-08-23 | filed | pynufft removal: unswept downstream residue (1 hard break + stale… | +| 2026-08-23 | filed | Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| -| 2026-08-23 | filed | Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right jax… | | 2026-08-22 | filed | Untrack the generated FITS test artifacts in autoarray | @@ -1229,12 +1220,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | | 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | +| 2026-08-19 | filed | status.sh --repos sources a file that no longer exists |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| -| 2026-08-19 | filed | status.sh --repos sources a file that no longer exists | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | @@ -1244,12 +1235,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | +| 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| -| 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | | 2026-08-16 | filed | LogGaussianPrior misreports its own support as (-inf, inf) | @@ -1259,12 +1250,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | +| 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| -| 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead | | 2026-08-06 | filed | Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two… | | 2026-08-05 | filed | Give PyAutoFit searches a seed — today no search can be made… | | 2026-08-04 | filed | dataset/imaging/jwst_lw is untracked because the gitignore was never… | @@ -1274,6 +1265,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | HowToLens ch4 tutorial 3: mask overlay is never actually drawn | | 2026-08-03 | filed | Un-park imaging/features/scaling_relation/slam — the PyAutoArray#431… | | 2026-07-31 | filed | interferometer/start_here.py OOM in nightly release-validation… | +| 2026-07-31 | filed | Quick-update plotting cost — minutes per update, and it is not JAX… |