diff --git a/active.md b/active.md index fb25a165..6b188a9d 100644 --- a/active.md +++ b/active.md @@ -13,3 +13,15 @@ - registered: 2026-08-19 by the wake_up session — the issuing session (claude/autofit-priors-messages-audit-ylvenv) filed the prompt + issue but not this entry, tripping Lifecycle Drift on main. - repos-none-claimed: this entry claims NO repos — one line deliberately, not 2-space bullets. + +## jax-stall-shared-preloads-retime-refutation +- issue: https://github.com/PyAutoLabs/PyAutoMind/issues/328 (issued 2026-08-25) +- issued: 2026-08-25 +- prompt: active/jax_stall_shared_preloads_retime_refutation.md +- epic: jax-compile-stall (evidence-recording only; not a phase) +- status: library-dev +- worktree: ~/Code/PyAutoLabs-wt/jax-stall-shared-preloads-retime-refutation +- repos: + - PyAutoMind (branch claude/weekly-smoke-timings-naming-012fuj) +- note: web-github session — no local worktree created. Mind-only record change; parks nothing. + diff --git a/active/jax_stall_shared_preloads_retime_refutation.md b/active/jax_stall_shared_preloads_retime_refutation.md new file mode 100644 index 00000000..77e759f2 --- /dev/null +++ b/active/jax_stall_shared_preloads_retime_refutation.md @@ -0,0 +1,119 @@ +# The 2026-08-24 retime returned `shared_preloads.py` to coverage; it stalled the next day + +Type: bug +Target: ci +Repos: +- @PyAutoMind +Epic: jax-compile-stall +Difficulty: small +Autonomy: safe +Priority: medium +Status: issued +Filed: 2026-08-25 +Issued: 2026-08-25 + +Evidence-recording task against the jax-stall ledger +[`jax_vmap_jit_compile_stall.md`](jax_vmap_jit_compile_stall.md). Surfaced +2026-08-25 by the `weekly-smoke-timings-naming` verification sweep +(PyAutoHeart#182), which dispatched a full `workspace-smoke.yml` run and turned +up a stall the ledger does not know about. + +## What was observed + +PyAutoHeart `Workspace Smoke` run **32902243623** (2026-08-25, `workflow_dispatch`), +job `smoke / run_scripts (3.12, autolens_test, multi_dataset)` (97978549465). +16 of 17 scripts passed: + +``` +scripts/multi_dataset/jax_likelihood/shared_preloads.py ... TIMEOUT (300s) +``` + +Its siblings in the same job, same commit, same runner: + +| Script | Result | +|---|---| +| `jax_likelihood/mge.py` | PASS 11.7s | +| `jax_likelihood/rectangular_mge.py` | PASS 20.7s | +| `jax_likelihood/mge_group.py` | PASS 49.3s | +| `jax_likelihood/delaunay_mge.py` | PASS 20.2s | +| `jax_likelihood/rectangular.py` | PASS 15.8s | +| `jax_likelihood/shared_preloads.py` | **TIMEOUT (300s)** | + +A >6x outlier against the slowest sibling, with every other member of the family +completing comfortably — the bimodal signature the ledger already uses to +separate a stall from slowness. + +## Why this is not just another occurrence + +**The 2026-08-24 retime refuted this entry's SLOW marker and put it back in +coverage. It hung on the next weekly run.** + +`shared_preloads.py` sat in the PyAutoHeart#74 "flakes at the 1800s cap" SLOW +family. On 2026-08-24 that family was retimed (5 repeats per Python leg, 300s +cap; runs 32741371308 + 32741386752) and eight entries "completed 5/5 on both +legs (slowest 54.5s) and were removed — marker refuted, scripts back in mega-run +coverage" (`autolens_workspace_test/config/build/no_run.yaml`, and +`complete/2026/08/smoke-surface-retime-sweep.md`). + +Verified 2026-08-25 against `autolens_workspace_test` at `7fc497d`: + +- `config/build/no_run.yaml` — **no `shared_preloads` entry**. It is in mega-run + and weekly coverage. +- `smoke_tests.txt` line 15 — still **commented out**, "disabled 2026-07-22: + exceeds the 300s smoke cap (measured 300s+ in CI, autolens_workspace_test#196) + … it is the heaviest entry here and does not belong in the fast PR gate." + +So the two exclusion lists now disagree about this script, and the list that +readmitted it did so on ~10 executions at ≤54.5s. **A 10-execution sample at +~50s cannot exclude a low-probability hang** — which is exactly the claim the +retime's "marker refuted" conclusion rests on. The retime was right that the +script is not *slow*; it was not powered to see that it *stalls*. + +That is the ledger's own phase-2 question (§ Task step 1: "a slow script has a +tight timing distribution, a stalling one is bimodal") landing on the other side +from where the retime put it. + +## Also new: the channel + +Every occurrence in the ledger comes from a workspace repo's own `Smoke Tests` +gate or from `release-integrate`. This is its **first from the weekly +`workspace-validation.yml` sweep** — a different harness (PyAutoHeart-owned, +`script_matrix.py`-driven, honouring `no_run.yaml` and *not* `smoke_tests.txt`) +reproducing the same signature. That is corroboration the stall is not an +artifact of one repo's runner configuration. + +It also means the weekly channel runs scripts the PR gate deliberately excludes, +which is how a script disabled in `smoke_tests.txt` since July was still able to +burn a 300s cap in CI. + +## Task + +Record this in the ledger — do not park the script. + +1. Append a `## New occurrence — 2026-08-25, multi_dataset/jax_likelihood/shared_preloads.py` + section to [`jax_vmap_jit_compile_stall.md`](jax_vmap_jit_compile_stall.md), + in the shape of the existing § "New occurrence — 2026-08-24" section: run and + job ids, the sibling timing table, and the affected-set consequence. +2. State the retime refutation explicitly, so phase 2's written answer is not + left contradicting the evidence: this entry was readmitted to coverage on a + sample too small to exclude a stall, and the readmission is now falsified. +3. Extend the ledger's affected-set shape to name `shared_preloads.py`, and note + the `smoke_tests.txt` / `no_run.yaml` disagreement about it. +4. Add the "first occurrence on the weekly `workspace-validation` channel" point, + with the cross-harness corroboration it gives. + +**Deliberately out of scope:** re-adding `shared_preloads.py` to `no_run.yaml`. +The ledger's own § "This is why quarantining is the wrong end state" argues +against whack-a-mole parking that strips coverage of the heaviest JAX paths, and +the 2026-08-24 precedent for `multi_dataset/jax_likelihood/mge.py` was to record +and deliberately **not** park. Any re-marking is phase 3's call with a root cause +in hand, not this task's. + +## Acceptance + +- The ledger names `shared_preloads.py` in its affected set, with the run/job + evidence and the sibling timings. +- The 2026-08-24 retime refutation for this entry is recorded in writing, so a + reader of phase 2's conclusion sees the counter-evidence beside it. +- No script is parked, disabled, or re-marked by this task. +- `lifecycle.py check` clean; PyAutoMind is the only repo touched. diff --git a/dashboard.html b/dashboard.html index 401dcefc..3b6eb0d6 100644 --- a/dashboard.html +++ b/dashboard.html @@ -185,7 +185,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- +

Last updated 2026-08-25. This page is generated from active/, draft/ and the registry files, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to complete/ keeps rendering here as pickable backlog. Reconciling those is the refresh below.

@PyAutoFit TransformedMessage.factor_gradient crashes on first callissue #1501 — issued 2026-08-19HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

+

The 2026-08-24 retime returned shared_preloads.py to coverage; it stalled the next dayissue #328 — issued 2026-08-25library-dev

Parked markdown version

3 task(s) @@ -394,6 +395,12 @@

Backlog The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from complete/index.md, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.

+ + + + + + @@ -447,7 +454,7 @@

Backlog pynufft removal: unswept downstream residue (1 hard break + stale…

- + @@ -687,12 +694,6 @@

Backlog interferometer/start_here.py OOM in nightly release-validation…

- - - - - -
2026-08-25issuedThe 2026-08-24 retime returned shared_preloads.py to coverage; it…
2026-08-24 filed wiki-currency's --check-version gate rots on every library main merge

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index 4b9aed47..bf9ff131 100644 --- a/dashboard.md +++ b/dashboard.md @@ -42,7 +42,7 @@ anything you could not verify. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 1 | +| [In flight](#in-flight) (`active/`) | 2 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 5 | | [Backlog](#backlog) (`draft/`) | 137 | @@ -163,6 +163,14 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
+
📋 The 2026-08-24 retime returned shared_preloads.py to coverage; it stalled the next dayissue #328 — issued 2026-08-25 — library-dev + +``` +/start_dev active/jax_stall_shared_preloads_retime_refutation.md +``` + +
+ ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -1196,6 +1204,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| +| 2026-08-25 | issued | The 2026-08-24 retime returned shared_preloads.py to coverage; it… | | 2026-08-24 | filed | wiki-currency's --check-version gate rots on every library main merge | | 2026-08-24 | filed | interferometer/jax_grad/gradient.py: eager and jitted likelihoods… | | 2026-08-24 | filed | euclid: CRLF has reached the HPC submit scripts AGENTS.md warns about | @@ -1205,12 +1214,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-24 | filed | Induct PyAutoReduce into the PyAutoHands release machinery (date… | | 2026-08-24 | filed | Heart's local smoke runner cannot run any CTI workspace — no autocti… | | 2026-08-23 | filed | pynufft removal: unswept downstream residue (1 hard break + stale… | -| 2026-08-23 | filed | Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| +| 2026-08-23 | filed | Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right jax… | | 2026-08-22 | filed | Untrack the generated FITS test artifacts in autoarray | @@ -1220,12 +1229,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | | 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | -| 2026-08-19 | filed | status.sh --repos sources a file that no longer exists |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| +| 2026-08-19 | filed | status.sh --repos sources a file that no longer exists | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | @@ -1235,12 +1244,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | -| 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| +| 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | | 2026-08-16 | filed | LogGaussianPrior misreports its own support as (-inf, inf) | @@ -1250,12 +1259,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | -| 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| +| 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead | | 2026-08-06 | filed | Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two… | | 2026-08-05 | filed | Give PyAutoFit searches a seed — today no search can be made… | | 2026-08-04 | filed | dataset/imaging/jwst_lw is untracked because the gitignore was never… | @@ -1265,7 +1274,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | HowToLens ch4 tutorial 3: mask overlay is never actually drawn | | 2026-08-03 | filed | Un-park imaging/features/scaling_relation/slam — the PyAutoArray#431… | | 2026-07-31 | filed | interferometer/start_here.py OOM in nightly release-validation… | -| 2026-07-31 | filed | Quick-update plotting cost — minutes per update, and it is not JAX… |
diff --git a/draft/bug/ci/jax_vmap_jit_compile_stall.md b/draft/bug/ci/jax_vmap_jit_compile_stall.md index 6b445412..629cd9e5 100644 --- a/draft/bug/ci/jax_vmap_jit_compile_stall.md +++ b/draft/bug/ci/jax_vmap_jit_compile_stall.md @@ -244,3 +244,97 @@ was re-run at ~14:00 UTC to clear the gate; the script was deliberately **not** parked, per #109's own warning against whack-a-mole parking that strips coverage of the heaviest JAX paths. The per-script cap behaved exactly as designed: a 300s TIMEOUT and exit 1 rather than a six-hour silent hang. + +## New occurrence — 2026-08-25, `multi_dataset/jax_likelihood/shared_preloads.py` + +**This one refutes a refutation.** The entry had its SLOW marker removed the day +before and was returned to coverage; it stalled on the next weekly run. + +Verified via the Actions API. PyAutoHeart `Workspace Smoke` run **32902243623** +(2026-08-25, `workflow_dispatch` on the `weekly-smoke-timings-naming` branch), +job `smoke / run_scripts (3.12, autolens_test, multi_dataset)` (**97978549465**). +16 of 17 scripts passed: + +``` +scripts/multi_dataset/jax_likelihood/shared_preloads.py ... TIMEOUT (300s) +``` + +Siblings in the same job, same commit, same runner: + +| Script | Result | +|---|---| +| `jax_likelihood/mge.py` | PASS 11.7s | +| `jax_likelihood/rectangular_mge.py` | PASS 20.7s | +| `jax_likelihood/mge_group.py` | PASS 49.3s | +| `jax_likelihood/delaunay_mge.py` | PASS 20.2s | +| `jax_likelihood/rectangular.py` | PASS 15.8s | +| `jax_likelihood/rectangular_mge_rtu.py` | PASS 21.9s | +| `jax_likelihood/rectangular_rtu.py` | PASS 16.3s | +| `jax_likelihood/lp.py` | PASS 8.6s | +| `jax_likelihood/dataset_model.py` | PASS 12.9s | +| **`jax_likelihood/shared_preloads.py`** | **TIMEOUT (300s)** | + +A >6x outlier against the slowest sibling, with every other family member +completing comfortably — the bimodal signature § "It is not one script" describes. + +### The 2026-08-24 retime put this script back in coverage + +State verified 2026-08-25 against `autolens_workspace_test` at `7fc497d`: + +- `config/build/no_run.yaml` — **no `shared_preloads` entry.** It is in mega-run + and weekly coverage. +- `smoke_tests.txt` line 15 — still commented out: *"disabled 2026-07-22: exceeds + the 300s smoke cap (measured 300s+ in CI, autolens_workspace_test#196) … it is + the heaviest entry here and does not belong in the fast PR gate."* + +`shared_preloads.py` was in the PyAutoHeart#74 "flakes at the 1800s cap" SLOW +family. The 2026-08-24 retime (5 repeats per Python leg, 300s cap; runs +32741371308 + 32741386752) removed eight of those entries because they "completed +5/5 on both legs (slowest 54.5s) and were removed — marker refuted, scripts back +in mega-run coverage". + +**~10 executions at ≤54.5s cannot exclude a low-probability hang.** The retime +was right that this script is not *slow*; it was not powered to see that it +*stalls*. Its readmission to coverage is falsified by the very next weekly run. + +This lands § Task step 1's slow-vs-stall question on the opposite side from where +the retime put this entry, and it is a caution about the retime protocol +generally: for a bimodal failure, N=5 per leg measures the fast mode and says +nothing about the tail. Any other entry readmitted by that sweep carries the same +uncertainty. + +### The two exclusion lists now disagree + +`smoke_tests.txt` (PR gate) excludes `shared_preloads.py` as too heavy; +`no_run.yaml` (mega-run / weekly) admits it. The gates read different lists, so a +script disabled since July was still able to burn a 300s cap in CI. Worth +resolving deliberately rather than by whichever list is edited next. + +### First occurrence on the weekly `workspace-validation` channel + +Every prior occurrence in this ledger comes from a workspace repo's own +`Smoke Tests` gate or from `release-integrate`. This is the first via +PyAutoHeart's weekly `workspace-validation.yml` sweep — a different harness +(PyAutoHeart-owned, `script_matrix.py`-driven, honouring `no_run.yaml` and not +`smoke_tests.txt`) reproducing the same signature. That is cross-harness +corroboration that the stall is not an artifact of one repo's runner +configuration. + +### Affected-set shape + +Extends the family to `multi_dataset/jax_likelihood/shared_preloads.py`, joining +`multi_dataset/` `mge.py` (2026-08-24) and `rectangular.py` (parked 2026-08-01, +autolens_workspace_test#245), and `imaging/` `mge_group.py` + `rectangular_mge.py` +(parked 2026-08-23, autogalaxy_workspace_test#109). Note `shared_preloads.py` is +not an MGE variant — it is the shared-preloads path — so the "composite MGE vmap +graph" common factor no longer covers the whole set. What the members share is +heavy `multi_dataset` vmap composition, not MGE specifically. + +**Consequence — nothing was parked.** Recorded and left in coverage, following +this file's § "This is why quarantining is the wrong end state" and the +2026-08-24 precedent, where `multi_dataset/jax_likelihood/mge.py` was deliberately +not parked. Re-marking is phase 3's call with a root cause in hand. + +Surfaced incidentally by `complete/2026/08/weekly-smoke-timings-naming.md` +(PyAutoHeart#182), whose verification sweep produced this run — the first +practical dividend of the weekly timing dataset that task shipped. diff --git a/epics.md b/epics.md index 1b883865..0898eada 100644 --- a/epics.md +++ b/epics.md @@ -32,6 +32,14 @@ epic, never picked standalone. - title: Intermittent XLA compile stall in the JAX vmap likelihood path - ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md - status: CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md +- NEW EVIDENCE 2026-08-25: multi_dataset/jax_likelihood/shared_preloads.py stalled at TIMEOUT (300s) in + PyAutoHeart Workspace Smoke run 32902243623 — one day after the 2026-08-24 retime refuted its SLOW + marker and returned it to mega-run coverage. N=5 per leg measures the fast mode of a bimodal failure + and says nothing about the tail, so every entry that sweep readmitted carries the same uncertainty. + Also the epic's first occurrence via the weekly workspace-validation channel (cross-harness + corroboration), and smoke_tests.txt vs no_run.yaml now disagree about this script. Nothing parked. + See the ledger's "New occurrence — 2026-08-25" section; surfaced by + complete/2026/08/weekly-smoke-timings-naming.md. - CORRECTION (post-close-out): the captured stack shows the hang is in jax.block_until_ready, NOT in compilation. The epic's name and every marker calling this an "XLA compile stall" are wrong. Resume from "why does block_until_ready never return", not from compiler behaviour.