mind: complete jax-stall-shared-preloads-retime-refutation (#328 → #329)

active.md

@@ -13,15 +13,3 @@
 - registered: 2026-08-19 by the wake_up session — the issuing session (claude/autofit-priors-messages-audit-ylvenv)
   filed the prompt + issue but not this entry, tripping Lifecycle Drift on main.
 - repos-none-claimed: this entry claims NO repos — one line deliberately, not 2-space bullets.
-
-## jax-stall-shared-preloads-retime-refutation
-- issue: https://github.com/PyAutoLabs/PyAutoMind/issues/328 (issued 2026-08-25)
-- issued: 2026-08-25
-- prompt: active/jax_stall_shared_preloads_retime_refutation.md
-- epic: jax-compile-stall (evidence-recording only; not a phase)
-- status: library-dev
-- worktree: ~/Code/PyAutoLabs-wt/jax-stall-shared-preloads-retime-refutation
-- repos:
-  - PyAutoMind (branch claude/weekly-smoke-timings-naming-012fuj)
-- note: web-github session — no local worktree created. Mind-only record change; parks nothing.
-

active/jax_stall_shared_preloads_retime_refutation.md → complete/2026/08/jax-stall-shared-preloads-retime-refutation.md

@@ -1,3 +1,78 @@
+# A retime refuted a stall marker; one day later the script stalled
+
+PyAutoMind#329 → `5eeee59` (closing PyAutoMind#328), merged 2026-08-25.
+Evidence-recording task against the `jax-compile-stall` ledger. Filed as a
+one-line follow-up ("add `shared_preloads.py` to the family list") from
+`complete/2026/08/weekly-smoke-timings-naming.md`; the investigation found
+something materially larger.
+
+## What shipped
+
+- **`draft/bug/ci/jax_vmap_jit_compile_stall.md`** gained a
+  `## New occurrence — 2026-08-25` section, mirroring the 2026-08-24 precedent:
+  run/job ids, a ten-row sibling timing table, the retime refutation, the
+  exclusion-list disagreement, the widened affected-set shape, and the channel
+  finding.
+- **`epics.md`** — the `jax-compile-stall` entry carries a
+  `NEW EVIDENCE 2026-08-25` note so the epic's state reflects it.
+- Nothing parked, disabled, or re-marked. `autolens_workspace_test` read for
+  evidence only.
+
+## The finding
+
+`multi_dataset/jax_likelihood/shared_preloads.py` hit `TIMEOUT (300s)` in
+PyAutoHeart `Workspace Smoke` run **32902243623** (job **97978549465**,
+2026-08-25) while all nine siblings passed in 8.6–49.3s — a >6x outlier with the
+bimodal signature the ledger uses to separate a stall from slowness.
+
+**The 2026-08-24 retime had removed this entry's SLOW marker and returned it to
+coverage the day before.** Verified against `autolens_workspace_test` at
+`7fc497d`: absent from `config/build/no_run.yaml` (in mega-run/weekly coverage),
+still commented out at `smoke_tests.txt` line 15 (out of the PR gate).
+
+## Key traps / findings
+
+- **N=5 cannot refute a bimodal failure.** The retime (5 repeats per leg, 300s
+  cap; runs 32741371308 + 32741386752) removed eight PyAutoHeart#74 entries on
+  "completed 5/5 on both legs, slowest 54.5s — marker refuted". ~10 executions
+  measure the fast mode and say nothing about the tail. The retime was right
+  that the script is not *slow*; it was not powered to see that it *stalls*.
+  **The other seven entries that sweep readmitted carry identical uncertainty**
+  and are in the weekly channel now — the standing follow-up below.
+- **The two exclusion lists disagree, and different gates read different ones.**
+  `smoke_tests.txt` gates the PR run; `no_run.yaml` gates mega-run/weekly. A
+  script disabled since 2026-07-22 was therefore still able to burn a 300s cap
+  in CI. Resolve deliberately rather than by whichever file is edited next.
+- **First occurrence via the weekly `workspace-validation` channel.** Every
+  prior one came from a workspace repo's own `Smoke Tests` gate or from
+  `release-integrate`. A different, PyAutoHeart-owned, `script_matrix.py`-driven
+  harness reproducing the same signature is cross-harness corroboration that the
+  stall is not one repo's runner configuration.
+- **The common factor is wider than MGE.** `shared_preloads.py` is not an MGE
+  variant, so the ledger's "composite MGE vmap graph" shape no longer covers the
+  affected set. What the members share is heavy `multi_dataset` vmap
+  composition.
+- **A one-line follow-up note was not the whole finding.** The note said "add it
+  to the family list". Checking the workspace's actual exclusion state — rather
+  than trusting the note — is what surfaced the refutation. Cheap to do, and it
+  changed the task's meaning.
+
+## Follow-ups
+
+- **Re-check the seven other entries the 2026-08-24 sweep readmitted.** One of
+  eight hung within a day; the rest were cleared by the same under-powered
+  protocol and sit in weekly coverage. Now cheaper to settle than before: the
+  weekly `smoke-timings-*` dataset (shipped same day, PyAutoHeart#182) makes
+  per-script weekly timings globbable, so a few weeks of data answers it without
+  a bespoke sweep.
+- **Reconcile `smoke_tests.txt` vs `no_run.yaml`** for the entries where they
+  disagree — a decision, not code.
+- `pyauto-brain bug` returns `owner: unresolved` / `fix locus: unresolved` for a
+  prompt whose only repo is `@PyAutoMind`, because Mind is absent from its
+  library-repo owner map. Cosmetic; classification was otherwise sound.
+
+## Original prompt
+
 # The 2026-08-24 retime returned `shared_preloads.py` to coverage; it stalled the next day
 
 Type: bug

complete/index.md

@@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema:
 only then grep a dated bucket. Curators: edit the band between the CURATED
 markers; everything below GENERATED is rebuilt.
 
-1139 records across 7 buckets.
+1140 records across 7 buckets.
 
 <!-- CURATED:START -->
 ## Highlights
@@ -122,6 +122,7 @@ _(curate hard-won records here — survives regeneration.)_
 - [jax-grad-local-vs-ci-assertions](2026/08/jax-grad-local-vs-ci-assertions.md)
 - [jax-grad-smoke-timeout-budget](2026/08/jax-grad-smoke-timeout-budget.md)
 - [jax-likelihood-mass-sensitivity](2026/08/jax-likelihood-mass-sensitivity.md) — Audit + fix for median-literal mass blindness across all jax_likelihood/datacube scripts (phase 2 of the findi…
+- [jax-stall-shared-preloads-retime-refutation](2026/08/jax-stall-shared-preloads-retime-refutation.md)
 - [knowledge-board](2026/08/knowledge-board.md) — auto-closed on merge
 - [lacosmic-cr-option-and-star-pass-decoupling](2026/08/lacosmic-cr-option-and-star-pass-decoupling.md) — both deliberately OPEN — closure is the default-flip decision, not this ship
 - [latex-docstrings-invalid-escape-warnings](2026/08/latex-docstrings-invalid-escape-warnings.md)

dashboard.html

@@ -185,7 +185,7 @@
 </head>
 <body>
 <header class="hero"><span class="orb"><svg class="mark" viewBox="0 0 48 48" fill="none" stroke="currentColor" stroke-width="1.3" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="24" cy="24" r="20.4"/><path d="M20.2,37.6 L20.2,33.6 C20.2,32.4 19.4,31.6 18.4,30.8 C16.8,29.6 15.8,28.2 15.6,26.6 C15.5,25.6 14.9,25.2 13.9,24.9 C12.6,24.5 12.3,23.7 13.0,22.7 C13.8,21.6 14.5,20.6 14.6,19.4 C15.0,15.4 18.2,12.2 22.4,11.8 C27.2,11.3 31.6,14.6 32.3,19.2 C32.8,22.4 31.6,25.0 30.6,27.0 C29.9,28.4 29.6,29.6 29.6,31.2 L29.6,37.6"/><circle cx="24.6" cy="19.6" r="4.4"/><path d="M22.5,19.7 L24.0,21.3 L26.8,18.1"/><path d="M34.2,16.5 L38.8,16.5 M35.2,21.0 L39.4,21.0 M33.6,25.5 L37.2,25.5"/><g fill="currentColor" stroke="none"><circle cx="40.1" cy="16.5" r="1.4"/><circle cx="40.7" cy="21.0" r="1.4"/><circle cx="38.5" cy="25.5" r="1.4"/></g></svg></span><h1>PyAuto<b>Mind</b><span class="kind">Dashboard</span></h1><div class="rule"></div><p class="tag">Intent. Priority. Flow.</p></header><p class="lede">Every task the Mind is holding. Tap a task's 📋 and its <code>/start_dev</code> command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. <a href="#recent">Recent</a> is the same work by date — what has been happening rather than what to do next.</p>
-<ul class="stats"><li><b>2</b><span>In flight</span></li><li><b>3</b><span>Parked</span></li><li><b>5</b><span>Planned</span></li><li><b>137</b><span>Backlog</span></li></ul>
+<ul class="stats"><li><b>1</b><span>In flight</span></li><li><b>3</b><span>Parked</span></li><li><b>5</b><span>Planned</span></li><li><b>137</b><span>Backlog</span></li></ul>
 <div class="fresh"><p><b>Last updated 2026-08-25.</b> This page is generated from <code>active/</code>, <code>draft/</code> and the registry files, so it is only as current as they are. <code>dashboard_refresh.yml</code> re-renders it on every push to <code>main</code> — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to <code>complete/</code> keeps rendering here as pickable backlog. Reconciling those is the refresh below.</p>
 <div class="task"><button class="copy" data-cmd="Bring the PyAutoMind dashboard up to date. Work in the PyAutoMind checkout:
 
@@ -233,7 +233,6 @@ <h3>Quick wins <span class="facets">(small enough, and safe enough to run unatte
 <h2>In flight <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active.md">markdown version</a></h2>
 <p class="muted">Issued — each has an open GitHub issue and usually a branch.</p>
 <div class="task"><button class="copy" data-cmd="/start_dev active/16_transformed_message_factor_gradient_unpack.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active/16_transformed_message_factor_gradient_unpack.md"><code>@PyAutoFit</code> <code>TransformedMessage.factor_gradient</code> crashes on first call</a> — <a href="https://github.com/PyAutoLabs/PyAutoFit/issues/1501">issue #1501</a><span class="facets"> — issued 2026-08-19</span> — <span class="facets">HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract</span></p></div>
-<div class="task"><button class="copy" data-cmd="/start_dev active/jax_stall_shared_preloads_retime_refutation.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active/jax_stall_shared_preloads_retime_refutation.md">The 2026-08-24 retime returned <code>shared_preloads.py</code> to coverage; it stalled the next day</a> — <a href="https://github.com/PyAutoLabs/PyAutoMind/issues/328">issue #328</a><span class="facets"> — issued 2026-08-25</span> — <span class="facets">library-dev</span></p></div>
 <h2>Parked <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/parked.md">markdown version</a></h2>
 <details>
 <summary>3 task(s)</summary>
@@ -395,12 +394,6 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <p class="muted">The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from <code>complete/index.md</code>, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.</p>
 <table class="recent">
 <tr>
-<td class="when">2026-08-25</td>
-<td class="what">issued</td>
-<td><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active/jax_stall_shared_preloads_retime_refutation.md">The 2026-08-24 retime returned <code>shared_preloads.py</code> to coverage; it…</a></td>
-<td class="pick"><button class="copy" data-cmd="/start_dev active/jax_stall_shared_preloads_retime_refutation.md" aria-label="Copy the Claude command">📋</button></td>
-</tr>
-<tr>
 <td class="when">2026-08-24</td>
 <td class="what">filed</td>
 <td><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/maintenance/ci/wiki_currency_check_version_gate.md">wiki-currency's --check-version gate rots on every library main merge</a></td>
@@ -454,7 +447,7 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <td><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/maintenance/workspaces/pynufft_removal_downstream_residue.md">pynufft removal: unswept downstream residue (1 hard break + stale…</a></td>
 <td class="pick"><button class="copy" data-cmd="/start_dev draft/maintenance/workspaces/pynufft_removal_downstream_residue.md" aria-label="Copy the Claude command">📋</button></td>
 </tr>
-<tr hidden>
+<tr>
 <td class="when">2026-08-23</td>
 <td class="what">filed</td>
 <td><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/maintenance/workspaces/pynufft_removal_downstream_residue_phase_3_ci_install_docs.md">Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI…</a></td>
@@ -694,6 +687,12 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <td><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autolens/interferometer_release_leg_oom.md">interferometer/start_here.py OOM in nightly release-validation…</a></td>
 <td class="pick"><button class="copy" data-cmd="/start_dev draft/bug/autolens/interferometer_release_leg_oom.md" aria-label="Copy the Claude command">📋</button></td>
 </tr>
+<tr hidden>
+<td class="when">2026-07-31</td>
+<td class="what">filed</td>
+<td><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/research/autolens/quick_update_plotting_cost.md">Quick-update plotting cost — minutes per update, and it is not JAX…</a></td>
+<td class="pick"><button class="copy" data-cmd="/start_dev draft/research/autolens/quick_update_plotting_cost.md" aria-label="Copy the Claude command">📋</button></td>
+</tr>
 </table>
 <button class="more" data-page="10">… 10 more (40 left)</button>
 <h2>Epics <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/epics.md">markdown version</a></h2>