PyAutoLabs · Jammy2211 · Aug 27, 2026 · Aug 27, 2026 · Aug 27, 2026 · Aug 27, 2026
diff --git a/complete/2026/08/jax-traceback-filtering-release-harness.md b/complete/2026/08/jax-traceback-filtering-release-harness.md
@@ -0,0 +1,178 @@
+# Release/smoke script harness stopped filtering JAX tracebacks
+
+Issue: PyAutoLabs/PyAutoHeart#186 (CLOSED 2026-08-27)
+
+## What was wrong
+
+JAX removes its own frames from a traceback by default and, in practice, takes
+the exception that names the failure with them. A workspace script failing under
+JAX inside Heart's script harness reported only:
+
+```
+scripts/interferometer/start_here.py ...  FAIL (19.5s)
+For simplicity, JAX has removed its internal frames from the traceback of the
+following exception. Set JAX_TRACEBACK_FILTERING=off to include these.
+```
+
+Which script failed, and nothing about why.
+
+The cost was measured, not hypothetical. Heart run 30516167217 (2026-07-30)
+failed **both** interferometer shards (`autogalaxy` and `autolens`) with exactly
+that placeholder, so that night read as an unidentified failure. The real cause —
+`jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating
+85898814480 bytes` — became legible only in run 30607596240 the following night,
+where the message happened to survive. One night was spent identifying a failure
+the harness already had the information to name.
+
+## What shipped
+
+| Repo | PR | Merged | Change |
+|---|---|---|---|
+| PyAutoHeart | [#187](https://github.com/PyAutoLabs/PyAutoHeart/pull/187) | 2026-08-27 (`c8b093e1`) | `JAX_TRACEBACK_FILTERING: "off"` — workflow-level `env:` in `workspace-validation.yml`, and the workspace-runner step's `env:` in `smoke-tests.yml`. 2 files, +18/−0 |
+
+Workflow-level `env:` reaches every job and step of `workspace-validation.yml`,
+so both `mode=smoke` and `mode=release`, and every project in the matrix, inherit
+it. `run_python.py`'s per-script env profiles inherit it too — a profile only
+*sets* the keys it is given and never clears inherited vars, which is the
+behaviour `profile_release.yaml`'s own header already relies on.
+`workspace-smoke.yml` needed no edit: it is a thin caller of
+`workspace-validation.yml`.
+
+CI before merge: Heart Tests run 33093776599, `pytest (3.12)` + `pytest (3.13)`
+both success, `mergeable_state: clean`. Locally `python3 -m pytest -q -n auto` →
+641 passed.
+
+## The exit criterion is deferred, not met
+
+The issue asked that a real JAX exception print its own type and message in a job
+log, "verified against a real (or deliberately induced) JAX failure in a CI run,
+not only by reading the diff". Nothing has failed under JAX since the merge, so
+that has not been observed. The next JAX failure in `run_scripts` or the smoke
+runner is the proof. This is recorded as shipped-but-unproven on purpose: the
+whole defect is that the diff looks obviously right and the log is what has to
+change.
+
+## Traps worth keeping
+
+- **`off` must be quoted in YAML.** Bare `off` is a YAML 1.1 boolean, so
+  `JAX_TRACEBACK_FILTERING: off` hands GitHub Actions `false`, not the string JAX
+  expects. Checked with `yaml.safe_load` rather than assumed.
+- **A stale prompt can outlive its own fix by a month.** This task exists only
+  because `/start_dev` on
+  `draft/bug/autolens/interferometer_release_leg_oom.md` found the prompt's
+  headline OOM already shipped —
+  `complete/2026/08/interferometer-start-here-integrate-oom.md`, 2026-07-31,
+  `MultiStartProdigy` running 48 starts as one unbatched `vmap`. That prompt sat
+  in `draft/` as pickable backlog for 27 days, and its triage notes pointed at
+  the wrong subsystem (NUFFT/transformer). It was retired to
+  `complete/archive/shelved/` in the same run. **Check `complete/` for the slug
+  before planning a bug prompt**, not after.
+- **A prompt's own amendment can be the only live part of it.** The retired
+  prompt's 2026-08-04 amendment carried two consequences. One ("it is not
+  autolens-specific") was already closed — both workspaces carry `batch_size=4`.
+  The other was this task. Reading only the headline would have lost it.
+
+## Original prompt
+
+`active/jax_traceback_filtering_release_harness.md`, filed and issued
+2026-08-27 by the same `/start_dev` run that retired its parent.
+
+## Not done
+
+12 workspace scripts still call `MultiStartProdigy` with no `batch_size`
+(autolens: `group/`, `point_source/`, `multi_dataset/`, `cluster/`, `weak/`, …;
+autogalaxy: `multi_galaxy/modeling.py`, `multi_dataset/start_here.py`,
+`cluster/modeling.py`). Cheaper datasets than a 108k-visibility interferometer
+fit, so likely fine — surfaced during this run, deliberately not filed.
+
+## Original prompt
+
+# Release/smoke script harness filters JAX tracebacks away, hiding the real exception
+
+Type: bug
+Target: ci
+Repos:
+- @PyAutoHeart
+Difficulty: small
+Autonomy: safe
+Priority: medium
+Status: formalised
+Filed: 2026-08-27
+Issued: 2026-08-27
+
+## Symptom
+
+When a workspace script fails under JAX inside PyAutoHeart's script harness
+(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the
+`smoke` and `release` legs), JAX's default traceback filter removes its own
+internal frames *and*, in practice, the exception that matters:
+
+```
+scripts/interferometer/start_here.py ...  FAIL (19.5s)
+For simplicity, JAX has removed its internal frames from the traceback of the
+following exception. Set JAX_TRACEBACK_FILTERING=off to include these.
+```
+
+That is the whole failure record. The reader is told which script failed and
+nothing about why.
+
+## Why it matters — the cost is measured, not hypothetical
+
+PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards
+(`autogalaxy` and `autolens`) with exactly the message above. Because the real
+exception was filtered away, that night read as an unidentified failure. The
+same defect surfaced its true cause only the following night, run
+30607596240 (2026-07-31), where the message survived intact:
+
+```
+jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes.
+```
+
+One night was spent identifying a failure the harness already had the
+information to name. The underlying OOM was root-caused and fixed the same day
+(`autolens_workspace#450`; record:
+`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is
+about the *harness*, which still filters, and will do the same to the next
+JAX failure.
+
+A gate whose error messages are filtered away is the same category of problem
+as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the
+harness discards it before a human sees it.
+
+## Current state (verified 2026-08-27)
+
+`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart:
+
+```
+$ grep -rn "JAX_TRACEBACK_FILTERING" .        # PyAutoHeart @ d576003 — no matches
+```
+
+Nor in the workspaces' `config/build/profile_release.yaml` /
+`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var
+they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly.
+
+## Proposed fix
+
+Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of
+`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that
+already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both
+the `smoke` and `release` legs and every workspace in the matrix, and
+`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the
+keys it is given; it never clears inherited vars).
+
+Consider whether the same belongs in the other script-running workflows
+(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason.
+
+## Exit criteria
+
+- A JAX exception raised inside a workspace script under `run_scripts` reports
+  its own exception type and message in the job log, not the
+  "JAX has removed its internal frames" placeholder.
+- Verified against a real (or deliberately induced) JAX failure in a CI run,
+  not only by reading the diff.
+
+## Origin
+
+Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its
+2026-08-04 amendment, consequence 2) when that prompt was retired as superseded
+— the OOM it described had already shipped, this harness gap had not.
diff --git a/...utolens/interferometer_release_leg_oom.md → ...shelved/interferometer_release_leg_oom.md b/...utolens/interferometer_release_leg_oom.md → ...shelved/interferometer_release_leg_oom.md
@@ -1,3 +1,28 @@
+> **RETIRED 2026-08-27 — superseded, not actioned as written.**
+>
+> The OOM this prompt describes was root-caused and fixed before the prompt's own
+> amendment was written. See `complete/2026/08/interferometer-start-here-integrate-oom.md`:
+> `MultiStartProdigy` ran 48 starts as a single unbatched `vmap` (~86 GB in one
+> allocation); `autolens_workspace#450` set `batch_size=4` and the leg was
+> discharged by Release Integrate run 30901054267 on 2026-08-04
+> (`scripts/interferometer/start_here.py ... PASS (173.5s)`).
+>
+> Re-verified at retirement (2026-08-27):
+> - `autolens_workspace` and `autogalaxy_workspace` **both** carry `batch_size=4`
+>   on the 48-start `MultiStartProdigy` in `scripts/interferometer/start_here.py`,
+>   so the amendment's consequence 1 ("it is not autolens-specific") is closed too.
+> - The interferometer shard is green in the two most recent nightly Release
+>   Integrate runs — 32804373015 (2026-08-25, `autogalaxy`) and 33073386315
+>   (2026-08-27, `autolens`). No recurrence.
+> - The NUFFT/transformer hypothesis in "Notes for triage" was wrong: the
+>   allocation was the multi-start batch, not a dense transformer matrix.
+>
+> The amendment's consequence 2 — `JAX_TRACEBACK_FILTERING=off` in the release
+> harness — was **not** done and is still open. It was split out as its own task:
+> `draft/bug/ci/jax_traceback_filtering_release_harness.md`.
+
+---
+
 # interferometer/start_here.py OOM in nightly release-validation integrate leg
 
 Filed: 2026-07-31 (backfilled from git)

diff --git a/complete/index.md b/complete/index.md
@@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema:
 only then grep a dated bucket. Curators: edit the band between the CURATED
 markers; everything below GENERATED is rebuilt.
 
-1153 records across 7 buckets.
+1154 records across 7 buckets.
 
 <!-- CURATED:START -->
 ## Highlights
@@ -125,6 +125,7 @@ _(curate hard-won records here — survives regeneration.)_
 - [jax-likelihood-mass-sensitivity](2026/08/jax-likelihood-mass-sensitivity.md) — Audit + fix for median-literal mass blindness across all jax_likelihood/datacube scripts (phase 2 of the findi…
 - [jax-likelihood-smoke-pins-stale](2026/08/jax-likelihood-smoke-pins-stale.md)
 - [jax-stall-shared-preloads-retime-refutation](2026/08/jax-stall-shared-preloads-retime-refutation.md)
+- [jax-traceback-filtering-release-harness](2026/08/jax-traceback-filtering-release-harness.md)
 - [knowledge-board](2026/08/knowledge-board.md) — auto-closed on merge
 - [lacosmic-cr-option-and-star-pass-decoupling](2026/08/lacosmic-cr-option-and-star-pass-decoupling.md) — both deliberately OPEN — closure is the default-flip decision, not this ship
 - [latex-docstrings-invalid-escape-warnings](2026/08/latex-docstrings-invalid-escape-warnings.md)

diff --git a/dashboard.html b/dashboard.html
@@ -204,7 +204,7 @@
 </head>
 <body>
 <header class="hero"><span class="orb"><svg class="mark" viewBox="0 0 48 48" fill="none" stroke="currentColor" stroke-width="1.3" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="24" cy="24" r="20.4"/><path d="M20.2,37.6 L20.2,33.6 C20.2,32.4 19.4,31.6 18.4,30.8 C16.8,29.6 15.8,28.2 15.6,26.6 C15.5,25.6 14.9,25.2 13.9,24.9 C12.6,24.5 12.3,23.7 13.0,22.7 C13.8,21.6 14.5,20.6 14.6,19.4 C15.0,15.4 18.2,12.2 22.4,11.8 C27.2,11.3 31.6,14.6 32.3,19.2 C32.8,22.4 31.6,25.0 30.6,27.0 C29.9,28.4 29.6,29.6 29.6,31.2 L29.6,37.6"/><circle cx="24.6" cy="19.6" r="4.4"/><path d="M22.5,19.7 L24.0,21.3 L26.8,18.1"/><path d="M34.2,16.5 L38.8,16.5 M35.2,21.0 L39.4,21.0 M33.6,25.5 L37.2,25.5"/><g fill="currentColor" stroke="none"><circle cx="40.1" cy="16.5" r="1.4"/><circle cx="40.7" cy="21.0" r="1.4"/><circle cx="38.5" cy="25.5" r="1.4"/></g></svg></span><h1>PyAuto<b>Mind</b><span class="kind">Dashboard</span></h1><div class="rule"></div><p class="tag">Intent. Priority. Flow.</p></header><p class="lede">Every task the Mind is holding. Tap a task's 📋 and its <code>/start_dev</code> command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. <a href="#recent">Recent</a> is the same work by date — what has been happening rather than what to do next.</p>
-<ul class="stats"><li><b>1</b><span>In flight</span></li><li><b>3</b><span>Parked</span></li><li><b>5</b><span>Planned</span></li><li><b>136</b><span>Backlog</span></li></ul>
+<ul class="stats"><li><b>1</b><span>In flight</span></li><li><b>3</b><span>Parked</span></li><li><b>5</b><span>Planned</span></li><li><b>135</b><span>Backlog</span></li></ul>
 <div class="fresh"><p><b>Last updated 2026-08-27.</b> This page is generated from <code>active/</code>, <code>draft/</code> and the registry files, so it is only as current as they are. <code>dashboard_refresh.yml</code> re-renders it on every push to <code>main</code> — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to <code>complete/</code> keeps rendering here as pickable backlog. Reconciling those is the refresh below.</p>
 <div class="task"><button class="copy" data-cmd="Bring the PyAutoMind dashboard up to date. Work in the PyAutoMind checkout:
 
@@ -269,7 +269,7 @@ <h2>Planned <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob
 <div class="task"><button class="copy" data-cmd="/route start the planned PyAutoMind task latent-nan-guard-honest-run — its record is in planned.md" aria-label="Copy the Claude command">📋</button><p><b>latent-nan-guard-honest-run</b><span class="facets"> — planned 2026-07-22</span></p></div>
 </details>
 <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree/main/draft">markdown version</a></h2>
-<p class="muted">136 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.</p>
+<p class="muted">135 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.</p>
 <details>
 <summary>feature — 28</summary>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md">Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching</a><span class="tags"><span class="pill w">✨ feature</span><span class="pill">autoarray</span><span class="pill n">medium</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
@@ -344,7 +344,7 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <div class="task"><button class="copy" data-cmd="/start_dev draft/maintenance/workspaces/notebook_setup_notebook_drift_siblings.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/maintenance/workspaces/notebook_setup_notebook_drift_siblings.md">Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces</a><span class="tags"><span class="pill w">🧹 maintenance</span><span class="pill">workspaces</span><span class="pill n">small</span><span class="pill n">supervised</span><span class="pill n">low</span></span></p></div>
 </details>
 <details>
-<summary>bug — 17</summary>
+<summary>bug — 16</summary>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autogalaxy/lenscalc_numpy_hessian_step_is_too_coarse.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autogalaxy/lenscalc_numpy_hessian_step_is_too_coarse.md">LensCalc NumPy Hessian step is too coarse for multi-plane tracers</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autogalaxy</span><span class="pill y">large</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/jax_runtime_and_parity.md">Fix release JAX runtime compatibility and likelihood parity</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">health_fixes</span><span class="pill r">too-large</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/jit_visualization_outputs.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/jit_visualization_outputs.md">Fix JIT quick-update visualization output regressions</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">health_fixes</span><span class="pill r">too-large</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
@@ -361,7 +361,6 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/priors/z_features.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/priors/z_features.md">Priors &amp; Messages cleanup — tracker</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">priors</span><span class="pill r">too-large</span><span class="pill n">supervised</span><span class="pill n">normal</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md">The reconstruction noise map describes a different estimator than the default solver</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autoarray</span><span class="pill n">medium</span><span class="pill r">human-required</span><span class="pill n">low</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md">Point-source JSON datasets record no resolution regime</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">pyautolens</span><span class="pill n">medium</span><span class="pill n">supervised</span><span class="pill n">low</span></span></p></div>
-<div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autolens/interferometer_release_leg_oom.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autolens/interferometer_release_leg_oom.md">interferometer/start_here.py OOM in nightly release-validation integrate leg</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autolens</span></span></p></div>
 </details>
 <details>
 <summary>docs — 12</summary>