diff --git a/complete/2026/08/jax-traceback-filtering-release-harness.md b/complete/2026/08/jax-traceback-filtering-release-harness.md new file mode 100644 index 00000000..f76d5899 --- /dev/null +++ b/complete/2026/08/jax-traceback-filtering-release-harness.md @@ -0,0 +1,178 @@ +# Release/smoke script harness stopped filtering JAX tracebacks + +Issue: PyAutoLabs/PyAutoHeart#186 (CLOSED 2026-08-27) + +## What was wrong + +JAX removes its own frames from a traceback by default and, in practice, takes +the exception that names the failure with them. A workspace script failing under +JAX inside Heart's script harness reported only: + +``` +scripts/interferometer/start_here.py ... FAIL (19.5s) +For simplicity, JAX has removed its internal frames from the traceback of the +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. +``` + +Which script failed, and nothing about why. + +The cost was measured, not hypothetical. Heart run 30516167217 (2026-07-30) +failed **both** interferometer shards (`autogalaxy` and `autolens`) with exactly +that placeholder, so that night read as an unidentified failure. The real cause — +`jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating +85898814480 bytes` — became legible only in run 30607596240 the following night, +where the message happened to survive. One night was spent identifying a failure +the harness already had the information to name. + +## What shipped + +| Repo | PR | Merged | Change | +|---|---|---|---| +| PyAutoHeart | [#187](https://github.com/PyAutoLabs/PyAutoHeart/pull/187) | 2026-08-27 (`c8b093e1`) | `JAX_TRACEBACK_FILTERING: "off"` — workflow-level `env:` in `workspace-validation.yml`, and the workspace-runner step's `env:` in `smoke-tests.yml`. 2 files, +18/−0 | + +Workflow-level `env:` reaches every job and step of `workspace-validation.yml`, +so both `mode=smoke` and `mode=release`, and every project in the matrix, inherit +it. `run_python.py`'s per-script env profiles inherit it too — a profile only +*sets* the keys it is given and never clears inherited vars, which is the +behaviour `profile_release.yaml`'s own header already relies on. +`workspace-smoke.yml` needed no edit: it is a thin caller of +`workspace-validation.yml`. + +CI before merge: Heart Tests run 33093776599, `pytest (3.12)` + `pytest (3.13)` +both success, `mergeable_state: clean`. Locally `python3 -m pytest -q -n auto` → +641 passed. + +## The exit criterion is deferred, not met + +The issue asked that a real JAX exception print its own type and message in a job +log, "verified against a real (or deliberately induced) JAX failure in a CI run, +not only by reading the diff". Nothing has failed under JAX since the merge, so +that has not been observed. The next JAX failure in `run_scripts` or the smoke +runner is the proof. This is recorded as shipped-but-unproven on purpose: the +whole defect is that the diff looks obviously right and the log is what has to +change. + +## Traps worth keeping + +- **`off` must be quoted in YAML.** Bare `off` is a YAML 1.1 boolean, so + `JAX_TRACEBACK_FILTERING: off` hands GitHub Actions `false`, not the string JAX + expects. Checked with `yaml.safe_load` rather than assumed. +- **A stale prompt can outlive its own fix by a month.** This task exists only + because `/start_dev` on + `draft/bug/autolens/interferometer_release_leg_oom.md` found the prompt's + headline OOM already shipped — + `complete/2026/08/interferometer-start-here-integrate-oom.md`, 2026-07-31, + `MultiStartProdigy` running 48 starts as one unbatched `vmap`. That prompt sat + in `draft/` as pickable backlog for 27 days, and its triage notes pointed at + the wrong subsystem (NUFFT/transformer). It was retired to + `complete/archive/shelved/` in the same run. **Check `complete/` for the slug + before planning a bug prompt**, not after. +- **A prompt's own amendment can be the only live part of it.** The retired + prompt's 2026-08-04 amendment carried two consequences. One ("it is not + autolens-specific") was already closed — both workspaces carry `batch_size=4`. + The other was this task. Reading only the headline would have lost it. + +## Original prompt + +`active/jax_traceback_filtering_release_harness.md`, filed and issued +2026-08-27 by the same `/start_dev` run that retired its parent. + +## Not done + +12 workspace scripts still call `MultiStartProdigy` with no `batch_size` +(autolens: `group/`, `point_source/`, `multi_dataset/`, `cluster/`, `weak/`, …; +autogalaxy: `multi_galaxy/modeling.py`, `multi_dataset/start_here.py`, +`cluster/modeling.py`). Cheaper datasets than a 108k-visibility interferometer +fit, so likely fine — surfaced during this run, deliberately not filed. + +## Original prompt + +# Release/smoke script harness filters JAX tracebacks away, hiding the real exception + +Type: bug +Target: ci +Repos: +- @PyAutoHeart +Difficulty: small +Autonomy: safe +Priority: medium +Status: formalised +Filed: 2026-08-27 +Issued: 2026-08-27 + +## Symptom + +When a workspace script fails under JAX inside PyAutoHeart's script harness +(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the +`smoke` and `release` legs), JAX's default traceback filter removes its own +internal frames *and*, in practice, the exception that matters: + +``` +scripts/interferometer/start_here.py ... FAIL (19.5s) +For simplicity, JAX has removed its internal frames from the traceback of the +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. +``` + +That is the whole failure record. The reader is told which script failed and +nothing about why. + +## Why it matters — the cost is measured, not hypothetical + +PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards +(`autogalaxy` and `autolens`) with exactly the message above. Because the real +exception was filtered away, that night read as an unidentified failure. The +same defect surfaced its true cause only the following night, run +30607596240 (2026-07-31), where the message survived intact: + +``` +jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes. +``` + +One night was spent identifying a failure the harness already had the +information to name. The underlying OOM was root-caused and fixed the same day +(`autolens_workspace#450`; record: +`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is +about the *harness*, which still filters, and will do the same to the next +JAX failure. + +A gate whose error messages are filtered away is the same category of problem +as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the +harness discards it before a human sees it. + +## Current state (verified 2026-08-27) + +`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart: + +``` +$ grep -rn "JAX_TRACEBACK_FILTERING" . # PyAutoHeart @ d576003 — no matches +``` + +Nor in the workspaces' `config/build/profile_release.yaml` / +`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var +they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly. + +## Proposed fix + +Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of +`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that +already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both +the `smoke` and `release` legs and every workspace in the matrix, and +`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the +keys it is given; it never clears inherited vars). + +Consider whether the same belongs in the other script-running workflows +(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason. + +## Exit criteria + +- A JAX exception raised inside a workspace script under `run_scripts` reports + its own exception type and message in the job log, not the + "JAX has removed its internal frames" placeholder. +- Verified against a real (or deliberately induced) JAX failure in a CI run, + not only by reading the diff. + +## Origin + +Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its +2026-08-04 amendment, consequence 2) when that prompt was retired as superseded +— the OOM it described had already shipped, this harness gap had not. diff --git a/draft/bug/autolens/interferometer_release_leg_oom.md b/complete/archive/shelved/interferometer_release_leg_oom.md similarity index 71% rename from draft/bug/autolens/interferometer_release_leg_oom.md rename to complete/archive/shelved/interferometer_release_leg_oom.md index 17e5d642..da630395 100644 --- a/draft/bug/autolens/interferometer_release_leg_oom.md +++ b/complete/archive/shelved/interferometer_release_leg_oom.md @@ -1,3 +1,28 @@ +> **RETIRED 2026-08-27 — superseded, not actioned as written.** +> +> The OOM this prompt describes was root-caused and fixed before the prompt's own +> amendment was written. See `complete/2026/08/interferometer-start-here-integrate-oom.md`: +> `MultiStartProdigy` ran 48 starts as a single unbatched `vmap` (~86 GB in one +> allocation); `autolens_workspace#450` set `batch_size=4` and the leg was +> discharged by Release Integrate run 30901054267 on 2026-08-04 +> (`scripts/interferometer/start_here.py ... PASS (173.5s)`). +> +> Re-verified at retirement (2026-08-27): +> - `autolens_workspace` and `autogalaxy_workspace` **both** carry `batch_size=4` +> on the 48-start `MultiStartProdigy` in `scripts/interferometer/start_here.py`, +> so the amendment's consequence 1 ("it is not autolens-specific") is closed too. +> - The interferometer shard is green in the two most recent nightly Release +> Integrate runs — 32804373015 (2026-08-25, `autogalaxy`) and 33073386315 +> (2026-08-27, `autolens`). No recurrence. +> - The NUFFT/transformer hypothesis in "Notes for triage" was wrong: the +> allocation was the multi-start batch, not a dense transformer matrix. +> +> The amendment's consequence 2 — `JAX_TRACEBACK_FILTERING=off` in the release +> harness — was **not** done and is still open. It was split out as its own task: +> `draft/bug/ci/jax_traceback_filtering_release_harness.md`. + +--- + # interferometer/start_here.py OOM in nightly release-validation integrate leg Filed: 2026-07-31 (backfilled from git) diff --git a/complete/index.md b/complete/index.md index 9b79da84..237a3b56 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1153 records across 7 buckets. +1154 records across 7 buckets. ## Highlights @@ -125,6 +125,7 @@ _(curate hard-won records here — survives regeneration.)_ - [jax-likelihood-mass-sensitivity](2026/08/jax-likelihood-mass-sensitivity.md) — Audit + fix for median-literal mass blindness across all jax_likelihood/datacube scripts (phase 2 of the findi… - [jax-likelihood-smoke-pins-stale](2026/08/jax-likelihood-smoke-pins-stale.md) - [jax-stall-shared-preloads-retime-refutation](2026/08/jax-stall-shared-preloads-retime-refutation.md) +- [jax-traceback-filtering-release-harness](2026/08/jax-traceback-filtering-release-harness.md) - [knowledge-board](2026/08/knowledge-board.md) — auto-closed on merge - [lacosmic-cr-option-and-star-pass-decoupling](2026/08/lacosmic-cr-option-and-star-pass-decoupling.md) — both deliberately OPEN — closure is the default-flip decision, not this ship - [latex-docstrings-invalid-escape-warnings](2026/08/latex-docstrings-invalid-escape-warnings.md) diff --git a/dashboard.html b/dashboard.html index 85f95f1a..ed2588ff 100644 --- a/dashboard.html +++ b/dashboard.html @@ -204,7 +204,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- +

Last updated 2026-08-27. This page is generated from active/, draft/ and the registry files, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to complete/ keeps rendering here as pickable backlog. Reconciling those is the refresh below.

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

136 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

135 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 28 @@ -344,7 +344,7 @@

Backlog

Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces🧹 maintenanceworkspacessmallsupervisedlow

-bug — 17 +bug — 16

LensCalc NumPy Hessian step is too coarse for multi-plane tracers🐛 bugautogalaxylargesupervisedhigh

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

@@ -361,7 +361,6 @@

Backlog

Priors & Messages cleanup — tracker🐛 bugpriorstoo-largesupervisednormal

Point-source JSON datasets record no resolution regime🐛 bugpyautolensmediumsupervisedlow

-

docs — 12 diff --git a/dashboard.md b/dashboard.md index cc89c9a0..26688199 100644 --- a/dashboard.md +++ b/dashboard.md @@ -45,7 +45,7 @@ anything you could not verify. | [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 5 | -| [Backlog](#backlog) (`draft/`) | 136 | +| [Backlog](#backlog) (`draft/`) | 135 | ## Start here @@ -247,7 +247,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**136** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below. +**135** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
feature — 28 @@ -777,7 +777,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-bug — 17 +bug — 16
📋 LensCalc NumPy Hessian step is too coarse for multi-plane tracers — autogalaxy · large · supervised · high @@ -907,14 +907,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-
📋 interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens - -``` -/start_dev draft/bug/autolens/interferometer_release_leg_oom.md -``` - -
-
@@ -1522,12 +1514,11 @@ Continue the 'Expectation propagation (EP) campaign' epic. Its canonical state l ## Hygiene -4 prompt(s) without a metadata header — they show no facets above. Re-home or re-run intake on them when touched. +3 prompt(s) without a metadata header — they show no facets above. Re-home or re-run intake on them when touched.
Headerless prompts -- `draft/bug/autolens/interferometer_release_leg_oom.md` - `draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md` - `draft/docs/workspaces/plot_coverage_followups.md` - `draft/research/autolens_profiling/cluster_gradient_search_benchmark.md` diff --git a/draft/triage/nightly_release_blocked_eight_nights.md b/draft/triage/nightly_release_blocked_eight_nights.md index 2d726090..075885d7 100644 --- a/draft/triage/nightly_release_blocked_eight_nights.md +++ b/draft/triage/nightly_release_blocked_eight_nights.md @@ -102,8 +102,8 @@ most of the set was already fixed or tracked by the time the streak was noticed. | Night | Failing legs | Cause | Status | |---|---|---|---| | 07-28 | 3 × `FileNotFoundError: dataset/…/data.fits` (autogalaxy/multi, autolens/imaging) | auto-simulate guards pointed at the wrong simulator | **FIXED** — see below | -| 07-30 | `interferometer/start_here.py` in **both** autolens and autogalaxy shards; `group/start_here.py` TIMEOUT | same OOM as 07-31, message eaten by the JAX traceback filter | `draft/bug/autolens/interferometer_release_leg_oom.md` | -| 07-31 | `interferometer/start_here.py` → `RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes`; `delaunay.py` TIMEOUT | ~86 GB single allocation; XLA hang | as above; autolens_workspace_test#245 | +| 07-30 | `interferometer/start_here.py` in **both** autolens and autogalaxy shards; `group/start_here.py` TIMEOUT | same OOM as 07-31, message eaten by the JAX traceback filter | **FIXED** — OOM `complete/2026/08/interferometer-start-here-integrate-oom.md`; filter `complete/2026/08/jax-traceback-filtering-release-harness.md` | +| 07-31 | `interferometer/start_here.py` → `RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes`; `delaunay.py` TIMEOUT | 48-start `MultiStartProdigy` vmapped unbatched; XLA hang | **FIXED** — autolens_workspace#450 (`batch_size=4`), discharged by run 30901054267; autolens_workspace_test#245 | | 08-01 | `guides/modeling/advanced/hierarchical.py` TIMEOUT; `delaunay.py` TIMEOUT | Nautilus deadlock; XLA hang | hierarchical **FIXED** (PyAutoFit#1443); #245 | | 08-03 | `graphical/ep.py` — "`log_likelihood_function` is always returning `nan`" | EP initializer | `draft/bug/autofit/graphical_ep_nan_likelihood_release_leg.md` | | 08-04 | `guides/results/database/start_here.py` `IndexError` | `samples_weight_threshold` pruning | **FIXED + MERGED** 08-04 09:40 — autolens_workspace#465, autogalaxy_workspace#202 | @@ -171,8 +171,13 @@ this needs its own prompt. - `draft/bug/autofit/covariance_interpolator_test_unseeded_rng.md` — Class B root cause. Smallest and highest value: it alone unblocks a live release. -- `draft/bug/autolens/interferometer_release_leg_oom.md` — filed 07-31, never - issued; now updated with the 07-30 both-workspaces recurrence. +- ~~`draft/bug/autolens/interferometer_release_leg_oom.md`~~ — **closed out + 2026-08-27.** Its OOM had already shipped on 07-31 (autolens_workspace#450, + `batch_size=4` on the 48-start `MultiStartProdigy`; + `complete/2026/08/interferometer-start-here-integrate-oom.md`), so the prompt + was retired to `complete/archive/shelved/`. The one live part of its 08-04 + amendment — the JAX traceback filter that ate the 07-30 message — shipped as + PyAutoHeart#187 (`complete/2026/08/jax-traceback-filtering-release-harness.md`). - `draft/bug/autofit/graphical_ep_nan_likelihood_release_leg.md` — new. - autolens_workspace_test#245 (delaunay hang) — already tracked, no action here. - Confirm on 08-05 that Check D self-healed and that PyAutoBrain#196 renders the