complete: jax-vmap-materialisation-hang — epic shipped, 7 quarantines cleared - #348
Merged
Merged
Conversation
…pic reopened for phase 3
… cleared Phase 3 of jax-compile-stall, closing the epic. PyAutoFit#1528 closed; autolens_workspace_test#245 closed. PRs merged 2026-08-27: PyAutoFit#1529 (4130b22), PyAutoHands#269 (ad64e12), autolens_workspace_test#281 (69ee97c), autogalaxy_workspace_test#114 (adf5ffe). The epic's name was wrong for a month. It is not an XLA compile stall: compilation completes in ~12-18s and what never returns is materialising the vmap result. epics.md and the campaign ledger are renamed and marked shipped rather than left asserting the old name, since every future reader starts from them. Trigger is XLA CPU's multithreaded Eigen thread pool; workaround is XLA_FLAGS=--xla_cpu_multi_thread_eigen=false in the smoke AND release profiles of both test workspaces. ABAB, 12 passes / 0 hangs with the flag against 2 passes / 14 hangs without, Fisher exact p ~ 3e-6. All seven quarantined entries restored, 42/42 completions. Filed draft/research/ci/xla_cpu_eigen_pool_deadlock.md for the part that is NOT done: why the pool wedges. It carries the refuted hypotheses so they are not re-run, and asks whether the pool is merely mis-sized against the runner's cgroup quota -- if so the ~15% cost of the flag is recoverable. active.md entry released by hand: lifecycle.py record keys on the record slug, and this record is filed under the corrected name while the task slug still carried the misnomer, so the two did not match. The record now names both. Co-Authored-By: Claude <noreply@anthropic.com>
The branch was 15 commits behind. All three conflicts were in GENERATED files -- complete/index.md, dashboard.md, dashboard.html -- so they were resolved by regenerating from the merged sources rather than hand-edited: lifecycle.py index --apply -> 1155 records pyauto-brain intake --apply dashboard -> 135 prompts Both --check legs confirm current, and lifecycle.py check is OK. No source-of-truth file conflicted: active.md, epics.md, the completion record and the new research prompt all merged cleanly. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mind close-out for PyAutoFit#1528 (phase 3 of the
jax-compile-stallepic). All four code PRs are already merged: PyAutoFit#1529 (4130b22), PyAutoHands#269 (ad64e12), autolens_workspace_test#281 (69ee97c), autogalaxy_workspace_test#114 (adf5ffe).What this carries
complete/2026/08/jax-vmap-materialisation-hang.md, filed under the corrected name and cross-referencing the task slugjax-stall-block-until-ready.active/prompt folded out,active.mdentry released.epics.mdand the campaign ledger both asserted "intermittent XLA compile stall". They are renamed and marked shipped — otherwise the next reader starts from the wrong name, which is exactly how this ran for a month.draft/research/ci/xla_cpu_eigen_pool_deadlock.md, for the part that is deliberately not done.complete/index.mdregenerated, both--checklegs current.The finding, in one line
Not a compile stall. Compilation completes in ~12–18s; materialising the vmap result never returns — caught at
jax.block_until_ready/try_to_blockand atjax.Array._value, two doors onto one failure. Trigger is XLA CPU's multithreaded Eigen thread pool; workaround isXLA_FLAGS=--xla_cpu_multi_thread_eigen=falsein the smoke and release profiles of both test workspaces.ABAB: 12 passes / 0 hangs with the flag against 2 passes / 14 hangs without, Fisher exact p ≈ 3e-6. All seven quarantined entries restored, 42/42 completions.
Honest about what is unfinished
A workaround, not a root cause — we do not know why the pool wedges. The follow-up prompt carries the refuted hypotheses (compilation cache, jax/jaxlib version, the 3.12-vs-3.13 split) so nobody re-runs them, and asks the question that could recover the ~15% cost: whether the pool is merely mis-sized against the runner's cgroup quota rather than genuinely deadlocked.
Merge note
The branch was 15 commits behind main. Merged (never rebased). All three conflicts were in generated files —
complete/index.md,dashboard.md,dashboard.html— and were resolved by regenerating from the merged sources, not hand-edited. No source-of-truth file conflicted.lifecycle.py checkOK ·lifecycle.py index --checkOK ·intake dashboard --checkcurrent.Generated by Claude Code