bug: record the shared_preloads.py stall that refutes its 08-24 retime - #329
Merged
Merged
Conversation
multi_dataset/jax_likelihood/shared_preloads.py hit TIMEOUT (300s) in PyAutoHeart Workspace Smoke run 32902243623 (job 97978549465, 2026-08-25) while all nine siblings passed in 8.6-49.3s — a >6x outlier with the bimodal signature the ledger uses to separate a stall from slowness. The occurrence matters beyond a tally mark: the 2026-08-24 retime removed this entry's PyAutoHeart#74 SLOW marker and returned it to mega-run coverage on 5 repeats per leg at <=54.5s. Verified against autolens_workspace_test 7fc497d, it is absent from no_run.yaml (in coverage) yet still commented out in smoke_tests.txt (out of the PR gate) — the two exclusion lists disagree. N=5 measures the fast mode of a bimodal failure and says nothing about the tail, so the readmission is falsified and every entry that sweep readmitted carries the same uncertainty. Also the epic's first occurrence via the weekly workspace-validation channel, which is cross-harness corroboration that the stall is not one repo's runner config. shared_preloads.py is not an MGE variant, so the 'composite MGE vmap graph' common factor no longer covers the whole affected set. Nothing parked, disabled or re-marked, per the ledger's own argument against whack-a-mole quarantining and the 2026-08-24 mge.py precedent. Closes #328
Jammy2211
pushed a commit
that referenced
this pull request
Aug 25, 2026
…ind#328 → #329) Shipped as 5eeee59. Prompt advanced active/ → complete/2026/08/, active.md entry released, index + dashboard regenerated. Record's standing follow-up: the 2026-08-24 retime readmitted eight entries to coverage on N=5 per leg; one hung the next day, so the other seven carry the same uncertainty. The weekly smoke-timings-* dataset shipped the same day makes that cheap to settle from accumulating data rather than a bespoke sweep.
This was referenced Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #328.
Summary
multi_dataset/jax_likelihood/shared_preloads.pyhitTIMEOUT (300s)in PyAutoHeartWorkspace Smokerun 32902243623 (job 97978549465, 2026-08-25) while all ninesiblings in the same job passed in 8.6–49.3s.
The point is not the tally mark. The 2026-08-24 retime removed this entry's SLOW
marker and returned it to coverage; it stalled on the next weekly run.
Why this is more than another occurrence
Verified 2026-08-25 against
autolens_workspace_testat7fc497d:shared_preloads.pyconfig/build/no_run.yamlsmoke_tests.txtline 15The retime (5 repeats per Python leg, 300s cap; runs 32741371308 + 32741386752)
removed eight PyAutoHeart#74 entries because they "completed 5/5 on both legs
(slowest 54.5s) — marker refuted".
~10 executions at ≤54.5s cannot exclude a low-probability hang. The retime was
right that this script is not slow; it was not powered to see that it stalls.
This lands § Task step 1's slow-vs-stall question on the opposite side from where
the retime put this entry — and the same uncertainty attaches to every other entry
that sweep readmitted.
What changed
draft/bug/ci/jax_vmap_jit_compile_stall.md— a## New occurrence — 2026-08-25section mirroring the existing 2026-08-24 precedent: run/job ids, the ten-row
sibling timing table, the retime refutation, the exclusion-list disagreement, and
the affected-set consequence.
epics.md— thejax-compile-stallentry gains aNEW EVIDENCE 2026-08-25noteso the epic's state carries it.
lifecycle check: OK.Two findings worth flagging to a reader of the epic
The channel is new. Every prior occurrence in the ledger comes from a workspace
repo's own
Smoke Testsgate or fromrelease-integrate. This is the first viaPyAutoHeart's weekly
workspace-validation.ymlsweep — a different, PyAutoHeart-owned,script_matrix.py-driven harness that honoursno_run.yamland notsmoke_tests.txt.Cross-harness corroboration that the stall is not one repo's runner configuration.
The common factor has widened.
shared_preloads.pyis not an MGE variant, so theledger's "composite MGE vmap graph" shape no longer covers the whole affected set.
What the members share is heavy
multi_datasetvmap composition, not MGE specifically.Deliberately not done
Nothing is parked, disabled, or re-marked. The ledger's own § "This is why
quarantining is the wrong end state" argues against whack-a-mole parking that strips
coverage of exactly the heaviest JAX paths, and the 2026-08-24 precedent for
multi_dataset/jax_likelihood/mge.pywas to record and deliberately not park.Re-marking is phase 3's call with a root cause in hand.
autolens_workspace_testwas read for evidence only and is not modified by this PR.Follow-up noticed
pyauto-brain bugreturnsowner: unresolved/fix locus: unresolvedfor a promptwhose only repo is
@PyAutoMind, because Mind is not in its library-repo owner map.The classification was otherwise sound. Not fixed here.
Generated by Claude Code