Skip to content

bug: record the shared_preloads.py stall that refutes its 08-24 retime - #329

Merged
Jammy2211 merged 1 commit into
mainfrom
claude/weekly-smoke-timings-naming-012fuj
Aug 25, 2026
Merged

bug: record the shared_preloads.py stall that refutes its 08-24 retime#329
Jammy2211 merged 1 commit into
mainfrom
claude/weekly-smoke-timings-naming-012fuj

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Closes #328.

Summary

multi_dataset/jax_likelihood/shared_preloads.py hit TIMEOUT (300s) in PyAutoHeart
Workspace Smoke run 32902243623 (job 97978549465, 2026-08-25) while all nine
siblings in the same job passed in 8.6–49.3s.

The point is not the tally mark. The 2026-08-24 retime removed this entry's SLOW
marker and returned it to coverage; it stalled on the next weekly run.

Why this is more than another occurrence

Verified 2026-08-25 against autolens_workspace_test at 7fc497d:

List State of shared_preloads.py Effect
config/build/no_run.yaml absent — SLOW marker removed 2026-08-24 in mega-run / weekly coverage
smoke_tests.txt line 15 still commented out (2026-07-22, "the heaviest entry here") out of the PR gate

The retime (5 repeats per Python leg, 300s cap; runs 32741371308 + 32741386752)
removed eight PyAutoHeart#74 entries because they "completed 5/5 on both legs
(slowest 54.5s) — marker refuted".

~10 executions at ≤54.5s cannot exclude a low-probability hang. The retime was
right that this script is not slow; it was not powered to see that it stalls.
This lands § Task step 1's slow-vs-stall question on the opposite side from where
the retime put this entry — and the same uncertainty attaches to every other entry
that sweep readmitted.

What changed

  • draft/bug/ci/jax_vmap_jit_compile_stall.md — a ## New occurrence — 2026-08-25
    section mirroring the existing 2026-08-24 precedent: run/job ids, the ten-row
    sibling timing table, the retime refutation, the exclusion-list disagreement, and
    the affected-set consequence.
  • epics.md — the jax-compile-stall entry gains a NEW EVIDENCE 2026-08-25 note
    so the epic's state carries it.
  • Dashboard regenerated. lifecycle check: OK.

Two findings worth flagging to a reader of the epic

The channel is new. Every prior occurrence in the ledger comes from a workspace
repo's own Smoke Tests gate or from release-integrate. This is the first via
PyAutoHeart's weekly workspace-validation.yml sweep — a different, PyAutoHeart-owned,
script_matrix.py-driven harness that honours no_run.yaml and not smoke_tests.txt.
Cross-harness corroboration that the stall is not one repo's runner configuration.

The common factor has widened. shared_preloads.py is not an MGE variant, so the
ledger's "composite MGE vmap graph" shape no longer covers the whole affected set.
What the members share is heavy multi_dataset vmap composition, not MGE specifically.

Deliberately not done

Nothing is parked, disabled, or re-marked. The ledger's own § "This is why
quarantining is the wrong end state" argues against whack-a-mole parking that strips
coverage of exactly the heaviest JAX paths, and the 2026-08-24 precedent for
multi_dataset/jax_likelihood/mge.py was to record and deliberately not park.
Re-marking is phase 3's call with a root cause in hand.

autolens_workspace_test was read for evidence only and is not modified by this PR.

Follow-up noticed

pyauto-brain bug returns owner: unresolved / fix locus: unresolved for a prompt
whose only repo is @PyAutoMind, because Mind is not in its library-repo owner map.
The classification was otherwise sound. Not fixed here.


Generated by Claude Code

multi_dataset/jax_likelihood/shared_preloads.py hit TIMEOUT (300s) in PyAutoHeart
Workspace Smoke run 32902243623 (job 97978549465, 2026-08-25) while all nine
siblings passed in 8.6-49.3s — a >6x outlier with the bimodal signature the
ledger uses to separate a stall from slowness.

The occurrence matters beyond a tally mark: the 2026-08-24 retime removed this
entry's PyAutoHeart#74 SLOW marker and returned it to mega-run coverage on 5
repeats per leg at <=54.5s. Verified against autolens_workspace_test 7fc497d, it
is absent from no_run.yaml (in coverage) yet still commented out in
smoke_tests.txt (out of the PR gate) — the two exclusion lists disagree. N=5
measures the fast mode of a bimodal failure and says nothing about the tail, so
the readmission is falsified and every entry that sweep readmitted carries the
same uncertainty.

Also the epic's first occurrence via the weekly workspace-validation channel,
which is cross-harness corroboration that the stall is not one repo's runner
config. shared_preloads.py is not an MGE variant, so the 'composite MGE vmap
graph' common factor no longer covers the whole affected set.

Nothing parked, disabled or re-marked, per the ledger's own argument against
whack-a-mole quarantining and the 2026-08-24 mge.py precedent.

Closes #328
@Jammy2211
Jammy2211 merged commit 5eeee59 into main Aug 25, 2026
4 checks passed
Jammy2211 pushed a commit that referenced this pull request Aug 25, 2026
…ind#328 → #329)

Shipped as 5eeee59. Prompt advanced active/ → complete/2026/08/, active.md entry
released, index + dashboard regenerated.

Record's standing follow-up: the 2026-08-24 retime readmitted eight entries to
coverage on N=5 per leg; one hung the next day, so the other seven carry the same
uncertainty. The weekly smoke-timings-* dataset shipped the same day makes that
cheap to settle from accumulating data rather than a bespoke sweep.
Jammy2211 added a commit that referenced this pull request Aug 25, 2026
…) (#330)

mind: complete jax-stall-shared-preloads-retime-refutation (#328#329)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: record the shared_preloads.py stall that refutes its 08-24 retime

2 participants