You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Phase 2 of the compile-axis arc (phase 1 is PyAutoBrain#218 / PyAutoBrain#219).
Tracking warm compile per cell is the point of the arc: the persistent
compilation cache turned 117.0 s into 2.3 s (CPU MGE vag) and 5517.8 s into
937.1 s (A100 end-to-end), --xla_gpu_autotune_level=0 cut the pathological cold
probe 17×, and nothing watches for either reverting. Both are settings, so a
config drift or an XLA_FLAGS clobber — the PyAutoNerves#127 failure that went
undetected for two months — puts the 70-minute worst case straight back.
The blocker
Warm rows were not machine-identifiable. Warmness lived only in the free-text tag, which carries ~40 ad-hoc spellings across the committed corpus
(census-warm, census-warm2, prodigy-census-warm-retry, cache-warm, mb_homo_cold, idle-check, smoke, …). A dashboard whose whole job is tracking
warm compile could not tell which rows were warm.
The obvious shortcut is a trap: cache_dir is non-empty on cold rows too,
because the cold run is the one that populates the cache.
Legs
cache_state derived from probe behaviour — count cache entries either
side of lowered.compile(). Wrote a new entry ⇒ cold (miss); wrote nothing
into a populated cache ⇒ warm (hit); no --cache-dir ⇒ none;
configured-but-empty-and-nothing-written ⇒ unknown, not silently warm.
Derived per transform, since one invocation can legitimately miss on one
transform and hit on another — something a per-run tag cannot express.
host_state (cpu_count, 1-minute load average). XLA compiles on the
host cores; README.md records the first measurements being wrong by up to 7× (851 s vs 117 s for the same compile) from host load alone, and the
corpus already mixes a 32-core RAL allocation with laptop rows.
record_kind so the schema is self-describing — export_probe.py and trace_profile.py append their own shape into the same results tree.
Backfill of the 89 existing compile records — exact where cache_dir was
empty (none), inferred only from an end-anchored cold/warm tag, left unknown otherwise. Result: 33 cold, 34 warm, 19 none, 3 unknown.
Warm-compile pins per cell, keyed on the comparability key (hardware, jax_version, mixed_precision, cache_state) — the compile
equivalent of the runtime results' pinned_expected.
Dashboard rows through build_baseline.py / build_readme.py.
Brain leg — ingest --axis compile, reporting unpinned and drifted rows
strictly within one comparability key. Blocked on PyAutoBrain#219 merging
(it needs that PR's load_compile_corpus / compile_tier_of).
The trap any pin comparison must not create
Never compare a cold row against a warm pin, an A100 row against a CPU pin, or
rows across a jax_version bump. Cache keys include the jax version, so a bump
recompiles once by design — expected behaviour, not drift. Phase 3 classifies
it as such; a tool that flags it is one people learn to ignore.
Note on CI
scripts/misc/test/ is not run by any workflow — lint.yml runs ruff, build_readme.py --check, lychee and a smoke script, but no pytest step. The new test_jax_compile_cache_state.py (9 tests) therefore passes locally but is not a
CI gate, and test_vram_config.py::test_aggregate_matches_cell_prefixed_stems is
currently failing on main unnoticed. Worth a separate small PR adding the step;
flagging rather than folding it in here.
Overview
Phase 2 of the compile-axis arc (phase 1 is PyAutoBrain#218 / PyAutoBrain#219).
Tracking warm compile per cell is the point of the arc: the persistent
compilation cache turned 117.0 s into 2.3 s (CPU MGE
vag) and 5517.8 s into937.1 s (A100 end-to-end),
--xla_gpu_autotune_level=0cut the pathological coldprobe 17×, and nothing watches for either reverting. Both are settings, so a
config drift or an
XLA_FLAGSclobber — the PyAutoNerves#127 failure that wentundetected for two months — puts the 70-minute worst case straight back.
The blocker
Warm rows were not machine-identifiable. Warmness lived only in the free-text
tag, which carries ~40 ad-hoc spellings across the committed corpus(
census-warm,census-warm2,prodigy-census-warm-retry,cache-warm,mb_homo_cold,idle-check,smoke, …). A dashboard whose whole job is trackingwarm compile could not tell which rows were warm.
The obvious shortcut is a trap:
cache_diris non-empty on cold rows too,because the cold run is the one that populates the cache.
Legs
cache_statederived from probe behaviour — count cache entries eitherside of
lowered.compile(). Wrote a new entry ⇒cold(miss); wrote nothinginto a populated cache ⇒
warm(hit); no--cache-dir⇒none;configured-but-empty-and-nothing-written ⇒
unknown, not silently warm.Derived per transform, since one invocation can legitimately miss on one
transform and hit on another — something a per-run tag cannot express.
host_state(cpu_count, 1-minute load average). XLA compiles on thehost cores;
README.mdrecords the first measurements being wrong by up to7× (851 s vs 117 s for the same compile) from host load alone, and the
corpus already mixes a 32-core RAL allocation with laptop rows.
record_kindso the schema is self-describing —export_probe.pyandtrace_profile.pyappend their own shape into the same results tree.cache_dirwasempty (
none), inferred only from an end-anchored cold/warm tag, leftunknownotherwise. Result: 33 cold, 34 warm, 19 none, 3 unknown.(hardware, jax_version, mixed_precision, cache_state)— the compileequivalent of the runtime results'
pinned_expected.build_baseline.py/build_readme.py.ingest --axis compile, reporting unpinned and drifted rowsstrictly within one comparability key. Blocked on PyAutoBrain#219 merging
(it needs that PR's
load_compile_corpus/compile_tier_of).The trap any pin comparison must not create
Never compare a
coldrow against awarmpin, an A100 row against a CPU pin, orrows across a
jax_versionbump. Cache keys include the jax version, so a bumprecompiles once by design — expected behaviour, not drift. Phase 3 classifies
it as such; a tool that flags it is one people learn to ignore.
Note on CI
scripts/misc/test/is not run by any workflow —lint.ymlruns ruff,build_readme.py --check, lychee and a smoke script, but no pytest step. The newtest_jax_compile_cache_state.py(9 tests) therefore passes locally but is not aCI gate, and
test_vram_config.py::test_aggregate_matches_cell_prefixed_stemsiscurrently failing on
mainunnoticed. Worth a separate small PR adding the step;flagging rather than folding it in here.