Skip to content

ci: publish the weekly sweep's smoke timings under a globbable name - #182

Merged
Jammy2211 merged 1 commit into
mainfrom
claude/weekly-smoke-timings-naming-012fuj
Aug 25, 2026
Merged

ci: publish the weekly sweep's smoke timings under a globbable name#182
Jammy2211 merged 1 commit into
mainfrom
claude/weekly-smoke-timings-naming-012fuj

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Closes #181.

Summary

RunReport.write() has emitted a consolidated smoke_timings.json into every
report dir since PyAutoHands d2a22f4, and #167 published it from the PR gate
(smoke-tests.yml) as smoke-timings-<python-version>. The weekly sweep runs
through a different body — workspace-validation.yml — which never got that
upload. So the widest timing sample the organism produces survived only inside
the per-leg results-* zips, under no name a consumer could glob for, and expired
with them at 30 days.

Verified against the 2026-08-24 weekly run 32688412626: 109 artifacts, not one
smoke-timings-*.

Decision: option (a)

The prompt offered two routes. This takes (a) — mirror the PR gate's named
upload — rather than (b), widening the deferred phase-3 board ingester's
contract to also glob results-*.

(b) was rejected because there is nothing to widen: outside smoke-tests.yml and
its tests, nothing in this repo references smoke-timings at all. The ingester
does not exist yet, so (b) would mean writing a promise instead of a fix — and
would leave the timings expiring with the 30-day results-* retention. (a) also
leaves that future ingester with one glob instead of two.

API Changes

None. No Python source is touched — this is CI wiring plus tests. smoke-tests.yml
is untouched, so PR-gate behaviour is unchanged.

The one externally-visible change is additive: a new family of artifact names,
smoke-timings-{scripts,notebooks}-<project>-<directory>, published by the weekly
sweep.

Design notes

  • A separate artifact, not a wider results-* payload. The artifact name is
    the contract a timing ingester globs on. results-* is a different contract —
    it is what the analyze job downloads and hands to aggregate_results.py,
    which globs **/*.json and skips this sidecar by name. Publishing timings
    under a results-* name would aim them at the one consumer that deliberately
    excludes them.
  • The name carries the leg. The PR gate has one leg per python version, so a
    fixed name is safe there; this body runs ~98 legs in a single weekly run. The
    (project, directory) pair is exactly what the sibling results-* names
    already rely on the script matrix to keep unique.
  • Only the JSON is uploaded. The report dir already ships as results-*;
    duplicating ~98 of them weekly buys nothing.
  • No retention-days, deliberately (the results-* siblings use 30). The
    dataset is the point, matching feat: upload the smoke report dir so per-script timings persist #167's reasoning — the timings now keep the
    repo's full default window and outlive the zips they were extracted from.
  • if: always() + if-no-files-found: ignore are both load-bearing: a
    failing leg still has timings worth keeping, and a leg that dies before the
    first report write must not turn "we collected no timings" into "the weekly
    sweep is red".

Testing

Heart suite on the branch: 641 passed. Six new tests in
tests/test_workflow_wiring.py pin the artifact names, both fail-open guards, the
notebook leg's no-notebooks gate, the absent retention-days, and the
results-* / smoke-timings-* namespace separation.

End-to-end: a real weekly sweep on this branch

Dispatched workspace-smoke.yml on this branch →
run 32902243623.
referenced_workflows resolves to workspace-validation.yml@5829bbb, so it
exercised this branch's body, not main's.

203 artifacts = 1   workspace-validation-report
              + 6   notebooks-<project>   (generate_notebooks runs nothing — no timings by design)
              + 196 → 98 results-* / 98 smoke-timings-*

One-to-one across every executed leg, both scripts and notebooks.

No artifact-name collisions — not one upload step failed in the whole run,
which is the proof, since upload-artifact@v4 hard-fails with a 409 on a
duplicate name.

Retention split confirmed live, e.g. the autolens-imaging scripts leg:

Artifact Size Expires
results-scripts-autolens-imaging 5,336 B 2026-09-24 (30d)
smoke-timings-scripts-autolens-imaging 1,452 B 2026-11-23 (default)

That sweep's four red legs, and why none is this change

Note this dispatch shares the head sha with the PR, so its checks are attached
here. It is the scheduled weekly channel, not a PR gate.

Failing job Cause Attributable to this diff?
run_scripts (3.12, autolens_test, multi_dataset) jax_likelihood/shared_preloads.pyTIMEOUT (300s) No — the known intermittent XLA/block_until_ready compile stall in that family
run_notebooks (3.12, howtofit, chapter_1_introduction) Failed to download action 'actions/download-artifact@v4' — DNS, internal-api.service.iad.github.net No — GitHub infrastructure
run_notebooks (3.12, howtogalaxy, chapter_2_modeling) same No
run_notebooks (3.12, howtogalaxy, chapter_4_scaling_up_galaxies) same No
smoke / analyze "Fail the run if validation is not ready" No — designed downstream consequence of the above

On the scripts failure: 16 of 17 scripts passed, and the bimodality is visible
in the same log — siblings mge.py (11.7s), rectangular_mge.py (20.7s) and
mge_group.py (49.3s) all passed comfortably. The failure is at step 9
(run_python.py); the new step 12 then ran anyway and succeeded
smoke-timings-scripts-autolens_test-multi_dataset, 690 bytes, artifact
9583834034. That is if: always() behaving as intended: a failing leg still
yields its timings, TIMEOUT row and cap included.

On the three notebook failures: all three died inside "Prepare all required
actions" — before checkout and before any step, mine included — within one
~3-minute window (21:53–21:57). actions/download-artifact is pre-existing in
that job (it fetches the generated notebooks); this diff adds only an
upload-artifact step, and upload-artifact downloaded fine in the same jobs'
siblings. Those three legs therefore produced neither a results-* nor a
smoke-timings-*, which is why the 98/98 pairing above is unaffected.

`RunReport.write()` emits a consolidated `smoke_timings.json` into every
report dir (PyAutoHands d2a22f4), and PyAutoHeart#167 published it from the
PR gate as `smoke-timings-<python-version>`. The weekly sweep runs through a
different body — workspace-validation.yml — which got no such upload, so the
widest timing sample the organism produces survived only inside the per-leg
`results-*` zips, under no name a consumer could glob for, and expired with
them at 30 days. Confirmed on weekly run 32688412626: 109 artifacts, not one
`smoke-timings-*`.

Mirror the gate's named upload into the `run_scripts` and `run_notebooks`
legs. The name carries the leg — `smoke-timings-<leg>-<project>-<directory>`
— because the gate has two legs per run and this body has ~50; that is the
same (project, directory) pair the sibling `results-*` names already rely on
to stay unique. Only the JSON is uploaded (the report dir already ships as
`results-*`) and deliberately with no `retention-days`, so the dataset keeps
the repo's full default window and outlives the zips it came from.

The new artifacts stay out of the `results-*` namespace on purpose: that is
what `analyze` downloads and hands to `aggregate_results.py`, which globs
`**/*.json` and skips this sidecar by name.

smoke-tests.yml is untouched — no change to PR-gate behaviour.

Six tests in test_workflow_wiring.py pin the names, the two fail-open guards,
the notebook leg's no-notebooks gate, the absent retention-days, and the
namespace separation. Full suite: 641 passed.

Closes #181
@Jammy2211 Jammy2211 added the pending-release Merged changes awaiting the next release cycle label Aug 25, 2026 — with Claude
@Jammy2211
Jammy2211 merged commit 9e3f1c8 into main Aug 25, 2026
132 of 138 checks passed
@Jammy2211
Jammy2211 deleted the claude/weekly-smoke-timings-naming-012fuj branch August 26, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release Merged changes awaiting the next release cycle

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: publish the weekly sweep's smoke timings under a globbable name

2 participants