Skip to content

release_validation.md

Latest commit

 

History

History
169 lines (139 loc) · 9.43 KB

release_validation.md

File metadata and controls

169 lines (139 loc) · 9.43 KB
Raw

Release validation — the release profile and acceptance criteria

This document is the spec half of Heart's release-validation tier. Heart owns the definition of what a release-grade validation run must do; it does not execute the build or the integration run (that is the Brain Release Agent dispatching Build's release.yml and Heart's release-integrate.yml channel).

M2 shipped the report schema, pyauto-heart validate --ingest, and the readiness hard gate. M3 wires the acceptance criteria below into the shared workspace-validation.yml body via a mode: release input. The two modes now have separate entry workflows (one run history per meaning): workspace-smoke.yml (the continuous smoke channel) and release-integrate.yml (this release-fidelity channel):

  • The release env profile lives in each workspace/*_workspace_test repo as config/build/profile_release.yaml — a self-contained sibling of profile_smoke.yaml (the smoke profile), passed to Build's run_python.py unmodified via its existing --env-config flag. No changes were needed in PyAutoHands's executor primitives to support this — --env-config already accepted an arbitrary path.

  • run_scripts, when mode: release, pip installs the Stage-2 TestPyPI wheels at the rehearsed version and puts no library source on PYTHONPATH, still executing from inside the workspace checkout.

  • verify_install_release runs heart/checks/verify_install.sh --testpypi --version <version> A–F against the same wheels.

  • emit_release_report reshapes Build's aggregate_results.py report.json (via the new heart/validate.py::to_stage_report / pyauto-heart validate --emit-stage-report) into the {"stage": "integrate", ...} contract --ingest expects, folding in the verify_install result and the Release-Agent-supplied commit_shas, and uploads it as the release-stage-report artifact for the Release Agent to feed into pyauto-heart validate --ingest.

    The verify_install result is folded in both directions: a failure forces the stage to fail, and the sidecar itself is carried into the stage report as evidence, which --ingest persists to ~/.pyauto-heart/verify_install.json — the install-verification readiness leg's input. Until 2026-07-15 only the failure direction existed, so a passing Stage 3 check was discarded and the leg reported install verification not run no matter how often it passed, holding Heart at YELLOW.

mode: release is scoped to the autofit/autogalaxy/autolens workspaces and their *_workspace_test siblings only — the HowTo* tutorial repos have no profile_release.yaml and stay out of the release-fidelity script matrix (they are still exercised under mode: smoke, unchanged).

Why a distinct profile

The per-PR smoke gate and the release gate are different jobs:

  • smoke — a fast structural / integration check: does the model compose and the script run end-to-end? Cheap, runs on every PR.
  • release — does the exact source about to ship, installed from the built wheel, pass at release fidelity? Slow, runs only for a release rehearsal.

Both tiers' config/build/profile_smoke.yaml today default to smoke values (PYAUTO_TEST_MODE=2, PYAUTO_SMALL_DATASETS=1, PYAUTO_DISABLE_JAX=1, PYAUTO_FAST_PLOTS=1). That is correct for a per-PR smoke and wrong for a release gate. The two must be named, distinct profiles so a release run cannot silently inherit smoke fidelity.

The release profile (wired in M3)

The intended release-fidelity env is already documented in PyAutoHands/.github/workflows/release.yml:

Tier PYAUTO_TEST_MODE PYAUTO_SMALL_DATASETS PYAUTO_FAST_PLOTS
user workspaces 1 (reduced iterations) 1 (capped grids) 1
*_workspace_test 0 (real searches, n_like_max) unset (full-res) unset

Per-script overrides: still layer on top of the selected profile — e.g. unset PYAUTO_SMALL_DATASETS for full-resolution FITS scripts, keep JAX on for jax_likelihood_functions/ and jax_substructure/. The profile sets the floor; overrides remain per-script.

The full library env-var surface (canonical entry: PyAutoNerves/autoconf/test_mode.py) is 13 PYAUTO_* vars, not the 4 smoke defaults:

PYAUTO_TEST_MODE, PYAUTO_SMALL_DATASETS, PYAUTO_FAST_PLOTS, PYAUTO_OUTPUT_MODE,
PYAUTO_DISABLE_JAX, PYAUTO_SKIP_FIT_OUTPUT, PYAUTO_SKIP_VISUALIZATION,
PYAUTO_SKIP_CHECKS, PYAUTO_SKIP_LATENTS, PYAUTO_SKIP_WORKSPACE_VERSION_CHECK,
PYAUTO_LATENT_NAN_INJECT, PYAUTO_DISABLE_IPYTHON_DISPLAY, PYAUTO_LIVE_VIEWER_LOG

plus a few per-script switches outside any yaml default (PYAUTO_MASS_MODE / PYAUTO_MASS_FAST, JAX_PILOT / JAX_PLATFORM_NAME / JAX_PLATFORMS).

Explicitly NOT Heart's to set. config/general.yaml's test: block (check_likelihood_function, lh_timeout_seconds, disable_positions_lh_inversion_check) and version: toggles are workspace-run/user settings. The release validation runs the scripts as the workspace ships them and does not mutate these. Heart's only version signal is the existing version_skew check. This release profile is an env-var profile, not a Heart-owned config mutation.

Wheel-install requirement (wired in M3)

Two verified gaps the M3 integration run MUST close (they are why the report carries profile and commit_shas, so the gate can enforce them):

  1. Test BUILDS, not SOURCE. Today workspace-validation.yml shadows the PyAuto packages with source checkouts via PYTHONPATH, so the gating run never touches a wheel — the exact blind spot that let a direct git-URL dependency break every TestPyPI upload for weeks. The release run MUST pip install the TestPyPI wheels published by the M1 rehearsal and put no source on PYTHONPATH.

  2. Wheel-based config resolution. autoconf resolves the workspace's config/ only when scripts run from inside the workspace checkout; a bare wheel falls back to the library's packaged defaults. So the run must pip install the wheels but still execute scripts from within the workspace checkout (for config/ + dataset/), with no source on PYTHONPATH.

The integration run also performs verify_install A–F against the same wheels. Check B reuses that exact TestPyPI version: it must install and import on Python 3.12 and 3.13, while Python 3.11 must reject it specifically because its Requires-Python metadata is >=3.12. A pinned rejection is the only evidence that this release holds the floor, because an unpinned install may select an older compatible one instead.

Check B then requires the unpinned install to be refused as well. That is a separate guarantee, and it was not met until 2026-08-19: pip install autolens on 3.11 backtracked to 2026.7.29.1 and installed a stale, JAX-less stack silently. The 2026.7.29.1.post1 tombstone refuses it, and this leg is what stops the backtrack reopening.

That TestPyPI A–F result feeds the install-verification readiness leg (see "How the gate enforces these"), tagged index: testpypi. It proves the wheels about to ship install, which is the right evidence for a release gate, and is reported as such rather than as proof that installing from PyPI works today.

For pre-publication development, verify_install B --version <version> --find-links <wheel-dir> exercises the same exact-version logic against local wheels while retaining PyPI for third-party dependencies. Its sidecar is labelled index: find-links, so it cannot be mistaken for PyPI or TestPyPI release evidence. Heart retains a fresh passing local result as development evidence but reports STALE until a PyPI or TestPyPI verification replaces it; a failing local artifact remains RED.

How the gate enforces these

heart/validate.py records profile and per-repo commit_shas in validation_report.json; heart/readiness.py then requires, for GREEN:

  • validation_outcome == pass (else RED for fail; STALE for incomplete, which means nothing failed and the rehearsal evidence is simply absent — a report predating the field falls back to release_ready == false → RED, so the gate fails closed on evidence it cannot classify),
  • profile == release (else YELLOW — a smoke-fidelity run is not a release gate),
  • commit_shas matching the current main HEADs (else YELLOW — stale source),
  • freshness (a rehearsal older than VALIDATION_STALE_DAYS is YELLOW).

The install-verification leg is separate and reads ~/.pyauto-heart/verify_install.json, written either by a local pyauto-heart verify_install --report-json invocation (index: pypi, testpypi, or find-links, according to its flags) or by --ingest folding the block out of a Stage 3 artifact (index: testpypi). Both release indexes satisfy the leg; the index is named in every reason line so the verdict states which install path it actually verified. Development-only index: find-links evidence does not satisfy this release gate.

Before M3 (or if the Release Agent only runs the M1 rehearsal and skips dispatching the release-integrate.yml channel), an ingested rehearsal-only report still (correctly) gates YELLOW: the source was built and TestPyPI-installed, but not yet exercised at release fidelity. mode: release is what supplies the integrate stage that flips this to GREEN-eligible.