Skip to content

internals.md

Latest commit

 

History

History
362 lines (290 loc) · 25.1 KB

internals.md

File metadata and controls

362 lines (290 loc) · 25.1 KB
Raw

PyAutoHands — internals

Operational detail for working inside this repo: the autohands CLI, the pre-build steps, workspace folder structure, config files, and release.yml. What PyAutoHands is and the Brain/Heart/Build boundary live in AGENTS.md — read that first; read this only when changing the build pipeline itself.

What the pipeline automates

PyAutoHands runs no release-readiness checks of its own (that is PyAutoHeart's job). It automates:

  1. Building and releasing packages to TestPyPI, then PyPI
  2. Running workspace Python scripts (integration tests)
  3. Converting Python scripts to Jupyter notebooks and executing them
  4. Committing generated notebooks to workspace main branches and tagging each workspace with a version matching the released library

The pipeline is triggered via GitHub Actions (release.yml) and is manually dispatched with configurable options. Release-readiness gating happens upstream: the PyAutoBrain release agent calls pyauto-heart readiness and only dispatches release.yml on a green verdict.

Bash CLI

Every operation in this repo is invokable from the shell via the autohands dispatcher at bin/autohands. List subcommands with autohands help; print the docstring for one with autohands help <subcommand> (or autohands <subcommand> --help).

Recommended alias for ~/.bashrc:

alias autohands-help='$HOME/Code/PyAutoLabs/PyAutoHands/bin/autohands help'

The dispatcher routes to the underlying bash script directly, or to the Python tool with PYTHONPATH already set so the internal build_util / result_collector / env_config imports resolve. The same operations remain callable as Claude skills (/pre_build, /verify_install, /review_release); use the skill when you want the validation + summary wrapper, the CLI when you just want to fire the underlying tool.

Pre-Build Steps

Before triggering a build, run:

bash $HOME/Code/PyAutoLabs/PyAutoHands/bin/autohands pre_build [minor_version]
# minor_version defaults to 1
# (equivalent to: bash $HOME/Code/PyAutoLabs/PyAutoHands/pre_build.sh [minor_version])

This script does the following for each repo:

Repo black generate.py commit & push
autofit_workspace yes yes (autofit) yes
autogalaxy_workspace yes yes (autogalaxy) yes
autolens_workspace yes yes (autolens) yes
autofit_workspace_test yes no yes
autogalaxy_workspace_test yes no yes
autolens_workspace_test yes no yes
euclid_strong_lens_modeling_pipeline yes no yes
HowToGalaxy yes yes (howtogalaxy) yes
HowToLens yes yes (howtolens) yes
HowToFit yes yes (howtofit) yes

Before any side effects, pre_build.sh requires PyAutoHands itself to be on clean main; it produces no PyAutoHands files and never stages or commits that repository. It then invokes PyAutoBrain/bin/ensure_workspace_labels.sh to assert the canonical pending-release label across every release-window repo (idempotent — a no-op when nothing has drifted).

Release-readiness checking is not Build's job — PyAutoHands is a pure executor. The version-skew check that used to live here (verify_workspace_versions.sh, a fail-fast guard against a workspace pinned ahead of its installed library, or a config/general.yaml ↔ version.txt disagreement) now lives in PyAutoHeart as the version_skew check feeding pyauto-heart readiness. The PyAutoBrain release agent gates on pyauto-heart readiness before invoking pre_build; a human running pre_build directly is trusted to have checked readiness first. See PyAutoHeart for the resolution precedence (config/general.yaml:version.workspace_version, then version.txt) — mirroring autoconf.workspace.check_version. Since PyAutoBuild#120, releases no longer write workspace version pins or commit __init__.py stamps back to library mains (wheels are stamped at build time; tags are the release anchor): the runtime check enforces a compatibility floor (version.minimum_library_version, bumped deliberately — PyAutoNerves#118), and Heart's version_skew check needs a follow-up rework to compare floors against release tags rather than stamp-vs-pin.

generate.py is run from the workspace root with PYTHONPATH pointing at PyAutoHands/autohands/. Only specific safe directories are committed — never output/, output_model/, or run-generated artefacts. After all workspaces are done, gh workflow run release.yml dispatches the GitHub Actions release; there is no PyAutoHands self-commit step.

Workspace Folder Structure

Each workspace repo (autofit_workspace, autogalaxy_workspace, autolens_workspace, their _test variants, and the lecture repos HowToGalaxy/HowToLens) has the following expected structure. Only these paths should ever be committed.

Folder / file autofit autogalaxy autolens Notes
config/ yes yes yes PyAutoNerves config files
dataset/ yes yes yes Allowlisted real observational data only; simulated datasets are never committed (#126/#150)
notebooks/ yes yes yes Generated from scripts/ by generate.py
scripts/ yes yes yes Source Python scripts
slam_pipeline/ no no yes autolens only
output/ — — — Always empty — kept under git with a .gitignore only
Root-level files yes yes yes README.md, llms-full.txt, workspace_index.json, requirements.txt, LICENSE* — committed by release.yml on the runner, not by pre_build.sh (#156)

Paths that must NEVER be committed

  • run_logs/ — local persistent smoke/release history generated by run_all.py
  • output/ contents — run results; the folder itself exists only via .gitignore
  • output_model/ — model JSON/pickle artefacts written during script execution
  • path/to/model/ or any nested model JSON files written at runtime
  • .fits files outside dataset/ (e.g. image.fits, dataset.fits generated by simulators into scripts/ or other subdirectories)

Running Tests

# Run all tests
pytest

# Run a single test
pytest tests/test_files_to_run.py::test_script_order

Codex / sandboxed runs

When running Python from Codex or any restricted environment, set writable cache directories so numba and matplotlib do not fail on unwritable home or source-tree paths:

NUMBA_CACHE_DIR=/tmp/numba_cache MPLCONFIGDIR=/tmp/matplotlib pytest

This workspace is often imported from /mnt/c/... and Codex may not be able to write to module __pycache__ directories or /home/jammy/.cache, which can cause import-time numba caching failures without this override.

Key Scripts

All scripts in autohands/ are run from within a checked-out workspace directory (not from this repo root). They rely on PYTHONPATH including the PyAutoHands directory.

  • run_python.py <project> <directory> [--report-dir DIR] [--env-config FILE] [--list FILE] — Executes Python scripts in a workspace folder, skipping files listed in config/no_run.yaml. Coverage is opt-out by default (recursive discovery); --list switches it to opt-in, running exactly the entries of a script list such as a workspace's smoke_tests.txt, in that list's own order. no_run.yaml filters discovery only — with --list the allowlist is authoritative, because the two are policy for different runs (no_run.yaml for the release mega-run and notebook generation, the allowlist for the PR smoke gate) and a script legitimately appears in both. A missing list file is an error, never an empty run; a missing no_run.yaml is an error for discovery but fine under --list.
  • run.py <project> <directory> [--visualise] [--report-dir DIR] [--env-config FILE] [--list FILE] [--no-write-back] [--retry-from DIR] — Executes Jupyter notebooks in a workspace folder, skipping files in config/no_run.yaml. --list switches coverage from opt-out discovery to opt-in (a workspace's smoke_notebooks.txt, in that list's order); no_run.yaml filters discovery only — with --list the allowlist is authoritative. --no-write-back executes a throwaway copy so the committed notebooks stay clean (a PR gate wants this; the release pipeline does not, since there the outputs are the product). --retry-from <scripts-dir> regenerates a failing notebook from its source .py and retries ONCE — the stale-notebook recovery; timeouts are never retried.
  • generate.py <project> — Converts Python scripts in scripts/ to .ipynb notebooks in notebooks/, run from within the workspace root
  • generate_markdown.py <project> [--only <substring>] [--optimize-only] — Renders the curated scripts listed in the workspace's config/build/markdown_examples.yaml to executed markdown pages with output images under markdown/, plus an index, committed so examples are readable on GitHub. Manual / at-release only, never per-commit; refuses PYAUTO_TEST_MODE (truncated searches make wrong images — model-fit reruns instead resume from the completed output/ cache); never renders features/ scripts; restores tracked files a script modifies (e.g. simulators rewriting dataset/). Extracted figures are optimized on the way out, but only for the render in progress — --optimize-only renders nothing and puts already-committed markdown/**/<page>_files/ PNGs through the same optimizer, for pages built before it shipped. Rules and rationale in the module docstring.
  • script_matrix.py <project1> [project2 ...] — Outputs a JSON matrix of {name, directory} pairs for GitHub Actions matrix strategy
  • tag_and_merge.sh --version <version> — Commits pending changes and tags library repos (PyAutoNerves, PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens) for release
  • url_check — URL hygiene moved to PyAutoHeart (Heart owns all health checking). autohands url_check is now a thin shim to pyauto-heart url_check; the ecosystem-wide sweep runs from PyAutoHeart's central url-check.yml workflow (replacing the old per-repo url_check.yml workflows). The runnable scripts live at PyAutoHeart/heart/checks/url_check*.{sh,py}.
  • bump_colab_urls.sh <new-tag> — Rewrites every colab.research.google.com/github/PyAutoLabs/<repo>/blob/<old-tag>/... URL in cwd to use <new-tag>, where <repo> is one of autofit_workspace, autogalaxy_workspace, autolens_workspace, HowToFit, HowToGalaxy, HowToLens. Called by the release_workspaces and bump_library_colab_urls jobs in release.yml so README/docs Colab links always pin to the just-released tag. Idempotent; skips URLs not in canonical PyAutoLabs/date-tagged form.

What --report-dir contains

Every RunReport.write() (so run_python.py, run.py and generate.py, on every leg that is given --report-dir) puts three things in that directory:

  • <project>__<dir>__<run_type>.json — the per-run report. Read by PyAutoHeart's script_timing (which globs *__script.json) and, via aggregate_results, by its test_run check. Its shape is a published interface — add fields to the timing dataset below, not to this.

  • <project>__<dir>__<run_type>.md — the human-readable form.

  • smoke_timings.json — the standing per-entry timing dataset: one row per entry, {entry, kind, status, seconds, cap_s, exit_code}, plus the run metadata (project, directory, run_type, env_profile, python, ts) and a legs list. seconds is the runner's own time.time() measurement — the same number the PASS/TIMEOUT console line prints, never re-derived — and cap_s is build_util.timeout_for's resolved cap, so a TIMEOUT row records the cap it hit. An entry that never ran (skipped, or listed but missing) carries "seconds": null rather than a fabricated zero.

    There is one timings file per report directory, not one per leg: a directory receives several runner invocations (a smoke gate's script and notebook legs; every directory of every workspace in the run_all mega-run), and each write() merges its rows into the existing file keyed on the entry path. So the script leg and notebook leg both survive, re-running a leg replaces its own rows, and the top-level metadata describes the leg that wrote last while legs records every contributor. aggregate_results skips the file by name — it is a sidecar, not a run report.

    When $GITHUB_STEP_SUMMARY is set (i.e. in Actions), each write() also appends a slowest-first markdown table of its own entries to the job summary, so a run's timings are readable without downloading an artifact. Off CI nothing is appended. PyAutoHeart's reusable smoke-tests.yml uploads the report directory as smoke-timings-<python-version>, which is what makes the dataset persist beyond the job.

Architecture

Script-to-Notebook Conversion Pipeline

generate.py → generate_autofit.py + build_util.py:

  1. add_notebook_quotes.py transforms triple-quoted docstrings into # %% cell markers in a temp .py file
  2. ipynb-py-convert converts the temp file to .ipynb
  3. build_util.uncomment_jupyter_magic() restores commented-out Jupyter magic commands (e.g. # %matplotlib → %matplotlib)
  4. build_util.inject_colab_setup() prepends the standard Google Colab setup cell pair (see "Google Colab architecture" below)
  5. Generated notebooks are git add -fed directly

Optional-dependency skip guards in notebooks

The workspaces guard optional-dependency examples with the script idiom

if importlib.util.find_spec("<optional-dep>") is None:
    print("Skipping ...")
    sys.exit(0)

As a .py script that is a clean exit 0. In a Jupyter kernel the same call raises SystemExit, nbclient marks the cell as errored and jupyter nbconvert --execute exits non-zero — so an intended skip is reported as a notebook failure. CI never sees it (its matrices install the optional extras); users and local runs without them do.

The fix is classification at the point of execution, not a source rewrite: build_util.is_clean_skip_exit(output) inspects the combined stdout/stderr of an nbconvert run and returns True only when the run's single CellExecutionError terminates in SystemExit: 0 (ANSI escapes stripped first — IPython colours the traceback). SystemExit: 1 and every other exception stay failures. build_util.execute_notebook consults it in its CalledProcessError branch and records a PASS, alongside the existing InversionException exemption.

Rewriting the guard at generation time was rejected: it would need a source-level transform of arbitrary "skip the rest of the notebook" control flow, would diverge notebook semantics from the script the user reads, and would require regenerating and committing notebooks across every workspace.

Propagation. Each workspace repo carries its own .github/scripts/run_smoke.py (no template sync — PyAutoHeart's reusable smoke-tests.yml deliberately leaves the runner in the workspace). The three notebook-capable copies each had their own execute_notebook, so the skip guard had to be adopted per repo — a two-line change, applied 2026-07-25:

from build_util import is_clean_skip_exit
...
rc, output = execute_notebook(nb_path, env)
if rc != 0 and is_clean_skip_exit(output):
    rc = 0

The run_smoke.py copies

Ten copies in three variants (measured 2026-08-24 across every repo's main). They are three structurally different programs, not revisions of one:

Variant Repos Lines Notebook leg Coverage
workspace autofit_workspace, autogalaxy_workspace, autolens_workspace 356 yes opt-in (smoke_tests.txt + smoke_notebooks.txt)
workspace_test autofit_workspace_test, autogalaxy_workspace_test, autolens_workspace_test, autocti_workspace_test 198 no opt-in (smoke_tests.txt)
HowTo HowToLens, HowToGalaxy, HowToFit 75 no opt-out (no_run.yaml)

There is no live drift inside any variant: the three workspace copies are byte-identical, the four workspace_test copies differ only in two docstring lines, and the three HowTo copies differ only in their PROJECT constant. That state was reached by three manual N-repo sweeps, not by a sync mechanism — #185 (env resolution collapsed onto env_config), #226/#227 (timeout_for and kill_group promoted here, so all ten honour BUILD_SCRIPT_TIMEOUT and the process-group kill), and the jupyter-guard fix. Only the skip guard is notebook-specific: is_clean_skip_exit belongs in exactly the three notebook-capable copies, since the other seven never shell out to jupyter.

Direction of travel: delegation. The HowTo tier is the target shape — 75 lines of PROJECT plus a subprocess.run into run_python.py — and it needed none of those three sweeps, precisely because it holds no logic. The obstacle was never behaviour but discovery model: run_python.py was opt-out only, while the other two variants are opt-in allowlists. --list <file> closes that gap for the script leg, so the workspace_test variant can collapse to a HowTo- shaped delegator.

Under --list the allowlist is authoritative and no_run.yaml is not consulted. The two are policy for different runs — no_run.yaml for the release mega-run and notebook generation, the allowlist for the PR smoke gate — and the vendored runners have never read no_run.yaml at all. Measured before this was settled: filtering the allowlists by no_run would have dropped 13 scripts across autogalaxy_workspace_test (9), autolens_workspace_test (2), autofit_workspace (1) and autolens_workspace (1), every one of which runs in smoke today. The notebook leg followed in the same change: run.py gained the matching --list, plus --no-write-back and --retry-from, so the workspace variant can collapse too.

Two of the three notebook behaviours that variant held needed real promotion, and one dissolved on inspection:

  • --no-write-back (promoted). run_notebook.py writes executed outputs back in place — right for generation, where the outputs are the product, but wrong for a PR gate, which must not dirty the tree it is testing. The flag executes a throwaway copy instead. The kernel cwd is pinned to the repo root either way, so this supersedes the workspace copy's staged-copy-at-root trick rather than porting it.
  • --retry-from (promoted). One regenerate-from-source retry recovers a stale notebook whose script moved on. Deliberately narrow: a TIMEOUT is never retried (a second full cap for the same answer), a clean skip-guard exit is already a PASS and never reaches it, and the retry's verdict REPLACES the first attempt's so one notebook contributes one result.
  • JUPYTER_MISSING_RC (not needed). That guard exists because the workspace copy shelled out to a bare jupyter binary, so an absent toolchain raised FileNotFoundError out of main() and aborted the run with no summary line. execute_notebook invokes sys.executable run_notebook.py, which always exists, so a missing toolchain is an ordinary non-zero exit — one FAIL, run continues. The failure mode is structurally absent here.

Google Colab architecture

Every published notebook must be runnable on Google Colab with zero local installation. Four pieces, spread over three organs plus PyAutoNerves:

  1. Runtime bootstrap — PyAutoNerves/autoconf/setup_colab.py. A _PROJECTS registry (autofit, autogalaxy, autolens, howtofit, howtogalaxy, howtolens) maps each notebook repo to its package stack, workspace repo and Colab directory. setup_colab.setup("<project>") is a no-op outside Colab; on Colab it pip-installs the stack (--no-deps — Colab ships the scientific base), shallow-clones the workspace at the tag matching the installed release (default branch as fallback) and points autoconf's config/output paths at it.
  2. Generation-time injection — build_util.inject_colab_setup(notebook, project), called by generate.py / generate_autofit.py after every py→ipynb conversion. It prepends a markdown explainer + code cell calling setup_colab.setup("<project>"), immediately after the notebook's title cell. Notebooks whose script already hand-writes a setup_colab call are left untouched. build_util.COLAB_PROJECTS must stay in sync with the PyAutoNerves registry; an unknown project fails generation loudly. Coverage is therefore guaranteed by construction — every generated notebook is Colab-ready, with no per-script maintenance.
  3. Release maintenance — bump_colab_urls.sh (above) re-pins every canonical Colab URL in READMEs/docs/notebooks to the just-released tag, from the release_workspaces and bump_library_colab_urls jobs. Only date-tagged PyAutoLabs/<repo> URLs are bumped — unpinned or wrong-owner URLs are invisible to it, which is why Heart forbids them (next item).
  4. Monitoring — PyAutoHeart's central url-check.yml (weekly): the offline guard (heart/checks/url_check.sh) forbids Colab URL forms that rot (Binder, Jammy2211 owner, /blob/release/, unpinned /blob/main/, chapter paths pointing at workspace repos instead of the HowTo repos); the live audit (url_check_live.py) converts Colab URLs to raw-GitHub form and 404-checks that each linked notebook actually exists at its pinned tag.

Script Execution Order

build_util.find_scripts_in_folder() enforces a specific ordering:

  1. Scripts with "simulator" in the path (data must be generated first)
  2. Scripts named start_here.py
  3. All other scripts

Config Files

Each workspace owns its own build config under <workspace>/config/build/:

  • no_run.yaml — flat list of script/notebook patterns to skip during execution. Required: every build target must own one (an empty file is valid and skips nothing). run.py raises FileNotFoundError if it is missing.
  • profile_smoke.yaml / profile_release.yaml — defaults + per-pattern overrides for environment variables
  • visualise_notebooks.yaml — flat list of notebook stems to run when the --visualise flag is used. Optional: a workspace without one simply has nothing marked for visualisation.

config/build/ is the single source of truth — autohands/config/ holds no per-project config fallbacks. The keyed-dict fallbacks (no_run.yaml, copy_files.yaml, visualise_notebooks.yaml) were removed once every build target owned its own files; the only file left in autohands/config/ is workspaces.yaml, which is build policy (the run matrix), not workspace config.

The copy_files.yaml mechanism — which copied listed scripts verbatim into notebooks/ instead of converting them — was removed entirely. It resolved to an empty list in every workspace and had produced no output in any build.

Environment Variables

  • BUILD_PYTHON_INTERPRETER — Python interpreter to use for script execution (defaults to python3)
  • PYAUTO_TEST_MODE — Set to 1 for workspace runs, 0 for *_test workspace runs
  • PYAUTO_SMALL_DATASETS — Set to 1 for workspace runs (caps grids to 15x15), not set for *_test runs
  • PYAUTO_FAST_PLOTS — Set to 1 for workspace runs (skips tight_layout() in subplots and critical curve/caustic overlays in plots), not set for *_test runs
  • JAX_ENABLE_X64 — Set to True during CI runs

GitHub Actions Workflow

The workflow (release.yml) is manually dispatched with inputs:

  • minor_version — appended to date-based version (format: YYYY.M.D.minor)
  • rehearsal — the one mode switch (default false = full real release). When true, it is TestPyPI-only rehearsal mode: build every package from source, publish to TestPyPI, emit the resolved version as the testpypi-rehearsal-version artifact, then STOP (no PyPI upload, no git tag, no notebook/version commits, no Colab bumps). This is the mode the Heart/Brain release-validation gate dispatches so it can install and validate the built wheels.

(The legacy skip_scripts / skip_notebooks / skip_release force-through knobs and the update_notebook_visualisations path were removed with the Heart/Build split — Build is a pure executor with no ad-hoc skip levers or inline notebook-visualisation job. "Build without releasing" is now exactly what rehearsal mode is for.)

release.yml is a pure executor: it builds, tests-the-install, publishes to PyPI, tags every library and workspace, and commits generated notebooks + Colab URL bumps to the workspaces (version stamps are build-tree-only since #120 — no __init__.py commit-backs to library mains, no workspace version pins). Workspace-integration validation (the old find_scripts / generate_notebooks / run_scripts / run_notebooks / analyze_results jobs) moved to PyAutoHeart's workspace-validation.yml; release readiness is gated upstream by the PyAutoBrain release agent via pyauto-heart readiness before this workflow is dispatched. The script_matrix.py / run_python.py / run.py / aggregate_results.py primitives remain here and are checked out + reused by the Heart workflow. The never-rewrite-history rules live in AGENTS.md and apply here as everywhere.