Skip to content

REFERENCE.md

Latest commit

 

History

History
536 lines (434 loc) · 25.8 KB

REFERENCE.md

File metadata and controls

536 lines (434 loc) · 25.8 KB
Raw

PyAutoMind reference

The registry schemas, prompt conventions, and workflow detail for this repo. Moved verbatim from README.md on 2026-07-10 — agent docs that point at README sections ("Prompt taxonomy", "Prompt file format", the active.md / completion-record schemas) resolve here, one link from the README.

What a prompt looks like

Here's a real prompt — the contents of autoarray/psf_oversampling.md — that became a tracked task. This is the level and style of detail to aim for in your own GitHub issue: free-form prose, with @RepoName/path/to/file.py references so the tooling knows which repo and files to target. No boilerplate.

A point spread function is used to blur images via 2d convolution.

This blurring occurs predominantly in the package @PyAutoArray/autoarray/operators/convolver.py.

The source code currently requires PSF blurring to occur at the same resolution (pixel scale) as the
image, meaning the PSF is always the same resolution as the image.

However, for modeling, convolution can be performed at a higher resolution than the image, which allows for more accurate
blurring and modeling of the image. This requires us to have an oversampled PSF, which is a PSF that has a higher
resolution than the image.

For modeling, where images are generated PSF blurring happens in @PyAutoGalaxy/autogalaxy/operate/image.py.

Modeling can always evaluate images using a hgiher resolition grid, blurring them with the PSF at high
resolution and then downsample to the observed image resolution. Oversampling is implemented in
@PyAutoArray/autoarray/operators/over_sampling.

Note that over sampling often uses an adaptive sub-szie, which means that 2D covnolution with a PSF is not
well defined. for now, we will assume adaptive over sampling is not used.

I want us to be able to append the Convolver class with a convolve_over_sample_size integer, which specifies the over sample size of the PSF.
This will allow us to perform convolution at a higher resolution than the image, which will improve the accuracy of the blurring and modeling of the image.
For example, if convolve_over_sample_size is 2, then the PSF will be oversampled by a factor of 2, meaning it will have a resolution that is 2 times higher than the image.

This, in turn, means out imaging object @PyAutoArray/autoarray/dataset/imaging/dataset.py will need to be
extended to include the convolve_over_sample_size_lp and convolve_over_sample_size_pixelization attributes, which will
specify the over sample size of the PSF for the lensing and pixelization operations, respectively.

class Imaging(AbstractDataset):
    def __init__(
        self,
        data: Array2D,
        noise_map: Optional[Array2D] = None,
        psf: Optional[Convolver] = None,
        psf_setup_state: bool = False,
        noise_covariance_matrix: Optional[np.ndarray] = None,
        over_sample_size_lp: Union[int, Array2D] = 4,
        over_sample_size_pixelization: Union[int, Array2D] = 4,
        use_normalized_psf: Optional[bool] = True,
        check_noise_map: bool = True,
        sparse_operator: Optional[ImagingSparseOperator] = None,
    ):


Also,read through the @PyAutoArray/autoarray/inversion/inversion/imaging package, and parents, to see
how PSF convolution enters this. I think we can get it to work in PyAutoArray/autoarray/inversion/inversion/imaging/mapping.py,
and will leave work in PyAutoArray/autoarray/inversion/inversion/imaging/sparse.py to future work.

This is a complex task, therefore I think we should extend @autolens_workspace_test/scripts/imaging/convolution.py
with a numerical test.

We should then build on this test in a separate test file using a simple over sampled PSF, to get a numerical
result we can test the source code against.

@autolens_workspace/scripts/imaging/simulator.py is a good example we can build on to show how to use
over sampled PSFs in a real simulation. We can extend this script to show how to use over sampled PSFs in a real simulation.

Come up with a plan to implement over sampled PSFs.

That prompt becomes a GitHub issue, gets routed to the affected repos (PyAutoArray, PyAutoGalaxy, the autolens workspaces), and lands as PRs against each. Typos, half-finished thoughts, and "I think we should…" are fine — write naturally, the AI fills in the rest.

How a prompt flows through the workflow

  idea               ── you write it in ideas.md
    │
    ▼
  draft prompt       ── you write a markdown file under
    │                   draft/<work-type>/<target>/<name>.md
    ▼
  /start_dev         ── reads the prompt, audits the code, drafts an issue,
    │                   creates the GitHub issue, registers the task in
    │                   active.md, moves the prompt draft/ → active/
    ▼
  active.md entry    ── the task is now tracked across machines and sessions
    │
    ▼
  /start_library     ── creates a worktree, branch, opens dev environment
    or                  (or workspace variant — chosen automatically)
  /start_workspace
    │
    ▼
  development        ── code, tests, run smoke tests, commit
    │
    ▼
  /ship_library      ── runs tests, opens PR, waits for merge
    or
  /ship_workspace
    │
    ▼
  PR merged          ── post-merge cleanup deletes the worktree, drops the
    │                   active.md entry, and writes the dated completion record
    │                   complete/<YYYY>/<MM>/<slug>.md (lifecycle.py record)
    ▼
  done

The slash commands above are skills hosted across the organism (Brain, Heart) but all read/write Mind's registry via workspace-root-anchored paths. One operates over the registry without starting work:

  • /health status — dashboard of active.md, planned.md, complete/ (a PyAutoHeart status view, reached through the single /health door). Continuity across execution environments needs no special step — any environment reads active.md and resumes an in-flight task.

Repository layout

PyAutoMind/
├── README.md                ← short front page
├── dashboard.md             ← GENERATED task page (picks / in flight / parked / planned / backlog)
│                              `pyauto-brain intake --apply dashboard`; CI self-heals it on main
├── REFERENCE.md             ← this file (schemas + conventions)
├── .gitignore
│
├── active.md                ← tasks currently in progress (one ## section per task)
├── ideas.md                 ← raw incubating ideas, no structure required
├── parked.md                ← started/scoped but not in flight (stashes, orphan worktrees, deferred)
├── planned.md               ← issued tasks blocked from starting (created on demand)

├── queue.md                 ← processing queue for /register_and_iterate
│
│   PROMPT-FILE LIFECYCLE (issue #71): draft/ → active/ → complete/YYYY/MM/.
│   Drafts are organised by WORK TYPE (first folder), then TARGET (second).
│   See "Prompt taxonomy" below and ROUTING.md.
├── draft/                   ← NOT STARTED (intaken, pre /start_dev)
│   ├── feature/             ← new user-facing or scientific capabilities
│   │   ├── autoarray/  autofit/  autogalaxy/  autolens/  workspaces/  pyautobrain/  …
│   ├── bug/                 ← incorrect behaviour, crashes, regressions
│   ├── refactor/            ← internal restructuring, no intended behaviour change
│   ├── docs/                ← documentation, tutorials, notebooks, examples
│   ├── test/  release/  maintenance/  research/  experiment/
│   └── triage/              ← classification still unclear; needs manual review
│
├── active/                  ← ISSUED, in flight (moved here by /start_dev)
│
├── complete/                ← SHIPPED — rich completion records (see complete/AGENTS.md)
│   ├── AGENTS.md            ← archive schema + how to look records up
│   └── 2026/07/<slug>.md    ← bucketed by completion date (zero-padded months)
│
│   (complete/archive/ holds retired non-record material — see below)
├── complete/archive/        ← skipped by lifecycle.py check/index
│   ├── epics/               ← retired multi-task epic trackers (former z_features/)
│   └── shelved/             ← deferred prompts + dev notes (former z_vault/)
│
├── scripts/
│   ├── status.sh            ← prompt inventory helper
│   ├── lifecycle.py         ← prompt-file lifecycle engine (move/split/check)
│   └── prompt_sync.sh       ← commit/push helpers sourced by skills
│
└── skills/                  ← Mind-owned skills + the ownership audit
    ├── OWNERSHIP.md          ← where every workflow skill lives, and why
    └── create_issue/         ← convert a prompt into a tracked GitHub issue

PyAutoMind/skills/ now holds only the Mind-owned create_issue skill (plus OWNERSHIP.md). The development-workflow skills were re-homed to the organs that own them — PyAutoBrain (start_dev, start_dev_for_user, plan_branches, start_library, start_workspace, ship_library, ship_workspace, register_and_iterate, health [the single health door, with check sweep, status dashboard, and full release-run legs]), PyAutoHeart (worktree_status, and the health-leg procedures health_sweep/, pyauto-status/, and pyauto-status-full/ that /health drives), and autolens_profiling (profile_likelihood). The handoff skill was retired (PyAutoBrain runs uniformly across execution environments — see OWNERSHIP.md). General PyAuto tooling (release prep, dependency audits, smoke tests, lint sweeps) lives in admin_jammy/skills/.

scripts/prompt_sync.sh is sourced by skills that mutate registry files (active.md, planned.md, etc.) to commit and push back to origin. It replaces a now-removed admin_sync.sh helper that formerly operated on admin_jammy/prompt/.

Prompt taxonomy

PyAutoMind organises intent by the kind of thinking required; PyAutoBrain uses that structure to choose the right reasoning agent.

Prompts start at draft/<work-type>/<target>/<name>.md (and advance draft/ → active/ → complete/YYYY/MM/; issue #71):

  • The first folder answers what kind of thinking or agent is needed? — the work type.
  • The second folder answers what domain or repository is affected? — the target repo (autoarray, autofit, autogalaxy, autolens, autolens_assistant, pyautobrain, …), a workspace bucket (workspaces), or a topic series (jax_substructure, weak, cluster, priors).

Work types → PyAutoBrain agents

Folder Holds Future PyAutoBrain agent
feature/ new user-facing or scientific capabilities feature planner
bug/ incorrect behaviour, crashes, regressions debugger
refactor/ internal restructuring, no intended behaviour change refactor architect
docs/ documentation, tutorials, notebooks, examples documentation agent
test/ test coverage, smoke tests, validation scripts test engineer
release/ packaging, versions, deployment, release readiness release engineer
maintenance/ dependency updates, hygiene, cleanup, small tech debt hygiene agent
research/ exploratory scientific / algorithmic investigation research analyst
experiment/ prototypes, spikes, proof-of-concept work prototype agent

triage/ holds prompts whose classification is still unclear — file there with a short note and re-home once the work type is obvious. The full mapping (and the note that the agents themselves live in PyAutoBrain, not here) is in ROUTING.md.

Good prompt paths

feature/autolens/potential_corrections.md
bug/autoarray/mask_edge_case.md
refactor/autofit/result_object_cleanup.md
docs/workspaces/pixelization_tutorial.md
research/autofit/sbi_design.md
experiment/autoarray/jax_sparse_mapping.md

Not work-types

active/ and complete/ are workflow lifecycle folders, not routed by work type. complete/archive/ holds retired non-record material — epics/ (former z_features/ trackers) and shelved/ (former z_vault/ deferred prompts + dev notes) — and is skipped by lifecycle.py check/index.

Migration note

The repository previously used the target repo as the first folder (autoarray/foo.md). Those prompts have moved to <work-type>/autoarray/foo.md. Routing always keyed off the @RepoName references in a prompt's body, not its folder, so the skills accept both old and new paths during the transition — but new prompts should use the work-type layout.

Conventions

Naming

  • Prompt filenames are lowercase kebab_or_snake_case.md.
  • Numbered series use a leading number: 0_docs.md, 1_simulator.md. Skipping a number (e.g. feature/weak/2_*.md not present) is fine — it usually means a step was consolidated or deferred.
  • First folder = work type (feature/, bug/, …); second folder = target repo or domain (lowercased, no Py prefix): feature/autoarray/, bug/autofit/, refactor/autogalaxy/. Workspace prompts go under <work-type>/workspaces/ regardless of which workspace. See "Prompt taxonomy".

Prompt file format

Free-form markdown. Strong conventions:

  • Reference repos and files with @RepoName/path/to/file.py (e.g. @PyAutoFit/autofit/non_linear/search.py). /start_dev parses these to identify the primary target repo.

  • One prompt = one task = one PR (ideally). If a prompt outlines several loosely-related changes, split before issuing.

  • No frontmatter required. Title in the first line is helpful but optional.

  • Optional metadata header. A prompt may carry a light, human-writable header near the top so both people and PyAutoBrain can see its type/target at a glance. This is a convention, not a schema — never required, no YAML frontmatter:

    # Short task title

Type: feature
Target: PyAutoLens
Repos:
- PyAutoLens
- autolens_workspace
Difficulty: medium        # small | medium | large | too-large
Autonomy: supervised      # safe | supervised | human-required
Priority: normal          # low | normal | high
Status: draft
Filed: 2026-07-09         # optional; the day the prompt was written
Issued: 2026-08-19        # optional; set when the prompt advances to active/
Blocked-by: PyAutoFit#1436          # optional; see "Declaring a gate" below

    When present, Type: should match the work-type folder. The goal is light structure, not bureaucracy — prompts stay free-form prose.

    Declaring a gate — Closes-when: / Blocked-by:. Both optional. A prompt that waits on something external can say so in a form lifecycle.py issues --drafts can grade:

    Closes-when: autolens_profiling#70    # this prompt is DONE when that closes
Blocked-by: PyAutoArray#431, PyAutoGalaxy#486   # READY TO START when all close

    The two readings are opposite, which is the whole point. Prose cannot be graded, so a cited issue could mean either and --drafts had to report every one as the same ambiguous question. With a declared key the tool reports the action instead: a closed Closes-when: says likely shipped, verify and retire; a closed Blocked-by: says ready to start. Prompts declaring a gate drop out of the ambiguous advisory list.

    Notes:

    • Accepts Repo#123 shorthand (assumed PyAutoLabs/) or a full URL, and PRs as well as issues. Several refs may be comma-separated.
    • Blocked-by: clears only when every ref closes; a partly-satisfied gate is reported in its own weaker band rather than as ready.
    • Keys inside fenced code blocks are documentation and are ignored, so a prompt may show the syntax without declaring a gate.
    • Advisory, never a gate on the exit code: retiring a prompt writes to complete/ and stays a human act.

    Motivated by the 2026-08-09 draft/ sweep, where five prompts' stated gates had closed without anyone noticing — including one whose exit condition was met the same day it was written.

    The optional Difficulty: / Autonomy: / Priority: keys let both people and PyAutoBrain see, at a glance, how hard a task is, whether an agent can safely take it on, and how urgent it is. What each Autonomy: level does at every workflow checkpoint is defined once in PyAutoBrain/AUTONOMY.md (the autonomy contract); levels bind only under an explicit --auto launch, and --auto runs append their outcome to autonomy_log.md (the calibration log). The Intake (Conception) Agent writes these automatically when it formalises a raw idea (/intake), sourcing Difficulty: from the shared sizing faculty the Feature Agent also uses — so the value shown up front is the one the Feature Agent later acts on. Still a convention, not a schema: all keys remain optional and there is no YAML frontmatter.

active.md schema

Each task is an H2 section:

## <task-name-kebab-case>
- issue: https://github.com/<owner>/<repo>/issues/<n>
- issued: YYYY-MM-DD                              # the day the task was issued
- session: claude --resume <session-id>           # optional
- status: <library-dev | workspace-dev | ready-to-ship | awaiting-input | …>
- location: <cli-in-progress | ready-for-mobile | …>   # optional, used by /handoff
- question: <issue-comment-url>   # optional; set when status is awaiting-input
                                  # (checkpoint-and-continue — PyAutoBrain/AUTONOMY.md)
- heart-ack:                      # optional; --auto launches: the exact YELLOW
  - <reason line acknowledged at launch>   # reason set the human acknowledged
- corrective-red:                 # optional; set when shipping under the
  reason: <exact Heart RED reason string>  # human-authorized corrective-PR
  authorization: <issue-comment-url>       # exception (PyAutoBrain/AUTONOMY.md
                                           # "Corrective-PR exception for Heart
                                           # RED"): names the one RED reason the
                                           # PR repairs + the human's live
                                           # authorization comment
- worktree: ~/Code/PyAutoLabs-wt/<task-name>
- repos:
  - <RepoName>: feature/<branch-name>
- summary: |
    Free-form summary of progress and next steps.

Task dates

The Mind used to date only what it finished — every completion record carries completed: — so it could answer "what shipped in July?" but not "what did we start?". Every task now carries a machine-readable date from the moment it leaves the backlog:

Where Field The event it dates
active.md - issued: YYYY-MM-DD the day the task got its GitHub issue
planned.md - filed: YYYY-MM-DD the day it was scoped
parked.md - parked: YYYY-MM-DD the day it stopped
draft/**/<name>.md Filed: YYYY-MM-DD the day the prompt was written
active/<name>.md Issued: YYYY-MM-DD the day it got its issue, in its light header
complete/<YYYY>/<MM>/<slug>.md - completed: YYYY-MM-DD unchanged — the ledger already did this

The backlog is the largest pool of tasks the Mind holds — 150 prompts against a handful of live rows — so draft/ carrying a date is what lets the dashboard's Recent feed see most of the work at all. A prompt keeps its Filed: when it advances to active/ and gains an Issued:; the later, more specific event is the one that dates the task.

The key names the event, so a merged feed can say what each date means rather than showing a bare timestamp. Reading is tolerant: registered:, started:, planned:, found: and shipped: are all read as dates too (the registries are hand-edited by many sessions, and an entry that says when it happened should count however it said it) — the table is what a writer should use. The most specific event wins when an entry carries several, so a task that was filed and later issued dates from its issue.

A date inside another field's prose (- issue: …/1501 (issued 2026-08-19)) is deliberately not read — that is the un-parseable habit this convention replaces. The prompt's Issued: header is its own copy of the registry date, so an issued prompt stays dated even if its registry row goes missing.

scripts/lifecycle.py dates reports every entry and issued prompt carrying no date; dates --write backfills them retroactively from the evidence the repo already holds, annotating each inferred date with where it came from:

Issued: 2026-08-18 (backfilled from parked.md `parked:`)

The sources, in order: git — the commit that introduced the entry, wrote the draft, or moved the prompt into active/; the prompt's own Intake trailer (<!-- formalised by the Intake (Conception) Agent on … -->); the dated registry entry that claims the prompt; a date the entry already stated in its own prose.

The two states want opposite readings of the same history, and the switch is --follow. An active/ prompt dates from the day it arrived there (being issued is a git mv, so following the rename back would report the wrong day); a draft/ prompt dates from the day it was written, wherever it lived then — the 2026-07-13 lifecycle migration git mv-ed 42 prompts in one commit, and without --follow all 42 would date from the migration rather than from themselves. Nothing is guessed — an entry with no evidence is reported for a human to date by hand. A shallow clone (CI, a cloud session) cannot see past its boundary commit, so git dates at or before it are discarded rather than stamping every task with the day the clone was made.

The dashboard's Recent table is the payoff: it holds the 50 newest events on the work in hand — issued, parked, filed — and shows 10, opening the next 10 on each tap of …. Shipped work stays out of it; complete/index.md is where the ledger is read.

Completion record (complete/<YYYY>/<MM>/<slug>.md) schema

The dated records are the completion ledger (complete/AGENTS.md; the monolithic complete.md was retired 2026-07-16, issue #81). Each record opens with the same fields the old ledger entries carried, then the rich narrative and, appended by lifecycle.py record, the original prompt:

## <task-name>
- issue: https://github.com/<owner>/<repo>/issues/<n>
- completed: YYYY-MM-DD
- library-pr: <url> [, <url>]
- workspace-pr: <url> [, <url>]
- summary: <what landed, gotchas, follow-ups — free-form bullets>

## Original prompt

<the active/ prompt the task started from>

Epic trackers (retired 2026-07-13)

Multi-task epic trackers (umbrella markdown files listing a sequence of sub-prompts) formerly lived in z_features/. That folder was retired into complete/archive/epics/ once the draft/ → active/ → complete/ lifecycle made its z_-prefixed home redundant. The per-task completion records live in complete/<YYYY>/<MM>/; the archived trackers keep the epic-level narrative.

Tracking and inspection

Quick inventory

bash scripts/status.sh

Prints counts per category, lists the active and recently-completed tasks, and and lists the recently-completed tasks.

From inside Claude Code

  • /health status — dashboard of registry state (active, planned, recent complete; PyAutoHeart, via the /health door)
  • /start_dev draft/<work-type>/<target>/<name>.md — read a prompt and route it (PyAutoBrain)
  • /worktree_status — cross-references registry with task worktrees (PyAutoHeart)

How this repo integrates with the rest

The PyAuto workflow has three repos with distinct roles:

Repo Purpose
PyAutoMind (this repo) The Mind: ideas, intent, goals, priorities, the prompt registry and prompt-coupled skills. The starting point.
admin_jammy Personal admin notes only (euclid.md, grants.md, week.md, travel.md, …). Formerly also held PyAuto tooling under software/; that has moved out (worktree/label scripts → PyAutoBrain/bin/, generic skills → Brain/Heart).
PyAutoMemory The Memory organ: topical LLM wikis (wiki/lensing/, wiki/smbh/, wiki/cti/, wiki/methods/, wiki/galaxies/) and a reading queue (reading-queue.md, moved from admin_jammy/papers.md).
PyAuto* libraries and *_workspace* repos Where the actual code work happens. Each task gets a feature branch + worktree under ~/Code/PyAutoLabs-wt/<task-name>/.

Helper scripts that this repo's skills source:

  • PyAutoBrain/bin/worktree.sh — task worktree management (create, remove, conflict check).

These live in PyAutoBrain/bin/ because they're general organism-wide tooling, not prompt-specific. The skills that need them source by absolute path.

Bootstrap on a new machine

cd ~/Code/PyAutoLabs
git clone git@github.com:PyAutoLabs/PyAutoMind.git    # the Mind (this repo)
git clone git@github.com:PyAutoLabs/PyAutoBrain.git   # dev-workflow skills
git clone git@github.com:PyAutoLabs/PyAutoHeart.git   # status / readiness skills
git clone git@github.com:Jammy2211/admin_jammy.git    # general tooling (optional)
bash PyAutoBrain/bin/install.sh                        # symlinks skills + commands

The local checkout directory must be named PyAutoMind. The skills and scripts reference PyAutoMind/... paths directly — e.g. source PyAutoMind/scripts/prompt_sync.sh and git -C PyAutoMind … — so a differently-named directory breaks those commands.

install.sh auto-discovers skills from every present discovery root (admin_jammy/skills/, PyAutoMind/skills/, PyAutoBrain/skills/, PyAutoHeart/skills/, autolens_profiling/skills/) and creates symlinks under ~/.claude/skills/ and ~/.claude/commands/. Roots that aren't checked out are skipped. Re-run any time after pulling new skills from any of those repos.