Skip to content

Commit cb20ecf

Browse files
committed
fix(hygiene): derive the conductor's repo sets from the body map
The hygiene conductor hardcoded the repos it scans: LIB_REPOS=(PyAutoNerves PyAutoFit PyAutoArray PyAutoGalaxy PyAutoLens) ORG_REPOS=(PyAutoBrain PyAutoHands PyAutoHeart PyAutoMind) DOC_REPOS=(PyAutoFit PyAutoGalaxy PyAutoLens) The body map declares six libraries and seven organs, so two libraries were skipped entirely, the config layer was classed as a library where the map calls it an organ, and three organs went uncovered. The drift was invisible by construction: a repo that is never scanned produces no findings, so the conductor reported clean and was believed. Measured, not inferred — `crlf` printed 5 .py with CRLF against a true 127, 122 of them in the one skipped library that has an LF-only rule nobody was enforcing. The sets now come from the map via _hygiene_repos.py, so adding a repo there adds it to the scan. Two modes take a narrower set from what a checkout CONTAINS rather than its category, because category alone gets it wrong: the config layer is an organ yet ships a real distribution, so keying `deps` off `category: library` would have dropped it — this same bug, one line down. `docs` was pinned to three named repos and never noticed a fourth acquiring Sphinx docs; `pyproject.toml` and `docs/api/` presence are the honest tests. Measured effect (17 repos scanned, was 9): crlf 5 -> 167 cosmetic, deps 5 -> 8 pyproject.toml, docs 3 -> 4 repos. COVERAGE ONLY — the backlog this exposes is a separate triage task and no finding is fixed here. Also fixes a third defect in the same surface: an empty scan root, or an unreachable body map, made every repo-array mode report `clean`. A zero from "nothing was examined" and a zero from "nothing was wrong" are indistinguishable to a reader and only one is good news, so those modes now report `unscanned` with the reason, and the default scan leads with a banner. Scoped to the modes that read the arrays: the helper-backed modes discover their own targets by walking the root, so suppressing them would hide real findings. The conductor now names no repository at all, so it carries no tenant-firewall allowlist entry — one less file an adopting fork must rewrite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzmzZidPBRqQZjEEw1d6ET
1 parent a2264fe commit cb20ecf

4 files changed

Lines changed: 529 additions & 44 deletions

File tree

agents/conductors/hygiene/AGENTS.md

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,22 +32,58 @@ kinds, which is what makes its count comparable (or not):
3232
delegated skill runs, so the count is **not** a problem count (`deps`, `docs`).
3333
- **advisory** — no cheap local signal at all (`noise`).
3434

35+
A mode also carries a **status**, and one of them is not a count at all:
36+
**`unscanned`** means the mode read *no repository* — the scan root holds no
37+
managed checkout, or the body map could not be reached. It is reported instead
38+
of `clean` because a zero from "nothing was examined" and a zero from "nothing
39+
was wrong" are indistinguishable to a reader, and the first is not good news.
40+
41+
## Which repositories it scans
42+
43+
**Derived from the body map (`repos.yaml`), never listed here or in the script.**
44+
The conductor takes `library`, `organ` and `workspace` from the map via
45+
`_hygiene_repos.py`; adding a repo to the map adds it to the scan.
46+
47+
This was a bash array once, and it drifted: five libraries where the map
48+
declared six, four organs of seven, and a CRLF count of **5** against a true
49+
**127**. The drift was invisible precisely because an unscanned repo yields no
50+
findings — the conductor reported clean and was believed. `repos_sync.py`'s
51+
`check_hygiene_coverage` now fails if the derived sets stop matching the map, or
52+
if a repo name is written back into a `*_REPOS=(…)` array.
53+
54+
Two modes need a narrower set, and both read what a checkout **contains** rather
55+
than its category — because category alone gets it wrong. The config layer is an
56+
*organ* in the map yet ships a real distribution, so keying `deps` off
57+
`category: library` would silently drop it:
58+
59+
| Set | Rule | Modes |
60+
|-----|------|-------|
61+
| code repos | `library` + `organ` | `tidy`, `deps`, `docs`, `packaging` |
62+
| scanned repos | code repos + `workspace` | `crlf`, `artifacts` |
63+
| ships a distribution | has a `pyproject.toml` | `deps` |
64+
| ships api docs | has a `docs/api/` tree | `docs` |
65+
66+
The helper-backed modes (`docstrings`, `refs`, `optdeps`, `extras`, `config`)
67+
are **not** on this list: they discover their own targets by walking the scan
68+
root for workspace-shaped directories, so they can find material the map never
69+
names — and they keep reporting even when the repo-array modes are `unscanned`.
70+
3571
| Mode | Pre-scan (kind) | Delegates to |
3672
|------|-----------------|--------------|
3773
| `perf` | dev-loop timing — prefers Heart's tracked timing legs when present (`import_time`, `unit_test_timing`, `workspace_testmode_timing`), else times `import <pkg>` per library in a **subprocess** (**timing**) | `/refactor` / `/bug` (+ Heart timing legs) |
3874
| `tidy` | git debris — stale branches, stashes, `[gone]` refs, dirty checkouts (**debris**) | **condemn** → files candidates into `condemned.md` async (PyAutoGut archives the fragile forms); no synchronous per-item gate |
3975
| `sweep` | reads `condemned.md`, classifies entries by their transit clock (**due** / pending / undated) | `pyauto-gut void` for past-due entries, behind the existing `repo_cleanup` safety gates |
4076
| `noise` | none — needs a pytest + workspace-script run (**advisory**) | `/cli_noise_clean` (Heart) |
41-
| `deps` | capped/pinned specifiers in library `pyproject.toml` (**surface**) | `/dep_audit` (Heart, hits PyPI) |
42-
| `docs` | `docs/api/*.rst` + `currentmodule` counts across the 3 doc repos (**surface**) | `/audit_docs` (Heart, imports) |
43-
| `crlf` | executable scripts (`.sh` + shebang-`755` `.py`) with CRLF — the shebang breaks on Linux/HPC (**debris**, the ranked count); library `.py` CRLF is reported separately as *cosmetic* (Python reads it fine — don't mass-normalise) | `/refactor` + `.gitattributes eol=lf` |
77+
| `deps` | capped/pinned specifiers in every managed repo that ships a `pyproject.toml` (**surface**) | `/dep_audit` (Heart, hits PyPI) |
78+
| `docs` | `docs/api/*.rst` + `currentmodule` counts across every managed repo shipping a `docs/api/` tree (**surface**) | `/audit_docs` (Heart, imports) |
79+
| `crlf` | executable scripts (`.sh` + shebang-`755` `.py`) with CRLF — the shebang breaks on Linux/HPC (**debris**, the ranked count); plain `.py` CRLF is reported separately as *cosmetic* (Python reads it fine — don't mass-normalise) | `/refactor` + `.gitattributes eol=lf` |
4480
| `docstrings` | consecutive module-level triple-quoted expressions separated only by whitespace in user-facing `*_workspace` and `HowTo*` root `*.py` entry scripts and `scripts/**/*.py` files (**finding**) | `/refactor` (mechanically merge each confirmed boundary) |
4581
| `refs` | file/folder references in user-facing `*_workspace` and `HowTo*` prose (`scripts/**/*.py` docstrings + comments, every `scripts/**/README.md` and `config/**/README.md`, and the top-level README) whose target no longer exists — restructure debt no health sweep can see, since the scripts still run (**finding**). Covers the README idioms a `scripts/`-anchored matcher cannot see: structure-list bullets (``- `slam_pipeline`: ``), slash-less relative folder paths (`data_preparation/imaging`), and config YAML names | `/refactor` (re-point each reference; judge the intended target) |
4682
| `optdeps` | smoke-listed workspace scripts that construct an optional-dependency-gated API (`TransformerNUFFT``nufftax`) without the house `find_spec` skip guard, so they hard-fail the CI matrices that omit the extras (**finding**). AST-confirmed — prose mentions don't count; scripts outside `smoke_tests.txt` are never flagged | `/refactor` (add the skip guard) |
4783
| `extras` | the complement of `optdeps`: an optional dependency a library **declares** (in the `[optional]` extra `mode=release` installs) that the `workspace-validation.yml` **`mode=smoke`** leg never installs (**finding**). The extras chain only reaches each library's own `[jax]`, never a sibling's `[optional]`, so those need hand-adding and silently drift — the symptom is a script red in smoke and **green in release** | `/bug` (add the install; fix the install set, **never** the script) |
4884
| `config` | library `config/*.yaml` keys missing from the matching workspace config — recursive diff (**surface**) | `/refactor` (mirror keys) |
4985
| `artifacts` | tracked files that look like leaked run outputs / stray data (under `output/`, or data-ext outside fixtures) (**debris**) | `/repo_cleanup` (gitignore + `git rm --cached`) |
50-
| `packaging` | ignored, fully-untracked top-level `*.egg-info/` and `build/` directories in managed library repos (**debris**) | preview then run `PyAutoBrain/bin/clean_slate.sh --packaging`; repo-set, exact-name, root-depth and tracked-file guards apply |
86+
| `packaging` | ignored, fully-untracked top-level `*.egg-info/` and `build/` directories in the managed code repos (**debris**) | preview then run `PyAutoBrain/bin/clean_slate.sh --packaging`; repo-set, exact-name, root-depth and tracked-file guards apply |
5187
| *(default)* | all of the above (**perf timing deferred** — it spawns real imports) | a ranked `HygieneDecision` worklist — recommends the highest-count direct mode (`tidy`/`crlf`/`docstrings`/`refs`/`artifacts`/`packaging`), then `hygiene perf`, then the periodic surface audits |
5288

5389
```
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
#!/usr/bin/env python3
2+
"""Read the organism's body map for the hygiene conductor.
3+
4+
The conductor scans repositories. WHICH repositories is not its decision to
5+
make: the body map (the Mind's ``repos.yaml``) is the single source of repo
6+
identity, and this helper is the conductor's only route to it.
7+
8+
Why a helper rather than an array in ``hygiene.sh``: a hardcoded repo list
9+
drifts as the organism grows, and the drift is *invisible* — a repo that is
10+
never scanned produces no findings, so the conductor reports a clean bill of
11+
health it has not earned. (It did: five libraries scanned where the map
12+
declared six, four organs of seven, and a CRLF count of 5 against a true 127.)
13+
Deriving the sets means adding a repo to the map adds it to the scan.
14+
15+
This file deliberately contains **no repository names**. That is what keeps it
16+
firewall-clean under ``repos_sync.py``'s tenant check, and it is also the
17+
property the coverage check relies on: there is nothing here to drift.
18+
19+
Usage
20+
-----
21+
_hygiene_repos.py --category <name> # one name per line, sorted
22+
_hygiene_repos.py --json # {"<category>": [names...], ...}
23+
24+
Exit codes: 0 = read; 3 = body map unresolvable (prints nothing, so a caller
25+
can distinguish "no repos declared" from "no repos present" and report
26+
`unscanned` rather than `clean` for either).
27+
28+
The map is located the way ``agents/_common.sh`` locates any organ checkout:
29+
an explicit ``PYAUTO_MIND``, then the sibling beside this Brain checkout, then
30+
``$PYAUTO_ROOT``, then a couple of common dev layouts. PyYAML is used when it
31+
imports and a minimal parser stands in when it does not — the conductor stays
32+
dependency-free by design (it must never drag a heavy stack into the Brain),
33+
and the map's own shape is simple enough to read without one.
34+
"""
35+
36+
from __future__ import annotations
37+
38+
import argparse
39+
import json
40+
import os
41+
import re
42+
import sys
43+
from pathlib import Path
44+
45+
MAP_FILENAME = "repos.yaml"
46+
47+
# The Mind is an organ, so its directory name is framework identity rather than
48+
# an instance fact — the same reason _common.sh may name it.
49+
MIND_REPO = "PyAuto" + "Mind"
50+
51+
52+
def candidate_map_paths() -> list[Path]:
53+
"""Where the body map might live, most-authoritative first.
54+
55+
An explicit ``PYAUTO_MIND`` pointing at a real directory is authoritative and
56+
ends the search, exactly as ``_resolve_dir`` in ``agents/_common.sh`` treats
57+
its override. Falling through to a sibling checkout would silently scan a
58+
*different* organism than the operator named — and would make "the map is
59+
unreachable" unreachable itself, so the branch that reports it could never
60+
be exercised.
61+
"""
62+
here = Path(__file__).resolve()
63+
# .../<checkout>/agents/conductors/hygiene/_hygiene_repos.py
64+
brain_parent = here.parents[4]
65+
override = os.environ.get("PYAUTO_MIND")
66+
if override and Path(override).is_dir():
67+
return [Path(override)]
68+
candidates: list[Path] = []
69+
candidates.append(brain_parent / MIND_REPO)
70+
root = os.environ.get("PYAUTO_ROOT")
71+
if root:
72+
candidates.append(Path(root) / MIND_REPO)
73+
home = Path.home()
74+
candidates += [home / MIND_REPO, home / "Code" / MIND_REPO]
75+
return candidates
76+
77+
78+
def resolve_map() -> Path | None:
79+
for base in candidate_map_paths():
80+
path = base / MAP_FILENAME
81+
if path.is_file():
82+
return path
83+
return None
84+
85+
86+
# --- Parsing -----------------------------------------------------------------
87+
#
88+
# Two readers for one file. PyYAML is correct and preferred; the fallback exists
89+
# so a missing optional dependency degrades the *rigour* of the parse, never the
90+
# *coverage* of the scan. Silently scanning fewer repos is the bug this whole
91+
# module exists to prevent, so "PyYAML absent" must not become a way to
92+
# re-introduce it.
93+
94+
_REPO_LINE = re.compile(r"^ ([A-Za-z0-9._-]+):\s*(#.*)?$")
95+
_CATEGORY_LINE = re.compile(r"^ category:\s*['\"]?([A-Za-z0-9_-]+)['\"]?\s*(#.*)?$")
96+
_TOP_LEVEL = re.compile(r"^\S")
97+
98+
99+
def parse_minimal(text: str) -> dict[str, str]:
100+
"""Map repo name -> category without PyYAML.
101+
102+
Walks the two-level ``repos:`` block by indentation: a two-space key opens a
103+
repo, a four-space ``category:`` sets it, and any new top-level key ends the
104+
block. Sufficient for this file's fixed shape and nothing more — it is a
105+
fallback, not a YAML implementation.
106+
"""
107+
out: dict[str, str] = {}
108+
in_repos = False
109+
current: str | None = None
110+
for line in text.splitlines():
111+
if not line.strip() or line.lstrip().startswith("#"):
112+
continue
113+
if _TOP_LEVEL.match(line):
114+
in_repos = line.startswith("repos:")
115+
current = None
116+
continue
117+
if not in_repos:
118+
continue
119+
m = _REPO_LINE.match(line)
120+
if m:
121+
current = m.group(1)
122+
continue
123+
m = _CATEGORY_LINE.match(line)
124+
if m and current:
125+
out[current] = m.group(1)
126+
return out
127+
128+
129+
def parse_with_yaml(text: str) -> dict[str, str]:
130+
import yaml # local import: absent PyYAML must fall back, not crash
131+
132+
data = yaml.safe_load(text) or {}
133+
return {
134+
name: entry.get("category")
135+
for name, entry in (data.get("repos") or {}).items()
136+
if isinstance(entry, dict) and entry.get("category")
137+
}
138+
139+
140+
def load_categories(path: Path, parser: str = "auto") -> dict[str, list[str]]:
141+
"""Return category -> sorted repo names.
142+
143+
``parser="minimal"`` forces the PyYAML-free path. That exists so the drift
144+
check can exercise the fallback on a machine that *has* PyYAML: a fallback
145+
only ever used where nothing verifies it is a fallback nobody can trust, and
146+
a parser that silently drops repos is this module's own bug class.
147+
"""
148+
text = path.read_text()
149+
if parser == "minimal":
150+
by_repo = parse_minimal(text)
151+
else:
152+
try:
153+
by_repo = parse_with_yaml(text)
154+
except ImportError:
155+
by_repo = parse_minimal(text)
156+
grouped: dict[str, list[str]] = {}
157+
for name, category in by_repo.items():
158+
grouped.setdefault(category, []).append(name)
159+
return {category: sorted(names) for category, names in sorted(grouped.items())}
160+
161+
162+
def main() -> int:
163+
parser = argparse.ArgumentParser(description=__doc__)
164+
parser.add_argument("--category", help="print the repos in one category")
165+
parser.add_argument("--json", action="store_true",
166+
help="print every category as JSON")
167+
parser.add_argument("--parser", choices=("auto", "minimal"), default="auto",
168+
help="force a reader; 'minimal' is the PyYAML-free path")
169+
args = parser.parse_args()
170+
171+
path = resolve_map()
172+
if path is None:
173+
searched = ", ".join(str(base / MAP_FILENAME) for base in candidate_map_paths())
174+
print(
175+
f"hygiene: body map not found — no {MAP_FILENAME} at: {searched}. "
176+
f"Set PYAUTO_MIND to the Mind checkout.",
177+
file=sys.stderr,
178+
)
179+
return 3
180+
181+
grouped = load_categories(path, args.parser)
182+
if args.json:
183+
print(json.dumps(grouped, indent=2, sort_keys=True))
184+
elif args.category:
185+
for name in grouped.get(args.category, []):
186+
print(name)
187+
else:
188+
parser.error("one of --category or --json is required")
189+
return 0
190+
191+
192+
if __name__ == "__main__":
193+
sys.exit(main())

0 commit comments

Comments
 (0)