docs: llms.txt census fixes (routing drift, group order, signposts)

[Jammy2211]

Overview

A census of the llms.txt layer (2026-07-31) ahead of the assistant paper going live found the generated catalogues (llms-full.txt / workspace_index.json) byte-identical and all navigator CI checks green, but the hand-curated routing layer has drifted behind recent workspace work: multi_galaxy/ is unrouted, the weak/ entry makes a false claim, a cross-repo link is dead, and the PyAutoHands GROUP_ORDER still lists the pre-rename multi folder. One approved performance change: scope llms-full.txt (168KB, ~30–40k tokens, grows linearly with script count) to non-chat harnesses so connector chats don't ingest it.

Plan

• Add multi_galaxy/ routing to autolens_workspace/llms.txt and fix its stale weak/ entry (a start_here.py exists now).
• Fix GROUP_ORDER in PyAutoHands' navigator (multi → multi_dataset, add multi_galaxy, ellipse) and regenerate both workspace catalogues.
• Repoint autocti_assistant's dead link to the non-existent autocti_workspace navigator.
• Signpost the autolens_assistant from PyAutoLens/llms.txt.
• Complete the scripts/README.md folder lists in both workspaces (autolens missing weak; autogalaxy missing multi_galaxy + cluster) and fix the shared "inculding" typo.
• Scope the llms-full.txt fetch instruction to non-chat harnesses in autolens_assistant/AGENTS.md and refresh the stale "(added in a later phase)" tail lines in both workspace llms.txt files.

Merge order: PyAutoHands first, then the workspaces — the workspaces' navigator_check CI regenerates against PyAutoHands main (same-named branches keep PR CI green in the meantime).

Detailed implementation plan

Work Classification

Workspace (docs/routing files; one tooling constant in PyAutoHands; no library API changes)

Affected Repositories

• autolens_workspace (primary)
• autogalaxy_workspace
• PyAutoHands
• autocti_assistant
• PyAutoLens
• autolens_assistant

Branch Survey

Repository	Current Branch	Dirty?
./autolens_workspace	main	clean (claimed by interferometer-start-here-integrate-oom, disjoint files)
./autogalaxy_workspace	main	clean
./PyAutoHands	main	clean
./autocti_assistant	main	clean
./PyAutoLens	main	clean (claimed by 2 active tasks, disjoint files — llms.txt only)
./autolens_assistant	main	1 uncommitted user edit in README.md (untouched by this task)

Suggested branch: feature/llms-txt-census-fixes

Worktree root: ~/Code/PyAutoLabs-wt/llms-txt-census-fixes/

Implementation Steps

1. PyAutoHands — autohands/navigator.py: in GROUP_ORDER, replace "multi" with "multi_dataset", add "multi_galaxy" and "ellipse" (after the dataset-type groups, before "guides"). PR merges FIRST.
2. autolens_workspace — llms.txt:
   • "Start here": add scripts/multi_galaxy/start_here.py entry (multi-galaxy lenses — 2+ co-dominant deflectors, no host halo).
   • "I want to…": add scripts/multi_galaxy/modeling.py entry.
   • Fix weak entry: route to scripts/weak/start_here.py (it exists now; drop the "no start_here.py" claim), keep fit.py/simulator.py as companions.
   • Tail line: drop "(added in a later phase)"; add "large file — local agents only; in a browser chat use this file's routing instead" caveat.
3. autolens_workspace — scripts/README.md: add weak to the folder list; fix "inculding" → "including".
4. autolens_workspace — regenerate catalogue with the fixed GROUP_ORDER (regenerate_navigator.py autolens with sibling PyAutoHands on the same-named branch) so llms-full.txt/workspace_index.json pick up the new group order. No notebook regeneration (no script changes).
5. autogalaxy_workspace — llms.txt tail line (same refresh + caveat as step 2); scripts/README.md: add multi_galaxy + cluster, fix typo; regenerate catalogue (regenerate_navigator.py autogalaxy).
6. autocti_assistant — llms.txt: repoint the "autocti_workspace navigator" link to https://github.com/PyAutoLabs/autocti_workspace with wording that doesn't promise an llms.txt routing file.
7. PyAutoLens — llms.txt: add one autolens_assistant line under "Use it".
8. autolens_assistant — AGENTS.md "Working with skills" step 3: qualify the "check the autolens_workspace catalogue (llms-full.txt)" instruction — local/coding agents read llms-full.txt; connector chats use the workspace llms.txt routing + targeted single-file reads (the full catalogue is ~30–40k tokens and slows every subsequent turn).

Key Files

• PyAutoHands/autohands/navigator.py — GROUP_ORDER constant
• autolens_workspace/llms.txt, scripts/README.md, llms-full.txt (generated), workspace_index.json (generated)
• autogalaxy_workspace/llms.txt, scripts/README.md, llms-full.txt (generated), workspace_index.json (generated)
• autocti_assistant/llms.txt
• PyAutoLens/llms.txt
• autolens_assistant/AGENTS.md

Acceptance

• check_navigator.py path + banner checks green in both workspaces after edits.
• regenerate_navigator.py zero-diff against the committed catalogues (run with patched PyAutoHands).
• Generated files change only via the generator; notebooks untouched.
• Workspace url_check CI green (all URLs in edited files resolve).

Out of scope (deliberately)

• autocti_workspace navigator layer (future task; today's fix is just the dead link).
• euclid_assistant llms.txt (possibly deliberate absence — collaboration-internal).
• autogalaxy_assistant birth checklist (workspace INTERIM science section, capability-boundary assistant link, PyAutoGalaxy/llms.txt signpost) — lands with that task.

Notes

• Brain sized this too-large (25) with a generic 4-phase split; overridden — the score is prose/repo-count driven, every edit is a few lines of docs with zero API risk. One task, per-repo PRs, PyAutoHands-first merge order.

Original Prompt

Click to expand starting prompt

See PyAutoMind/active/llms_txt_census_fixes.md — census findings (2026-07-31), verified green baseline, full scope. Original user request (verbatim) is embedded in the prompt file.

The llms.txt file is used by the autolens_assistant to support conversation AI like ChatGPT. Can you do a census of this file against all the workspace updates we have done recently, and make sure it is up to date and fit for purpose. In particular, also make sure its paired to the autolens_assistant correctly (e.g. they are in sync). Soon, I will make the autogalaxy_assistant, therefore also make sure the llms.txt file in autogalaxy_workspace and other repos is ok. These may include making sure README.md files with contents are up to date, or any other API / docs / structure checks. Can you also do an assessment of whether the llms.txt file structure and general design might be leading to some sort of slow down?

[Jammy2211]

Progress: all six repos implemented and committed locally on feature/llms-txt-census-fixes (worktree). Navigator path/banner checks green in both workspaces; catalogues regenerated with the fixed GROUP_ORDER (multi_galaxy + multi_dataset now ordered with the science-case groups, ellipse for autogalaxy).

Held at the ship gate: Heart is RED for reasons unrelated to this task (nightly integrate failure — corrective #449/#450 in flight; tenant-firewall manifest drift — PyAutoHeart#132). Awaiting human acknowledgement to push branches and open the six PRs. Merge order once approved: PyAutoHands first, then the workspaces.

[Jammy2211]

Shipped 2026-08-01 — all six PRs merged under a human-acknowledged Heart RED (reasons quoted in-session; both blockers unrelated and in flight: corrective #450, PyAutoHeart#132):

• fix: navigator GROUP_ORDER — multi→multi_dataset, add multi_galaxy/ellipse PyAutoHands#221 — GROUP_ORDER fix (merged first)
• docs: llms.txt census fixes — route multi_galaxy, fix weak entry #452 — multi_galaxy routing, weak entry fix, llms-full scoping, README weak entry, catalogue regen
• docs: llms.txt census fixes — README folder list, catalogue group order autogalaxy_workspace#199 — README folder list, tail scoping, catalogue regen
• docs: repoint dead autocti_workspace navigator link autocti_assistant#20 — dead navigator link repointed
• docs: signpost autolens_assistant from llms.txt PyAutoLens#684 — assistant signpost
• docs: scope llms-full.txt fetch to local harnesses autolens_assistant#106 — llms-full.txt scoped to local harnesses (chat-slowdown fix)

CI green everywhere (navigator paths + staleness, smoke 3.12/3.13, url check, assistant boundary + wiki-currency). Branches deleted, worktree removed, completion record: PyAutoMind complete/2026/08/llms-txt-census-fixes.md.

Deliberately left open for future tasks: autocti_workspace navigator layer; euclid_assistant llms.txt (absence may be deliberate); autogalaxy_assistant birth checklist (workspace INTERIM science section, capability-boundary assistant link, PyAutoGalaxy/llms.txt signpost).