diff --git a/complete/2026/08/test-mode-bypass-assertion-ties.md b/complete/2026/08/test-mode-bypass-assertion-ties.md new file mode 100644 index 00000000..5d17f9de --- /dev/null +++ b/complete/2026/08/test-mode-bypass-assertion-ties.md @@ -0,0 +1,140 @@ +## test-mode-bypass-assertion-ties +- issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1519 (closed) +- completed: 2026-08-24 +- library-pr: https://github.com/PyAutoLabs/PyAutoFit/pull/1520 (merged 438f56fac) +- summary: The PYAUTO_TEST_MODE=2/3 bypass evaluated the model at the prior + medians, so a model with identical priors plus an ordering assertion (the + exchange-degeneracy idiom, e.g. PyAutoCTI trap models) tied exactly there and + check_assertions hard-failed the run. The bypass now picks its point through a + shared `_test_mode_valid_parameter_vector` — prior medians first, then + `default_rng(seed=0)` prior draws, each candidate validated — so the vector it + evaluates AND stores satisfies the assertions. +- key-finding: **the prompt's own suggested fix would not have worked.** The + 2026-08-09 note had narrowed this to a one-liner (move `instance_from_vector` + inside the existing try, or pass `ignore_assertions=True`). Reading main + d3625a8 found THREE sites, and both one-liners fix only the first: + 1. `abstract_search.py:1007` — instantiation outside the FitException guard; + 2. `_build_fake_samples:1112` — the perturbed samples are the median vector + scaled UNIFORMLY (1.001/0.999/1.002), and a uniform scale preserves an + ordering tie, so every stored sample fails the same assertion; + 3. `SamplesSummary.max_log_likelihood` (`interface.py:122`) is `@to_instance()` + with `recover="raise"` — it extends SamplesInterface directly so it does NOT + inherit Samples' next-valid recovery — so `result.max_log_likelihood_instance` + raises SamplesException, and `Result.instance` catches only AttributeError. +- key-finding: **TEST_MODE=3 was broken too and nobody had noticed.** It never + calls instance_from_vector in the bypass, so it survived the fit and died at + the first `result.max_log_likelihood_instance`. Confirmed by reproduction, not + just by reading. Fixing the stored vector fixes modes 2 and 3 together. +- trap: do NOT re-run a bypass reproduction without clearing `output/` first. A + bypassed fit calls `paths.completed()`, so a second run with the same + unique_tag takes `result_via_completed_fit` and replays the OLD (broken) + samples — which reads exactly like "the fix didn't work". Cost one false + negative during verification. +- trap: `af.m.MockAnalysis` maps its likelihood over the model and returns a + LIST for an `af.Collection`, which the bypass's `float()` rejects. Regression + tests needed a small float-returning analysis instead. +- behaviour-change: mode 3 now instantiates the model once (previously zero + times). A model whose constructor raises a non-FitException at the medians now + fails at fit time rather than result time — same failure, surfaced earlier. + Flagged in the PR body for downstream repos. +- verification: reproduced on clean main first (mode 2 raised FitException in the + fit; mode 3 raised SamplesException at result access), then both modes complete + and select the identical vector after. All 5 new tests in + `TestBypassToleratesAssertionTies` fail against the un-patched source. Full + suite 2016 passed / 34 skipped / 0 failed; CI green on all three legs + (unittest 3.12, unittest 3.13, unittest-nojax) plus Docs. +- gate-caveat: shipped from a web-github session where `pyauto-heart` is + unreachable, so the readiness gate ran in the WORKFLOW.md fallback form (full + library suite as the gate). No Heart verdict was recorded for this task; CI + green at merge is the stronger confirmation that stands in its place. The + workspace-impact grep was likewise not run (workspace clones absent) — API + Changes are "none, internal", so option (iii) was inferred, not measured. +- follow-up: `autocti_workspace` documents this artifact in its AGENTS.md as a + workaround. Delete that note now the fix has shipped — the testmode-env-drift + precedent ("delete the trap, don't document it"). Separate repo, separate task. +- follow-up: re-enable autocti_workspace smoke coverage of the + `modeling/start_here.py`-class scripts (CTI epic Phase 5) that this unblocks. +- environment: web-github; no worktree was ever created, so there is none to + remove. PyAutoFit was worked in a session clone at /home/user/pyautofit. + +## Original prompt + +# TEST_MODE bypass crashes on ordered-parameter assertion ties + +Type: bug +Target: PyAutoFit +Repos: +- @PyAutoFit +Difficulty: small +Autonomy: supervised +Priority: normal +Status: formalised — STILL REPRODUCES; see the 2026-08-09 note before grading this against main +Filed: 2026-07-17 (backfilled from git) +Issued: 2026-08-24 + +## 2026-08-09 — do NOT mistake the adjacent FitException catch for this fix + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`). The bypass path +in `abstract_search.py` **now catches `exc.FitException`** and continues with the +`-1e99` sentinel, logging "TEST MODE 2: likelihood verification raised +FitException … treating as a resample-rejected instance". That reads exactly like +this prompt's suggested fix. **It is not.** The bug below still reproduces. + +The catch wraps only the likelihood call. The model instantiation is on the line +*before* the `try`: + +```python +if call_likelihood: + instance = model.instance_from_vector(vector=parameter_vector) # <-- outside + try: + log_likelihood = float(analysis.log_likelihood_function(instance)) + except exc.FitException as e: + ... +``` + +and `instance_from_vector` → `instance_for_arguments` → `check_assertions` +(`autofit/mapper/prior_model/abstract.py:193`) is precisely what raises +`exc.FitException("N assertions failed!")` when an ordering assertion ties at the +prior medians. `ignore_assertions` defaults to `False` and the bypass does not +pass it. So the assertion exception escapes the guard entirely and still +hard-fails the run. + +The upside: the fix is now a one-liner rather than the "catch and retry with a +perturbation" design sketched below. Two options, both cheap and both +deterministic: + +- move the `instance_from_vector` call inside the existing `try` — the sentinel + path already does the right thing for a rejected instance; or +- pass `ignore_assertions=True` at the bypass instantiation, on the grounds that + a verification eval at the medians is not a sampled point and assertions exist + to steer sampling. + +The second is probably the better semantics (a tied median is not a pathological +model), but it changes what the verification eval attests to — pick deliberately. +Prefer either over adding perturbation logic. + +`Difficulty:` stays small. The § Blocks note below still holds. + +--- + +Found during the CTI resurrection epic (Phase 4, 2026-07-17). `PYAUTO_TEST_MODE=2/3` +bypass evaluates the model at the **prior medians**. A model whose components have +identical priors plus an ordering assertion (the standard idiom for breaking +exchange degeneracy, e.g. PyAutoCTI trap models with +`model.add_assertion(trap_0.release_timescale < trap_1.release_timescale)`) +ties exactly at the medians, so the bypass evaluation raises +`autofit.exc.FitException: GreaterThanLessThanAssertion` and the script crashes. + +Real samplers resample assertion-failing points gracefully — this is purely a +bypass-path artifact, and it makes every ordered-trap CTI workspace script +un-smokeable at TEST_MODE=2 (reproduced with a bare +`model.instance_from_prior_medians()`; TEST_MODE=1 passes). + +Suggested fix: at the bypass evaluation, catch `FitException` from assertions +and retry with a small deterministic perturbation of the unit-cube point (or a +seeded random draw), mirroring what a real sampler does. Keep it deterministic +so smoke runs stay reproducible. + +Blocks: autocti_workspace smoke coverage of `modeling/start_here.py`-class +scripts (CTI epic Phase 5); the workspace documents the artifact in its +AGENTS.md meanwhile. diff --git a/complete/2026/08/testmode-assertion-note-removal.md b/complete/2026/08/testmode-assertion-note-removal.md new file mode 100644 index 00000000..619e37d3 --- /dev/null +++ b/complete/2026/08/testmode-assertion-note-removal.md @@ -0,0 +1,78 @@ +## testmode-assertion-note-removal +- issue: https://github.com/PyAutoLabs/autocti_workspace/issues/24 (closed) +- completed: 2026-08-24 +- workspace-pr: https://github.com/PyAutoLabs/autocti_workspace/pull/25 (merged 2933cddd) +- workspace-pr: https://github.com/PyAutoLabs/autocti_workspace_test/pull/17 (merged bfdb876b) +- workspace-pr: https://github.com/PyAutoLabs/autocti_assistant/pull/21 (merged 6a0f645c) +- summary: Follow-up to test-mode-bypass-assertion-ties (PyAutoFit#1520, merged + 438f56fac). Three CTI repos documented the now-fixed TEST_MODE bypass crash as a + live artifact readers must work around. Deleted per the testmode-env-drift + precedent ("delete the trap, don't document it") rather than updated. All three + branches proven ancestors of their main (0 unmerged commits each). +- key-finding: the prompt said "if a sibling note exists" — BOTH siblings existed, + and the second was not a note. In autocti_workspace_test/AGENTS.md the claim was + a parenthetical RATIONALE for a convention ("integration scripts are ... + single-trap (because ordered traps tie under the bypass)"). Deleting the whole + bullet would have quietly repealed a test-design rule. Removed only the + rationale; left the convention standing and flagged it for a maintainer. +- open-question: autocti_workspace_test's single-trap convention now has no stated + reason. If it existed only to dodge the bypass crash it can be dropped, and + multi-trap ordered models exercised — arguably better coverage, since ordered + traps are the realistic CTI case. If it also exists for runtime/simplicity the + bullet needs that reason written in. Deliberately not guessed. +- key-finding: the third site was in a SKILL (autocti_assistant + skills/ac_fit_cti_model.md), i.e. what the assistant tells users — a stale + "known artifact" there is active advice to work around a bug that no longer + exists. Worse than a stale AGENTS.md note. Its .claude/skills/ copy is a + SYMLINK (git mode 120000), so one edit covered both discovery surfaces; do not + assume the mirrored skills dirs are copies. +- deliberately-untouched: the add_assertion example at ac_fit_cti_model.md:54. + The ordering-assertion API is unchanged and still the right idiom for breaking + exchange degeneracy — only the bypass's handling of it was broken. +- trap: autocti_assistant's `wiki-currency` check is RED ON MAIN, independently of + any PR. Confirmed by dispatching it on main at 960fdd1c (run 32762029277): + failed identically. Proof it is not the PR's: the drift-report artifacts are + 964 bytes (PR) vs 961 (main), and the report header's `assistant_ref` line + differs by exactly 3 chars (refs/pull/21/merge vs refs/heads/main) — so every + drift FINDING is byte-identical. Artifact byte-size diffing is a cheap way to + prove two CI failures are the same failure when the log hides the detail. +- trap: that workflow redirects each sub-check into drift-report.md + (`>> "$REPORT" 2>&1`), so the job log NEVER names which of its five checks + failed. Read the artifact; do not try to infer it from the log tail. +- decision: PR#21 was merged deliberately over that known-bad base (human- + authorised at /prm) rather than regenerating an API baseline inside a docs PR. + The drift is filed separately as draft/bug/autocti/wiki_currency_baseline_drift.md. +- gate-note: autocti_workspace has NO CI configured at all (zero workflow runs, + zero checks). #25 was merged on explicit human authorisation per /prm's + no-checks guard. autocti_workspace_test's smoke ran `changes` green and skipped + `smoke` by path filter — correct for a docs-only diff, not a pending check. +- follow-up: draft/test/autocti/phase5_smoke_reenable_ordered_trap_scripts.md — + re-enable the smoke coverage the PyAutoFit fix unblocks (CTI epic Phase 5). +- environment: web-github; no worktree. Clones at /home/user/autocti_{workspace, + workspace_test,assistant}. + +## Original prompt + +# Delete the TEST_MODE ordered-assertion workaround note from @autocti_workspace AGENTS.md + +Type: docs +Target: autocti_workspace +Repos: +- @autocti_workspace +- @autocti_workspace_test +- @autocti_assistant +Difficulty: small +Autonomy: supervised +Priority: normal +Status: formalised +Filed: 2026-08-24 +Issued: 2026-08-24 + +Re-homed from `draft/triage/` by the filing session: intake classified this +`triage` / `Target: PyAutoFit` / `too-large` on low confidence, which is wrong on +all three counts — the PyAutoFit fix has already shipped, so nothing here touches +library source. This is a prose deletion across up to three workspace repos. + +Delete the TEST_MODE ordered-assertion workaround note from @autocti_workspace AGENTS.md. PyAutoFit#1520 (merged 438f56fac, 2026-08-24) fixed the bypass so a model with identical priors plus an ordering assertion no longer ties at the prior medians — the bypass now picks a deterministic assertion-valid point via _test_mode_valid_parameter_vector, at TEST_MODE 2 and 3. The workspace AGENTS.md documents the crash as a live artifact and tells readers to work around it; that text is now wrong. Delete the note rather than update it — the testmode-env-drift precedent is delete the trap, do not document it. Verify the note's exact wording and location in autocti_workspace first; if a sibling note exists in autocti_workspace_test or autocti_assistant, remove those too. + + diff --git a/complete/index.md b/complete/index.md index 706a098d..06b4e14d 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1100 records across 7 buckets. +1102 records across 7 buckets. ## Highlights @@ -211,9 +211,11 @@ _(curate hard-won records here — survives regeneration.)_ - [tenant-firewall-drift-aug](2026/08/tenant-firewall-drift-aug.md) — issue #198; 9 → OK, and a green `--check` is only evidence for the organs actually checked out - [tenant-firewall-hygiene-extras](2026/08/tenant-firewall-hygiene-extras.md) — auto-closed by the merge - [tenth-sample-hardcoded-index](2026/08/tenth-sample-hardcoded-index.md) — results/database/aggregator tutorials hardcoded stored-sample index 9 ("the tenth +- [test-mode-bypass-assertion-ties](2026/08/test-mode-bypass-assertion-ties.md) — The PYAUTO_TEST_MODE=2/3 bypass evaluated the model at the prior - [test-mode-fit-exception-finalization](2026/08/test-mode-fit-exception-finalization.md) — `PYAUTO_TEST_MODE=1` no longer finalizes a model point whose reconstruction raises `FitException`; it substitu… - [test-mode-samples-info-hook-contract](2026/08/test-mode-samples-info-hook-contract.md) — `NonLinearSearch._test_mode_samples_info()`'s docstring told subclasses - [test-performance-board](2026/08/test-performance-board.md) +- [testmode-assertion-note-removal](2026/08/testmode-assertion-note-removal.md) — Follow-up to test-mode-bypass-assertion-ties (PyAutoFit#1520, merged - [transformed-message-semantics-doc](2026/08/transformed-message-semantics-doc.md) - [undo-community-file-declutter](2026/08/undo-community-file-declutter.md) — Undid the 2026-08-19 community-file declutter (#248 Mind, #32 Memory): - [uniform-prior-bounds-numpy-path](2026/08/uniform-prior-bounds-numpy-path.md) — auto-closed by the merge's `Closes` line diff --git a/dashboard.html b/dashboard.html index 19930dc3..18d54ae3 100644 --- a/dashboard.html +++ b/dashboard.html @@ -138,7 +138,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- +

markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

@@ -179,7 +179,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

155 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

157 filed prompts, not started — sorted most-pickable first (priority, then size). 26 of them belong to an epic and are listed only under Epics below.

feature — 29

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

@@ -218,9 +218,9 @@

Backlog

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

Fix release result/sample parameter-path regressions🐛 bughealth_fixestoo-largesupervisedhigh

jax 0.11 breaks beta/gamma message log_partition under jit…🐛 bugautofitsmallsupervisedmedium

+

wiki-currency is red on autocti_assistant main — API baseline has drifted🐛 bugautoctismallsupervisednormal

LogGaussianPrior misreports its own support as (-inf, inf)🐛 bugautofitsmallsupervisednormal

autofit.plot functions accept **kwargs and silently discard them🐛 bugautofitsmallsupervisednormal

-

TEST_MODE bypass crashes on ordered-parameter assertion ties🐛 bugautofitsmallsupervisednormal

point.py JAX-vmap parity assert is non-deterministic under the smoke env🐛 bugautolenssmallsupervisednormal

HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX)🐛 bughowtogalaxysmallsupervisednormal

PyAutoConf rename leftovers in Brain functional surfaces🐛 bugpyautobrainsmallsupervisednormal

@@ -267,7 +267,7 @@

Backlog

Adopt Python 3.12 as the PyAuto ecosystem minimum🔬 researchlibraries

-maintenance — 24 +maintenance — 25

Untrack the generated FITS test artifacts in autoarray🧹 maintenancelibrariessmallsupervisedmedium

smoke_install.sh's stale jax<0.7 pin — CI is on the right jax…🧹 maintenancecilowsupervisedmedium

autolens_workspace_developer rectangular experiments — Gut stash + rename🧹 maintenanceautolens_workspace_developersmallsupervisednormal

@@ -276,6 +276,7 @@

Backlog

Un-park imaging/features/scaling_relation/slam once PyAutoArray#431 merges🧹 maintenanceworkspacessmallsupervisednormal

Un-park multi_galaxy/features/scaling_relation/slam once a capped run passes🧹 maintenanceworkspacessmallsupervisednormal

autolens_workspace_developer: broad stale-API rot (56 symbols, no CI)🧹 maintenanceautolens_workspace_developermediumsupervisednormal

+

Standardise the arcticpy CI install across every CTI repo🧹 maintenancecimediumsupervisednormal

Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two dead deps🧹 maintenancelibrariesmediumsupervisednormal

PyAutoMemory canonical-key TODO sweep🧹 maintenancepyautomemorymediumsupervisednormal

Single-source the "Never rewrite history" policy as a generated AGENTS.md block🧹 maintenancepyautomindmediumsupervisednormal

@@ -311,14 +312,6 @@

Backlog

Phase 2: Make workspace READMEs assistant-first📖 docsworkspaces

-refactor — 5 -

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder♻️ refactorautogalaxytoo-largesupervisedhigh

-

Vendor bessel_kve into autoarray and drop the tensorflow-probability dependency♻️ refactorautoarraylargesupervisedmedium

-

Split Fitness.batch_size into lh_batch_size and latent_batch_size♻️ refactorautofitsmallsupervisednormal

-

Remove the dead EDEN packaging tooling from PyAutoFit♻️ refactorpyautofitmediumsupervisednormal

-

Deduplicate repos_sync.py's check/write pairs♻️ refactorpyautomindmediumsafelow

-
-
test — 5

Re-baseline the MGE imaging JIT profiling regression value🧪 testautolens_workspace_developertoo-largesupervisedhigh

Restore absolute NumPy likelihood regression baselines in the _workspace_test🧪 testworkspacestoo-largesupervisedhigh

@@ -327,6 +320,14 @@

Backlog

The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced🧪 testworkspacestoo-largesupervisednormal

+refactor — 5 +

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder♻️ refactorautogalaxytoo-largesupervisedhigh

+

Vendor bessel_kve into autoarray and drop the tensorflow-probability dependency♻️ refactorautoarraylargesupervisedmedium

+

Split Fitness.batch_size into lh_batch_size and latent_batch_size♻️ refactorautofitsmallsupervisednormal

+

Remove the dead EDEN packaging tooling from PyAutoFit♻️ refactorpyautofitmediumsupervisednormal

+

Deduplicate repos_sync.py's check/write pairs♻️ refactorpyautomindmediumsafelow

+
+
triage — 4

TRIAGE: needs manual review before routing❓ triagemediumsafehigh

Triage: Convolver "No blurring_image provided" warning in canonical workspace scripts❓ triagesmallsupervisednormal

@@ -343,6 +344,12 @@

Backlog 2026-08-24 filed +wiki-currency is red on autocti_assistant main — API baseline has… + + + +2026-08-24 +filed interferometer/jax_grad/gradient.py: eager and jitted likelihoods… @@ -365,6 +372,12 @@

Backlog +2026-08-24 +filed +Standardise the arcticpy CI install across every CTI repo + + + 2026-08-23 filed pynufft removal: unswept downstream residue (1 hard break + stale… @@ -388,13 +401,13 @@

Backlog smoke_install.sh's stale jax<0.7 pin — CI is on the right jax… - + 2026-08-22 filed multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA… - + 2026-08-22 filed Untrack the generated FITS test artifacts in autoarray @@ -628,18 +641,6 @@

Backlog Give PyAutoFit searches a seed — today no search can be made… - -2026-08-04 -filed -dataset/imaging/jwst_lw is untracked because the gitignore was never… - - - -2026-08-04 -filed -cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of… - -

Epics markdown version

@@ -682,6 +683,10 @@

Epics

Expectation Propagation Scale-Up — Scoping🔬 researchgraphical_eptoo-largesupervisedhigh

slope_hierarchy: methods write-up (NUTS headline, EP cautionary)🔬 researchgraphical_epmediumsupervisednormal

+
+CTI resurrection — Phase 5 — 1 queued prompt(s) — ⚠️ not in epics.md +

Re-enable autocti_workspace smoke coverage of the ordered-trap modeling scripts🧪 testautoctimediumsupervisednormal

+