test: Check B must assert the unpinned sub-3.12 install is refused (#155)

* test: Check B must assert the UNPINNED sub-3.12 install is refused

Check B already knew about this gap and wrote it down — "An unpinned 3.11
install is not evidence because pip may select an older compatible release" —
then tested only the pinned rejection. The unpinned path was the one users
actually took, and until 2026-08-19 it silently installed the stale
2026.7.29.1 stack with no JAX and no warning.

Adds check_b_unpinned_refused: an unpinned `pip install autolens` on 3.11 must
fail, and fail for the right reason. A successful install is now a FAIL that
says the sub-floor backtrack is back.

verify_install_unpinned_refusal accepts the two shapes that mean refused — the
tombstone raising during metadata preparation (the live path), and pip running
out of candidates if the sub-floor catalogue is ever withdrawn. Anything else
(network, resolver, dependency failure) is not floor evidence.

Classifier verified against real captured pip output in both directions: a real
3.11 refusal classifies as a refusal, a real 3.12 success does not.

Issue: PyAutoLabs/PyAutoHands#238

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: release_validation must describe both Check B refusals

The page repeated the old caveat — "An unpinned install is not sufficient
evidence because pip may select an older compatible release" — which was
written when the unpinned path was untested. It now has its own required leg.

Restates it as two distinct guarantees: a pinned rejection proves this release
holds the floor, and the unpinned install must be refused as well. The second
was not met until 2026-08-19, when pip on 3.11 backtracked to 2026.7.29.1 and
installed a stale JAX-less stack silently.

Issue: PyAutoLabs/PyAutoHands#238

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Jammy2211 <JNightingale2211@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Author: 3 people

docs/release_validation.md

@@ -117,8 +117,15 @@ carries `profile` and `commit_shas`, so the gate can enforce them):
 The integration run also performs `verify_install` A–F against the same wheels.
 Check B reuses that exact TestPyPI version: it must install and import on Python
 3.12 and 3.13, while Python 3.11 must reject it specifically because its
-`Requires-Python` metadata is `>=3.12`. An unpinned install is not sufficient
-evidence because pip may select an older compatible release.
+`Requires-Python` metadata is `>=3.12`. A *pinned* rejection is the only evidence
+that this release holds the floor, because an unpinned install may select an
+older compatible one instead.
+
+Check B then requires the unpinned install to be refused as well. That is a
+separate guarantee, and it was not met until 2026-08-19: `pip install autolens`
+on 3.11 backtracked to `2026.7.29.1` and installed a stale, JAX-less stack
+silently. The `2026.7.29.1.post1` tombstone refuses it, and this leg is what
+stops the backtrack reopening.
 
 That TestPyPI A–F result feeds the install-verification readiness leg (see
 "How the gate enforces these"), tagged `index: testpypi`. It proves the wheels

heart/checks/verify_install.sh

@@ -11,7 +11,8 @@
 # Runs a suite of independent install-path checks:
 #
 #   A  pip install autolens (default Python) + start_here.py + welcome.py
-#   B  one exact autolens release installs on 3.12/3.13 and rejects on 3.11
+#   B  one exact autolens release installs on 3.12/3.13, and 3.11 refuses it
+#      both pinned (Requires-Python) and unpinned (the tombstone release)
 #   C  conda install flow (python=3.12) + start_here.py + welcome.py
 #   D  pip install "autolens[optional]" resolves
 #   E  pip install autolens==2026.2.26.4 on Python 3.12 by explicit pin
@@ -58,8 +59,9 @@ Usage:
 
 Checks:
   A   pip install autolens (default python3) + start_here.py + welcome.py
-  B   one exact autolens release installs on python3.12 and python3.13,
-      while python3.11 rejects it because Requires-Python is >=3.12
+  B   one exact autolens release installs on python3.12 and python3.13, while
+      python3.11 refuses it both pinned (Requires-Python >=3.12) and unpinned
+      (the 2026.7.29.1.post1 tombstone, so pip cannot backtrack to a stale one)
   C   conda install flow (python=3.12) + start_here.py + welcome.py
   D   pip install "autolens[optional]" resolves and imports
   E   pip install autolens==2026.2.26.4 (yanked) installs on python3.12 by explicit pin
@@ -449,12 +451,53 @@ check_b_rejected() {
     fi
 }
 
+check_b_unpinned_refused() {
+    local pybin="python3.11"
+
+    if ! command -v "$pybin" > /dev/null 2>&1; then
+        step "$pybin not installed — FAIL (required floor-rejection interpreter)"
+        RESULTS+=("B|FAIL|$pybin not installed")
+        return
+    fi
+
+    local venv="/tmp/autolens_verify_B_unpinned_3.11_$TS"
+    ARTEFACTS+=("$venv")
+
+    step "$pybin: creating unpinned-refusal venv at $venv"
+    if ! make_venv "$venv" "$pybin"; then
+        RESULTS+=("B|FAIL|$pybin could not create venv")
+        return
+    fi
+
+    # shellcheck source=/dev/null
+    source "$venv/bin/activate"
+    pip install --upgrade pip > /dev/null 2>&1 || true
+
+    step "$pybin: expecting an UNPINNED install to be refused"
+    local pip_out pip_rc=0
+    pip_out=$(pip install "${PIP_INDEX_ARGS[@]}" autolens 2>&1) || pip_rc=$?
+    deactivate
+
+    if [ "$pip_rc" -eq 0 ]; then
+        RESULTS+=("B|FAIL|$pybin silently installed an unpinned autolens — the sub-floor backtrack is back")
+        tail_log "Check B ($pybin) unpinned install unexpectedly succeeded" "$pip_out"
+        return
+    fi
+    if verify_install_unpinned_refusal "$pip_out" autolens; then
+        RESULTS+=("B|PASS|$pybin refused the unpinned install")
+    else
+        RESULTS+=("B|FAIL|$pybin unpinned install failed without a floor refusal (rc=$pip_rc)")
+        tail_log "Check B ($pybin) unexpected unpinned pip output" "$pip_out"
+    fi
+}
+
 check_b() {
     echo
-    echo "=== Check B: exact release on 3.12/3.13; rejected on 3.11 ==="
+    echo "=== Check B: exact release on 3.12/3.13; pinned and unpinned refused on 3.11 ==="
     check_b_supported python3.12
     check_b_supported python3.13
     check_b_rejected
+    check_b_unpinned_refused
 }
 
 # ----- check C: conda flow -----

heart/checks/verify_install_helpers.sh

@@ -44,3 +44,32 @@ except InvalidVersion:
 raise SystemExit(0 if equivalent else 1)
 PY
 }
+
+verify_install_unpinned_refusal() {
+    local pip_output="$1"
+    local package="$2"
+
+    # Tombstone path (the live one): releases at or below 2026.7.29.1 declare
+    # Requires-Python >=3.9 and stay valid candidates forever, so an unpinned
+    # install below the floor is refused by 2026.7.29.1.post1 raising during
+    # metadata preparation. That text arrives from the build subprocess, not as
+    # a pip resolver diagnostic, so it needs its own classifier.
+    if printf '%s\n' "$pip_output" | grep -qE \
+            "$package requires Python 3\.[0-9]+ or later" \
+        && printf '%s\n' "$pip_output" | grep -qE \
+            "you are running Python 3\.[0-9]+"; then
+        return 0
+    fi
+
+    # Retraction path: if the sub-floor catalogue is ever withdrawn, pip runs
+    # out of candidates instead and says so. Both shapes mean refused; anything
+    # else (network, resolver, dependency failure) is not floor evidence.
+    if printf '%s\n' "$pip_output" | grep -qi \
+            "Ignored the following versions that require a different python version:" \
+        && printf '%s\n' "$pip_output" | grep -qiE \
+            "No matching distribution found for $package"; then
+        return 0
+    fi
+
+    return 1
+}

skills/verify_install/verify_install.md

@@ -14,7 +14,7 @@ about cleanup if they ran with `--keep`.
 | Check | What it verifies |
 |-------|------------------|
 | A | `pip install autolens` in a venv on default `python3`; `start_here.py` and `welcome.py` both run cleanly. |
-| B | One exact `autolens` version installs and imports cleanly on `python3.12` and `python3.13`, then the same exact version is rejected by `python3.11` specifically because `Requires-Python` is `>=3.12`. An unpinned 3.11 install is not evidence because pip may select an older compatible release. |
+| B | One exact `autolens` version installs and imports cleanly on `python3.12` and `python3.13`, then the same exact version is rejected by `python3.11` specifically because `Requires-Python` is `>=3.12`. An **unpinned** `pip install autolens` on `python3.11` is then required to be refused too — it was not, until 2026-08-19: pip backtracked to `2026.7.29.1` and installed a stale JAX-less stack silently. The `2026.7.29.1.post1` tombstone closes that, and this leg is what stops it reopening. |
 | C | The conda flow from `installation/conda.rst` works end-to-end (`conda create … python=3.12` → `pip install autolens` → clone workspace → run `welcome.py` + `start_here.py`). |
 | D | `pip install "autolens[optional]"` resolves cleanly and imports. |
 | E | `pip install autolens==2026.2.26.4` (a yanked release the docs reference) still installs on `python3.12` by explicit pin. |

tests/test_verify_install_script.py

@@ -39,6 +39,23 @@ def classify_rejection(output, version="2026.7.29.1"):
     )
 
 
+def classify_unpinned(output, package="autolens"):
+    return subprocess.run(
+        [
+            "bash",
+            "-c",
+            'source "$1"; pip_output=$(cat); '
+            'verify_install_unpinned_refusal "$pip_output" "$2"',
+            "classifier",
+            str(HELPERS),
+            package,
+        ],
+        input=output,
+        capture_output=True,
+        text=True,
+    )
+
+
 def test_bash_syntax():
     for path in (SCRIPT, HELPERS):
         result = subprocess.run(
@@ -225,5 +242,59 @@ def test_help_describes_supported_success_and_below_floor_rejection():
 
     assert result.returncode == 0
     assert "installs on python3.12 and python3.13" in result.stdout
-    assert "python3.11 rejects it because Requires-Python is >=3.12" in result.stdout
+    assert "pinned (Requires-Python >=3.12)" in result.stdout
+    # The unpinned leg has to be advertised too, or the help understates what
+    # Check B now guarantees.
+    assert "unpinned" in result.stdout
+    assert "2026.7.29.1.post1 tombstone" in result.stdout
     assert "applies to A, B, C, D" in result.stdout
+
+
+def test_unpinned_classifier_accepts_the_tombstone_and_retraction_forms():
+    """The two shapes that mean "refused below the floor"."""
+    tombstone_output = (
+        "        autolens requires Python 3.12 or later — you are running "
+        "Python 3.11.\n"
+        "ERROR: Failed to build 'autolens' when getting requirements to "
+        "build wheel\n"
+    )
+    retraction_output = (
+        "ERROR: Ignored the following versions that require a different "
+        "python version: 2026.8.17.1 Requires-Python >=3.12\n"
+        "ERROR: No matching distribution found for autolens\n"
+    )
+
+    assert classify_unpinned(tombstone_output).returncode == 0
+    assert classify_unpinned(retraction_output).returncode == 0
+
+
+def test_unpinned_classifier_rejects_unrelated_pip_failures():
+    """A refusal for any other reason is not floor evidence."""
+    network_failure = (
+        "WARNING: Retrying after connection broken by NewConnectionError\n"
+        "ERROR: No matching distribution found for autolens\n"
+    )
+    dependency_conflict = (
+        "ERROR: Cannot install autolens because these package versions have "
+        "conflicting dependencies.\n"
+    )
+    other_package_tombstone = (
+        "        autofit requires Python 3.12 or later — you are running "
+        "Python 3.11.\n"
+    )
+
+    for output in (network_failure, dependency_conflict, other_package_tombstone):
+        assert classify_unpinned(output).returncode != 0
+
+
+def test_check_b_asserts_the_unpinned_install_is_refused():
+    """The gap Check B documented but never tested: until 2026-08-19 an
+    unpinned 3.11 install silently resolved to the stale 2026.7.29.1 stack."""
+    body = SCRIPT.read_text()
+
+    assert "check_b_unpinned_refused" in body
+    # Wired into the runner, not merely defined.
+    assert body.count("check_b_unpinned_refused") >= 2
+    assert "verify_install_unpinned_refusal" in body
+    # A successful unpinned install below the floor is the bug returning.
+    assert "the sub-floor backtrack is back" in body