Two fail-open paths in the grading pipeline: enclave no-regression guard and bidir label selection

[atoz96]

Filing as an issue rather than PRs: eval/ is maintainer-only per sensitive-paths-guard.yml, so #672 and #673 are blocked by design. Both findings are reproducible in pure Python with no GPU — happy for a maintainer to take them, or to close the PRs if the issue is the more useful artifact.

Both are the same class: a guard defeated by incomplete evidence rather than by wrong evidence. In each case the surrounding code fails closed on missing data and on bad data, but a partial record slips between them and reads as a pass.

---

1. eval/polaris/scoring.py — the no-regression guard passes when no context was measured

present = [k for k in GUARD_CTX_KEYS if k in guard]
speed_ok = all(guard.get(k, True) for k in present)   # all([]) is True

Running score() over the gate matrix:

input	verdict
empty primary	REJECT — fail-closed
empty guard	REJECT — fail-closed
guard_4k_pass: false	REJECT — names the context
{"top1": 0.99, "kl": 0.01} — accuracy ran, speed did not	label M, pass: true, speed_ok: true

The last shape is what a partial infra failure produces: the accuracy pass completed, the speed pass did not, so the guard_*_pass keys are simply absent. The comment above the block says "Every measured context must hold >= tolerance vs main branch baseline" — with nothing measured, "every" is satisfied vacuously.

This is the script the enclave runs, so per EVAL-TRUST.md ("The DCAP quote proves this exact script, with these exact inputs, produced this exact output") a hardware-signed receipt would attest "no regression" backed by zero speed evidence.

Suggested fix: speed_ok = bool(present) and all(guard[k] for k in present), plus a distinct reason for "measured no decode context". One measured context stays sufficient. (The get(k, True) default is already unreachable — present only holds keys that are in guard.)

Also worth noting: scoring.py is the only module under eval/polaris/ without a test file, and it is the one that decides rewards.

---

2. eval/pr_eval_bot.py — _public_eval_label can headline the passing side

The docstring states the invariant: "a passing Qwen3.6 XL cannot paper over a Qwen3.5 regression (PR #555)". Both selection loops require the failing side to carry a label, so a side with pass: False and no label falls past them to return res.get("label") — on a bidir run, the passing headline:

score_qwen35	score_qwen36	headline
S / pass	XL / pass	XL
REJECT / fail	XL / pass	REJECT
none / fail	XL / pass	none
None / fail	XL / pass	XL

Reward-visible rather than cosmetic: in the SN74 registry eval:XL is ×4.0 and eval:REJECT is ×0.0, so a bidir run whose failing side lost its label pays four times instead of nothing.

Suggested fix: after both loops, return "REJECT" instead of falling through — a failing side with no label of its own should fail closed. The existing preference order is unaffected: a side carrying REJECT still wins, and a side carrying a real none still headlines as none.

---

Verification

Both were confirmed by running the current code directly, and each has a regression test that fails on main and passes with the fix (in #672 / #673 if useful):

# scoring.py
AssertionError: True is not false        # speed_ok on zero measured contexts

# pr_eval_bot.py
AssertionError: 'XL' != 'REJECT'

python3 -m pytest eval/ -q is green with both fixes (100 → 111 and 100 → 103 with the added tests). No new dependencies; scoring.py's stdlib-only constraint is preserved.

Note on the two PRs

They were first opened with the template's proof-of-speedup section left with an unticked box and auto-closed by rtx5090-required. I did not tick the box — these are harness fixes with no speedup claim, and the attestation would have been false. I reopened them per the bot's own guidance (remove the section), but reopening a closed PR 422s here, hence #672/#673 as fresh PRs. Both then hit the sensitive-paths guard, which is working as intended.

[atoz96]

Maintainer-applyable patches

sensitive-paths-guard blocks #672 and #673 by design — ^eval/ is maintainer-only and I am atoz96 (NONE). That check cannot go green for a non-maintainer, and it should not: it is guarding the code that decides rewards. So rather than leave the fixes stranded behind two red PRs, here they are as patches you can apply directly.

Source-only (the two test files are in the PRs if you want them; the fixes stand alone):

1-scoring-fail-closed.patch — eval/polaris/scoring.py

diff --git a/eval/polaris/scoring.py b/eval/polaris/scoring.py
index 963ae35..fd8033b 100644
--- a/eval/polaris/scoring.py
+++ b/eval/polaris/scoring.py
@@ -87,8 +87,15 @@ def score(result: dict) -> dict:
 
     # ---- No-regression guard ----
     # Every measured context must hold >= tolerance vs main branch baseline.
+    #
+    # `all([])` is vacuously true, so an empty measurement set must not read as
+    # a pass: a guard carrying accuracy fields but no per-context speed flags is
+    # a partial infra failure, and this script fails closed on those (see the
+    # empty-primary branch above and the `bool(guard)` term below). Requiring at
+    # least one measured context keeps "no regression" a claim about evidence
+    # rather than about its absence.
     present = [k for k in GUARD_CTX_KEYS if k in guard]
-    speed_ok = all(guard.get(k, True) for k in present)
+    speed_ok = bool(present) and all(guard[k] for k in present)
     g_top1 = float(guard.get("top1", 0))
     g_kl = float(guard.get("kl", 99))
     g_acc_ok = g_top1 >= TOP1_BAR and g_kl <= KL_BAR
@@ -125,6 +132,10 @@ def score(result: dict) -> dict:
             )
         if not bool(guard):
             reasons.append("Qwen3-30B guard produced no verdict (infra error)")
+        elif not present:
+            reasons.append(
+                "Qwen3-30B guard measured no decode context (infra error)"
+            )
         if not reasons:
             reasons.append("Qwen3-30B no-regression guard failed")
         final["label"] = "REJECT"

2-bidir-label-fail-closed.patch — eval/pr_eval_bot.py

diff --git a/eval/pr_eval_bot.py b/eval/pr_eval_bot.py
index e9eaec4..d4d6a99 100755
--- a/eval/pr_eval_bot.py
+++ b/eval/pr_eval_bot.py
@@ -638,6 +638,11 @@ def _public_eval_label(res):
             for s in sides:
                 if s.get("pass") is False and s.get("label"):
                     return s["label"]
+            # A side failed but carried no label of its own (partial record).
+            # Falling through here would headline the top-level label, which on
+            # a bidir run is the passing side — the exact papering-over this
+            # function exists to prevent. Fail closed instead.
+            return "REJECT"
     return res.get("label")
 
 

Apply with:

git apply 1-scoring-fail-closed.patch 2-bidir-label-fail-closed.patch
python3 -m pytest eval/ -q

Reproducing each without the patches, if you want to see them fail first:

# 1. no-regression guard passes on zero measured contexts
from eval.polaris.scoring import score
score({"primary": {"top1": 0.99, "kl": 0.01, "label": "M", "pass": True},
       "guard":   {"top1": 0.99, "kl": 0.01}})
# -> label 'M', pass True, guard.speed_ok True   (expected: REJECT)

# 2. bidir headline takes the passing side
import eval.pr_eval_bot as bot
bot._public_eval_label({"mode": "bidir", "label": "XL", "pass": True,
    "score_qwen35": {"label": None, "pass": False},
    "score_qwen36": {"label": "XL",  "pass": True}})
# -> 'XL'   (expected: REJECT — eval:XL is x4.0, eval:REJECT is x0.0)

Both are the same shape: the surrounding code fails closed on missing evidence and on bad evidence, but a partial record slips between the two and reads as a pass.

Happy for #672/#673 to be closed once these are in (or now, if the patches are more useful than the branches) — no need to keep two red PRs open on my account.

[atoz96]

@ai-hpc — a decision request on #672 and #673, since sensitive-paths-guard is the only thing red on both and it is not something I can resolve from a fork.

The guard passes on two conditions: no changed file matching ^(eval/|bench/scripts/|...), or an author in MAINTAINERS/OWNER/MEMBER. Both PRs are entirely eval/, so the first would mean deleting the fix, and the second is not mine to satisfy. That is the guard doing its job on the code that decides rewards, so I have not tried to work around it.

Three options, whichever suits you:

1. Take the patches — both fixes are git apply-able in the comment above, with a repro for each. The PRs can then be closed as superseded.
2. Carry the branches — atoz96:fix/enclave-scoring-requires-measured-context and atoz96:fix/bidir-label-fails-closed-without-side-label, both with regression tests that fail on main.
3. Tell me to close them and keep this issue as the record.

No urgency from my side. Flagging mainly because both are fail-open paths in the grading pipeline rather than cosmetic: the enclave one lets a "no regression" verdict be signed with zero speed measurements, and the bidir one can headline eval:XL (x4.0) where the failing side should force eval:REJECT (x0.0).

[atoz96]

The auto-close is firing on PRs that follow its own documented remedy

#672 and #673 were both closed by rtx5090-required at 00:01 with:

If this PR does not need GPU eval (e.g. docs-only), remove the proof-of-speedup section from the description instead of leaving an unchecked box.

That is exactly what both PR bodies do — the section is absent, and neither body contains a checkbox line at all. Running the repo's own predicate from main against the live bodies:

>>> import eval.pr_eval_bot as bot
>>> bot.rtx5090_has_checkbox(body_672), bot.rtx5090_box_checked(body_672)
(False, False)
>>> bot.rtx5090_should_close(body_672, set())
False
>>> bot.rtx5090_should_close(body_672, {"eval"})
False

Same for #673. Both should have hit the continue in close_unchecked_rtx5090_prs:

if not legacy and not rtx5090_should_close(body, areas):
    continue

Neither carries the legacy not-tested label (labels are empty on both), and areas_for_pr returns set() for them — AREAS = {"kernels", "runtime", "moe", "bench"} does not include eval, so the runtime branch cannot fire either.

The only line in either body containing 5090 is prose with no markdown checkbox:

> speedup. I am not ticking the RTX 5090 box, because that attestation would be false.

_rtx5090_line is "5090" in ln, but rtx5090_has_checkbox additionally requires - [ ] or - [x] on that same line, which is absent.

So either the sweep that ran is on a revision older than the "remove the section" allowance, or it is closing on a predicate other than the one on main. Worth a look — as written, the documented escape hatch for a non-GPU PR does not actually work, which will hit any contributor submitting docs-only or tooling changes.

Not asking for #672/#673 to be reopened on my account — the sensitive-paths-guard block on them is correct and separate, and the patches above stand on their own. Flagging this because the misfire is independent of my PRs and affects the general case.