Skip to content

fix: retire the permanently red no-mistakes body check - #160

Open
quinnbot-ai wants to merge 1 commit into
mainfrom
fm/fm-retire-permanent-false-red
Open

fix: retire the permanently red no-mistakes body check#160
quinnbot-ai wants to merge 1 commit into
mainfrom
fm/fm-retire-permanent-false-red

Conversation

@quinnbot-ai

Copy link
Copy Markdown
Owner

no-mistakes-bypass: direct-PR - internal contributor-process change to the PR compliance check itself, shipped direct-PR

Summary

The PR must be raised via no-mistakes check accepted exactly one thing: the deterministic signature the no-mistakes pipeline writes into a PR body. Firstmate's direct-PR delivery mode legitimately ships without that pipeline, so every direct-PR pull request failed this check permanently and by construction. PR #152 merged with it red twice and every other check green.

That stopped being cosmetic when an absent check set began blocking a merge exactly as a red one does. A check that is red on correct work teaches every reader to dismiss red, and it is what forced bin/fm-pr-verify-lib.sh's merge gate to answer only "is a check set present" rather than "is it green".

What changed

The check was not deleted - it exists to stop pipeline-bypassing deliveries going unnoticed. Instead, direct-PR got a legitimate way to be green. Two things now satisfy it:

  1. The pipeline raised it. The body carries no-mistakes' signature. Unchanged.
  2. A maintainer declared the bypass. The body carries no-mistakes-bypass: direct-PR - <reason> and GitHub reports the author as OWNER, MEMBER, or COLLABORATOR.

Why the author association is the gate on path 2

Any body marker is hand-writable, so a marker alone would only move the gaming target - which is exactly the objection a worker raised when it declined to hand-write the pipeline signature. GitHub computes author_association from repository membership; a PR author cannot assert it about themselves. So an outside contributor who copies the bypass line still fails, and only people actually entitled to choose the delivery mode can take path 2.

Two shapes were deliberately rejected:

  • Exempting maintainers wholesale. That makes the bypass silent, which is the thing the check exists to prevent. An undeclared bypass by an OWNER still fails.
  • Deleting the check. Same problem, repo-wide.

The declared reason is published to the check run's summary, so a bypass is a recorded, reviewable fact.

Residual limitation, stated honestly: a maintainer could still hand-write path 1's pipeline signature. No body-content check can catch that, and pinning the script to the base ref would not help either - pull_request workflows run from the PR's own merge ref. What changed is that nobody needs to: faking path 1 is now a deliberate misrepresentation rather than the only route past a check that could never pass.

Structure

The verdict moved out of inline workflow run: bash into bin/fm-pr-body-compliance.sh, so a portable regression can drive it through a real interface instead of asserting YAML bytes, and so the marker strings have one owner. The job name is byte-stable because it may be a required status check on a protecting branch.

Doctrinal cleanup

bin/fm-pr-verify-lib.sh and docs/verification/pr-check-set-gate.md both documented the false red as a standing exception. Both are corrected: no check here is expected to be red on a healthy delivery now, so a red check is a stop-and-read result.

Verification

Base point, real CI. On current main, PR #152 (author_association: OWNER, direct-PR): PR must be raised via no-mistakesfail, twice, with all 12 other checks green. That is the symptom.

Fixed head, real CI. This PR's body carries the declaration and its author is OWNER, so the same check should report green on this run. pull_request workflows run from the merge ref, so this PR validates itself.

Smallest-mutation broken heads. One single-site mutation per load-bearing part, each reproducing a different symptom:

# Single-site mutation Symptom restored Test that fails
M1 BYPASS_VALID branch → if false the original bug: direct-PR permanently red declared bypass rejected for OWNER
M2 fm_has_write_accessif true bypass self-grantable by any contributor granted itself write access as CONTRIBUTOR
M3 early exit 0 for write access silent, undeclared bypass undeclared bypass passed for OWNER
M4 --author-association defaulted, not required miswired caller degrades quietly instead of erroring omitting the flag was not a usage error
M5 fm_trim class narrowed to space/tab GitHub's CRLF bodies mis-parse; stray CR in the published verdict verdict carried a stray carriage return
M6 drop the non-empty reason guard a bare magic word passes with no recorded justification declaration with no reason accepted
M7 drop the delivery-mode allowlist any invented mode is a valid bypass unrecognized mode accepted

One decoration found and removed. An explicit BODY=${BODY//$'\r'/} normalization survived every mutation: fm_trim's [[:space:]] class already covers the carriage return, and with both removed the suite still passed - meaning the CRLF test was vacuous. The redundant line is gone, fm_trim is documented as the real owner of CR handling, and two cases were added that make that class load-bearing (a CRLF declaration with an empty reason, and no CR in the published verdict). M5 above is the re-run against the real owner.

Gates run locally: bin/fm-lint.sh (shellcheck 0.11.0 + actionlint 1.7.12, 3 workflows valid) exit 0 · bin/fm-test-run.sh --check-coverage ok (155 total) · bin/fm-doc-audience-check.sh ok · tests/fm-pr-body-compliance.test.sh, tests/fm-pr-merge.test.sh, tests/fm-documentation-audiences.test.sh pass · full suite re-run under stock macOS bash 3.2.57 passes. The new test lands in CI serial shard 3 automatically.

The `PR must be raised via no-mistakes` check accepted exactly one thing:
the signature the no-mistakes pipeline writes into a PR body. Firstmate's
`direct-PR` delivery mode legitimately ships without that pipeline, so
every direct-PR pull request failed the check permanently and by
construction. PR #152 merged with it red twice and every other check
green.

That is no longer cosmetic. An absent check set now blocks a merge
exactly as a red one does, while a check that is red on correct work
teaches every reader to dismiss red - and it is what forced
bin/fm-pr-verify-lib.sh's merge gate to answer only "is a check set
present" rather than "is it green".

Give direct-PR a legitimate way to be green instead of deleting the
check. Two things now satisfy it:

  1. the pipeline raised it (unchanged), or
  2. a maintainer declared the bypass: the body carries
     `no-mistakes-bypass: direct-PR - <reason>` AND GitHub reports the
     author as OWNER, MEMBER, or COLLABORATOR.

The author association is the gate on path 2 because any body marker is
hand-writable, so a marker alone would only move the gaming target.
GitHub computes the association from repository membership; a PR author
cannot assert it about themselves. An outside contributor who copies the
line still fails, and the declared reason is published to the check run,
so a bypass is a recorded fact rather than a silent one. Exempting
maintainers wholesale would have made it silent, which is what the check
exists to prevent.

The verdict moves out of inline workflow bash into
bin/fm-pr-body-compliance.sh so a portable regression can drive it
through a real interface rather than asserting YAML bytes. The job name
stays byte-stable because it may be a required status check on a
protecting branch.

Correct the two places that documented the false red as a standing
exception, so neither keeps licensing "that one is always red".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant