Skip to content

Required checks count only from the App or workflow pinned for their name [PROTECTED: owner merges] - #788

Merged
jayden1711 merged 34 commits into
mainfrom
issue-727-pin-checks
Oct 21, 2026
Merged

jayden1711 merged 34 commits into
mainfrom
issue-727-pin-checks

Conversation

@jayden1711

@jayden1711 jayden1711 commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Intent: a required check can be pinned to its source: the GitHub App that posts it, or the Actions workflow file that runs it. For a workflow pin, the run must be in this repo and for the very commit judged, and the check run must be one of that run's own jobs: its page is the run's page plus /job/<id>, over https on the run's host.

  • A pinned check counts only from its source. A same-name check run from another App or workflow, one posted onto the commit from another run or through the API, and every commit status count as missing for it.
  • The owner's rule (2026-10-19), as tightened in rounds 5 and 6.
    • A pinned check's own Actions runs are grouped by workflow file. In every group, the latest run must pass, by the runs' own order (the later workflow run, then the later check run, as a re-run attempt is). The newest check run must pass too.
    • So a later run of a file supersedes that file's earlier failed or cancelled runs, but under an Actions-App pin one workflow file never supersedes another. A re-run of an older run that failed after a newer run passed still blocks. A run for another head never supersedes.
    • Round 8, the owner's decision: a pin to an App that isn't Actions. It can't be split by file, so that App's latest check run of the name on the head (the highest id) decides, in whatever check suite.
      • A later pipeline or re-run of that App supersedes its earlier ones, so the owner's cases 1 and 2 hold for such an App across suites.
      • A later failing run blocks, and another source's failure still blocks (tests).
      • Correction: rounds 6 and 7 required every one of that App's listed runs to pass, so a failure in another suite blocked. That was the coordinator's tightening, true through 0623667 and now replaced by the owner's decision.
      • Also: through 961ecba the round-6 text said only "a re-run in the same suite replaces its failure".
    • Round 8, the owner's decision: a token that can't read commit statuses. With a pinned name being judged, a status read GitHub refuses at all waits, a page-1 403 that isn't a rate limit and a page-1 404 included. Readiness throws, and the watch asks again. With no judged name pinned, statuses are read as on main (a 403 or 404 is none), whatever else is pinned (round 10).
      • Correction: in round 9 (f33956d) readiness made the pinned reads whenever any pin existed, so a bare-only PR waited where main and the watch didn't (the reviewer, measured: 1,064 of 2,536 scenarios).
      • Correction: through 0623667 a page-1 permission 403 or 404 meant no statuses even with a pin.
    • Round 6: readiness and the main watch judge a pinned check through one function (pinnedVerdict) over the same facts.
    • Any check of the same name from any other source that isn't a pass blocks it: another App, another workflow, a run for another head, a commit status that isn't success.
    • The two loosenings, both the owner's decisions:
      • a pinned file's later run on the head supersedes its own earlier failed or cancelled runs;
      • since round 8, a pinned non-Actions App's later run supersedes its earlier ones, in any suite.
      • Judging by name would have held both.
      • Everywhere else this is stricter than main: another App's failing check holds a pinned PR.
      • Correction: through 0623667 this named one loosening; the second is new in round 8.
    • Correction: through e8107eb this said "It is never looser". That was false: the same-file supersede is looser. Under an Actions-App pin, any later Actions run superseded any other file's run.
  • A check nothing pins is judged by name, exactly as before. This is the owner's choice for existing installs, option (c), 2026-10-19.
    • Since round 8, with no judged name pinned (and GitHub's rules read), readiness and the main watch also make main's reads of the commit's checks, whatever other check is pinned (round 10). So they reach main's verdict past every page cap and refused page (measured: the round-8 pass, 288 of 288; round 10's pass with an unrelated pin, below).
      • Correction: through f33956d, any pin anywhere turned readiness's pinned reads on (round 9), so this held only with no pin at all.
    • Correction: through 9c1c8bd this was false past the caps. For example, a bare name passing on page 1 with 1,001+ check runs, or with more than 300 status contexts, waited for good in readiness, where main is READY (the reviewer, measured).
  • Until GitHub's rules first answer: an unpinned check is missing (fail closed), and the main watch waits at a merge's checks. A direct push, an unknown actor, a bot or a non-writer is still flagged as on main.
  • The owner's decision (a), 2026-10-19: a branch read GitHub refuses for permission, or answers 404. For a judgement with no pinned name (a bare-names install), GET /branches/<b> answered that way means classic protection names no App, so readiness and the main watch give main's verdict.
    • A 5xx or a rate limit keeps waiting, because it is transient.
    • With a pinned name being judged (by config.yaml or GitHub's rules), the fail-closed rules stand: the rules stay unknown.
    • Tests cover a bare and a pinned install under a permission 403, a 404, a 500 and a rate-limited 403, for readiness and the main watch (measured, below).
  • Covered paths: PR readiness, the check of main after a merge, fix-forward, the PR batch's fast checks, the docs-only ruleset hold and the main watch.
  • Split out:

Risk class: protected. src/** is engine; src/backlog/**, src/config/schema.ts, src/pr-watch.ts, the merge path and src/hook-imports.ts are self-modification; docs/design.md is engine-guards. The owner merges.

Closes #727

How it works

  • Where a pin comes from, in order:
    1. config.yaml required_check_sources: { <name>: { app_id: N } | { workflow: .github/workflows/<file>.yml } }. It is a separate key, so the dashboard's required_checks editor, which writes a list of names, can't drop a pin. A key __proto__ is refused.
    2. Else the App GitHub's classic protection (app_id, from GET /branches/{b}) or a ruleset (integration_id, from GET /rules/branches/{b}) names for it. -1, null or absent means any source; two different Apps for one name means none. The answer is cached for 10 minutes, and an error keeps the last answer. Never answered: unknown, so an unpinned check is missing.
    3. Else no pin: judged by name, as before.
  • Matching (src/check-source.ts):
    • A status never matches.
    • An App pin needs app.id.
    • A workflow pin needs the suite's workflow run to have that path, the project's repo and head_sha equal to the commit. The check run's html_url must be that run's html_url plus /job/<its id>, over https on that host (isJobOfRun).
  • The owner's rule in code:
    • pinnedVerdict (src/check-source.ts) is the one judgement of a pinned name.
      • Its own source's runs: ownVerdict groups them by workflow file (file <path>). An Actions run whose workflow run wasn't read is a group of its own. A non-Actions App is one app group.
      • A file's group: the latest (latestOwn: workflow run id, then check run id) and the newest (highest check run id) must pass.
      • The app group: every run must pass.
      • The run reported is the same whatever the list order: a finished non-pass first, then the highest id.
      • When that passes but any check of the name from another source doesn't, that check's state stands, without its id or link.
    • Readiness (pinnedChecks, from PinnedChecks.of) and the main watch (judge()) both call it.
      • The facts: GitHub's latest check run per check suite and name (readiness reads filter=latest; the watch reduces its filter=all list the same way), the workflow run behind each Actions suite, and the latest status per context.
      • The watch builds them as they stood at merge time (checksAt). It reads the workflow runs whenever a pinned name has an Actions run, as readiness does.
      • An Actions run is told by its App id (15368) on both sides.
      • pinnedAt is gone.
    • Paging (rounds 6 and 7): readiness and the watch read every page of check runs and statuses, up to 10, reading on past any full page whatever total_count says.
      • Pages past the ones main read are marked extra, and only a pinned name is judged with them.
      • Rounds 8 and 10: all of this applies only when a name being judged is pinned.
        • PinnedChecks.of(sha, { judging }) passes pinned to the backlog, and judge() asks for runs and statuses with pinned.
        • The default judged names are the PR's required checks, the batch's fast and full checks, and the docs-only ones.
        • Fix-forward passes its episode's own names, and the fast checks, the ruleset hold and the flaky gate pass theirs.
        • Correction: through f33956d readiness turned these on for any pin (round 9).
        • Readiness, with none pinned: GitHubBacklog.checks reads exactly as main: 9 pages of check runs judged as read, the combined status's first page (a 403 or 404: none), and no workflow runs.
        • The watch, with none pinned: GitHubMainWatchApi reads one page each, as main.
        • Correction: through ff971e5 this added "ci-metrics, which calls checkRuns with no option, is back to main's one page as well". That was false. ci-metrics reads through its own GitHubCiMetricsApi (src/github/ci-metrics-api.ts), which this PR has never changed, so it never read more than one page here (the reviewer, measured).
        • So judging by name is exactly as before, reads included.
        • Correction: through 9c1c8bd "judging by name is exactly as before" was false past the caps and on refused later pages (the reviewer, measured).
      • Past 10 pages, readiness throws (the PR waits). The watch judges a merge with a pinned required check unaccounted (TooManyPages).
      • A status read refused at all throws in both, a page-1 403 or 404 included (round 8, the owner's decision; through 0623667 a page-1 permission 403 or 404 meant no statuses).
      • See the round-7 adversarial table.
    • Correction: through bdfcc29 this said "The main watch does the same at merge time (pinnedAt), with the same groups". That was false.
      • The watch read the workflow runs only under a workflow pin. Under an App pin (an Actions-App pin from GitHub's rules included), every check run became its own group.
      • So the owner's cases 1 and 2, and a normal passing re-run, were unaccounted at merge time while readiness said ready. The reviewer measured this in 8 of 40 cases; on this PR's table it was 7 of 22.
  • Before GitHub's rules first answer: PinnedChecks.githubApps() is null, so an unpinned check is missing. The main watch passes rulesUnknown into judge(), which waits only at the required checks; everything before them is judged as on main.
  • The owner's decision (a) in code:
    • checkApps still reads the rules when the branch read is refused for permission or 404, then throws BranchDenied carrying what the rules name. A rate limit or a 5xx throws as before.
    • PinnedChecks keeps that apart from an answer (asked again after 10 minutes). githubAppsFor(judging, configured) returns the rules' answer only when no judged name is pinned by config.yaml or those rules; otherwise null, as before.
    • Readiness (PinnedChecks.of) and the main watch (pins.github(judgedBy, sources), wired in merge-flow.ts) both ask through it.
  • A run-time brand (round 5).
    • judged() records every array it makes in a module-private WeakSet and freezes the array and each check.
    • readiness and requiredOutcomes call assertJudged first. It throws on an array not made in check-source.ts, or one no longer frozen. A cast, an any, JSON.parse data, a @ts-expect-error value, Object.assign or a copy all throw. A push or a changed check throws TypeError. judged() copies with Array.from, so it always makes a plain array.
    • mayMerge and fix-forward's checks reach requiredOutcomes (inferred, from code).
    • Test fixtures go through the one maker: test/helpers.ts asJudged is PinnedChecks.of over a stand-in backlog (now async).
  • Round 6: one maker, which reads its own facts. PinnedChecks is the only maker.
    • Under No CI it reads the event log's test runs itself (local: the live config and the coordinator's log), in place of any check of their names.
    • withTestRuns and ci-none's headChecks, which took a caller's checks or events, are gone. Neither can be imported (tests).
    • Size (measured, git diff --stat bdfcc294 961ecbad): about 70 changed lines in src/ (check-source.ts, ci-none.ts, merge-flow.ts) and about 115 in tests. The tests' share is mostly asJudged becoming async.
    • That is a little over the coordinator's 150. I kept it because it removes both maker routes rather than linting them.
  • What is left for the lint: a new PinnedChecks over a stand-in backlog (e7).
    • Making PinnedChecks build its own backlog (from the instance's repo and token) would take it off the coordinator's backlog. Every coordinator test that feeds checks through a FileBacklog would then need a GitHub stand-in.
    • Guess: 300 or more changed lines, mostly tests. Not done; named here.
    • Instead makerUses flags these routes to PinnedChecks in src/ outside MAKERS_ALLOWED (merge-flow.ts), through the compiler's own symbols and module resolution:
      • a name resolving through any alias, re-export or barrel;
      • an export * or export * as, or a namespace or dynamic import, of any module that exposes it;
      • a dynamic import the compiler can't resolve;
      • a require() call;
      • createRequire named as such.
    • The reviewer's m1 to m8 and m10, and a computed dynamic import, are fixtures, each flagged. e6 and m9 (headChecks) no longer compile.
    • Not flagged, named here as the residual with e7 (the reviewer, measured READY over a stand-in backlog):
      • m11, createRequire reached by a computed key on node:module;
      • m12, process.getBuiltinModule;
      • m13, an import inside new Function;
      • m14, new (this.pinned.constructor)(standIn) in a MergeFlow subclass.
      • I didn't chase these in the lint (the coordinator: diminishing returns).
      • m14 at run time: a constructor refusing anything but a GitHubBacklog isn't a one-line fix. PinnedChecks takes the coordinator's FileBacklog (no GitHub: judged by name) and the tests' stand-ins, so it would need the backlog-building refactor above. Not done.
    • Correction: through 961ecba this said makerUses flags "every src/ use of PinnedChecks" and "any require() or createRequire". Both were too broad.
  • Routes around the type are also flagged by a lint (test/raw-check-reads.ts, fixtures as data):
    • judgedCasts flags, outside check-source.ts:
      • any type assertion into JudgedChecks;
      • any value typed any or never where JudgedChecks is expected (JSON.parse data, (b as any).checks(…), an as never const);
      • a generic call returning a type parameter there;
      • a type predicate or type argument naming JudgedChecks.
    • testImports flags any src/ import from test/.
    • rawCheckReads catches a raw read however spelled: a const key, Reflect.get, a structural alias. It also catches a Backlog put into any, unknown, an empty type or a structural type, by cast, annotation or argument.
    • All 16 of the reviewer's fixtures are caught, and src/ itself is clean.
  • githubCheckApps builds its map with no prototype, so a check GitHub's rules pin under the name __proto__ keeps its pin.
  • A raw read can't be judged.
  • The main watch: requiredAt reads the pins committed at a merge's first parent. judge() judges by name, then pinned names by source.
  • doctor: a plain required check sources warning naming each unpinned required check and how to pin it.
  • Governance: governanceRefusal is Governance mode: the engine runs only its evaluator, merge policy and locks in front of PRs other agents or people open, with no agents of its own; external reviews and CI are policy inputs, never the decision #704's gate. Gating PRs nobody here wrote is Governance mode: the engine runs only its evaluator, merge policy and locks in front of PRs other agents or people open, with no agents of its own; external reviews and CI are policy inputs, never the decision #704's job.

Exemptions (decision guide 9)

  • The new raw-read scan exempts three files in RAW_READS_ALLOWED. The scan is new in this PR, so no existing check is loosened.
    • src/check-source.ts: PinnedChecks.of is the one place that makes JudgedChecks.
    • src/main-watch.ts: it reads runs at a merge and judges them by pins in judge().
    • src/canary.ts: the updater keeps or rolls back an install on its verdict, but a pass needs the PR merged by the canary instance (whose merge judged its checks by their pins) and every check green, so a spoofed check can only turn a pass into a fail (inferred).
  • The new cast scan (judgedCasts) exempts src/check-source.ts (JUDGED_CASTS_ALLOWED): PinnedChecks.of makes JudgedChecks, so the brand is its to give. The scan is new, so this loosens no existing check.
  • The kept regex test keeps its round-0 list (canary.ts, check-source.ts).
  • The new makers scan (makerUses, round 5) exempts check-source.ts itself and MAKERS_ALLOWED, which since round 6 is one entry. The scan is new, so this loosens no existing check.
    • src/coordinator/merge-flow.ts may construct PinnedChecks. It is the one place the coordinator reads GitHub's checks, and it builds the maker over the coordinator's own backlog, config and event log.
    • ci-none.ts's round-5 exemption is gone with withTestRuns.
  • test/helpers.ts asJudged is not a cast. Since round 6 it is PinnedChecks.of over a stand-in backlog, so fixtures pass the run-time brand as real checks do.
  • Round 6, two tests narrowed with the rule they test:
    • Two App-pin assertions changed from "the App's later run supersedes" to the fail-closed result: required-check-source.test.ts and required-check-source-coordinator.test.ts. Each now also asserts that a same-suite re-run passes. That is the coordinator's change 2, a tightening.
    • One line left the raw-read compile test, the headChecks call, because headChecks no longer exists. The makers test now asserts that it can't be imported.

Existing installs (owner rule: nothing reopens a route)

  • An install with no pins, whose GitHub rules name no App, behaves exactly as on main: its PR is ready, and the main watch accounts the merge (measured: the fail-without).
  • A pin from GitHub's protection or ruleset applies at once. It tightens, except for the same-file supersede the owner decided. Through e8107eb this said "only tightens", which was false.
  • The one change at upgrade: while GitHub's rules have never answered (an error from the start), an unpinned check is missing and the main watch waits. This is stricter, never looser.
    • Except, by the owner's decision (a): a bare-names install whose token is refused the branch read for permission, or gets a 404, gets main's verdict (measured: the fail-without on cf32770, below). A 5xx or a rate limit still waits.
  • To protect an install: pin its checks in config.yaml through a PR. doctor prints a paste-ready required_check_sources block; a new project's config starts pinned (after #788) [PROTECTED: owner merges] #803 makes that a paste.

Adversarial pass (decision guide)

Fixtures and stand-ins only, on this branch's build (scratchpad/work/issue-727/adversarial.mjs and the tests).

Attack Principle Result (measured) Fixed
Another App's failing same-name run before the pinned App's passing one (the reviewer's fixture) 3 was test pass, ready; now test fail, not ready round 1 (pinnedChecks, main watch)
An API-made check run in the pinned workflow's own suite (page /o/r/runs/<id>) 1 was accepted; now missing round 1 (isJobOfRun)
The right job path on another host (https://evil.example/o/r/actions/runs/700/job/1), or over http: 1 isJobOfRun returned true; now false, the check is missing yes: the page must be the run's own html_url plus /job/<id>, over https on that host
GitHub's rules unreadable from the start (an error on every read) 1 an unpinned check, which the rules might pin, was judged by name; now it is missing, and the main watch waits instead of judging. On the old head the main watch's stand-in returning null crashed (Cannot convert undefined or null to object) yes: githubApps() returns null until GitHub answers
A pin keyed __proto__ in config.yaml 1 the record dropped it (pinOf null, the spoof was kept: judged by name); now the config is refused yes: schema preprocess
A pin keyed constructor 1 pin held (pinOf {"app_id":1111}, spoof dropped) not needed
A raw Backlog.checks answer passed to the 6 judging calls 4 refused by the compiler at 6 of 6 (0 of 6 on the old head) round 1 (JudgedChecks)
12 spellings of a raw read (alias, .bind, destructuring, ?., ['checks'], other receivers, MainWatchApi.checkRuns) 4 the type scan catches 12 of 12; the regex catches 1 round 1 (scan)
Can the dashboard write a pin? 6 its editable keys are writers required_checks daily_budget_usd workers_count workers_model ci (from src/config-change.ts WHERE); no required_check_sources not needed
Does pinning touch any OFF path (auto-merge stop, main-watch off, policy)? 2 no OFF path reads the pins. An invalid required_check_sources makes the config invalid, and the last good one keeps running, as for any invalid config (inferred, code) not needed
Round 3: GitHub's rules never answered, then a direct push or a bot's merge 3 on d614853 the watch returned before judging, so neither was flagged (expected: 1 actual: 0); now each is unaccounted, as on main, and only the checks wait yes (judge(…, { rulesUnknown }))
Round 3: raw as never or as unknown as JudgedChecks at a judging call; a structural alias; a const computed key; Reflect.get 4 the old scan caught 1, 1, 0, 0, 0 and 0 of them, and the casts compiled; now the casts are flagged by judgedCasts and the reads by rawCheckReads (reviewer's fixtures: 6 of 6). src/ has no cast into JudgedChecks outside check-source.ts and no import from test/ yes
Round 3: a ruleset pins a check named __proto__ 1 on d614853 the pin was dropped: another App's __proto__ gave pass; now missing yes (Object.create(null))
Round 3: a pinned pass listed before another App's failure, or a newer pinned pass over an older failure (main watch) 3 on 5c4ef10 test pass / accounted; now test fail / unaccounted (the owner's rule) yes (round 4)
Round 4: cancelled-then-passed and failed-then-passed by the same pinned workflow on the head; a re-run attempt; list order reversed 1 ready / accounted: the latest own run decides, by run order; on 5c4ef10 the main watch said ci fail for cancelled-then-passed (it took the latest started) yes
Round 4: a later run of the pinned file for another head 1 never supersedes; its failure blocks, its pass never rescues (tests) yes
Round 4: the reviewer's n1 to n10 (as any, an any var, a structural parameter, as never consts, JSON.parse, a generic identity, a type predicate, Object.entries, as unknown as {…}) 4, 5 on 5c4ef10 none of n1 to n10 was flagged (caught: n1 … failed); now each is (the reviewer's script: raw-read or cast hits ≥ 1 for all 16) lint extended; the claims say lint, not run-time
Round 5: an Actions-App pin (GitHub's rules pin test to App 15368), ci.yml failed and a PR-added evil.yml passed later 1, 3 on e8107eb ready / main watch accounted (the later Actions run superseded the other file); now test fail / ci fail, unaccounted yes (ownVerdict, pinnedAt groups)
Round 5: the same with ci.yml still running, or the PR-added evil.yml failing 1 now pending / fail; both files passing is ready yes
Round 5: run 200 passed, then run 100 was re-run and its newest attempt failed (or was cancelled) 1 on e8107eb ready / accounted (run 200 is "later"); now fail / unaccounted. The owner's cases (cancelled or failed, then a later run passed) stay ready / accounted yes (newest check run must pass too)
Round 5: the reviewer's e1 to e8 at a judging call (a cast, JSON.parse, a @ts-expect-error value, Object.assign, a copy, withTestRuns over raw data, a stray withTestRuns call, a stray new PinnedChecks) 4 e1 to e5 and e8 throw at run time now (the brand); on e8107eb none threw. e6 and e7 (a maker over made-up data) can't be told apart at run time: makerUses flags them, as it does an alias, a namespace import with a computed key and a dynamic import. e8107eb's scans gave 0 hits for both e6 and e7 brand: yes. e6/e7: a lint, as the coordinator allowed
Round 6: the main watch under an App pin (Actions-App from GitHub's rules, or app_id: 1111) on the owner's cases 1 and 2, a normal re-run, a re-run still running 1, 3 on bdfcc29 7 of the shared table's 22 rows differed between readiness and the watch (watch fail where readiness said pass or pending: every run its own group); now 22 of 22 agree yes (pinnedVerdict, checksAt, sources read for any Actions run of a pinned name)
Round 6: a non-Actions App pin, its first suite cancelled or failed and its second passed 1 was ready (its later run superseded); now fail. A same-suite re-run that passed is ready yes (fail closed)
Round 6: the reviewer's m1 to m10 (renamed import, barrel re-export, export * with a computed key, export * as, dynamic import of a barrel, createRequire, literal dynamic import, destructured dynamic import, headChecks over made-up events, Object.values) 4 on bdfcc29 makerUses missed m3 to m6, m9 and m10; now each is flagged, or no longer compiles (m9, e6) yes
Round 6: a failing status of a pinned name past the combined status's first page (30 contexts); a failing run or status past the watch's first page of 100 1 was never read (the pinned check passed); now every page is read, up to 10, and a pinned name is judged with them (by name, as before) yes
A check removed or weakened? 7 none: the regex test kept, every existing test still passes. Existing tests were changed only to cast fixtures to JudgedChecks, and the (a) refusal test became (c) as the owner decided. In round 5, asJudged stopped being a cast. Round 6: two App-pin assertions now expect the fail-closed result, and one headChecks line went with headChecks (Exemptions above) n/a

Adversarial pass, round 5: what else could make a pinned check pass

These are the ways a pinned check could pass from something other than its pinned workflow file's own latest run on the head. Each is checked on bdfcc29.

  • Another workflow file.
    • Under a workflow pin, its run is another source: missing, and a non-pass of it blocks (test: the reviewer's fixture, round 1).
    • Under an Actions-App pin, each file is its own group, and every group's latest must pass (test above). Through e8107eb a later file superseded another (measured).
  • Another App: another source. Missing, and its non-pass blocks (tests, rounds 1 and 4) (measured).
  • A commit status: never matches a pin. A non-success status of the name blocks at readiness (pinnedChecks) and in the main watch (pinnedVerdict since round 6; tests) (measured).
  • Re-runs.
    • A re-run attempt of the latest run is "later" and decides (the owner's case).
    • A re-run of an older run that ends worse blocks, because the newest check run must pass (test above) (measured).
  • Another commit's run: fromPinnedSource needs the workflow run's head_sha to equal the commit. A run for another head is another source, so it never supersedes, and its failure blocks (tests, round 4) (measured).
  • API-created check runs:
    • one made into the pinned run's suite isn't one of that run's jobs (isJobOfRun: the run's own page plus /job/<id>, https, same host), so it is another source (tests, round 1) (measured on fixtures);
    • its real page is a guess (/<repo>/runs/<id>), and the check fails closed either way.
  • What is left:

Adversarial pass, round 6: every place that judges a pinned check gives the same answer

  • The places (measured, grep over src/ for pinOf, pinnedVerdict, pinnedChecks, readiness and requiredOutcomes):
    • PR readiness (prReadiness);
    • the check of main after a merge (the flaky gate's outcomes);
    • fix-forward's three reads;
    • the PR batch's fast checks (mayMerge, and the merge flow's own fast check);
    • the docs-only ruleset hold;
    • the main watch.
  • How they share it: all but the main watch take PinnedChecks.of's checks (pinnedChecks → pinnedVerdict) and judge them with requiredOutcomes. The main watch calls pinnedVerdict itself, over checksAt's facts at merge time.
  • The canary (src/canary.ts) needs every check green by name, which is stricter (Exemptions).
  • One shared table (test/main-watch.test.ts, measured): 22 rows, each the same facts under a workflow pin, an Actions-App pin and a non-Actions App pin.
    • Readiness runs through PinnedChecks.of over GitHubBacklog's real requests.
    • The watch runs through its own request loop, asking for the runs, workflow runs and statuses itself.
    • Readiness and the watch agree on all 22. On bdfcc29, 7 rows differed: under the Actions-App pin, owner 1, owner 2, a normal re-run and a re-run still running; under the non-Actions App pin, owner 1, owner 2 and a same-suite re-run.
  • Same facts:
    • Readiness reads filter=latest. The watch reads filter=all and keeps the newest check run per suite started by merge time.
    • That GitHub's latest is the newest per check suite and name is measured by the reviewer: on 17 of 17 of this repo's commits it is exactly the highest id per (suite, name) of filter=all.
  • What can still differ:
    • the time (readiness judges now; the watch at merge time, by design);
    • the page cap is now the same, 10, on both sides. Past it, readiness throws (wait) and the watch judges a pinned merge unaccounted.
    • Correction: through 961ecba this line said "Past either cap, the read is not judged … readiness reads as on main", which said both things. Readiness read 9 pages and judged them: a failing run of another source on page 10 gave READY while the watch said ci fail (the reviewer, measured).

Adversarial pass, round 7: every GitHub read on the readiness and main-watch paths

A truncated or refused read can only end as wait (readiness: not ready this tick; the watch: asks again), not ready, or unaccounted. It never ends as ready or accounted.

Readiness (src/backlog/github.ts, through PinnedChecks.of):

Read Full page, missing or wrong total_count Past the cap 403 / 404 Rate limit, other errors Ends as
Check runs (filter=latest, 100 a page) reads the next page throws past 10 pages (961ecba read 9 and judged them) throws throws wait
Combined status (30 a page) reads the next page (total_count no longer trusted) throws past 10 pages throws, page 1 too (round 8, the owner's decision; through 0623667 a page-1 permission 403 or 404 was no statuses; 961ecba also dropped page 1's statuses on a later page's refusal) throws (961ecba: no statuses) wait
Workflow runs (one page of 100) a suite past it has no workflow run n/a none read throws missing suites only make it stricter: a workflow pin finds no own run (missing), and an Actions-App pin's run is a group of its own that must pass (inferred, code)
Branch protection and rules rules: every page while full, up to 10 (round 9) rules: throws past 10 branch read: throws (the last answer is kept; never answered: unknown), except a permission 403 or a 404 with no judged name pinned, which is "no App named" (the owner's decision (a)). Rules: a first page refused for permission, or 404, is "no rulesets", as before; a later page's refusal throws branch read and rules: throw, so the last answer is kept (never answered: unknown). Correction: through ff971e5 a rate-limited or secondary 403 on the rules read was "no rulesets", cached 10 minutes, which dropped a ruleset's pin (the reviewer, measured: another App's passing ci was READY) never answered: an unpinned check is missing (wait)

The main watch (src/github/main-watch-api.ts, judge()):

Read Full page Past the cap 403 / 404 Rate limit, other errors Ends as
Check runs (filter=all, 100 a page) reads the next page TooManyPages: a merge with a pinned required check is unaccounted (a trip, an alert); by name judged as before (961ecba threw every step and never judged, holding every later merge) throws throws unaccounted, or asks again
Statuses (100 a page) reads the next page as check runs throws, page 1 too (round 8, the owner's decision; 961ecba returned the pages read) throws (961ecba: none) asks again
Workflow runs (one page of 100) as readiness n/a none (null) throws stricter only, as readiness
GitHub's rules (pins.github) n/a n/a never answered: waits at the checks the same wait

Measured (test/pin-pages.test.ts and test/main-watch.test.ts, each failing on 961ecba):

  • page 10 and page 11 of check runs in readiness;
  • a failing pinned status on page 1 with a rate-limited or 404 page 2;
  • a rate-limited page 1;
  • a permission 403 or 404 on page 1, which throws when a required check is pinned (round 8) and is no statuses when none is;
  • the watch's statuses with the same refusals;
  • the watch's check runs past 10 pages (TooManyPages carrying the 1000 read);
  • the watch judging such a merge unaccounted, then accounting the next one.

Left, inferred:

  • The watch judges commits in order. A merge whose read keeps throwing for another reason (a rate limit, a 5xx) holds the merges after it until the read answers. That is a wait, as on main for any read error, and it never accounts a merge. Only the page cap, which would never clear, is now judged.
  • A token that can't read commit statuses: decided by the owner in round 8.
    • With a required check pinned, it waits: every pinned check on such an install is held until the token can read statuses. That includes an existing install whose GitHub rules name an App.
    • With none pinned, statuses are read as on main.

Adversarial pass, round 8: a config with only bare names, against main

Method (measured, scratchpad/work/issue-727/bare8.mjs):

  • Main 178c44e, built in a scratch worktree at that commit (not a merge), and this head's build, over the same stand-in GitHub answers.
  • Readiness: GitHubBacklog.checks and readiness on main, against PinnedChecks.of (required: ['ci'], nothing pinned, GitHub's rules naming no App) and readiness here.
  • The watch: each build's GitHubMainWatchApi and judge(), driven through the need loop for a harness merge of ci.
  • 288 scenarios:
    • 1, 9, 10 or 11 pages of check runs;
    • ci passing or failing on page 1, failing on page 10, or absent;
    • statuses: none; failing on page 1; a full page 1 and failing on page 2; page 1 rate-limited, permission-403, 404 or 500; a full page 1 and page 2 rate-limited or 404;
    • with and without a 500 on check-run page 2.
  • Compared: the verdict, and the number of reads of the commit's checks.

Results:

Head The same verdict The same verdict and reads
9c1c8bd 72 of 288 40 of 288
f00b731 288 of 288 288 of 288

The other judging paths: fix-forward, the PR batch's fast checks, mayMerge, the ruleset hold and the flaky gate all read through the same PinnedChecks.of (headChecksOf) (inferred, code; the round-6 table of judging places). Its required covers:

  • the PR's required checks;
  • the batch's fast and full checks;
  • review.yaml's docs-only checks;
  • every name config.yaml pins.

So a pin on any name those paths judge turns the pinned reads on, and with none they are main's.

Left, justified:

  • GitHub's rules: readiness reads the branch protection and rules (2 GETs, cached 10 minutes) for every config. That is how a pin from GitHub's rules is found, and it has been in this PR since round 1. Until they first answer, a check config.yaml doesn't pin counts as missing (the documented upgrade window), and the pinned reads are used.
  • The canary: it calls backlog.checks with no option, so it reads as main (inferred).

Round 9: two fail-open edges closed

  • A rate-limited rules read kept a ruleset's pin out (the reviewer, measured).
    • checkApps now reads every rules page.
    • Only a first page refused for permission, or a 404, means "no rulesets", as before (GitHub Free).
    • A rate-limited or secondary 403, or any refusal of a later page, throws. The last answer is then kept, or the rules stay unknown.
    • Test: after an answer, a rate-limited read keeps the pin (ci: missing, where ff971e5 was READY). From the start, the rules stay unknown.
  • An Actions-App pin whose workflow runs weren't read was judged like a non-Actions App. That happened when the judged name wasn't in required(), such as fix-forward's episode names (the reviewer, measured at the API level).
    • Now only a non-Actions App's runs can be in the App group.
    • An Actions run whose workflow runs weren't read at all waits (reported as still running), never passes.
    • PinnedChecks.of also asks for the pinned reads whenever any pin exists. Round 10 replaced that, which went past its brief: the pinned reads turn on when a name being judged is pinned, and fix-forward passes its episode's names (checks(sha, judging)).
    • Tests:
      • through PinnedChecks.of and requiredOutcomes(['ci']), the judgement fix-forward makes, with required: ['lint']: ci.yml failed and evil.yml passed gives fail (ff971e5: pass);
      • ownVerdict directly.
    • I didn't drive fix-forward's whole episode loop. The route through it is inferred, as the reviewer says.

Adversarial pass, round 9: every read whose error becomes a reassuring answer

Each place where a GitHub error turns into "none", "no pin", "no rulesets" or "no statuses", and why it is safe (inferred, code, with the tests named):

Read Error Becomes Why it's safe
Combined status, no judged name pinned (firstStatusPage) any 403 or 404 no statuses main's exact code (the bare-name comparison: 288 of 288 with no pin; round 10's with an unrelated pin). Correction: through f33956d this row held only with no pin anywhere
Combined status, a check pinned (statusPages) any refusal throws (wait) the owner's decision, round 8 (tests)
Rules, GET /rules/branches/{b} (checkApps) page 1: a permission 403 or a 404 no rulesets GitHub Free has none; main read no rules at all
a rate-limited or secondary 403; any refusal of a later page throws: the last answer kept, or unknown round 9 (test)
Branch, GET /branches/{b} (checkApps) a permission 403 or a 404 BranchDenied: for a judgement with no pinned name, classic protection names no App; with one, unknown (unpinned checks missing) the owner's decision (a), 2026-10-19 (tests, both sides)
a 5xx, a rate-limited or secondary 403, any other error throws: the last answer kept, or unknown (unpinned checks missing) fails closed: transient, so it waits
Workflow runs (workflowRuns, readiness; runSources, the watch) any 403 or 404 no workflow runs read: each Actions run's workflow is null stricter only. A workflow pin finds no own run, so it is missing. An Actions-App pin makes each Actions run its own group that must pass, which a full read never makes stricter
The watch's statuses, nothing pinned any 403 or 404 none main's exact code
The watch's statuses, pinned any refusal throws (asks again) the owner's decision (tests)
GitHub's rules in the watch (pins.github, from PinnedChecks.githubAppsFor) as checkApps as above the same answer readiness has
protection(), the watch's mode 403 or 404 "none" (the watch is on) main's code, unchanged, and it turns the guard on, not off

No other read on these paths turns an error into an answer. Every other one throws: readiness isn't judged, and the watch asks again.

Round 10: the pinned reads only for a judged name's pin

  • What was wrong: round 9 turned readiness's pinned reads on whenever any pin existed. That went past its brief: a PR whose checks are all bare names waited where main and the watch don't, once some other check was pinned (the reviewer, measured: 1,064 of 2,536 scenarios, all to "waits").
  • The fix:
    • PinnedChecks.of(sha, { judging }) makes the pinned reads only when a name being judged is pinned, or GitHub's rules haven't answered.
    • The default judged names are the PR's required checks, the batch's fast and full checks, and the docs-only ones. merge-flow.ts no longer adds every config.yaml pin.
    • Fix-forward passes its episode's own names (checks(sha, judging)). The fast checks, the ruleset hold and the flaky gate pass theirs.
    • ownVerdict's round-9 change stays: an Actions run with no workflow read waits, never passes.
  • Tests (test/pin-pages.test.ts):
    • a bare ci with other pinned by the ruleset or config.yaml is READY with main's reads, on a page-1 status 403, 301 status contexts and 1,001 check runs;
    • fix-forward's judgement (judging: ['ci']) still blocks ci.yml failed then evil.yml passed, and without judging the unread runs wait;
    • the rules paging: a pin on page 2 is kept; a refused page 2 keeps the last answer or leaves the rules unknown; 11 full pages are no answer.

Adversarial pass, round 10: a bare-names config with one unrelated pin, against main

Method (measured):

  • bare8.mjs, main 178c44e built in a scratch worktree at that commit, and the same 288 scenarios as round 8.
  • other, a check nothing requires, is pinned to App 1111, by GitHub's ruleset or by config.yaml.
  • Readiness and the watch as before. Fix-forward is its judgement over PinnedChecks.of(sha, { judging: ['ci'] }), against main's requiredOutcomes over the backlog.
  • The updater is bare8-789.cjs (135 scenarios), with the branch rules or updates.json pinning other.

Results:

Path No pin other in the ruleset other in config.yaml
Readiness, the same verdict and reads as main (f33956d) 288 / 288 288 / 288 (132) 288 / 288 (132)
The watch 288 / 288 288 / 288 288 / 288
Fix-forward 288 / 288 288 / 288 288 / 288
The updater (#789) 135 / 135 135 / 135 135 / 135
  • On f33956d, readiness gave the same verdict in 156 of 288 with an unrelated pin; now 288 of 288.

Package size (the owner's practice, 2026-10-19, as revised late that day)

This PR doesn't change PACKAGE_BUDGET: it fits main's { size: 1_950_000, unpackedSize: 6_060_000 } (#823), so test/package-files.test.ts is main's file exactly.

  • Measured (npm pack --dry-run --json after a clean npm ci, npm run build and npm run build:web, in throwaway worktrees under .worktrees/, since removed):
    • main d5674d0: 1,902,158 packed, 5,934,411 unpacked;
    • this branch at 9b9f846 (main d5674d0 merged in): 1,912,424 packed, 5,972,971 unpacked. That is +10,266 / +38,560, leaving 37,576 / 87,029 under the budget.
  • What grew (the per-file diff of the two pack lists, unpacked bytes): only this PR's code. The deltas add up to the +38,560.
    • dist/src/check-source.js, new: +16,061.
    • backlog/github.js +7,602, main-watch.js +7,104, github/main-watch-api.js +3,840, config/schema.js +1,514, coordinator/merge-flow.js +1,460.
    • pr-watch.js +334, canary.js +333, doctor.js +285, hook-imports.js +27, coordinator/fix-forward.js +21, ci-none.js −460.
    • templates/project/config.yaml +439 (the commented required_check_sources key).
    • No new file besides check-source.js; no sourcemap, test or fixture is packed.
  • History: 6b1594e and 20b912b changed the budget under the earlier practice; the merge of d5674d0 (9b9f846) takes main's line back.

How it was verified

What this can never do

  • Turn a check that isn't a pass into a pass, except the two loosenings the owner decided (inferred, from ownVerdict/pinnedVerdict and their tests):
    • a pinned workflow file's later run on the head supersedes that same file's earlier failed or cancelled runs;
    • a pinned non-Actions App's later run supersedes its earlier ones (round 8).
    • Otherwise a pinned check passes only when it passes in readiness and in the main watch alike, by the same function. For that:
      • in every workflow-file group of its source, the latest run and the newest check run pass;
      • under a non-Actions App pin, that App's latest check run of the name passes (the owner's decision, round 8);
      • no check or status of its name from any other source is a non-pass.
    • An unpinned check is judged exactly as before; with no required check pinned, from main's reads too (round 8). While GitHub's rules are unknown, it is missing.
    • Correction: through 9c1c8bd "an unpinned check is judged exactly as before" was false past the caps and on refused later pages, for a bare-names-only config.
    • Correction: through e8107eb this line had no exception, which was false. Under an Actions-App pin it was also looser than that: a later run of any workflow file superseded another file's failure.
  • Let a commit status satisfy a pinned check.
  • Let a raw GitHub checks answer reach readiness or requiredOutcomes without a throw, however it was cast. The run-time brand refuses any array not made in check-source.ts, or one changed since. The one maker, PinnedChecks, takes no checks, runs or events from a caller.
    • A PinnedChecks built over a stand-in backlog outside merge-flow.ts is not refused at run time. The lint (makerUses) flags the routes it lists; m11 to m14 are named gaps.
  • Judge a truncated or part-refused read of check runs or statuses as ready or accounted. It waits, or the watch judges it unaccounted (round 7; the adversarial table).
    • Correction: through bdfcc29 withTestRuns and headChecks took a caller's checks or events, and makerUses missed m3 to m6, m9 and m10 (the reviewer, measured).
    • Correction: through e8107eb this said a raw read "can never … compile without a cast or an any". That was a lint-only claim: a cast or an any compiled, and only the test-suite scan flagged it.
  • Write to GitHub. The new calls are GETs.

Compromised component

  • Agent user / agent-written PR:
  • Coordinator: reads the pins from main's config and GitHub's rules. No new reach.
  • Dashboard: can't edit required_check_sources (measured, above).
  • Other instances, hub viewer: no new reach.
  • Someone with push access to main: can change pins, as they can change required_checks.
  • Another App on the repo: its same-name check no longer passes a pinned check, and any non-pass of it blocks the pinned check. It can hold a pinned PR; it can't pass one. That is the owner's accepted cost.

New dependencies

None.

…or its name, not any same-name check

Readiness, the check of main after a merge, fix-forward, the fast checks and the main watch matched a
required check by name alone, so anything that can post a check run or a commit status under that name
(another App on the repo, a workflow a PR adds, a run posting onto another commit through the API, a
status) could turn it green. Governance mode (#704) would gate PRs nobody here wrote on that.

Now every judgement reads GitHub's checks through one place (PinnedChecks in src/check-source.ts) and
keeps a check only from its name's pinned source: config.yaml required_check_sources ({ app_id } or
{ workflow: .github/workflows/<file> }, the workflow run in this repo and for the very commit judged),
else the App GitHub's branch protection or ruleset requires for it, else none, and the check is refused
(missing) with a doctor line naming it. A commit status never satisfies a pin. The main watch reads the
pins committed with the required checks at a merge's first parent. FixForward's checks reader is now
required, so nothing judges GitHub's raw answer; a test pins that only check-source.ts reads it.

Test (fails without): test/required-check-source-coordinator.test.ts (a PR whose check is posted by
another App is never ready; the merge flow's checks of a commit drop it) and the main-watch test "a
merge whose required checks came from another App or workflow than their pins, or a status, is
unaccounted"; test/required-check-source.test.ts covers the stand-in GitHub API answers.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
…hose import tree now reaches it

src/backlog/types.ts, in the guard's import tree, takes CommitCheck's source types from
src/check-source.ts, so that module is part of the tree and test/review.test.ts requires it on the
guard's own list (locked as self-modification, L3). It decides which checks count, so the lock fits.

Test (fails without): test/review.test.ts "the guard's own list (src/hook-imports.ts) is exactly the
tree's modules no other glob locks" (red on #788's CI run 39199536673, shard 3/4).

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 1/2)

Reviewed head 1a4a97f3 against #727. Every claim is labelled measured (a command I ran, or CI output I cite), inferred (reasoned from code, not run) or guess.

Acceptance criteria

Criterion (#727) Status Evidence
A check pinned to app_id: A and posted by App B is missing, and the PR isn't ready met measured: required-check-source.test.ts "a check pinned to app A…" and the coordinator test "a PR whose required check is posted by another App is never marked ready" pass on the head and fail on main (below)
A check pinned to a workflow path and posted by another workflow is missing met, with a gap (required change 3) measured: the workflow-pin test passes on the head and fails on main. guess: a check run made through the API might join an existing Actions suite on the commit; see change 3
A commit status with a pinned check's name is missing met measured: the status test. inferred: fromPinnedSource returns false for any non-check_run (src/check-source.ts:34), and the main watch sets statuses = [] once pins apply (src/main-watch.ts:283), so outcomesAt never returns unknown and the raw-statuses branch at line 288 is never reached
A bare name matches exactly as today not met as built. This is the open owner decision, so it is not counted as a required change measured: with this repo's real GitHub answers, all 9 required checks are missing on this PR's own head (table below). The "exactly as before" test covers only the file backlog (checkApps absent)
Turning governance on with any bare-name check is refused, naming it partly met measured: the governanceRefusal test passes. inferred: nothing calls it yet; #704 must call it
The main watch uses the same pinned matching met measured: main-watch test 3 fails on main (expected: 1 actual: 0) and passes on the head
done_when: test/required-check-source.test.ts, npm run typecheck met measured: 53/53 pass in the three new or changed files; typecheck is clean

Fail-without (measured)

  • Method: in my worktree I put main's copy back for the 9 changed src/ files and kept the new tests and src/check-source.ts. Then npx tsc (31 type errors, emitted anyway) and node --import ./dist/test/isolate.js --test dist/test/required-check-source-coordinator.test.js dist/test/main-watch.test.js dist/test/required-check-source.test.js, run with a scratch HOME and state dirs and a fake claude on PATH.
  • Result: # tests 53 # pass 42 # fail 11, the same as the builder's count.
  • Key failing lines:
    • not ready on a spoofed check 1 !== 0;
    • expected: 0: 'test missing' actual: 0: 'test pass';
    • main watch expected: 1 actual: 0.
  • On the head: the same 3 files plus fix-forward, pr-watch, review and doctor give # tests 95 # pass 95 # fail 0. npm run typecheck is clean. npm run check:denylist gives denylist: 1282 files clean.

Every judging path goes through PinnedChecks

  • inferred (grep of src/ on the head): merge-flow.ts reads GitHub's checks only through this.pinned.of:
    • readiness: lines 299 and 626;
    • headChecksOf (1220), which feeds fix-forward (104), the fast checks (212), the ruleset hold (524), mayMerge (581) and the check of main after a merge (763);
    • the main watch gets pins (956) whenever the API has runSources, which the real one always has.
  • inferred: the only other readers are starter-ci.ts (a setup view), ci-metrics (timings), dashboard.ts:2181 (a list of names) and canary.ts. The canary is not report-only (see change 4).
  • The "no other backlog.checks(" test is not sound (measured): I fed 12 fixture sources that read the raw checks to the test's own predicate /backlog\.checks\(/, and it missed 11. Missed: an alias variable, .bind, an unbound method reference, destructuring, ?., ['checks'], a line break before .checks(, a space before (, another receiver name, a GitHubBacklog named gh, and the main watch API's raw checkRuns. See change 2.
  • A status, a check from another repo, a check for another commit:
    • A status never satisfies a pin (inferred, above).
    • Another repo, or a workflow run for another head_sha, doesn't either (measured: the fork and other-commit cases in the workflow-pin test).
    • A check run made through the API onto this commit may (guess; change 3).

Workflow pins and pull_request runs (the known gap for #704)

Inferred: this is acceptable for #788 alone.

required_check_sources as its own key

  • The reasoning holds (inferred). src/config-change.ts edits only required_checks, through setIn, with a string-list validator (lines 33, 91, 94). Object entries in that list would be refused or stripped on the next dashboard edit. The new key isn't in the editor's KEYS, so the dashboard can't edit it ("not editable here").
  • It is protected like required_checks (inferred): both live in .worklane/config.yaml, under BUILTIN_PROTECTED (src/guardrails/engine.ts:17) and the engine's .worklane/** review category.

Owner decision: what (a), (b) and (c) do to an existing install (this repo, measured)

How I measured it: with #788's own built githubCheckApps, pinnedChecks and readiness, on this repo's real answers:

  • gh api repos/jayden1711/worklane/branches/main and .../rules/branches/main;

  • the check runs and workflow runs of this PR's head 1a4a97f3 and of main's tip 0b294b53;

  • the 9 required_checks in main's .worklane/config.yaml.

  • GitHub-derived pins: {} (measured). Classic protection is off, and none of the ruleset's 8 required checks has an integration_id.

  • Classic protection: GET /branches/main/protection gives 404 (measured), so the main watch reads the branch as unprotected and is on unless the policy says off (inferred from src/github/main-watch-api.ts:29-34; I didn't read the instance policy).

Option This PR's head (all 9 green by name) Effect on this install
(a) fail closed, as built ready=false: all 9 missing (measured) No PR becomes ready. Every merge is unaccounted (stop plus revert PR) until a hand-merged, pre-accepted config PR pins them (inferred). A machine with an updater canary would roll back and hold every engine install that contains #788, because the canary instance's PR never becomes ready (inferred from scripts/machine/worklane-update.cjs:72-77)
(b) learn from main's tip 8 of 9 learnable, each from exactly one workflow file (ci.yml, or gitleaks.yml for scan). new-deps isn't posted on main at all: it runs on pull_request only (.github/workflows/new-deps.yml). Result: ready=false, new-deps missing (measured) It still blocks every PR here unless it also learns from main's workflow files or falls back to (c) for names it can't learn. The pins would also have to live outside the repo, since agents can't write .worklane/**, so a local state file would hold a security setting nobody reviews (inferred)
(c) bare name as today, with a doctor warning ready=true, the same as today (measured) No change until pinned. Pins set in config, or named by GitHub, apply at once

Protection

Measured, from the PR metadata and diff:

  • It touches engine paths (src/**, templates/**), self-modification (src/coordinator/merge-flow*, src/main-watch*, src/github/main-watch-api*), the guard's own list (src/hook-imports.ts) and engine-guards (test/main-watch.test.ts).
  • It is marked correctly: the title has [PROTECTED: owner merges], the body has **Risk class:** protected, it carries the needs-owner label, and it has real What this can never do and Compromised component sections.
  • The "can never do" claims:
    • "Turn a check that was … failed … into a pass" is false as built (measured, the fixture above).
    • "canary.ts reports, never gates" is inaccurate: the updater keeps or rolls back an install on the canary's verdict (inferred, scripts/machine/worklane-update.cjs:72-77).
    • The rest hold (inferred): the new calls are GETs only, and the updater is untouched.

Scope, hygiene, CI

Required changes

  1. Pinning must only ever turn a match into missing. This is the issue's constraint and this PR's own "can never do". Today it can also turn a fail into a pass (measured above). Fix it as Engine updater counts a required check only from its pinned workflow or App [PROTECTED: owner merges] #789's updater does with its all-green check over every run: a same-name non-pass from any source still blocks. One way: a check is pass only when it passes both by name, as today, and from its pin. Apply it in the merge flow (requiredOutcomes over pinned.of) and in the main watch (outcomesAt over the filtered runs). Add a test with that fixture, for readiness and for the main watch, that fails on the current head.
  2. Make a raw read of GitHub's checks impossible, not grep-able. The predicate misses 11 of 12 raw reads (measured). Preferred: PinnedChecks.of returns a branded type, and prReadiness, headChecks, mayMerge, rulesetHold and FixForwardDeps.checks accept only that type, so a raw Backlog.checks result doesn't compile. At minimum: a TypeScript AST scan that flags any checks property access on a Backlog-typed expression, tested with fixtures like those above passed to the scanner's own function.
  3. Close or prove the API-made check run case for workflow pins. Guess: a check run created through the API with a workflow's GITHUB_TOKEN may be filed under an existing Actions suite on the target commit, and that suite maps to the pinned file. If so, "a run posting onto another commit through the API" passes the pin. Proving the negative needs a write to GitHub, so add the cheap positive check instead:
    • require an Actions check run's html_url to be /<repo>/actions/runs/<the mapped workflow run's id>/job/<this check run's id>. On this repo a genuine run's check-run id is its job id, e.g. .../actions/runs/39197227793/job/117883963166 (measured);
    • or check the run's jobs with GET /actions/runs/{id}/jobs;
    • do it in both src/backlog/github.ts and the main watch API, with a test where a suite mapped to ci.yml carries a check run whose html_url is /runs/<id>, and that run is missing.
  4. Correct the claims.
    • "What this can never do" must match the code after change 1.
    • Fix "canary.ts reports, never gates": it decides whether an install stands. Either route it through PinnedChecks or say so.
    • If (a) stays, add the canary rollback to "Existing installs".

… a raw read can't be judged

Review round 1 and the owner's choice for existing installs (option c, 2026-10-19):
- A pinned check passes only when its pinned source passes and the check as judged by name (any
  source) passes too: another App's failing or pending same-name run still blocks it, in readiness,
  the merge flow and the main watch. Before, pinning could turn a fail into a pass.
- A check nothing pins (no config.yaml entry, no App named by GitHub's rules) is judged by name,
  exactly as before, so an existing install changes nothing on upgrade.
- Every judging function (readiness, requiredOutcomes, headChecks, mayMerge, prReadiness,
  fix-forward's reader) takes JudgedChecks, which only PinnedChecks.of makes: a raw Backlog.checks
  answer doesn't compile there. test/raw-check-reads.ts scans src/ by type for raw reads elsewhere
  (check-source, the main watch, the canary, each with why); the regex check stays.
- A workflow pin also needs the check run to be one of its workflow run's own jobs (its page
  /<repo>/actions/runs/<run>/job/<id>), so a check run made through the API into that suite doesn't count.
- doctor prints a paste-ready required_check_sources block (src/check-pin-block.ts): each name
  mapped to the workflow file that posts it, confirmed against main's tip where it can be, ambiguous
  ones listed for a manual choice. auto-config (init, add-project) writes it into a new config.

Test (fails without): test/required-check-source-coordinator.test.ts (another App's failing run
blocks a pinned pass; an unpinned install's PR is ready as on main), test/main-watch.test.ts (#727
cases: fail-to-pass, API-made run, unpinned as before), test/required-check-source.test.ts (raw
answer doesn't compile, the 12 raw-read spellings), test/auto-config.test.ts (new config pinned),
test/check-pin-block.test.ts.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Round 1: changes at 65fa328

Each claim is labelled measured, inferred or guess.

  1. A pin never turns a fail into a pass. A pinned check passes only when its source passes and the check judged by name (any source) passes too. This applies in pinnedChecks (readiness, the merge flow, fix-forward, fast checks, the ruleset hold) and in the main watch over outcomesAt. The blocking check carries no id or link, so no fix run touches another App's job.
    • Measured: your fixture (an unpinned App's failing test listed before the pinned App's passing one) gives test fail, not ready.
    • Fail-without on 1a4a97f: the coordinator test was made ready (expected: 0 actual: 1), and the main watch accounted the merge.
  2. A raw read can't be judged. readiness, requiredOutcomes, headChecks, mayMerge, prReadiness and fix-forward's checks take JudgedChecks, which only PinnedChecks.of makes.
    • Measured: a fixture passing a raw Backlog.checks answer to the six calls is refused at 6 of 6 on 65fa328 and at 0 of 6 on 1a4a97f.
    • The scan: test/raw-check-reads.ts finds raw reads by type. It catches all 12 of your spellings, and the regex catches 1; both are asserted. A checks that isn't GitHub's isn't flagged.
    • On src/: only check-source.ts, main-watch.ts and canary.ts read them raw, each with its reason in RAW_READS_ALLOWED.
    • The regex test stays (rule 10).
  3. A workflow pin needs one of its run's own jobs. The check run's html_url must be /<repo>/actions/runs/<the mapped run's id>/job/<its id>, in src/backlog/github.ts and in the main watch API (the run id comes from actions/runs).
    • Measured with fixtures: an API-made check run in a ci.yml-mapped suite, whose page is /o/r/runs/<id>, is missing. The main watch counts it unaccounted; on 1a4a97f it was accounted.
    • Guess: that a real API-made check run has such a page.
  4. Claims corrected.
    • "Can never do" now matches change 1.
    • The canary is no longer described as reporting only. Its verdict keeps or rolls back an install. Its raw read stays, justified (inferred): a pass needs the PR merged by the canary instance, whose own merge judged the checks by their pins, and every check green, so a spoofed check can only turn a pass into a fail.
  5. The owner's option (c):

Verification (measured):

  • every compiled test except mcp-* and plugin-fetch gives # tests 3539 # pass 3482 # fail 0 # skipped 57;
  • typecheck is clean, denylist: 1285 files clean, and DCO: all commits signed off.

The full fail-without table is in the body.

… own PR, keep doctor's plain line

#788 grew to about twice the 600-line limit, so the doctor pin block (src/check-pin-block.ts) and the
pins auto-config writes into a new project's config move to a PR stacked on this branch. What stays:
the engine pinning, option (c) matching, the JudgedChecks type, the job-page check for workflow pins
and the corrected claims. doctor keeps its plain line (checkSourceDoctor): each required check with
no pin, judged by name, and how to pin it.

Test (fails without): test/required-check-source.test.ts "doctor names each required check with no
recorded source, judged by name, and how to pin it".

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
…page on its run's host; no __proto__ pin

From an adversarial pass against the owner's decision guide:
- GitHub's protection and rules never answered (an error from the start): a check config.yaml doesn't
  pin was judged by name, though the rules may pin it. Now it is missing until GitHub answers, and the
  main watch waits to judge a merge (never unaccounted for a failed read).
- A check run's page with the right path on another host, or over http, passed the job-page check. Now
  it must be the workflow run's own page (its html_url) plus /job/<id>, over https on that host.
- A config.yaml pin keyed __proto__ was dropped by the record, leaving that check judged by name.
  Now the config is refused.

Test (fails without): test/required-check-source.test.ts (unknown rules: missing; the other-host and
http pages; the __proto__ pin) and test/main-watch.test.ts (a merge waits while the rules are unknown).

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Split, and fixes from an adversarial pass (head d614853)

@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 2)

I reviewed head d6148530 against #727, the round-1 verdict and the owner's option (c). Every claim is labelled measured (a command I ran, or CI output I cite), inferred (reasoned from the code, not run) or guess. The fixtures are stand-ins passed as data, run with a scratch HOME and state dirs and a fake claude on PATH. No repo file was edited.

Round-1 required changes

# Change Status Evidence
1 A pin never turns a fail into a pass met in the code, but the claim overstates it (change 4) measured: the round-1 fixture (another App's failing test listed before the pinned App's pass) now gives test=fail ready=false; it was pass/true. Across 5 readiness orderings and 2 main-watch orderings, the pinned result is never a pass where judging by name isn't
2 A raw read can't be judged partly met (change 2) measured: a raw Backlog.checks answer passed to the 6 judging calls fails to compile, and so does a direct as JudgedChecks. The type scan catches the 12 round-1 spellings. The gaps are below
3 A workflow pin needs the run's own html_url plus /job/<id>, over https on that host met measured: isJobOfRun refuses another host, http:, another port, /runs/<id>, a trailing slash, job/01 and //job. That covers readiness (fromPinnedSource) and the main watch (judge passes r.url)
4 Corrected claims, including the canary partly met (change 4) The canary is now described as deciding whether an install stands, with the reason it is safe (src/canary.ts:108-113; inferred true from canaryProgress at canary.ts:91-97, which needs GitHub to say merged and every check on the head green). One overclaim remains (change 4)

Option (c): an install without pins behaves as main

  • When GitHub has answered (measured):
    • an unpinned check judged through pinnedChecks equals the by-name judgement in 256 of 256 subsets and orders of 7 check runs and statuses;
    • on this repo's real answers from round 1 (9 required checks, rules naming no App), the PR head is ready=true, as on main;
    • in the main watch, a direct push with pins and GitHub's answer {} is unaccounted, exactly as without pins.
  • Pins only tighten (measured): a config or GitHub App pin turns another App's test into missing. A GitHub-named pin keyed __proto__ is lost (change 3).
  • When GitHub's rules have never answered:
    • Readiness is fine (measured): an unpinned check is missing, so no PR becomes ready. That is fail closed (principle 1), and it is disclosed in the PR body.
    • The main watch is not. It skips the whole commit, not only the check part (measured, MainWatch.step with stand-ins):
      • no pins (as main): a direct push gives unaccounted=["22222222 was pushed to main directly: no merged PR has it"];
      • with pins and github: async () => null: unaccounted=[], and not even the pulls call is made (calls=protection,protection).
    • Why that matters: src/main-watch.ts:737 returns before judge(). While the rules are unread, it holds back verdicts that don't depend on the rules at all: a direct push, an unknown actor, a bot merge, a merger who isn't a writer. Those verdicts are main's guard against a bypass, and the stop that follows is delayed until GitHub answers. That is fail open, against principles 1 and 3 (change 1).
    • The window is bounded (inferred): githubApps() keeps the last answer after any error, so this applies only until the first answer after a coordinator start, for example during a budget pause or an outage. A branch name with a slash is not a lasting trigger: gh api repos/jayden1711/worklane/branches/batch%2F1 answers batch/1 (measured).
  • Could "waits" block turning OFF or the stop? No (inferred):
    • the policy's main_watch: off and direct mode return before step() (src/coordinator/merge-flow.ts:978-979);
    • the stop file and the dashboard's OFF don't go through the watch;
    • every backlog request has a timeout (src/backlog/github.ts:66).

Adversarial checks (measured)

  • Config pin keys:
    • __proto__ is refused;
    • constructor, toString, hasOwnProperty and valueOf parse, and the pin holds (pinOf gives {"app_id":1111}).
  • GitHub-named pins (classic protection or a ruleset):
    • constructor and toString hold, and a spoof is missing;
    • __proto__ is dropped: githubCheckApps builds a plain {}, so out["__proto__"] = id is ignored, and another App's __proto__ check gives pass (change 3).
  • Unicode lookalikes:
    • a Cyrillic tеst or test (trailing space) from another App never satisfies test: missing pinned, missing by name;
    • a lookalike pin key leaves test unpinned, and doctor warns no source recorded for test.
  • isJobOfRun also accepts userinfo (https://x@github.com/...), a query or fragment, and an upper-case host or owner. Inferred harmless: html_url is computed by GitHub, and an App can set only details_url.

The brand and the raw-read scan (principle 4)

Fixtures were passed as data to the PR's own typeErrors and rawCheckReads, as src/ files (measured):

Fixture in src/ Compiles Scan hits
raw as JudgedChecks no 1
raw as never passed to readiness yes 1
alias through a structural type { checks(s): … }, then as never yes 0
const k = 'checks' as const; backlog[k](sha) yes 0
Reflect.get(backlog, 'checks') yes 0
GitHub data from another source, as unknown as JudgedChecks yes 0
  • Is asJudged test-only? No test or lint keeps it, or any cast to JudgedChecks, out of src/ (measured: no such check in test/). Only packaging stops it: package.json files ships dist/src only, so a src/ import of test/helpers.js would fail at runtime but pass the tests (inferred).
  • Accidental raw reads are now caught. A deliberate cast plus an unusual spelling passes both layers (change 2).

Principle 9: the raw-read allowlist

  • Naming: each entry in RAW_READS_ALLOWED names the check it exempts (the new raw-read scan) and why, and the PR body repeats this under "Exemptions (decision guide 9)". The scan is new, so no existing check is loosened (measured, diff).
  • Each reason (inferred):
    • check-source.ts holds: it is the judge itself;
    • main-watch.ts holds: its runs are judged by pins in judge();
    • canary.ts holds as stated: a spoofed check can only turn a pass into a fail, because a pass needs GitHub to say merged and every check green.
  • Not blocking: the exemption is per file, so any later raw read in main-watch.ts or canary.ts passes unseen. A per-function entry would be tighter.
  • Gap: asJudged is a fourth exemption in all but name, and nothing guards it (change 2).

Fail-without (measured)

  • This round:
    • Method: a scratch worktree at 77f3c345, with test/required-check-source.test.ts and test/main-watch.test.ts from d6148530, then npx tsc (3 type errors, emitted anyway) and node --import ./dist/test/isolate.js --test dist/test/required-check-source.test.js dist/test/main-watch.test.js.
    • Result: # tests 59 # pass 53 # fail 6, matching the builder.
    • Key lines:
      • https://evil.example/o/r/actions/runs/700/job/1 was accepted;
      • the __proto__ pin case failed with true !== false;
      • the main watch with unknown rules failed with Cannot convert undefined or null to object.
  • On the head:
    • every compiled test except mcp-* and plugin-fetch (concurrency 4) gives # tests 3511 # pass 3453 # fail 1 # skipped 57. The one failure is dashboard-speed.test.js "event cache…", with read ECONNRESET under load. Run alone it gives # tests 9 # pass 9 (measured). This PR doesn't touch the dashboard (inferred: a flake, not this PR);
    • npm run typecheck is clean;
    • npm run check:denylist gives denylist: 1283 files clean.

Rule 10, size, hygiene, protection, CI

Required changes

  1. The main watch must wait only where the unknown rules matter (principles 1 and 3).

    • Don't return before judge() when pins.github() is null. Pass "rules unknown" into judge and wait only at the required-checks step. Everything before it must reach its verdict as on main: accepted, direct push, unknown actor, bot, a merger who isn't a writer.
    • Add a test that fails on this head: a direct push while the rules have never answered is unaccounted. Also test a bot merge.
  2. Close the cast route into JudgedChecks (principle 4; it answers "is asJudged test-only": not yet). Extend test/raw-check-reads.ts:

    • flag any type assertion in src/ outside check-source.ts whose target or contextual type is JudgedChecks (this catches as never and as unknown as JudgedChecks passed where JudgedChecks is expected);
    • flag any src/ import that resolves under test/.
    • Add fixture tests, as data, for the table's rows. That also covers the three spellings the scan misses, since each needs such a cast to reach a judge.
  3. Fix the __proto__ class for GitHub-named pins as well.

    • Build githubCheckApps' result with Object.create(null) or a Map, so a check GitHub's rules pin under that name keeps its pin.
    • Add a test: a ruleset pinning __proto__ to App A, another App's __proto__ is missing.
  4. Correct the remaining overclaim (principle 5). "A same-name check from any source that isn't a pass still blocks the pinned one" appears in:

    • the PR body's intent bullet and "Compromised component";
    • CHANGELOG.md (both entries);
    • docs/design.md.

    It holds only when that check is the one judging by name picks: listed first in readiness, or the latest started in the main watch. Measured:

    • a pinned pass listed before another App's failure gives by name=pass pinned=pass;
    • in the main watch, a newer pinned pass with an older failure from another App is accounted.

    This is never looser than main. Say what the code does: "pinning never turns a check that isn't a pass, as judged by name, into a pass".

    Also, CHANGELOG.md's "nothing changes on upgrade for an install without pins (no PR held, no main-watch stop…)" should name the window before GitHub's rules first answer, as the PR body already does.

A genuinely new trade-off for the owner

Should a pinned name also be blocked by any same-name non-pass, from any source, not only the one judging by name picks? Inferred: this is stricter than main, and it has a new cost. Any App on the repo, or a cancelled duplicate suite of the pinned workflow (push and pull_request runs on one head), could then hold a pinned PR until someone acts. As built, nothing is looser than main, so this is not required.

…edChecks caught; no dropped __proto__ rule

Review round 2:
- While GitHub's rules have never answered, the main watch returned before judging at all, so a direct
  push or a bot's merge waited too. Now judge() waits only at the required checks (need 'rules', no
  call); everything before them is judged as on main.
- A cast reached a judging function past the brand (`as never`, `as unknown as JudgedChecks`), and a
  raw read spelled with a const key, Reflect.get or a structural alias went unseen. The scan now flags
  any type assertion into JudgedChecks outside check-source.ts, those raw-read spellings, and any src/
  import from test/.
- githubCheckApps built a {}, so a check GitHub's rules pin under the name __proto__ lost its pin.
  It now has no prototype.
- The claim that another source's non-pass always blocks a pinned check is corrected: pinning never
  turns a check that isn't a pass, as judged by name, into a pass. The CHANGELOG names the window
  before GitHub's rules first answer.

Test (fails without): test/main-watch.test.ts (a direct push and a bot's merge while the rules are
unknown are unaccounted), test/required-check-source.test.ts (the cast and raw-read fixtures; the
ruleset's __proto__ pin).

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Round 3: changes at 5c4ef10

Each claim is labelled measured or inferred.

  1. The main watch waits only at the checks while GitHub's rules have never answered. judge() takes rulesUnknown and returns a rules need (no call) at the required-checks step. A direct push, an unknown actor, a bot and a non-writer are judged as on main.
    • Measured on d614853: a direct push and a bot's merge under unknown rules weren't flagged (expected: 1 actual: 0). Both are unaccounted now.
  2. The cast route into JudgedChecks is closed.
    • judgedCasts flags any type assertion in src/ whose target, or expected type at a judging call, is JudgedChecks. That covers as never and as unknown as. Only check-source.ts is allowed.
    • testImports flags any src/ import from test/.
    • rawCheckReads now also catches a const computed key, Reflect.get and a structural alias of a Backlog.
    • Measured: your six fixtures, as data, are all caught. With d614853's own scan they gave hits 1, 1, 0, 0, 0 and 0, with five compiling. src/ has no cast outside check-source.ts and no import from test/.
  3. githubCheckApps builds its map with no prototype.
    • Measured on d614853: a ruleset's __proto__ pin was dropped ('__proto__ pass'). Now another App's __proto__ is missing.
  4. Overclaim corrected in the body, both CHANGELOG entries, design.md and the code comment. It now says: "pinning never turns a check that isn't a pass, as judged by name (the one judging by name picks), into a pass". The CHANGELOG names the window before GitHub's rules first answer. Behaviour is unchanged: making it stricter is the owner's call.

Verification (measured):

  • the fail-without on d614853 gave # tests 63 # pass 61 # fail 2;
  • every compiled test except mcp-* and plugin-fetch gives # tests 3539 # pass 3482 # fail 0;
  • typecheck is clean, denylist: 1283 files clean, and DCO: all commits signed off.

#803 and #794 have merged this head (signed off, no rewrite), and both build.

@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 3)

I reviewed head 5c4ef109 against #727, the round-2 verdict and the owner's decisions. Every claim is labelled measured (a command I ran, or CI output I cite), inferred (reasoned from the code, not run) or guess. Fixtures are stand-ins passed as data, run with a scratch HOME and state dirs and a fake claude on PATH. No repo file was edited.

Two things block this round:

  • A new owner decision (relayed by the orchestrator, 2026-10-19) makes the stricter rule required (change 1).
  • A route around the brand: a value typed any reaches a judging function with no cast and no scan hit (change 2).

Everything round 2 asked for is in place.

Round-2 required changes

# Change Status Evidence
1 The main watch waits only at the checks while GitHub's rules have never answered met measured, judge() with rulesUnknown: true against the same facts without it. A direct push, an unknown actor, a bot merge and a non-writer each give the same unaccounted reason as on main, and an accepted sha is accounted. Only a writer's merge and a harness merge with green checks wait (need: rules, no call). MainWatch.step with github: async () => null flags the direct push (22222222 was pushed to main directly); round 2 gave unaccounted=[]. The rules need returns before take(), so it spends no call budget (src/main-watch.ts:773)
2 Close the cast route into JudgedChecks; no src/ import from test/ met for casts and imports. A wider route stays open (change 2) measured, my six round-2 fixtures as data: every one is caught by rawCheckReads or judgedCasts (raw-read hits 1,1,1,1,1,0; cast hits 1,1,1,0,0,2). testImports flags static, re-export and dynamic imports from test/ (the builder's test)
3 githubCheckApps with no prototype met measured: a ruleset pinning __proto__, constructor or toString to App 1111 gives pinOf={"app_id":1111}, and another App's check of that name is missing (round 2: __proto__ gave pass). The test now compares { ...githubCheckApps(...) }: own keys are compared as before, so it is not weaker
4 Corrected wording met for round 2's wording Body, both CHANGELOG entries, docs/design.md and the src/check-source.ts comment now say "never turns a check that isn't a pass, as judged by name (the one judging by name picks), into a pass". The CHANGELOG names the window before GitHub's rules first answer. This wording must change again under change 1

Option (c) and the pins (measured, r2-788.mjs re-run on this head)

  • An unpinned check judged through pinnedChecks equals the by-name judgement in 256 of 256 subsets and orders.
  • While GitHub's rules have never answered, an unpinned passing check is missing.
  • isJobOfRun refuses another host, http:, another port, a trailing slash, job/01, //job and /runs/<id>, as in round 2.
  • Today's behaviour, which change 1 replaces:
    • readiness: a pinned pass listed first, with another App's failure second, gives by name=pass pinned=pass;
    • main watch: a newer pinned pass over an older failure from another App is accounted.

New attacks on the brand (measured; fixtures as src/ files, passed to this head's typeErrors, rawCheckReads and judgedCasts)

Fixture Compiles Raw-read hits Cast hits
n1 (backlog as any).checks(sha) passed to readiness yes 0 0
n2 const b: any = backlog; b.checks(sha) passed to readiness yes 0 0
n3 structural alias through a parameter: g = (b: { checks(s): Promise<any> }) => b.checks(sha); g(backlog) yes 0 0
n4 const raw = (await backlog.checks(sha)) as never, then checks: raw yes 1 0
n5 other GitHub data as never into a const, then checks: raw yes 0 0
n6 other data through JSON.parse (type any), no cast at all yes 0 0
n7 a generic id<JudgedChecks>(x) whose body is x as T yes 0 0
n8 a type predicate (x): x is JudgedChecks => true yes 0 0
n9 Object.entries(backlog), the member called through as any yes 0 0
n10 backlog as unknown as { checks(s): Promise<any> } yes 0 0
  • The cast scan: it looks at the assertion's target type and its contextual type. A cast to any, to never in a const with no annotation, or to a type parameter has neither.
  • The raw-read scan finds nothing in n1, n2, n3, n9 or n10:
    • an any receiver has no member to trace;
    • as unknown as hides the Backlog type before the second cast;
    • a parameter alias isn't a variable declaration or an assertion.
  • So two claims are false (measured):
    • "What this can never do: Let a raw GitHub checks answer reach a judging function and compile";
    • the scan "lists every read of Backlog.checks … in src/".
  • Cost of the gap (inferred): src/ is protected, so the owner reviews any such change. These are deliberate spellings, not slips. The judging still needs the brand on the normal path.

Fail-without (measured)

  • Method:
    • a scratch worktree at d6148530, with test/required-check-source.test.ts, test/main-watch.test.ts and test/raw-check-reads.ts from 5c4ef109;
    • npx tsc, then node --import ./dist/test/isolate.js --test dist/test/required-check-source.test.js dist/test/main-watch.test.js.
  • Result: # tests 63 # pass 61 # fail 2, matching the builder.
    • main watch (#727): while GitHub's rules have never answered, a direct push and a bot's merge are still unaccounted: expected: 1 actual: 0.
    • a check GitHub's ruleset pins under the name __proto__ keeps its pin: failed.
  • On the head:
    • every compiled test except mcp-* and plugin-fetch gives # tests 3539 # pass 3482 # fail 0 # skipped 57;
    • npm run typecheck: exit 0;
    • npm run check:denylist: denylist: 1283 files clean.

Rule 10, hygiene, protection, CI, merge (measured)

Required changes

  1. Owner decision (2026-10-19, relayed by the orchestrator): any same-name non-pass, from any source, blocks a pinned check.

    • What blocks: any App, any workflow, and any run of that name on the commit, including a cancelled duplicate.
    • Where: readiness, the merge flow (the check of main after a merge, fix-forward, the PR batch's fast checks, the docs-only hold) and the main watch.
    • The owner accepts the cost: this is stricter than main, so another App or a cancelled duplicate run can hold a pinned PR.
    • Unpinned checks stay exactly as main (option c).
    • Statuses: say whether a same-name commit status that isn't a pass also blocks. "Any source" and principle 1 suggest yes. Test whichever you choose.
    • Tests to add, each failing on 5c4ef109:
      • readiness: App 1111's pinned pass listed first, then App 2222's same-name failure: not ready (test fail). Today it gives pass.
      • readiness: a pinned workflow's pass plus a cancelled duplicate run of the same name, in another suite on the same head: not ready.
      • main watch: a newer pinned pass with an older same-name failure from another App: unaccounted. Today it is accounted.
      • main watch: a pinned pass plus a cancelled duplicate: unaccounted.
      • one merge-flow site, through pinnedChecks: the same pair blocks.
      • option (c) is unchanged: an unpinned check equals the by-name judgement over subsets and orders, as the 256/256 fixture shows.
    • Update the claims: the body (intent, "How it works", "What this can never do", "Compromised component: Another App"), both CHANGELOG entries, docs/design.md and the src/check-source.ts comment.
  2. Close the any and never route into a judging function (principles 4 and 5; the class of round 2's change 2). Extend test/raw-check-reads.ts so that every n1–n10 fixture above is flagged, and add them as fixture tests. For example:

    • outside check-source.ts, flag any expression whose contextual type is JudgedChecks and whose own type is any or never (n1–n6, and round 2's as never);
    • flag a type predicate or a type argument that names JudgedChecks (n7, n8);
    • for rawCheckReads, flag any value whose type has a raw member when it is asserted, annotated or passed as a type that isn't a raw owner, any and unknown included (n1, n2, n3, n9, n10).

    If any part stays open, narrow the two claims to what is caught.

For the owner (inferred; not a required change)

On this repo, new-deps.yml runs on edited with cancel-in-progress: true. On all four PR heads here it ran twice on the same commit (measured, filter=all), and filter=latest, which the engine reads, keeps both (measured on bdb92eac). So once new-deps is pinned (#794), a PR body edited while new-deps runs leaves a cancelled duplicate on the head. Under change 1, that holds the PR until the next push. This is the cost you accepted; it names where it will show first.

… source's non-pass blocks it

The owner's rule (2026-10-19), in readiness, the merge flow and the main watch:
- For a pinned check's own source (its App or workflow), only the latest run on the head counts, by
  the runs' own order: the later workflow run, then the later check run (a re-run attempt). A later
  run of that source supersedes its earlier or cancelled ones. A run for another head is not the
  pinned source, so it never supersedes.
- Any check of the same name from any other source that isn't a pass blocks it: another App, another
  workflow, a run for another head, a commit status that isn't success. The main watch now reads the
  statuses of a pinned merge for that.
- Unpinned checks stay exactly as main (option c).

Also, from review round 3: the scans in test/raw-check-reads.ts now flag any `any` or `never` value,
generic return or type predicate where JudgedChecks is expected, and a Backlog put into an `any`,
`unknown`, empty or structural type (a cast, an annotation or an argument). The claims say this is a
lint, not a run-time guarantee.

Test (fails without): test/required-check-source.test.ts (cancelled-then-passed and
failed-then-passed are ready; a passed pin plus another App's failure is not; run order, not list
order; another head never supersedes; a status blocks; n1 to n10), test/main-watch.test.ts (the same
cases at merge time), test/required-check-source-coordinator.test.ts (the merge flow's checks).

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Round 4: changes at e8107eb

Each claim is labelled measured or inferred.

  1. The owner's rule (2026-10-19), in readiness, the merge flow and the main watch.
    • What counts: for a pinned check's own source, only its latest run on the head counts, by the runs' own order: the later workflow run, then the later check run, as a re-run attempt is. A later run of that source supersedes its earlier or cancelled ones. A run for another head never does.
    • What blocks: any non-pass of the name from any other source: another App, another workflow, a run for another head, a commit status that isn't success. The main watch now reads a pinned merge's statuses.
    • Unpinned: exactly as main. Measured: a test compares every subset and order of four checks, 65 cases.
    • Tests cover cancelled-then-passed, failed-then-passed, passed then cancelled in the later run, a pinned pass plus a foreign failure, another head, re-run attempts, list order and a status, in readiness, the merge flow and the main watch.
    • Fail-without on 5c4ef10 (measured): # tests 71 # pass 67 # fail 4.
  2. The any and never routes are closed by the lint, not at run time.
    • judgedCasts flags any/never values, generic returns and type predicates where JudgedChecks is expected.
    • rawCheckReads flags a Backlog put into any, unknown, an empty or a structural type, by cast, annotation or argument.
    • Measured: your n1 to n10 are each caught. All 16 of your fixtures hit, and src/ stays clean. On 5c4ef10 n1 wasn't caught.
    • The claims in the body, CHANGELOG, design.md and the code now say "a lint, not a run-time guarantee". I didn't build the run-time WeakSet brand; the body says why.
  3. Wording updated everywhere, including "Another App" in "Compromised component" and "What this can never do".

Verification (measured):

  • every compiled test except mcp-* and plugin-fetch gives # tests 3543 # pass 3486 # fail 0;
  • typecheck is clean, denylist: 1283 files clean, and DCO: all commits signed off.

new-deps.yml is unchanged, per the owner.

@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 4)

I reviewed head e8107eb4 against #727, the round-3 verdict and the owner's refined rule (2026-10-19). Every claim is labelled measured (a command I ran, or CI output I cite), inferred (reasoned from the code, not run) or guess. Fixtures are stand-ins passed as data. Each ran with a scratch HOME, scratch state dirs and a fake claude on PATH. No repo file was edited.

Where it stands:

  • The owner's rule works: all three of the owner's cases and every edge case hold in readiness and the main watch, in every list order.
  • Unpinned checks equal main.
  • Three things block:
    1. A run-time brand is cheap, so principle 4 makes it required.
    2. An Actions-App pin lets another workflow file supersede the pinned failure.
    3. Several claims say "never looser", but the owner's own case 1 is looser than main.

The refined rule (measured, rule-788.mjs: PinnedChecks.of → readiness, and judge(), each case over every permutation of its runs)

Case Readiness Main watch
Owner 1: cancelled, then passed, same pinned workflow ready accounted
Owner 2: failed, then passed, same pinned workflow ready accounted
Owner 3: pinned pass + another App's same-name failure (workflow pin; App pin) ci: fail ci fail
Passed, then a later own run failed / still running fail / pending fail / pending
A later run of the pinned file for another head: its pass over the head's failure; its failure under the head's pass fail; fail fail; fail
Re-run attempt (same workflow run, newer check run): pass over fail; fail over pass ready; fail accounted; fail
Same-name status failure / error / pending / success beside a pinned pass fail / fail / pending / ready same
Foreign App posts a newer same-name pass over the pinned failure (workflow pin; App pin; with a forged /actions/runs/100/job/999 page) fail fail
An API-made Actions check in the pinned run's suite (/runs/<id> page), newer pass fail fail
Another workflow file's newer pass over the pinned file's failure (workflow pin) fail fail
GitHub's rules pin ci to the Actions App (15368): evil.yml's later pass over ci.yml's failure ready accounted
ci.yml run 100 re-run (attempt 2, the newest check run) fails after run 200 passed ready accounted
  • Order independence: every case gave a single verdict across all list orders, so the ordering is by the runs' own ids, never list order.
  • Unpinned equals main: 326 of 326 subsets and orders of a 5-check pool (two Apps, a running Actions job, two statuses) give the same readiness through PinnedChecks.of as by name. The builder's 65-subset test also passes.
  • The supersede comes only from the pinned source (inferred, code):
    • an App pin needs c.app === app_id;
    • a workflow pin needs the suite's workflow run (Actions slug, check_suite_id), that path, repo and head, and the run's own /job/<id> page.
    • A foreign App's run therefore lands in "other" and can only block. The one exception is the Actions-App pin (change 2).

JudgedChecks: lint or run-time brand (item 2)

The lint on the 16 reviewer fixtures (measured, scan.mjs): all 16 are caught (round 2's six and round 3's n1 to n10). src/ itself is clean.

My own 8 fixtures, each a src/ file fed as data (measured): every one compiles, and every one gets 0 raw-read hits and 0 cast hits.

# Route into a judging call
e1 // @ts-expect-error over other data (this repo has no ts-comment ban)
e2 Object.assign(otherData, judged) (type CommitCheck[] & JudgedChecks)
e3 a generic returning T & {}, with T inferred from context
e4 an overload whose implementation returns any
e5 a judged array mutated through a CommitCheck[] cast
e8 Array.prototype.push.apply(judged, otherData)
e6 withTestRuns(judged, new Set(), otherData)
e7 new PinnedChecks({ backlog: { checks: () => otherData }, … }).of(sha) with no checkApps (so judged by name)

Is a run-time brand cheap? Yes (measured, sync-brand.mjs):

  • PinnedChecks.of is async (AsyncFunction), but the brand is made by the private, sync judged(), and the exported withTestRuns is sync.
  • A test helper needs no async construction. One top-level await in test/helpers.ts makes EMPTY = await new PinnedChecks({ backlog: { checks: async () => [] }, … }).of(sha). Then asJudged = (c) => withTestRuns(EMPTY, new Set(), [...c]) stays sync with the same signature. On a 3-check fixture it gave the same items in the same order, and the same readiness.
  • So no test call site changes. The package is ESM ("type": "module", NodeNext, ES2023), so top-level await works (inferred).
  • Where the check goes: every judging path ends in readiness or requiredOutcomes (src/pr-watch.ts), since prReadiness, fix-forward's three calls, mayMerge and the merge flow's fast-check, hold and flaky-gate calls all go through them (measured, grep), plus withTestRuns' github input. That makes it about three checks in src/.

A brand stops e1 to e5, e8 and all 16 reviewer routes at run time (inferred). e6 and e7 go through the real makers, so they need a narrow lint or a named claim (change 1).

Fail-without (measured)

  • Method: a scratch worktree at 5c4ef109, with test/required-check-source.test.ts, test/main-watch.test.ts and test/required-check-source-coordinator.test.ts from e8107eb4. test/raw-check-reads.ts stays at 5c4ef109's, since the lint is the fix for the n-fixtures. Then npx tsc and node --import ./dist/test/isolate.js --test on the three files.
  • Result: # tests 71 # pass 67 # fail 4, matching the builder.
    • main watch: expected: 'accounted' actual: 'ci fail';
    • merge flow: the pinned App's later run didn't supersede;
    • readiness: 'test pass', expected 'test fail';
    • caught: n1 backlog as any, read, pass.
  • On the head:
    • every compiled test except mcp-* and plugin-fetch: # tests 3543 # pass 3486 # fail 0 # skipped 57;
    • npm run typecheck: exit 0;
    • npm run check:denylist: denylist: 1283 files clean.

Size (item 5, measured: git diff --shortstat origin/main...e8107eb4)

  • Totals: +1,317/−48 in 27 files.
    • src/ is +449 in 14 files. 316 of those lines are in check-source.ts (209, all new) and main-watch.ts (98); the rest are call-site edits of 2 to 49 lines.
    • Tests are +858, including the 182-line type-flow lint.
  • Judgement: reviewable as one PR. It is one feature with one module and narrow call sites. The hardest part to review is the lint. Change 1 makes that lint secondary to a few lines of run-time check, which helps.

Usual checks (measured)

Required changes

  1. Add the run-time brand (principle 4; cheap, measured above). Keep the lint (rule 10).
    • A module-private WeakSet in src/check-source.ts, filled only by judged(), which also freezes the array.
    • readiness, requiredOutcomes and withTestRuns (on its github input) throw on any array not in it. That fails closed; judging is on no OFF path (inferred; confirm it in the Adversarial pass).
    • test/helpers.ts asJudged goes through withTestRuns over one top-level-awaited EMPTY, so no call site changes.
    • Tests:
      • a cast, any or @ts-expect-error value at readiness and at requiredOutcomes throws;
      • pushing onto a judged array throws.
    • For e6 and e7: add a callers allowlist to the lint (withTestRuns only in ci-none.ts, new PinnedChecks only in merge-flow.ts), or name both routes in the claims.
    • Fix the false claims:
      • check-source.ts:20 says "which only PinnedChecks.of makes";
      • the body's "can never … compile without a cast or an any" is false: e1 and e2 have neither.
  2. An Actions-App pin must not supersede across workflow files (principle 1; the owner's rule says "the latest run of that workflow").
    • Measured: with GitHub's rules pinning ci to App 15368, ci.yml failed (run 100) and a PR-added evil.yml passed (run 101) gives ready and accounted. So does ci.yml still running with evil.yml passed.
    • This also makes the body's "Compromised component: Agent … can't make its PR ready with a workflow it adds" false for such pins.
    • Guess: GitHub's protection UI often records the Actions App as a check's source, so this may be common on existing installs.
    • Fix: under an App pin, take the latest run per workflow file (an Actions run whose workflow can't be read is its own group), and every group's latest must pass. Do this in pinnedChecks and pinnedAt.
    • Tests: the two cases above, each failing on e8107eb4.
  3. Correct the "never looser" claims (principle 5).
    • Measured (vs-main.mjs): the owner's case 1 is looser than main. With ci listed cancelled first, then passed, main's by-name readiness gives ci: cancelled and this head gives ready.
    • False now:
      • the body's "It is never looser" and "Turn a check that isn't a pass into a pass";
      • CHANGELOG's "stricter than judging by name";
      • src/check-source.ts:12's "Pinning only ever turns a match into missing".
    • Say instead: a pinned source's later run on the head supersedes its own earlier failed or cancelled runs, which judging by name could block. That is the owner's decision (2026-10-19), and it is the only way it is looser.

For the owner (a genuinely new trade-off; not a required change)

Which run counts as "latest"? The specified order is workflow run, then check run.

  • The case: run 200 passed, then someone re-ran the older run 100, and attempt 2 (the newest check run) failed. That gives ready / accounted (measured above).
  • Fail-closed alternative: require the latest by run order and the newest check run of that source to pass.
  • Re-running needs write access (inferred).

Resolved from round 3

Round 3's owner note on new-deps' cancelled duplicates no longer applies. A same-workflow cancelled run is now superseded by the later run (owner case 1, measured above).

… re-run blocks (#727)

- JudgedChecks is now checked at run time: judged() records each array it
  makes in a module-private WeakSet and freezes it and its checks, and
  readiness, requiredOutcomes and withTestRuns throw on an array not made
  there or since changed. Test fixtures go through the real makers
  (test/helpers.ts asJudged, one top-level await).
- The lint flags any src/ use of the two makers (withTestRuns,
  PinnedChecks) outside ci-none.ts and merge-flow.ts, including through an
  alias, a namespace import or a dynamic import (makerUses).
- A pinned source's own runs are grouped by workflow file (one group for
  an App that isn't Actions). In every group both the latest run and the
  newest check run must pass, so one workflow file never supersedes
  another under an Actions-App pin, and a re-run of an older run that
  failed after a newer run passed blocks. Same in the main watch.
- CHANGELOG and design.md name the one loosening the owner decided: a
  pinned file's later run supersedes its own earlier failed or cancelled
  runs.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Round 5 at bdfcc29 (one signed-off commit on e8107eb, no rewrite).

  1. A run-time brand, and the lint kept.
    • judged() records each array it makes in a module-private WeakSet, and freezes the array and its checks.
    • readiness, requiredOutcomes and withTestRuns throw (assertJudged) on an array that wasn't made there or has been mutated.
    • test/helpers.ts asJudged goes through withTestRuns over one empty judged set, with one top-level await, so it stays sync.
    • The reviewer's 8 fixtures: e1 to e5 and e8 now throw (measured, tests).
    • e6 and e7 (a maker called over made-up data) can't be told apart at run time. Making them impossible would mean taking ci-none.ts's test runs and the coordinator's backlog off the shared makers, a larger refactor. So makerUses flags any src/ use of withTestRuns or PinnedChecks outside MAKERS_ALLOWED (ci-none.ts, merge-flow.ts). That includes aliases, a namespace import and a dynamic import. The claims name this.
    • The body's "can never … compile without a cast or an any" line is corrected, with a "false through e8107eb" note.
  2. An Actions-App pin never lets one workflow file supersede another.
    • ownVerdict (readiness) and pinnedAt (main watch) group the pinned source's runs by workflow file, and every group's latest run must pass. A group that is pending or missing blocks.
    • Tests: the reviewer's two cases (ci.yml failed then evil.yml passed; ci.yml running), a failing workflow added in the PR itself, and both passing.
    • The Compromised component claim is corrected, with a "false through e8107eb" note.
  3. "Latest" fails closed. In each file's group, the newest check run must pass too.
    • Test: run 200 passes, then run 100 is re-run and fails, so it is blocked (readiness and the main watch).
    • The owner's three cases are unchanged and pass.
  4. The "never looser" claims are corrected in the body (both lines), the CHANGELOG (both entries), docs/design.md and src/check-source.ts. Each names the same-file supersede as the one loosening, decided by the owner.

Fail-without (measured): the new test files on e8107eb gave # tests 72 # pass 68 # fail 4:

  • the main watch Actions-App case (expected: 'ci fail' actual: 'accounted');
  • the readiness Actions-App case;
  • "latest" fails closed;
  • the brand (Missing expected exception: a cast).

With e8107eb's own scans, e6 and e7 got 0 hits each.

After (measured):

  • the broad suite: # tests 3548 # pass 3491 # fail 0 # skipped 57;
  • typecheck clean, denylist: 1283 files clean, DCO: all commits signed off.

The body has a new Adversarial pass, round 5 section: other files, other Apps, statuses, re-runs, other commits and API-created runs.

#803 and #794 have this head merged in.

@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 5)

I reviewed head bdfcc294 against #727, the round-4 verdict, the owner's rule and the orchestrator's round-5 tightening. Every claim is labelled measured (a command I ran, or CI output I cite), inferred (reasoned from the code, not run) or guess. Fixtures are stand-ins passed as data. Each ran with a scratch HOME, scratch state dirs and a fake claude on PATH. No repo file was edited.

Where it stands:

  • Readiness is right. The owner's three cases, the tightening and every round-4 attack hold there, in every list order. The run-time brand holds against every forgery I tried.
  • Two things block:
    1. The main watch splits every run into its own group under an App pin. So the owner's cases 1 and 2, and a normal passing re-run, trip the main watch (stop, alert, revert PR) on a merge that readiness called ready.
    2. makerUses misses four routes. Its claim of covering re-exports and dynamic imports is too broad.

1. The owner's rule and the tightening (measured, rule-788.mjs)

Method:

  • Each case runs over every permutation of its runs, through PinnedChecks.of → readiness, and through judge().
  • judge() is driven the way the watcher drives it. Facts start without sources, and they are supplied only when judge returns need: sources (src/main-watch.ts:335, :839).

Result: 40 cases with an expected result, plus one informational. Readiness matches in all 40, with one verdict per case across all orders. The main watch differs from it in 8.

Case Readiness Main watch
Owner 1 / 2: cancelled or failed, then passed (workflow pin) ready accounted
Owner 3: pinned pass + another App's same-name failure (workflow, App and Actions-App pins) ci: fail ci fail
Actions-App pin: ci.yml failed or still running, then a PR-added evil.yml passed later fail / pending fail / pending
Actions-App pin: ci.yml failed in the newest run, evil.yml passed; ci.yml passed, evil.yml failed fail; fail fail; fail
Actions-App pin: both files passed ready accounted
Re-run of the older run fails or is still running after the newer run passed (workflow pin; Actions-App pin) fail / pending fail / pending
Re-run of the older run passes; a normal re-run of the latest run passes (older run cancelled) (workflow pin) ready accounted
Round 4's set: a later own run failed or is running; another head's run; a re-run attempt; statuses failure/error/pending/success; a foreign App's newer pass (with a forged job page); an API-made run in the suite; another file under a workflow pin as round 4 as round 4
Owner 1 / 2 under an App pin (app_id: 1111) ready ci fail, unaccounted
Owner 1 / 2 under an Actions-App pin (GitHub's rules name 15368) ready ci fail, unaccounted
A re-run attempt of the latest run passes over its failure (App pin; Actions-App pin) ready ci fail, unaccounted
Actions-App pin: a normal re-run of the latest run passes, older run cancelled ready ci fail, unaccounted
App pin: the App fails ci, then passes another check (lint) later ci: fail ci fail
  • Unpinned equals by-name: 326 of 326 subsets and orders of a 5-check pool (measured).
  • Does the newest-check-run rule block a legitimately green PR? Not in readiness. A normal re-run of the latest run that passes is ready (table).
    • Real data: I took 120 commits of this repo where a workflow ran twice or more (read-only gh api, realids.mjs). They give 439 (file, check name) pairs over two or more runs. In 0 of them is the newest check run outside the latest workflow run (measured). So on this repo the rule has never blocked a green repeated run.

Why the main watch differs (inferred, code; confirmed by the measurement above):

  • The watch asks for sources only when some required name has a workflow pin (main-watch.ts:335).
  • Under an App pin, f.sources stays undefined. Every check run has a suite, so workflowOf (:337) returns null, and pinnedAt makes each run its own unread <id> group (:229-233).
    • The main watch reads filter=all (main-watch-api.ts:53), so a failed first attempt is there too.
    • Each own run must then pass, and no later run supersedes anything.
  • Consequences:
    • An unaccounted merge "trips (stop, alert)" and opens a revert PR (main-watch.ts:848-855). The harness merges such a PR on readiness, and then its own main watch trips on it.
    • Existing installs: GitHub's rules naming the Actions App (or any App) is enough to hit this, with no config change. Main judged these merges by name and accounted them, so the upgrade adds false trips (loss of function).
    • Guess: GitHub's protection UI often records "GitHub Actions" as a check's source.
    • Inferred: this repo isn't hit today. Its ruleset names no integration_id, and branches/main protection lists no checks (gh api).
  • Also inferred: even when sources is read because another name has a workflow pin, a run from an App that isn't Actions has a suite missing from sources. So it still goes to unread (:337), not to the one app group that ownVerdict uses.
  • The new test main watch (#727): an Actions-App pin never lets one workflow file supersede another sets s.gh.sources, but the watcher never asks for them under that pin (inferred). Its ci fail cases pass because each run is its own group, not because of per-file grouping.

2. The brand

Forgery attempts (measured, brand-788.mjs, against a real judged array whose ci failed):

Attempt Result
structuredClone (changed or not), a Proxy over it, Object.create(it), Reflect.construct with its constructor, a subclass instance, my own array frozen (elements frozen too) each throws checks to judge must come from PinnedChecks.of
Object.setPrototypeOf, Object.defineProperty(…, 'find'), real[0] = …, length = 0 the forging step itself throws TypeError (frozen)
A second module instance (check-source.js?second) its arrays throw at the first instance's readiness (own WeakSet)
withTestRuns(real, {ci}, a pass) (e6), a fresh PinnedChecks over a stand-in backlog (e7) READY: through a maker, as the body says
headChecks (ci-none.ts) with project: { ci: 'none' }, a tests mapping a tier to ci, and made-up testrun.finished events READY: a third maker, callable from any src/ file
e7 with the backlog returning a Proxy whose map returns another Proxy READY, and judged() recorded the attacker's Proxy itself. That is still e7's route (not required, see below)
WeakSet.prototype.has patched to return true READY. In-process code can always patch builtins (not required)

My 8 fixtures:

  • At run time (measured, require-788.mjs):
    • e1, e2, e3 and e4 throw;
    • e5 and e8 throw TypeError: Cannot add property 1, object is not extensible;
    • e6 and e7 are READY, through the makers.
  • Through the lints (measured, scan.mjs): all 8 compile and are flagged by makerUses.
    • e6 and e7 are flagged for the maker call.
    • e1 to e5 and e8 are flagged only because the fixture imports both makers. The brand is what stops them.
  • Real callers (measured): every compiled test except mcp-* and plugin-fetch gives # tests 3548 # pass 3491 # fail 0 # skipped 57.
    • No src/ code mutates a judged array or check (inferred, grep for assignments).
    • mayMerge and fix-forward go through requiredOutcomes (inferred, code).
    • So the brand changes no real caller's behaviour.

Is makerUses complete? (measured, scan.mjs; virtual src/ files passed as data)

Route Hits
m1 renamed import; m2 named re-export in a barrel; m7 literal import('./check-source.js'); m8 destructured from a dynamic import of a barrel 3, 3, 2, 1 (caught)
m3 export * from './check-source.js' barrel + namespace import + computed key 0
m4 export * as cs from './check-source.js' barrel + computed key 0
m5 dynamic import of an export * barrel + computed key 0
m6 createRequire(import.meta.url)('./check-' + 'source.js') + computed key 0. At run time require(esm) returns the same instance (withTestRuns === import's: true), and withTestRuns through it gives READY
m10 Object.values over an export * barrel's namespace 0
m9 headChecks over made-up events (above) 0

src/ has no export * and no createRequire today (grep; testrun/affected.ts only names them in a regex). So these are routes a future change could take, which is what the lint is for.

Fail-without (measured)

  • Method: a scratch worktree at e8107eb4, with test/main-watch.test.ts, test/required-check-source.test.ts, test/helpers.ts and test/raw-check-reads.ts from bdfcc294. Then npm run build and node --import ./dist/test/isolate.js --test on the first two.
  • Result: # tests 72 # pass 68 # fail 4, matching the builder:
    • main watch, Actions-App pin: + 'accounted' - 'ci fail';
    • readiness, Actions-App pin: + 'test pass' - 'test fail';
    • "latest" fails closed: + 'test pass' - 'test fail';
    • the brand: Missing expected exception: a cast.
  • Adding required-check-source-coordinator.test.js gives 76 / 72 / 4, the same four failures.
  • On the head: suite as above; npm run typecheck exit 0; npm run check:denylist: denylist: 1283 files clean.

Usual checks (measured)

Required changes

  1. Main watch under an App pin (principle 1 as the owner's rule defines it, and principle 3's spirit for existing installs; this is a loss of function, not a loosening).
    • Read the workflow runs when any required name is pinned to the Actions App (configured app_id: 15368, or GitHub's rules naming 15368), as readiness and Engine updater counts a required check only from its pinned workflow or App [PROTECTED: owner merges] #789 do.
    • In judge()'s workflowOf:
      • a run whose App isn't Actions has no workflow (the one app group, as ownVerdict does by slug);
      • only an Actions run whose suite isn't in sources is unread.
    • Tests that drive the watcher's real need loop (no sources handed in when the watcher wouldn't ask). Under an Actions-App pin and under an app_id pin:
      • owner cases 1 and 2 are accounted;
      • a passing re-run attempt of the latest run is accounted;
      • evil.yml passing after ci.yml failed or while it runs is not accounted.
      • Each must fail on bdfcc294.
    • Correct the body's "The main watch does the same at merge time (pinnedAt), with the same groups" (principle 5).
  2. Close makerUses' gaps, or narrow its claims (principles 4 and 5). The body says "Uses through an alias or re-export count, and so do a namespace import and a dynamic import of check-source.ts". m3 to m6 and m10 aren't caught.
    • Flag in src/:
      • any export * / export * as whose module resolves (through the program, not a regex) to src/check-source.ts;
      • any namespace or dynamic import of a module that re-exports it;
      • any require/createRequire(...)(...) call.
    • headChecks: add it to the makers (allowed only in src/coordinator/merge-flow.ts), or stop it taking a caller's tests and events. Name it in check-source.ts's header, which says "The two makers are PinnedChecks.of and withTestRuns".
    • Fixtures: m3 to m6, m9 and m10 as data, each flagged.

Not required (noted)

  • judged() maps the input with its own .map, so a stand-in backlog that returns a Proxy gets its own Proxy branded (measured above).
    • This is e7's route, which the lint covers.
    • Array.from(c, …) would always make a plain array.
  • Patching WeakSet.prototype.has or Object.isFrozen defeats any in-process brand. That's out of scope.

For the owner (a genuinely new trade-off; not a required change)

A pin to an App that isn't Actions is one group (the builder's "what's left").

  • Measured (readiness): the App posts ci failed (check run 11), then ci passed (12). That is ready, as the owner's rule says for an App source. A different check name from the same App never supersedes: ci failed, then lint passed, stays ci: fail.
  • The hole: a third-party CI App that runs pipeline config from the PR's own head (as CircleCI or Buildkite can; guess for any given App) lets a PR add a second pipeline posting the same name. Its later pass then supersedes the real pipeline's failure. That is the evil.yml case, for an App the engine can't split by file.
  • Fail-closed alternative: under a non-Actions App pin, every run of the name must pass (no supersede). That gives up the owner's cases 1 and 2 for that App.
  • doctor prints a paste-ready required_check_sources block; a new project's config starts pinned (after #788) [PROTECTED: owner merges] #803's block proposes such an app_id pin only when no workflow file posts the name and exactly one provable App does.

… App pins fail closed; one maker (#727)

- Readiness and the main watch judge a pinned name through one function,
  pinnedVerdict, over the same facts: GitHub's latest check run per check
  suite and name, the workflow run behind each Actions suite, and the
  latest status per context. The watch builds them as they stood at merge
  time (checksAt), and now reads the workflow runs whenever a pinned name
  has an Actions run, so under an Actions-App pin (or an App pin) the
  owner's cases and a normal re-run are no longer split into one group
  per run. pinnedAt is gone.
- An Actions run is told by its App id (15368) on both sides.
- A pin to an App that isn't Actions fails closed: every one of that App's
  check runs of the name on the head must pass (a re-run GitHub lists in
  place of a failure, in the same suite, is the one there is).
- The deciding non-pass is the same whatever the list order (a finished
  one before a running one, then the highest id).
- PinnedChecks is the one maker: under No CI it reads the event log's
  test runs itself. withTestRuns and ci-none's headChecks, which took a
  caller's checks or events, are gone; asJudged in the tests goes
  through PinnedChecks.of.
- makerUses finds uses through the compiler's module resolution: barrels,
  export *, export * as, namespace and dynamic imports of a module that
  exposes the maker, unresolvable dynamic imports, require and
  createRequire. Fixtures for each route.
- Readiness reads every page of the combined status, and the main watch
  every page of check runs and statuses; later pages are marked extra and
  only pinned names are judged with them (by name, exactly as before).
- One table test: 22 cases under a workflow pin, an Actions-App pin and a
  non-Actions App pin, through readiness's real requests and the watch's
  own request loop; both give the expected verdict.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
…788 brings in

#788 makes src/backlog/types.ts, a module the guard already reaches, import its types from src/check-source.ts,
so the guard's import tree is the 56 pinned modules plus that one (57). #788 lists it in src/hook-imports.ts, and the
merge keeps it there; this adds it to the pinned tree in test/review.test.ts. One addition, nothing removed.
src/bundle.ts stays out: the guard still doesn't reach it.

Test (fails without): the static pin test, "the agents' tool guard's import tree is exactly its pinned modules", and
"every module in the agents' tool guard's import tree is self-modification, L3", both named src/check-source.ts as
outside the pin on the merged tree before this change.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
No conflicts. GUARD_TREE keeps #839's list, which now has src/check-source.ts, and this PR's src/os/own-env.ts: 58 modules.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
…s pinned-source verdict

The owner's decision on #834 x #788, option (c). The dashboard makes no check read: on a stop the default branch caused, Clear and resume records a request (merge.resume_requested, with an acknowledgement if typed) and answers 202 with the rule and the coordinator's last measurement, or that there is none. The coordinator answers the request in watchMerges: it measures main's tip with headChecksOf(sha, requiredChecks()) and requiredOutcomes (JudgedChecks, so a pinned check from another source is missing), records each look (merge.main_measured), and resumes only if green or on a recorded acknowledgement. A request for an older stop, or a stop that changes while main is measured, resumes nothing; a pending request is measured again every RESUME_MEASURE_MS. Tests: green only from an unpinned same-name check does not resume, green from the pinned source does, a request not yet measured stays stopped, and the dashboard's backlog sees no check read.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
…in-required-checks.md

Main's CHANGELOG.md holds released sections only (#870). The two bullets this PR added are the fragment's, byte for byte, under the same headings; CHANGELOG.md is main's exactly.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
This PR's only note was its edit of #788's Upgrading bullet (the paste-ready block). With notes in fragments (#870), that edit is made in changelog.d/788-pin-required-checks.md, byte for byte as before; CHANGELOG.md is #788's exactly, and this PR adds no fragment of its own (it adds no note line of its own).

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
…loads 57 modules

Since #870, release notes live in changelog.d/, one file per PR, and CHANGELOG.md holds only released sections.
This PR's one note, under "Guardrail changes: what each one does", moves to changelog.d/839-guard-loads-only-its-modules.md
word for word, with one change the stack-delta review asked for: the guard loads 57 modules, not 56. Stacked on #788,
whose src/check-source.ts is in the guard's tree through src/backlog/types.ts, the pinned tree is 57 modules.

Test (fails without): test/changelog-fragments.test.ts and test/changelog.test.ts read every fragment; the note itself
changes no behaviour.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
jayden1711 added a commit that referenced this pull request Oct 20, 2026
The combined check of 2026-10-20 (main f9422c4, clean dist with
build:web) measured the queue in loop order: #881, #788, #803, #789,
#839, #844, #834 and #878 merged give a 1,931,687-byte tarball, under
1,950,000; adding #769 gives 1,951,238, over by 1,238 (the full queue is
1,960,374). Per the owner's practice the raise goes on the PR that
crosses, at the new size plus 5%, rounded up to 10,000:
1,951,238 * 1.05 = 2,048,800, so 2,050,000. The unpacked budget stays at
6,400,000.

Part of #726.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711
jayden1711 merged commit 61858dc into main Oct 21, 2026
26 checks passed
@jayden1711
jayden1711 deleted the issue-727-pin-checks branch October 21, 2026 00:59
jayden1711 added a commit that referenced this pull request Oct 21, 2026
doctor prints a paste-ready required_check_sources block; a new project's config starts pinned (after #788) [PROTECTED: owner merges]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Required checks: pin each one to the App or workflow that posts it, before governance mode (today a same-name check from anywhere counts)

1 participant