Skip to content

Route trusted Linux builds to managed runners - #508

Merged
wesm merged 4 commits into
mainfrom
feat/trusted-linux-ci
Jul 26, 2026
Merged

Route trusted Linux builds to managed runners#508
wesm merged 4 commits into
mainfrom
feat/trusted-linux-ci

Conversation

@wesm

@wesm wesm commented Jul 25, 2026

Copy link
Copy Markdown
Member

Summary

  • route canonical same-repository Linux pull requests and main-branch builds to managed Linux runners, with forks and noncanonical copies falling back to GitHub-hosted runners
  • invoke reusable workflows from main and let checkout use the caller event's immutable SHA without accepting a dispatcher-supplied ref
  • lint proposed workflow revisions on GitHub-hosted infrastructure

The managed runner group admits only the listed reusable workflow files from main; PR-controlled dispatcher changes therefore cannot target it directly.

Merge #511 first so main accepts reusable-workflow calls before this PR replaces the existing pull-request trigger.

Same-repository branches are controlled by repository collaborators and should use organization capacity, while fork pull requests remain arbitrary code. Main-branch reusable workflows make that trust decision enforceable even when a fork edits its own workflow files.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Jul 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (5b99e4c)

Verdict: One high-severity issue—the managed-runner trust gate can be bypassed by fork PRs.

High

Fork PRs can bypass the managed-runner trust gate

Files: .github/workflows/ci-pr.yml:6, .github/workflows/docker-pr.yml:6

A fork contributor controls the workflow definition used by a pull_request run because GitHub loads it from the PR merge commit. An attacker can replace the guarded reusable-workflow call with a job specifying runs-on: kenn-linux-x64-public. If the workflow runs automatically or receives maintainer approval, arbitrary commands execute on the paid managed fleet, exposing its compute capacity and any configured network access to untrusted code.

Restrict the runner group to jobs defined only in:

  • kenn-io/msgvault/.github/workflows/ci.yml@refs/heads/main
  • kenn-io/msgvault/.github/workflows/docker.yml@refs/heads/main

Alternatively, move runner selection into a base-branch-controlled dispatcher that fork PRs cannot modify.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 8m56s

pull_request_target associates validation with the base commit, so it cannot reliably gate updates to a same-repository PR. Use ordinary pull_request dispatchers and keep the runner trust decision inside the main-pinned reusable workflows, where both head and base repository identities must match before selecting the managed fleet.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Jul 26, 2026

Copy link
Copy Markdown

roborev: Combined Review (3feea52)

Verdict: Changes requested — one high-severity runner trust-boundary issue and one medium-severity checkout race remain.

High

  • Managed runner selection can be altered by fork PRs.github/workflows/ci-pr.yml:6, .github/workflows/docker-pr.yml:6
    pull_request workflows are evaluated from the PR merge revision, so a fork can modify these dispatchers and add a job targeting kenn-linux-x64-public. Pinning only the called workflow to main does not enforce the intended runner trust boundary. Restrict the managed runner group to the main-pinned reusable workflows, or move runner selection behind an immutable, base-controlled dispatcher.

Medium

  • Mutable merge ref can test a different revision than the associated check.github/workflows/ci-pr.yml:15, .github/workflows/docker-pr.yml:28
    refs/pull/<number>/merge can change while a run is queued or retried, causing it to test a newer merge revision while checks remain associated with the original event SHA. Pass ${{ github.sha }} as checkout_ref, or omit the input so the reusable workflows use their existing github.sha fallback.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 8m31s

Bind managed runner selection to the canonical repository and use the immutable pull request event SHA, so forks fall back to hosted runners and checks cannot drift to a newer merge ref.

Lint proposed workflow revisions on hosted infrastructure before merge.

Assisted-by: OpenAI
@roborev-ci

roborev-ci Bot commented Jul 26, 2026

Copy link
Copy Markdown

roborev: Combined Review (bc31bab)

Verdict: High-severity workflow trust-boundary vulnerability found.

High

  • Location: .github/workflows/ci-pr.yml:7, .github/workflows/docker-pr.yml:7
  • Problem: pull_request workflows load from the PR merge revision, allowing a fork to modify the dispatchers before execution. An attacker could bypass the main-pinned workflow, request kenn-linux-x64-public directly, or provide a known-good SHA as checkout_ref, defeating runner isolation and PR validation.
  • Fix: Enforce managed-runner access outside PR-controlled workflow files, such as through a runner-group/required-workflow policy or a base-controlled trigger. Remove the caller-controlled checkout input and bind reusable workflows directly to the immutable event SHA.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 6m26s

Let the reusable workflow derive its checkout target directly from the caller event instead of accepting a ref from the pull request-controlled dispatcher. This removes the remaining caller-controlled input at the runner trust boundary and keeps each check tied to the revision GitHub associated with the run.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Jul 26, 2026

Copy link
Copy Markdown

roborev: Combined Review (39ccd7e)

Verdict: High-severity managed-runner access bypass must be fixed before merge.

High

  • PR-controlled dispatchers can bypass managed-runner admission
    Locations: .github/workflows/ci-pr.yml:6-13, .github/workflows/docker-pr.yml:6-26

    Because pull_request workflows execute the PR merge revision, a fork author can replace either reusable-workflow call with a direct job targeting kenn-linux-x64-public. This bypasses the repository-origin checks in the main-pinned reusable workflows and permits arbitrary untrusted code to run on the managed fleet.

    Recommended fix: Administratively restrict the runner group to trusted reusable workflows pinned at immutable SHAs. Alternatively, use a base-branch-controlled dispatcher such as pull_request_target, reject fork heads before selecting the managed runner, and explicitly check out the intended immutable PR revision. Pinning only the reusable workflow does not protect a caller that the pull request can rewrite.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 7m37s

@wesm

wesm commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

false positive

@wesm
wesm merged commit aaba110 into main Jul 26, 2026
6 checks passed
@wesm
wesm deleted the feat/trusted-linux-ci branch July 26, 2026 20:34
@wesm

wesm commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

This finding is a false positive under the runner policy currently enforced by GitHub.

The runner group that provides kenn-linux-x64-public has selected-repository visibility and restricted_to_workflows: true. For this repository, the only admitted workflow identities are:

  • kenn-io/msgvault/.github/workflows/ci.yml@refs/heads/main
  • kenn-io/msgvault/.github/workflows/docker.yml@refs/heads/main

A fork can rewrite ci-pr.yml or docker-pr.yml and request the label, but GitHub will not assign a runner because that PR-controlled caller workflow identity is not in the group allowlist. The admitted main-branch reusable workflows also derive the head and base repository identities themselves before selecting the managed label; the dispatcher does not supply the trust decision.

This would become a real vulnerability if the group stopped being restricted to selected workflows, if a PR-ref dispatcher were added to the allowlist, or if the same label were exposed through another unrestricted runner group. None of those conditions is present.

wesm pushed a commit that referenced this pull request Jul 27, 2026
Three consecutive main builds (aaba110, a8b4f4b, 930a891) failed in
test-postgres and test-pgvector after #508 routed trusted Linux jobs to the
managed public fleet. Both die on Go's default 10m per-package timeout in
cmd/msgvault/cmd, and it is not a hung test: each failure has a different
test in flight when the alarm fires, so the package as a whole is simply
running too slowly to finish.

The cause is the runner, not the tests. These two lanes are the only ones
driving the full Go suite against a live PostgreSQL service container, and
that container is badly I/O-bound on the managed fleet -- a routine
checkpoint logged write=116.7s sync=55.7s over 11169 files, and individual
CREATE TABLE statements crossed the store's slow-query threshold at 1.3-1.9s
apiece. Same commits on GitHub-hosted runners: test-postgres 5m11s,
test-pgvector 1m48s.

Raising -timeout would have kept main green while leaving both lanes running
20+ minutes on every push and trusted PR, so pin them back instead and leave
the rest of #508's routing untouched. setup-go regains Actions caching here
because #513's cache: false exists only to reuse a self-hosted machine-local
GOMODCACHE, which no longer applies once the job is hosted.

This is a workaround, not a fix -- the fleet's service-container disk I/O
still needs attention before these lanes can move back.

Generated with Claude Code (claude-opus-5)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wesm pushed a commit that referenced this pull request Jul 27, 2026
`test-postgres` and `test-pgvector` have failed on every managed-runner main build since #508. Host telemetry confirms the lanes are storage-bound: PostgreSQL checkpoint syncs take minutes while CPU remains mostly idle.

Move both lanes back to GitHub-hosted runners, restore Actions caching, and restore the test commands from their last known-green hosted configuration. This keeps main reliable without hiding the managed fleet's service-container I/O problem behind longer timeouts.

The workflow passes actionlint and all repository hooks. The fleet storage bottleneck remains a separate infrastructure follow-up.

Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
jesserobbins pushed a commit to jesserobbins/msgvault that referenced this pull request Aug 6, 2026
## Summary

- route canonical same-repository Linux pull requests and main-branch builds to managed Linux runners, with forks and noncanonical copies falling back to GitHub-hosted runners
- invoke reusable workflows from `main` and let checkout use the caller event's immutable SHA without accepting a dispatcher-supplied ref
- lint proposed workflow revisions on GitHub-hosted infrastructure

The managed runner group admits only the listed reusable workflow files from `main`; PR-controlled dispatcher changes therefore cannot target it directly.

Merge kenn-io#511 first so `main` accepts reusable-workflow calls before this PR replaces the existing pull-request trigger.

Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
jesserobbins pushed a commit to jesserobbins/msgvault that referenced this pull request Aug 6, 2026
`test-postgres` and `test-pgvector` have failed on every managed-runner main build since kenn-io#508. Host telemetry confirms the lanes are storage-bound: PostgreSQL checkpoint syncs take minutes while CPU remains mostly idle.

Move both lanes back to GitHub-hosted runners, restore Actions caching, and restore the test commands from their last known-green hosted configuration. This keeps main reliable without hiding the managed fleet's service-container I/O problem behind longer timeouts.

The workflow passes actionlint and all repository hooks. The fleet storage bottleneck remains a separate infrastructure follow-up.

Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant