Skip to content

Delegate sticky BuildKit to setup-buildx #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
crohr wants to merge 10 commits into
base: feature/sticky
Choose a base branch
from
Open

Delegate sticky BuildKit to setup-buildx #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d246a92
Delegate sticky BuildKit to setup-buildx
crohr Jul 10, 2026
d49592e
feat: add structured sticky cache input
crohr Jul 13, 2026
29a3df3
dist: rebuild binaries
crohr Jul 13, 2026
a32326a
ci: run sticky buildkit test on dev fleet
crohr Jul 13, 2026
3f763d0
ci: separate deploy-time sticky integration
crohr Jul 13, 2026
d6099d8
feat: rename snap runner label to sticky
crohr Jul 13, 2026
515da5f
dist: rebuild binaries
crohr Jul 13, 2026
ea79945
feat: make sticky cache availability strict
crohr Jul 14, 2026
8782bcb
Fail sticky cache on unavailable disk
crohr Jul 21, 2026
fe267ad
dist: rebuild binaries
crohr Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions .github/workflows/test-sticky-buildkit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
name: Test / Sticky BuildKit

on:
push:
branches:
- main
- v*
- feature/*
- fix/*
- auto/*
workflow_dispatch:

jobs:
cold:
if: github.event_name == 'workflow_dispatch'
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- id: pinned
version: v0.34.1
- id: latest
version: latest
runs-on: runs-on=${{ github.run_id }}/cpu=2/family=m7/image=ubuntu24-full-x64/sticky=buildkit-${{ github.run_id }}:20gb
steps:
- uses: actions/checkout@v6
- id: runs-on
uses: ./
with:
sticky_cache: |
buildkit
custom,path=.runs-on-custom-cache/${{ matrix.id }}
- name: Populate custom cache
run: echo "${{ matrix.id }}" > .runs-on-custom-cache/${{ matrix.id }}/marker
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
with:
name: ${{ steps.runs-on.outputs.buildkit-builder }}
version: ${{ matrix.version }}
driver: docker-container
driver-opts: |
image=moby/buildkit:v0.31.1
buildkitd-config-inline: ${{ steps.runs-on.outputs.buildkit-inline-config }}
cleanup: false
- name: Populate BuildKit cache
shell: bash
run: |
mkdir -p /tmp/runs-on-buildkit-test
printf 'FROM busybox:1.36\nRUN dd if=/dev/zero of=/layer.bin bs=1M count=16\n' > /tmp/runs-on-buildkit-test/Dockerfile
docker buildx build --builder "${{ steps.runs-on.outputs.buildkit-builder }}" --progress=plain --load --tag runs-on-buildkit-test:cold /tmp/runs-on-buildkit-test

wait-for-snapshot:
needs: cold
runs-on: ubuntu-slim
steps:
- name: Wait for the cold snapshot to become selectable
run: sleep 45

warm:
needs: wait-for-snapshot
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- id: pinned
version: v0.34.1
- id: latest
version: latest
runs-on: runs-on=${{ github.run_id }}/cpu=2/family=m7/image=ubuntu24-full-x64/sticky=buildkit-${{ github.run_id }}:20gb
steps:
- uses: actions/checkout@v6
- id: runs-on
uses: ./
with:
sticky_cache: |
buildkit
custom,path=.runs-on-custom-cache/${{ matrix.id }}
- name: Verify restored state
env:
CACHE_HIT: ${{ steps.runs-on.outputs.cache-hit }}
run: |
test "$CACHE_HIT" = true
test "$(cat .runs-on-custom-cache/${{ matrix.id }}/marker)" = "${{ matrix.id }}"
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
with:
name: ${{ steps.runs-on.outputs.buildkit-builder }}
version: ${{ matrix.version }}
driver: docker-container
driver-opts: |
image=moby/buildkit:v0.31.1
buildkitd-config-inline: ${{ steps.runs-on.outputs.buildkit-inline-config }}
cleanup: false
- name: Verify cached build
shell: bash
run: |
mkdir -p /tmp/runs-on-buildkit-test
printf 'FROM busybox:1.36\nRUN dd if=/dev/zero of=/layer.bin bs=1M count=16\n' > /tmp/runs-on-buildkit-test/Dockerfile
docker buildx build --builder "${{ steps.runs-on.outputs.buildkit-builder }}" --progress=plain --load --tag runs-on-buildkit-test:warm /tmp/runs-on-buildkit-test 2>&1 | tee /tmp/runs-on-buildkit.log
grep -q 'CACHED' /tmp/runs-on-buildkit.log

missing-sticky-disk:
runs-on: runs-on=${{ github.run_id }}/cpu=2/family=m7/image=ubuntu24-full-x64
steps:
- uses: actions/checkout@v6
- id: sticky_cache
continue-on-error: true
uses: ./
with:
sticky_cache: buildkit
- name: Verify missing sticky disk failed
env:
OUTCOME: ${{ steps.sticky_cache.outcome }}
run: test "$OUTCOME" = failure
90 changes: 76 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,9 +324,9 @@ echo "SCCACHE_S3_KEY_PREFIX=cache/sccache" >> $GITHUB_ENV
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
```

### `cache` / `path`
### `sticky_cache`

Available for Linux and Windows runners on jobs with a sticky-disk label. Use `snap=<size>` for the default snapshot lineage or `snap=<name>:<size>` for a named lineage; the optional name must come first. Volume settings follow the size, for example `snap=go-cache:20gb:gp3:750mbs:6000iops`. The `apt`, `buildkit`, and `git` cache modes are Linux only.
Available for Linux and Windows runners on jobs with a sticky-disk label. Use `sticky=<size>` for the default snapshot lineage or `sticky=<name>:<size>` for a named lineage; the optional name must come first. Volume settings follow the size, for example `sticky=go-cache:20gb:gp3:750mbs:6000iops`. The `apt`, `buildkit`, and `git` cache modes are Linux only.

Persists package manager caches across jobs by bind-mounting them onto the job's sticky disk — a dedicated EBS volume that is snapshotted at job completion and restored (per repo, name, architecture, and branch) on the next job. No tarball upload/download: caches are available at native disk speed, with no size penalty on job duration.

Expand All @@ -335,16 +335,35 @@ Example:
```yaml
jobs:
build:
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/snap=20gb
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/sticky=20gb
steps:
- uses: actions/checkout@v4
- uses: runs-on/action@v2
with:
cache: |
sticky_cache: |
go
node
```

Each non-empty line is one cache record. A record starts with a mode and may
include comma-separated `key=value` options. Use one line per mode; the old
comma-separated mode list is not supported.

```yaml
with:
sticky_cache: |
go
node
buildkit
custom,path=vendor/custom-cache,path=~/.cache/my-tool
```

The action fails if the sticky disk is absent or does not become ready before
`sticky_wait_timeout`. The `custom` mode requires one or more `path=` options;
repeat the record or option to persist several paths. Relative paths resolve
from `GITHUB_WORKSPACE`, `~/` resolves from the runner home, and absolute paths
are preserved. Literal commas in paths are unsupported.

Supported cache modes and the directories they persist:

| Mode | Aliases | Cached paths |
Expand All @@ -359,13 +378,57 @@ Supported cache modes and the directories they persist:
| `uv` | | `~/.cache/uv` |
| `poetry` | | `~/.cache/pypoetry` |
| `apt` | | `/var/cache/apt/archives` |
| `buildkit` | `buildx` | BuildKit layer cache (dedicated `buildkitd` with state on the sticky disk, registered as the current buildx builder) |
| `buildkit` | `buildx` | BuildKit layer cache (the official setup-buildx builder stores its state on the sticky disk) |
| `git` | `checkout` | Git repository mirrors (local git proxy serving github.com fetches from the sticky disk) |
| `gradle` | | `~/.gradle/caches`, `~/.gradle/wrapper` |
| `maven` | | `~/.m2/repository` |
| `playwright` | | `~/.cache/ms-playwright` |
| `custom` | | One or more paths supplied with `path=` |

#### `buildkit` mode (Docker layer cache)

The `buildkit` mode prepares the state volume used by Docker's official `setup-buildx-action`, backed by the sticky disk. RunsOn does not download or start its own BuildKit daemon. The actions must run in this order, with the fixed builder topology and cleanup settings shown below:

```yaml
jobs:
build:
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/sticky=docker:20gb
steps:
- uses: actions/checkout@v4
- id: runs-on
uses: runs-on/action@v2
with:
sticky_cache: buildkit
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
with:
name: ${{ steps.runs-on.outputs.buildkit-builder }}
version: v0.34.1
driver: docker-container
driver-opts: |
image=moby/buildkit:v0.31.1
buildkitd-config-inline: ${{ steps.runs-on.outputs.buildkit-inline-config }}
cleanup: false
- uses: docker/build-push-action@v7
with:
builder: ${{ steps.runs-on.outputs.buildkit-builder }}
context: .
load: true
```

Sticky BuildKit caching supports one `docker-container` node named by the `buildkit-builder` output. The RunsOn post step verifies that setup-buildx mounted the expected sticky volume, then stops and removes the builder before the disk is snapshotted. A missing setup step, reversed action order, different builder name, appended node, or setup-buildx cleanup causes a clear failure instead of silently using ephemeral cache storage.

The action always emits `buildkit-builder` and `buildkit-inline-config`, even without a sticky disk. When the RunsOn `ecr-pull-through` extra transparently mirrors Docker Hub, the inline config contains the matching BuildKit registry mirror; otherwise it is empty. This also supports a regular non-sticky builder:

```yaml
- id: runs-on
uses: runs-on/action@v2
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
with:
name: ${{ steps.runs-on.outputs.buildkit-builder }}
buildkitd-config-inline: ${{ steps.runs-on.outputs.buildkit-inline-config }}
```

The `buildkit` mode follows the dedicated-builder pattern: it starts a `buildkitd` daemon whose state (and the buildkit binaries themselves) live on the sticky disk, and registers it as the current buildx builder. The docker daemon is never touched. Use `docker buildx build` (or `docker/build-push-action`); add `--load` when you need the built image available to the local docker daemon (e.g. for `docker run`). `docker pull` and plain `docker build` on the default builder are not cached by this mode.
Use `docker buildx build` (or `docker/build-push-action`) with the emitted builder; add `--load` when you need the built image in the local Docker daemon. `docker pull` and plain `docker build` do not use this cache.

#### `git` mode (fast checkouts)

Expand All @@ -376,32 +439,31 @@ Unlike other modes, the `git` mode must run **before** `actions/checkout`:
```yaml
jobs:
build:
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/snap=20gb
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/sticky=20gb
steps:
- uses: runs-on/action@v2
with:
cache: git
sticky_cache: git
- uses: actions/checkout@v4
```

To combine it with workspace-relative `path:` caching (which must run after checkout), run the action twice — the second invocation reuses the already-running proxy.
To combine it with workspace-relative `custom,path=...` caching (which must run after checkout), run the action twice — the second invocation reuses the already-running proxy.

Notes and limitations:

* Mirror syncs authenticate with the `token` input (default: `${{ github.token }}`). Checking out **other private repositories** requires passing a PAT with access to them, since the rewritten URLs no longer match the credentials configured by `actions/checkout`.
* `git push` is pinned to upstream (`pushInsteadOf`) and never goes through the proxy; Git LFS and anything else the proxy cannot serve is transparently forwarded to github.com. If mirroring fails for any reason, fetches fall back to upstream — the mode never breaks a build.
* SSH remotes (`git@github.com:`) are not rewritten, and container jobs (`container:`) are not accelerated (the proxy listens on the host's loopback). GitHub Enterprise Server is not supported. Linux only.

Use the `path` input to persist arbitrary additional paths (newline separated). Relative paths are resolved against the workspace, so run this action **after** `actions/checkout` when caching workspace-relative paths (e.g. `vendor/bundle`).
Use `custom,path=...` records to persist arbitrary additional paths. Relative paths are resolved against the workspace, so run this action **after** `actions/checkout` when caching workspace-relative paths (e.g. `custom,path=vendor/bundle`).

**Disk pressure:** the buildkit cache is bounded by BuildKit's own garbage collection (capped at ~75% of the volume, keeping 20% free; least-recently-used layers are pruned automatically during builds). Other cache modes have no native GC: when the volume drops below 20% free space (or 10% free inodes), the post step emits a warning and a job summary with a per-cache breakdown — increase the `snap=` label size to fix. If a volume is ever critically full at job start (<5% free space or inodes), all caches on it are automatically reset so the job runs cold instead of failing with "no space left on device", and the next snapshot starts clean.
**Disk pressure:** the buildkit cache uses BuildKit's default garbage collection. Other cache modes have no native GC: when the volume drops below 20% free space (or 10% free inodes), the post step emits a warning and a job summary with a per-cache breakdown — increase the `sticky=` label size to fix. If a volume is ever critically full at job start (<5% free space or inodes), all caches on it are automatically reset so the job runs cold instead of failing with "no space left on device", and the next snapshot starts clean.

Other related inputs:

* `wait_timeout` - how long to wait for the sticky disk to be ready (default `5m`)
* `fail_on_missing` - fail the step when no sticky disk is available instead of continuing without cache (default `false`)
* `sticky_wait_timeout` - how long to wait for the sticky disk to be ready, as a Go duration (default `5m`)

The action sets a `cache-hit` output: `true` when every requested path was restored from a previous snapshot.
The action sets a `cache-hit` output: `true` when every requested path was restored from a previous snapshot. It also sets `buildkit-builder` to the stable builder name and `buildkit-inline-config` to the ECR mirror TOML described above.

## Development

Expand Down
18 changes: 7 additions & 11 deletions action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,23 @@ inputs:
description: 'Enable sccache. Can take either "s3" (RunsOn S3 cache bucket) or be empty (disabled). You still need to setup sccache in your workflow, for instance with mozilla-actions/sccache-action.'
required: false
default: ''
cache:
description: 'Newline/comma separated list of cache modes to persist on the sticky disk (go, node, yarn, pnpm, ruby, rust, python, uv, poetry, apt, buildkit, git, gradle, maven, playwright). Requires a snap=<size> or snap=<name>:<size> label on the job; the optional name must come first, before the size and any volume settings. Linux and Windows (apt, buildkit and git modes are Linux only). Run this action after actions/checkout when caching workspace-relative paths. The git mode is the exception: it must run BEFORE actions/checkout, since it accelerates github.com fetches with git mirrors persisted on the sticky disk (combine both by running the action twice).'
sticky_cache:
description: 'Newline-separated sticky-disk cache records. Each line is a mode followed by comma-separated key=value options, for example buildkit or custom,path=vendor/cache. Supported modes: go, node, yarn, pnpm, ruby, rust, python, uv, poetry, apt, buildkit, git, gradle, maven, playwright, custom. Requires a sticky=<size> or sticky=<name>:<size> label; the action fails if the disk is absent or not ready before sticky_wait_timeout. Linux and Windows (apt, buildkit and git modes are Linux only). Run after actions/checkout for workspace-relative custom paths; git must run before checkout; buildkit must run before docker/setup-buildx-action.'
required: false
default: ''
token:
description: 'GitHub token used by the git cache mode to sync repository mirrors from github.com. Supply a PAT if the job checks out private repositories other than the workflow repository.'
required: false
default: ${{ github.token }}
path:
description: 'Newline separated list of arbitrary paths to persist on the sticky disk. Requires a snap=<size> or snap=<name>:<size> label on the job; the optional name must come first, before the size and any volume settings. Linux and Windows.'
required: false
default: ''
wait_timeout:
sticky_wait_timeout:
description: 'How long to wait for the sticky disk to be ready (Go duration, e.g. "5m")'
required: false
default: '5m'
fail_on_missing:
description: 'Fail the step if no sticky disk is available (no snap= label, or not ready before wait_timeout)'
required: false
default: 'false'

outputs:
cache-hit:
description: 'Set to "true" when every requested cache path was restored from a previous snapshot, "false" otherwise'
buildkit-builder:
description: 'Stable Buildx builder name to pass to docker/setup-buildx-action and docker/build-push-action'
buildkit-inline-config:
description: 'BuildKit TOML for the RunsOn ECR Docker Hub pull-through mirror, or an empty string when unavailable'
Loading
Loading