Skip to content

CI: gate and isolate image publishing, pin actions, harden tooling - #35

Draft
ahma wants to merge 9 commits into
feat/terraform-clients-token-ttlfrom
ci/release-and-supply-hardening
Draft

ahma wants to merge 9 commits into
feat/terraform-clients-token-ttlfrom
ci/release-and-supply-hardening

Conversation

@ahma

@ahma ahma commented Oct 28, 2026 •

Copy link
Copy Markdown

Summary

Publishing and signing a container image now happens in a dedicated job that runs only after the test, e2e,
lint and license checks, under a release environment, with the write scopes confined to that job. Every
action is pinned to a commit, the verify step takes its inputs through the environment, the debug stage
pins Delve and binds it to loopback, the build context is an allowlist, and two tooling recipes stop
doing the wrong thing silently.

Why

The security review of the repository at d515e17 (main plus #29 to #32) recorded that any push of a
tag matching the release glob, or of a release-* branch, published and keyless-signed an official image
with no quality gate and with write scopes granted to every image build, including pull requests. The
repository settings add no protection today (no rulesets, no tag protection). The review also noted
floating action tags, an unpinned Delve on an unauthenticated port, no .dockerignore, a minikube-deploy
recipe that continued on the default Docker daemon when minikube docker-env failed, and an e2e suite that
drove whatever process held port 8080.

What changed

  • ci(image): the image workflow has a build job (read-only token; builds both platforms, scans, uploads
    the report) and a publish job (if: inputs.publish, needs: build, environment: release; rebuilds
    from the layer cache, pushes, signs and verifies). QEMU is dropped because the Dockerfile cross-compiles
    and the final stage has no RUN.
  • ci: a read-only token by default; the image build runs after the four checks on every push.
  • ci(release): a tag push now runs the same four checks before it publishes.
  • build(docker): Delve pinned to v1.27.2 and bound to 127.0.0.1; .dockerignore allows only the six
    source paths.
  • chore(git): ignore signing-key.pem (the README example path) and the user-database temp files.
  • build(just): minikube-deploy stops when minikube docker-env fails and pins --context minikube.
  • test(e2e): the suite refuses a listener on the port that is not the server it started.
  • docs(readme): a "Releases" section with what publishes, what gates it, and the settings the owner sets.
  • ci: the minikube profile is pinned for docker-env as it is for kubectl; the e2e issuer check normalises
    the trailing slash and states what it can tell apart.

Test plan

  • just verify, just test-race and the e2e suite (in a Linux container) green on the branch head.
  • Workflow files parsed and the job graph checked: every needs: target exists, every uses: is pinned.
  • Negative check of the e2e guard: with another listener on the port, every case fails in setup with a clear
    message; without it the class passes.
  • First CI run of this PR (run 40779970641): Test, E2E, Lint, License check and the build job of the
    split workflow passed; the build job compiled linux/amd64 and linux/arm64 by cross-compilation
    ([linux/amd64->arm64 builder] stages, provenance for both platforms) with no QEMU step; the publish
    job was skipped as designed for a pull request. The environment gate and the push, sign and verify steps
    run only on a publishing push and are still to be observed there.

Review record and residuals

Reviewed twice by an independent reviewer (plan and diff); every finding accepted. Residuals stated on
purpose: the quality gates are a property of the two current callers, not of the reusable workflow (a new
caller could skip them); the Trivy report describes the prepublication build, not the rebuilt image that
is signed; the e2e listener check does not distinguish a foreign IdP serving the same issuer (accepted on a
private runner); loopback-only binding of the e2e server waits for the server's LISTEN_ADDR support in a
later PR; the buildx BuildKit image is not digest-pinned; the digest output falls back to the build digest
and does not certify publication.

Compatibility

  • Publishing from main now waits for the release environment's rules once they exist.
  • The image that is signed is rebuilt from cache in the publish job; its digest may differ from the
    tarball the scan step examined. The scan remains a report, not a gate.

Settings this PR cannot provide

Until the owner configures them, anyone with push access can still publish a signed image:

  1. A ruleset on tags [0-9]+.[0-9]+.[0-9]+* and branches release-* that restricts creation, update and
    deletion to the release team.
  2. Protection rules on the release environment (required reviewers, deployment branches and tags).
  3. Optionally, required status checks on main.

Not in this PR

A USER directive in the Dockerfile (needs a chart change for the user-database volume ownership);
Trivy as a blocking gate.

Stacked on #32 (feat/terraform-clients-token-ttl). Draft until the stack is merged.

ahma added 9 commits October 28, 2026 23:05
One job built, pushed, signed and scanned, and it held packages: write
and id-token: write on every caller, a pull request build included,
where publish is false and no step needs them (audit run-2,
release.publish-and-sign-gated-only-by-ref-push and the hunter-supply
notes). The OIDC request token and the ghcr login were then in reach of
every step, Trivy's database download among them.

build runs everywhere with contents: read only. It builds both
platforms into the OCI tarball (no push), fills the gha layer cache,
runs Trivy, uploads the SARIF and summarises it, and hands on the image
name, digest, tag, tags, labels and description. publish runs only
with inputs.publish, after build, in the release environment, and is
the only job with write scopes: it rebuilds from the gha cache, pushes,
and signs and verifies as before. The environment is where the owner's
protection rules attach; the rules themselves are GitHub settings, so
this is a partial fix (README "Releases" follows). The rebuilt image
uses the cached layers, but its manifest digest need not match the
scanned tarball's, and Trivy stays a report, not a gate.

The verify step took github.ref and github.repository by template
interpolation into the script, so a crafted ref ran in bash; both now
arrive through env and are quoted.

QEMU is dropped: the builder stage runs on $BUILDPLATFORM and
cross-compiles, and the final stage has only COPY, so no step executes
an arm64 binary. Not run on a GitHub runner yet; if the arm64 leg fails
there, the step comes back.

Every action is pinned to a commit SHA (gh api
repos/<owner>/<repo>/git/ref/tags/<tag>, annotated tags dereferenced).
Each exact tag resolves to the commit its floating major tag pointed
at on 28 October 2026, so behaviour is unchanged:

  actions/checkout             v5.1.0   fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09
  actions/setup-go             v6.5.0   924ae3a1cded613372ab5595356fb5720e22ba16
  extractions/setup-just       v3.1.0   f8a3cce218d9f83db3a2ecd90e41ac3de6cdfd9b
  actions/cache                v4.3.0   0057852bfaa89a56745cba8c7296529d2fc39830
  docker/setup-buildx-action   v3.12.0  8d2750c68a42422c14e847fe6c8ac0403b4cbd6f
  docker/metadata-action       v5.10.0  c299e40c65443455700f0fdfc63efafe5b349051
  docker/login-action          v3.7.0   c94ce9fb468520275223c153574b00df6fe4bcc9
  docker/build-push-action     v6.19.2  10e90e3645eae34f1e60eeb005ba3a3d33f178e8
  sigstore/cosign-installer    v3.9.1   398d4b0eeef1380460a10c8013a76f728fb906ac
  actions/upload-artifact      v4.6.2   ea165f8d65b6e75b540449e92b4886f43607fa02
  aquasecurity/trivy-action    v0.36.0  ed142fd0673e97e23eac54620cfb913e5ce36c25 (already pinned)

cosign-installer's v3 tag sits on v3.9.1, not on the newest v3.10.1,
so v3.9.1 is pinned to keep the cosign it installs.
ci.yaml had no permissions block, so test, e2e, lint and license-check
ran with the repository's default token scope on push, and every step
after actions/checkout could use it, the third-party setup-just among
them (audit run-2 hunter-supply notes). The workflow now defaults to
contents: read; licensei and setup-just only read from GitHub.

image-build now needs test, e2e, lint and license-check, so a push to
main or a release branch publishes only an image whose source passed
them. publish stays true on push only, and a pull request never
reaches the release environment.

The caller job keeps contents: read, packages: write and id-token:
write, with a comment saying why: a called workflow's jobs cannot exceed
the caller's grant, so the publish job needs them here. The build job
in image-build.yaml narrows itself to contents: read.
A tag push ran only the image build, so a tag on a commit that never
passed CI (one pushed only as a tag, or a branch head whose checks
failed) was published and signed without a single test. release.yaml
now runs test, e2e, lint and license-check with ci.yaml's steps and
pins, and image-build needs all four.

A tag push is a push event, for which ci.yaml runs the race tests, so
the test job runs just test-race without the pull_request branch.

This is a quality gate, not an authority check: whoever can push a tag
still chooses the workflow code it runs. Tag and environment rules are
owner settings (README "Releases").
The debug stage installed dlv@latest and served Delve's headless,
unauthenticated, multi-client API on every interface (audit run-2
hunter-supply and hardening notes). Anyone who built --target debug
and exposed the port handed out code execution in the process. dlv is
pinned to v1.27.2 (go.mod says go 1.25.0 and it supports Go 1.25
to 1.27, so the image's Go 1.26.6 is in range; not built here) and
listens on 127.0.0.1:40000, which kubectl port-forward reaches. A
comment says the stage is never to be deployed.

There was no .dockerignore, so the build context carried .git (with
the token actions/checkout persists in CI) and any local config.json,
users.json and signing keys to the builder; only the COPY allowlist
kept them out of the image. The new .dockerignore admits exactly what
the Dockerfile copies: go.mod, go.sum, main.go, internal, pkg and
templates. Checked by exporting a FROM scratch / COPY . / build of
the context with dummy config.json, users.json and signing-key.pem
present: only those six entries arrived.
The README Quickstart stores the generated signing key at
signing-key.pem, which .gitignore did not cover, so following the
README left a private key one git add away from a commit (audit run-2
hardening notes). SaveUsers writes users.json~ today, and lane F1's
temp-file save writes .users-*.json next to the database; a crash
leaves either behind with every password hash in it. All three are
ignored now.
eval "$(minikube docker-env)" exits 0 even when the substitution
fails, so set -e did not stop the recipe: it built on the default
Docker daemon and then patched axoflow-local/axoidp in whatever
kubectl context was current (audit run-2 hunter-supply notes, observed
there). The recipe now captures the output, checks the exit status
and stops with a message, and both kubectl calls pass --context
minikube.

Checked with stub minikube, docker and kubectl on PATH: a failing
docker-env stops the recipe with exit 1 before docker runs, and a
working one reaches kubectl with --context minikube.
setUp treated any listener on 127.0.0.1:8080 as the server it had just
started. With another process on the port, the new server failed to
bind and the suite sent its logins and password changes to that
process and reported assertion failures (audit run-2 hunter-supply
notes, observed there). On this Mac Tailscale serve holds :8080.

After the port opens, setUp now requires the started process to be
alive and /.well-known/openid-configuration, under the config's path
prefix, to return the issuer of the config it wrote; otherwise it
stops the server and fails with the exit code, discovery status and
issuer it saw. Nothing else in the suite changes.

Checked in the e2e container with python3 -m http.server on :8080:
ChangePasswordTest fails in setUp with "the listener on :8080 is not
the server under test (server exit code None, discovery status 404,
issuer None, want 'http://localhost:8080')". Without it, all 25 tests
pass.
Nothing said which pushes publish and sign an image, what gates it, or
that the gate on who may publish lives in repository settings rather
than in the workflows (audit run-2,
release.publish-and-sign-gated-only-by-ref-push; Astra round-1
finding 13). The new Releases section covers the build and publish
jobs, the triggers and resulting tags, the four check jobs the build
waits for, the release environment, how to verify a signature, and the
settings the owner must configure: tag and release-* rulesets limited
to the release team and protection rules on the release environment.
It says plainly that the checks are a quality gate, that Trivy only
reports, and that until those settings exist anyone with push access
can publish.
minikube docker-env without a profile can point at another profile's daemon
while kubectl is pinned to the minikube context, so the recipe would build in
one cluster and patch another. Pass -p minikube to both.

The e2e suite's new listener check compares the configured baseUrl with the
served issuer; the server trims trailing slashes before it becomes the issuer,
so the expected value is normalised the same way. The comment states what the
check can and cannot tell apart: a foreign IdP serving the same issuer would
still pass, which is accepted on a private runner.

The README says that the Trivy report describes the prepublication build and
not the rebuilt image that is signed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cited PR run skipped the privileged publish job, so release-environment approval enforcement and the registry push/sign/verify flow still require maintainer validation on an authorized publishing push.

No open issues 🎉

What changed in this PR

This PR hardens container publishing for the OIDC identity provider by separating read-only builds from privileged publishing and requiring quality checks before either current workflow publishes.

Changes:

  • Gate image builds on tests, e2e tests, lint and license checks; isolate publishing and signing in a release environment job.
  • Pin actions and Delve, restrict the Docker build context, and ignore additional local key and database files.
  • Add deployment and e2e safeguards, and document release behavior and required repository protections.
File Description
scripts/​e2e.py Check server process state and discovery issuer before running tests.
README.md Document publishing gates, signature verification and owner-configured protections.
justfile Stop on minikube environment failures and pin deployment context.
Dockerfile Pin Delve and bind its debug API to loopback.
.gitignore Ignore the PEM signing key and temporary user-database files.
.github/​workflows/​release.yaml Add four quality gates before release image publishing and pin actions.
.github/​workflows/​image-build.yaml Separate build and publish jobs, isolate write permissions and pin actions.
.github/​workflows/​ci.yaml Default to read-only permissions, gate image builds and pin actions.
.dockerignore Allow only required source paths into the build context.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants