overflow.service in this directory is the systemd unit the production
deployment runs. It starts Next as the dedicated overflow system account, with
the filesystem read-only apart from one cache directory, no capabilities, and no
path under /root. tests/deploy/unit-file.test.ts fails if any of that is
removed from the unit — and also if a [Service] directive is added to it or
given a value other than the reviewed one, because the reviewed set there is
closed on both, and if the file uses a shape of systemd's grammar the guard does
not model rather than guessing at it.
This file is the procedure that makes a host match what the unit expects. The commands are run as root. Everything except the running service is root-owned: the service account can read the code it executes and cannot write it, so code execution inside the web process cannot rewrite what the next restart runs.
The optional container route — an alternative to section 10's host procedure, not used in production on this host — lives in container.md.
Backing the production database up and restoring it — the least-privilege backup role, the daily overflow-backup.timer, the exact restore steps and the drill procedure — lives in backup-restore.md.
Containing and scoping an incident, preserving evidence and recording the maintainer's decisions — incident-response.md.
Screening the contributor scoring for data-protection impact — the activities, the human-review routes and the dated conclusion — dpia-screening.md.
Values used throughout: deployment tree /srv/overflow, service account
overflow:overflow, Node 24.17.0 at /usr/local/lib/nodejs/node-v24.17.0,
secrets in /etc/overflow/overflow.env, listener 127.0.0.1:3000 behind nginx.
The unit assumes the database is local. It carries Requires=postgresql.service
alongside After=postgresql.service, as the unit it replaces did.
Requires= propagates stops, so taking Postgres down for maintenance takes
Overflow down with it, and Overflow does not come back on its own when Postgres
returns — restart it. On a host whose database is remote there is no
postgresql.service to require and the unit refuses to start at all: drop both
postgresql.service references from [Unit] and keep
After=network-online.target.
On a host that already runs Overflow, save the current unit before touching
anything. This copy is the rollback in section 9, and it is the only one: the
unit lives only in /etc/systemd/system, so systemctl revert — which exists to
drop overrides of a vendor-supplied unit under /usr/lib/systemd/system — has
nothing to revert to here.
cp -a /etc/systemd/system/overflow.service /root/overflow.service.pre-hardeningLeave the previous checkout and Node installation in place until the hardened service has been running long enough to trust. Rollback needs them.
A system account with no login shell and no password. Its home is the
deployment tree, which is why the tree lives under /srv: the unit sets
ProtectHome=yes, which makes /home, /root and /run/user unreachable to
the service, and a home directory under /home would be hidden from the process
that owns it.
groupadd --system overflow
useradd --system --gid overflow --home-dir /srv/overflow \
--shell /usr/sbin/nologin --no-create-home overflow/root is mode 0700, so an unprivileged account cannot reach a runtime
installed under it — including an nvm installation in root's home. Install the
pinned version system-wide and symlink the binary the unit names.
cd /tmp
curl -fsSLO https://nodejs.org/dist/v24.17.0/node-v24.17.0-linux-x64.tar.xz
curl -fsSLO https://nodejs.org/dist/v24.17.0/SHASUMS256.txt
grep node-v24.17.0-linux-x64.tar.xz SHASUMS256.txt | sha256sum --check
mkdir -p /usr/local/lib/nodejs
tar -xJf node-v24.17.0-linux-x64.tar.xz -C /usr/local/lib/nodejs
mv /usr/local/lib/nodejs/node-v24.17.0-linux-x64 /usr/local/lib/nodejs/node-v24.17.0
ln -sfn /usr/local/lib/nodejs/node-v24.17.0/bin/node /usr/local/bin/node
/usr/local/bin/node --versionThe last command must print v24.17.0.
The package manager is needed for installs, migrations and builds, all of which
run as root. The service never runs it, so the corepack shims go in
/usr/local/sbin, which the unit's PATH of /usr/local/bin:/usr/bin:/bin does
not reach and root's default PATH does.
ln -sfn /usr/local/lib/nodejs/node-v24.17.0/bin/corepack /usr/local/sbin/corepack
corepack enable --install-directory /usr/local/sbin
corepack prepare pnpm@10.33.0 --activate
pnpm --versionThe last command must print 10.33.0, the version package.json pins.
The environment file holds OVERFLOW_DATABASE_URL, AUTH_SECRET, the OAuth credentials,
OVERFLOW_TOKEN_ENCRYPTION_KEY (with OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS beside it only
while section 11's key rotation is under way), and the public callback URLs in
OVERFLOW_GITHUB_WEBHOOK_URL and OVERFLOW_GITLAB_WEBHOOK_URL. Registrations using repository
webhooks get their own secret, generated at registration time; App-based
registration uses the separate App secret described below.
The Environment reference in OPERATING.md
describes each variable. This section is only about where the file lives and who
may read it.
The reverse proxy must allow request bodies of at least 25 MiB on
/api/github/webhooks, /api/github/app-webhook and /api/gitlab/webhooks, matching the application's
delivery limit. The reverse-proxy requirements
below cover body caps and the address, origin and resource bounds that must
accompany them.
systemd reads EnvironmentFile= as PID 1, before it drops to User=overflow,
so the service account does not need to read overflow.env directly. Keep that
file 0600 root:root: the process receives its values in the environment,
but cannot reread the file. This restriction applies to overflow.env.
install -d -o root -g root -m 0700 /etc/overflow
[ -e /etc/overflow/overflow.env ] \
|| install -o root -g root -m 0600 /dev/null /etc/overflow/overflow.env
chown root:root /etc/overflow/overflow.env
chmod 0600 /etc/overflow/overflow.envThe [ -e ] guard is what makes this safe on a host that already runs
Overflow: install would otherwise truncate the secrets that are already
there. Populate the file before section 5 — its migration step reads
OVERFLOW_DATABASE_URL out of it.
A host that configured the read-only reconciliation App retires it: remove
GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_PATH and OVERFLOW_GITHUB_APP_SLUG
from /etc/overflow/overflow.env, delete /etc/overflow/github-reconcile-app/,
return /etc/overflow to the mode this section sets above, and delete the App
in GitHub's developer settings (Settings → Developer settings → GitHub Apps →
the app → Delete). The fold reconciles every repository with its sponsor's
OAuth token and reads no App credential.
App-based registration uses a GitHub App with three optional variables:
| Variable | Value |
|---|---|
GITHUB_REGISTRATION_APP_ID |
The registration App's id. |
GITHUB_REGISTRATION_APP_PRIVATE_KEY_PATH |
Path to its PEM RSA private key, readable by the overflow service account. |
OVERFLOW_GITHUB_REGISTRATION_APP_SLUG |
Optional public slug of the registration App; the landing and registration pages link its install page. Unset or invalid shows no link. |
OVERFLOW_GITHUB_REGISTRATION_APP_WEBHOOK_SECRET |
The webhook secret configured on that App, used to authenticate its deliveries. |
Leave these unset to keep the feature off: App-based registration is unavailable,
and App webhook deliveries receive HTTP 503 (fail-closed). Either the App id
or key path unset or empty disables App-based registration; an unset or empty
webhook secret makes the receiver answer 503. With the id and key path
configured, an unreadable, unparseable or non-RSA key throws at wiring time.
Configure the registration App's webhook URL as
https://<public-host>/api/github/app-webhook (receiver:
POST /api/github/app-webhook). Subscribe to exactly issues, issue_comment,
pull_request and pull_request_review, the events exported as
githubWebhookEvents in webhook-schema.ts.
Grant repository permissions metadata: read, issues: read and
pull_requests: read. Install the App on repositories that will register
through it; those registrations use App deliveries instead of creating a
repository webhook.
Before a recovery or credential-maintenance block that uses the database,
paste this block into the same shell. Repeat it after editing the environment
file. It exports the canonical database URL for direct clients and derived
restore URLs: a set canonical value wins, including an empty string; only an
unset canonical variable falls back to the legacy value. The file is read the
way the service's EnvironmentFile= reads it, by the deploy script's own
reader, so a value holding $ or quotes loads exactly as the service sees it.
The file itself is unchanged.
. /srv/overflow/scripts/systemd-env-file.sh; export_env_file /etc/overflow/overflow.env
export OVERFLOW_DATABASE_URL="${OVERFLOW_DATABASE_URL-${DATABASE_URL-}}"The unit binds Next to 127.0.0.1:3000. The documented deployment puts nginx
behind Cloudflare; these controls belong to the proxy, not to the readiness
probe. Preserve them when constructing or changing a vhost:
- Client-address trust. Set
real_ip_header CF-Connecting-IPand restrictset_real_ip_fromto Cloudflare's IPv4 and IPv6 edge ranges; keep that list current. Never trust this header from arbitrary peers. In every proxying location, overwriteX-Real-IPwith$remote_addrafter that real-ip step, rather than forwarding the caller's header. The privileged-action logger reads onlyX-Real-IP, requires one valid IP, and never uses the appendableX-Forwarded-Forchain. The host's range list is/etc/nginx/snippets/cloudflare-realip.conf. - Origin lock and HTTPS. Require authenticated origin pulls: the host
configures
ssl_client_certificatewith the Cloudflare origin-pull CA andssl_verify_client on. Enable the corresponding authenticated-origin-pull setting at Cloudflare too; do not remove certificate verification to make a direct origin request succeed. The HTTP vhost redirects to HTTPS, and the HTTPS vhost emitsStrict-Transport-Security: max-age=31536000; includeSubDomainswithalways. These are deployment controls; the application does not supply the origin lock or this HSTS policy. - Forwarded origin. Each proxying location sets
HostandX-Forwarded-Hostto$host,X-Forwarded-Prototo$scheme, andX-Forwarded-Portto$server_port. KeepOVERFLOW_APP_URLset to the public HTTPS origin; the application's cookie-authenticated mutation guard compares the request'sOriginto that configured origin. - Body caps. The required proxy body caps are 25 MiB
(
client_max_body_size 25m) on/api/github/webhooks,/api/github/app-webhookand/api/gitlab/webhooks, and 64 KiB (64k) on/api/moderationand/api/overrides, including their subpaths, matching the host configuration. The application's limits on moderation and overrides are 8 or 32 KiB. Other routes retain their own application streaming body limits: 4 KiB on/api/accountand/api/forge-identities, 128 KiB on/api/repositories, and 1 MiB on/api/mcp. The server-level 25m cap is acceptable for these other routes because the application enforces those smaller limits. A smaller proxy cap there is optional defence-in-depth and must not fall below the corresponding application limit. - Proxy timeouts. All three proxying locations on the host use
proxy_connect_timeout 5s,proxy_read_timeout 60s, andproxy_send_timeout 60s. Preserve these bounds. Read/send timeouts bound inactivity between operations, not the entire request duration; do not treat them as a 60-second reconciliation deadline. Webhooks enqueue durable work, and the worker folds it separately from the delivery request. - Origin rate limits. Keep a rate limit at nginx, keyed by the trusted
real client address after the real-ip step. Application limits on expensive
routes run after authentication and use account/token identities; they do
not replace the origin's request bound. The host's
/etc/nginx/conf.d/overflow-ratelimit.confdefines expensive writes and account export at 12 requests/minute (burst 20), MCP at 60/minute (burst 30), and general traffic at 20/second (burst 200), all withnodelayand rejection status429. MCP and expensive requests also encounter the general bound. Loopback and the host's own addresses are exempt; webhook receivers are exempt from the general class and retain the application's shared receiver buckets. Review exemptions for the deployment's own callers rather than copying another host's trusted addresses. - Address attestation in every location. Follow section 13
to set
OVERFLOW_PRIVILEGED_PROXY_SECRETand put the matchingX-Privileged-Proxy-Secretdirective in a root-only include. Include it in every proxyinglocation, including/api/overrides,/api/moderationand/on the host. nginx's header inheritance rule means that a location setting anyproxy_set_headerinherits none from an outer level. A missing or mismatched attestation leaves logged addresses markedclientAddressVerified: false; it does not stop the privileged action.
The host's vhost is /etc/nginx/sites-enabled/overflow.nitjsefni.eu. Review the
vhost and its nonsecret includes against this list; keep the attestation
include root-only and do not copy its value into deployment records. Section 13
provides the validation/reload and privileged-journal check after a change.
Use the operational state map
to distinguish a proxy failure from a database or reconciliation failure.
Clone, install, migrate and build as root, with the production settings loaded
from the environment file so the db:migrate script reaches the right database.
Run these blocks in Bash and stop on a failed command; set -e makes a pasted
block stop too, before a failed build can be switched into service.
A first installation has no serving release: nothing serves while this section runs, and the release built here first serves after its switch, when section 6 starts the service. The migrate-then-build window that section 10's migration check guards does not exist on this path — there is no previous release to serve against the new schema — so the check is not part of this flow. A host being hardened by sections 1 and 6 is the exception: its old deployment still serves from the previous checkout while this section runs, so section 10's migration check applies on that path too. Carry the check out before this section's migrate step, treat whatever is serving at migrate time as the previous release for the test, and stop to split the change across deploys instead of running the block below if a migration fails it.
set -e
git clone https://github.com/Nitjsefnie/Overflow.git /srv/overflow
cd /srv/overflow
npm_config_package_import_method=copy pnpm install --frozen-lockfile
. /srv/overflow/scripts/systemd-env-file.sh; export_env_file /etc/overflow/overflow.env
pnpm db:migrate
release=".next-release-$(date -u +%Y%m%dT%H%M%SZ)-$(git rev-parse --short=7 HEAD)"
mkdir "$release"
node scripts/release.ts prepare /srv/overflow "$release"
OVERFLOW_NEXT_DIST_DIR="$release" pnpm buildEach build gets a new directory at the tree root, alongside .next, named
.next-release-<YYYYMMDDTHHMMSSZ>-<7 to 40 lowercase hex characters>.
The UTC timestamp makes the names sort in deployment order, and the SHA
identifies the source revision. The explicit --short=7 requests at least seven
hexadecimal characters even when core.abbrev is shorter. prune deletes
directories and the <release>.tsconfig.json sidecars beside them, so it only
touches names matching this grammar to be certain they are ones the deployment
procedure created. A directory named .next-release-notes is safe beside the releases:
prune ignores it entirely, including when counting retention slots. Reserve
matching names for releases; the name check does not prove a build succeeded.
mkdir deliberately has no -p: a collision must stop the deploy, not reuse an
existing build. Never build into .next on a serving host, or reuse a release
directory, even to retry a failed build.
The release directory must be one path segment deep. The tracked tsconfig.json
includes .next/types/**/*.ts, and Next generates <distDir>/types/validator.ts
with relative imports back to the project's src directory. TypeScript resolves
those imports lexically through the .next include, without following the
symlink. A nested layout such as .next-releases/<id> generates ../../../src/...
imports, which land one directory above the project when read as
.next/types/validator.ts. Keeping the release beside .next gives both paths
the same depth. Release builds also isolate their generated types as described
below, so a validator from the serving release cannot prevent a route from
being removed.
next.config.ts reads OVERFLOW_NEXT_DIST_DIR for this build only. It trims the value
and rejects either path separator (/ or \), absolute paths, . (zero depth),
.. and an existing symlink at the output path. Use a single directory name inside
the tree, as above, with no ./ prefix or trailing slash. With the variable unset or blank,
the local build script still uses .next. Do not export
OVERFLOW_NEXT_DIST_DIR for the service or add it to /etc/overflow/overflow.env:
next start uses .next at runtime, with the variable unset.
Before each release build, run
node scripts/release.ts prepare <tree> <releaseDir> with the same release name
used for OVERFLOW_NEXT_DIST_DIR. It generates the ignored <releaseDir>.tsconfig.json
beside the release directory, preserving the tracked tsconfig.json compiler
settings and source includes while removing all .next* includes. Each release
has its own config; for example, .next-release-20260907T101500Z-abc1234 uses
.next-release-20260907T101500Z-abc1234.tsconfig.json at the tree root.
Skipping preparation now fails loudly. With OVERFLOW_NEXT_DIST_DIR set,
next.config.ts refuses a missing, stale or invalid generated config before
Next can create defaults or consume it for type checking. The error prints the
preparation command to run. A config left by a previous release cannot satisfy
this check. The generated file must be a regular file, name the intended release,
and match fingerprints of both the tracked input and the prepared configuration.
Changing tracked compiler options after preparation or editing the generated
settings requires running preparation again.
Next uses the checked file through typescript.tsconfigPath, appends the current
release's type entries, and type-checks those validators. The fingerprint allows
those specific additions. Preparation accepts TypeScript config comments and
trailing commas using the installed TypeScript parser. It requires an explicit
include array of strings and rejects extends and references with a diagnostic
naming the unsupported property. Keep the tracked config self-contained; the
pinned Next 16.3.8 updates the generated type includes in the prepared config.
releaseConfig is reserved for the generated preparation metadata.
Next also regenerates the ignored next-env.d.ts. Release builds leave the
tracked tsconfig.json untouched, including on failure, so no restore step is
needed. With OVERFLOW_NEXT_DIST_DIR unset or blank, the preparation check does no
filesystem work; local development uses the tracked config and the running
service continues to use .next. Start a deploy with a clean tracked tree.
Then set the ownership the unit assumes. The tree is root-owned and readable by
the overflow group; nothing in it is group-writable.
chown -R root:overflow /srv/overflow
chmod -R u=rwX,g=rX,o= /srv/overflowThe install commands here and in section 10 set
npm_config_package_import_method=copy for each invocation, importing package
files into private inodes instead of hardlinking the shared store.
Ownership and mode resets therefore leave the store and unrelated checkouts'
package files untouched. Existing deployments must first complete section 10's
one-time dependency migration: an unchanged install does not replace old
hardlinks. The service's permissions inside the deployment tree stay as they are.
next start writes inside .next/cache — the image optimizer's output and the
.previewinfo and .rscinfo files — and that directory is the only one the
unit makes writable. The cache lives inside the new release. Create it and
hand it to overflow before switching and starting the service: ReadWritePaths
naming a missing path is a start failure, and a root-owned cache is not writable
by the service account.
mkdir -p "$release/cache"
chown -R overflow:overflow "$release/cache"
chmod -R u=rwX,g=rX,o= "$release/cache"
pnpm release:switch /srv/overflow "$release" --expect-current absentThe release:switch package script takes <tree> <releaseDir> and runs
node scripts/release.ts switch <tree> <releaseDir>; a relative release argument
is relative to the tree. Passing --expect-current <absent|path> makes the
switch conditional: it refuses unless .next still resolves to <path> at
switch time, or to nothing at all when the value is absent; the deploy
procedure always passes it. The resolved directory must be a direct child of the
canonical tree, whether the argument is relative, absolute or a symlink alias.
A symlink to a nested or outside build is refused; an alias resolving to a
valid direct child is accepted.
switch also refuses a resolved directory name that does not match the release
grammar above. Inventing a name therefore produces a clear failure instead of
selecting a release that silently accumulates because prune cannot manage it.
The script requires a real BUILD_ID file and a real cache directory, refuses
symlinks for those two markers, then renames a temporary relative symlink over
.next and prints the resolved release path.
That rename keeps an existing .next symlink resolvable throughout the swap.
The marker checks do not validate every manifest or prove that a build succeeded;
the successful build and the service verification remain required.
node scripts/release.ts check <tree> <releaseDir> runs the same candidate
checks as switch: resolved depth, release name, BUILD_ID file and cache
directory, without following symlinks for either marker. It prints the resolved
path and changes nothing on disk. It can check a candidate while .next is
still a real directory; switch continues to require the one-time migration
below before replacing that directory.
The unit stays unchanged for this layout. Every path it names still reads
/srv/overflow/..., including ReadWritePaths=/srv/overflow/.next/cache.
On this host, next start served routes, static assets and RSC requests with
HTTP 200 and no missing-manifest errors through the .next symlink, with
OVERFLOW_NEXT_DIST_DIR unset at runtime. systemd resolved the writable cache path through
the symlink too: the sandboxed process could write in that cache and got EROFS
everywhere else. No service environment change or unit edit is needed.
An existing host has a real /srv/overflow/.next directory. A symlink cannot
be renamed over a real directory; scripts/release.ts switch refuses it with a
one-time migration message rather than deleting the serving build silently.
Unlike a first installation, this host has a serving release throughout: section 10's block migrates and builds while the current release keeps serving, and the release built here starts serving only at this subsection's switch. Section 10's migration check therefore applies on this path too — carry it out before running section 10's block, and stop to split the change across deploys instead of running the block if a migration fails it.
Use the same Bash shell for section 10's preparation and this migration block.
Follow section 10 through config preparation, the build, ownership reset,
new cache handover and previous/new build printout, stopping before its switch
and restart lines. The reset excludes the serving cache, which is still in the
real .next directory on this host; leave that directory in place while
preparing the new release. Replace section 10's switch and restart lines with
the block below, pasted into that same shell so $release still names the
completed new build. In a separate shell, ${release:?} stops with parameter null or not set before removal and leaves the old .next intact:
set -e
cd /srv/overflow
test -d /srv/overflow/.next
test ! -L /srv/overflow/.next
node scripts/release.ts check /srv/overflow "${release:?}"
rm -rf -- /srv/overflow/.next
pnpm release:switch /srv/overflow "$release" --expect-current absent
systemctl restart overflow.serviceThe candidate check must succeed before removal, so a malformed name, wrong
depth or invalid build marker leaves the serving directory in place.
This one deploy still hits the old missing-build window: removing the real
.next can break requests until the symlink is installed and the service
restarted. It is the last deploy that needs that removal; later builds leave
the serving release in place. Build and prepare the new release first to keep
this window to the removal, switch and restart, rather than the whole build.
The removed directory provides no release-level rollback. If switching fails
after removal, keep the completed new release, correct the reported failure
and rerun the switch and restart; do not start another build into .next.
A refused switch can leave .next a symlink, so re-running this block aborts
at test ! -L /srv/overflow/.next; that abort is the signal to recover through
the standing section 10 procedure, which expects the symlink layout. Continue
with section 10's verification before pruning.
systemctl show overflow.service -p MainPID --value > /run/overflow-preswitch-mainpid
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow.service /etc/systemd/system/overflow.service
systemd-analyze verify /etc/systemd/system/overflow.service
systemctl daemon-reload
systemctl enable overflow.service
systemctl restart overflow.servicesystemd-analyze verify prints nothing and exits 0 for a well-formed unit; it
names any directive systemd does not recognise.
The restart is the switchover, and it is separate from enable on purpose.
--now on enable means start — man systemctl: "also start/stop/try-restart
the units after the specified unit file operations succeed" — and start on a
unit that is already active is a no-op with no job and no message. On the host
section 1 is written for, the old root process would keep serving while systemd
held the new unit file loaded and unapplied, and section 7 would report the new
unit's User= beside a ps line owned by root. The recorded MainPID is
what section 7 compares against; on a host that has never run Overflow it is
0, which is the same evidence read the same way.
Wait for the HTTP readiness check to succeed before inspecting the process owner. Keep that order when pasting the commands separately too.
set -e
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/auth/providers
printf 'MainPID before the switch: %s\nMainPID now: %s\n' \
"$(cat /run/overflow-preswitch-mainpid)" \
"$(systemctl show overflow.service -p MainPID --value)"
systemctl show overflow.service \
-p MainPID -p User -p Group -p NoNewPrivileges -p ProtectSystem
ps -o user=,pid=,args= -p "$(systemctl show overflow.service -p MainPID --value)"Expected: active; 200 from curl — the readiness endpoint answers 200 only
when PostgreSQL is reachable and every migration this build bundles is recorded
applied in schema_migrations (a bounded probe: at most a few seconds, ~3 s
worst case), so a 200 here is a real dependency check, not a bare port probe.
An ahead schema answers 200 only when its extra migrations are not flagged
for mixed-version review. A missing bundled migration or an extra migration
flagged for that review answers 503; inspect Readiness probe failed: in the
journal before continuing a deployment or rollback. The second 200 is the
sign-in smoke: GET
/api/auth/providers runs the Auth.js configuration, and answers 500
([auth][error] UntrustedHost in the journal) when the deployment's
environment cannot sign in — a failure class the readiness endpoint cannot
see. The two MainPID values differ and the current
one is not 0; User=overflow, Group=overflow, NoNewPrivileges=yes,
ProtectSystem=strict; one ps line, owned by overflow and never root.
The two MainPID values are the check that distinguishes a switch from a
reload. systemctl show reports the merged effective configuration, so it answers
User=overflow from the moment daemon-reload runs, whether or not anything
restarted; only a new PID says the process serving requests is the one the
hardened unit started. A pair that has not moved means the old process is still
serving and section 6's restart did not run.
A host drop-in under /etc/systemd/system/overflow.service.d/ can override the
installed unit. Section 6 installs only overflow.service; the repository guard
does not inspect host drop-ins. The systemctl show check above reveals overrides
to the properties it lists because it reports the merged configuration, including
drop-ins, rather than just the unit file.
Two details in those commands are the point of them. Ask systemctl for the
PID instead of asking ps for Node processes: ps -C node matches on comm,
and Next rewrites its process title to next-server (v…, so ps -o user= -C node never sees this service at all and answers root from whatever unrelated
Node processes the host happens to run — the alarming answer whether the
hardening worked or not. And Type=simple has no readiness barrier, so
systemctl start returns as soon as the process is forked, before the drop to
overflow and before Next binds the port. An early ps can therefore print
root for a process still starting. The curl retry absorbs that readiness race;
only inspect the PID and owner after it succeeds, so a connection refusal or
the transient startup owner is not mistaken for a failed deploy.
systemd-analyze security overflow.service reports the remaining exposure and
is worth reading after any change to the unit.
Restrictions only bite on the code paths that use them, so exercise the
application through the browser before calling the switch done: sign in with
GitHub, open the dashboard, and register a repository. That is what makes DNS
resolution, an outbound HTTPS call to GitHub and a database write happen inside
the sandboxed process; the reconcile package script does not, because it runs
as root outside the unit and so is subject to none of these restrictions. The unit sets
SystemCallErrorNumber=EPERM, so a syscall the filter blocks returns an error
to the process instead of killing it. That buys resilience, not visibility: a
seccomp errno action logs nothing of its own, and it is the default kill action
— the one EPERM replaces — that would have reached the journal, as
status=31/SYS. So what you are looking for is the application's own report of
an operation that failed:
journalctl -u overflow.service -n 100 --no-pagerDo this once, immediately, while the previous unit and checkout are still on the host. Restore the saved copy, confirm the service comes back on it, then reinstall the hardened unit and confirm again with section 7.
The rollback has three preconditions and nothing on the host keeps them alive —
a cleanup of /root voids the rollback silently, months later. Check them
first, here and in section 9; if any fails, there is no rollback and a broken
hardened unit has to be fixed forward instead.
missing=0
for path in /root/overflow.service.pre-hardening /root/overflow \
/root/.nvm/versions/node/v24.17.0/bin/pnpm; do
if [ -e "$path" ]; then
echo "present: $path"
else
echo "MISSING: $path" >&2
missing=1
fi
done
[ "$missing" = 0 ] \
|| echo "No rollback is available. Fix the hardened unit forward instead." >&2Every path prints, present or missing, because test prints nothing either way:
three blank results is what an operator sees whether the rollback is intact or
gone, and a check that cannot be read has replaced the silent expiry it exists
to catch.
systemctl show overflow.service -p MainPID --value > /run/overflow-preswitch-mainpid
systemctl stop overflow.service
cp -a /root/overflow.service.pre-hardening /etc/systemd/system/overflow.service
systemctl daemon-reload
systemctl start overflow.service
systemctl is-active overflow.service
printf 'MainPID before the rollback: %s\nMainPID now: %s\n' \
"$(cat /run/overflow-preswitch-mainpid)" \
"$(systemctl show overflow.service -p MainPID --value)"
ps -o user=,pid=,args= -p "$(systemctl show overflow.service -p MainPID --value)"
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/active, a MainPID that moved, a ps line owned by root again and 200
mean the rollback path works — the root is the point of it, since that is the
state the saved unit runs in. Then repeat section 6 and section 7 to get back to
the hardened unit; section 6's restart is what makes that return leg real, and
its MainPID pair is what proves it happened. A rollback that has never been run
is an assumption. This block keeps the landing-page URL on purpose: the restored
old checkout predates the readiness endpoint, and what the block verifies is
unit-rollback serving, not dependency reachability.
If a deploy fails, capture why before restoring anything:
systemctl status overflow.service --no-pager
journalctl -u overflow.service -n 100 --no-pagerThe failures this configuration produces:
ReadWritePathsnames a path that does not exist —/srv/overflow/.next/cachenow resolves through the.nextsymlink into the selected release. A missing or dangling link, or a missingcacheinside that release, prevents startup. The cache must also be owned byoverflowfor runtime writes to work./etc/overflow/overflow.envdoes not exist — section 4 was skipped. Its mode is not a start failure: systemd reads the file as root, before the drop toUser=overflow, so0600 root:rootis correct and a service that cannot read the file itself is the design, not a fault.postgresql.servicedoes not exist on this host, so the unit'sRequires=postgresql.servicerefuses the start outright. That is the remote-database case; see the note at the top of this file./usr/local/bin/nodeis missing or is a dangling symlink into/root.- The tree is unreadable to the group, so
nextcannot load its own build.
For a failed release, switch back to the retained previous build and restart
the same hardened unit. Section 10 prints the previous release path before
switching; record it with the deploy. Replace the value below with that recorded
path, and confirm the directory still exists. The ownership reset on a later
deploy also resets retained caches other than the serving cache, so hand the
previous cache back before the restart, even if it was writable when that
release last ran. The block takes the same deploy lock as section 10, so a
rollback cannot interleave with a deploy, and its switch is conditional on
.next still resolving to the release the rollback started from. Paste the
block into a fresh shell, never one that already holds the deploy lock: its
exec 9> re-opens fd 9, which would release the lock that deploy still holds.
set -e
cd /srv/overflow
exec 9>/run/overflow-deploy.lock
flock -w 900 9 || { echo "Could not acquire the deploy lock on /run/overflow-deploy.lock; refusing to deploy. Consult the deploy procedure's serialization notes before re-running." >&2; exit 1; }
expected_serving=$(readlink -f /srv/overflow/.next || printf absent)
previous_release='.next-release-REPLACE-WITH-RECORDED-ID'
test -f "$previous_release/BUILD_ID"
test -d "$previous_release/cache"
chown -R overflow:overflow "$previous_release/cache"
chmod -R u=rwX,g=rX,o= "$previous_release/cache"
pnpm release:switch /srv/overflow "$previous_release" --expect-current "$expected_serving"
systemctl restart overflow.service
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readinessExpect active and HTTP 200, then exercise the application and inspect its
journal as in section 7. Restart immediately after switching: a process running
across the swap retains its old writable cache mount, while the fresh start
picks up the selected release's cache. A symlink switch alone is not a deploy
or a rollback.
A retained release built before the readiness endpoint existed answers 404
there. curl -f treats 404 as a failure, so the verification fails —
readiness unknown rather than confirmed — and that is the intended fail-safe:
never report a rollback healthy on a signal that cannot see the database. The
caveat goes moot once every retained release postdates the endpoint.
The endpoint refuses a database missing a migration the release bundles with
503. A rollback never undoes migrations, so the database can also be ahead
of the retained release. A release that checks the review flag answers 503
when an applied migration recorded as review is absent from its bundled list;
extra unflagged migrations still allow 200. An older release that does not
check the flag still answers 200 against an ahead schema, even after a flagged
migration, so its readiness signal cannot establish compatibility. The
classifier's limits below also mean that a 200 does not replace review of
the retained release against the schema as it stands.
This rolls back the build, not the revision. The checkout and node_modules
are shared with the current revision, and database migrations are not undone.
The retained build must work with those dependencies, configuration and schema;
the switch script does not check compatibility. If those need reverting too,
restore the intended revision and its dependencies and build it into a new
release using section 10's build, ownership, switch and verification steps;
assess the database schema separately.
If the hardened unit itself is broken and needs the old unit restored, check
section 8's three preconditions, then restore with the same commands. That
restored state runs the application as root again, which is the state this
procedure exists to leave behind, so fix the failure rather than forgetting it.
systemctl is-active reporting active and the HTTP check returning 200 are
the evidence the rollback worked; a journal that shows the process starting as
root:root is what tells you the old unit — not the new one — is the one now
running.
The old-unit rollback decays with every deploy, and neither of those two checks
reports it. Section 10 migrates the production database and does not touch
/root/overflow, so from the first revision deploy onwards the saved unit runs
older code against a newer schema. Once the hardened service is trusted, stop
treating the old checkout as the rollback. Retaining previous releases partially
restores the path: a compatible previous build can be selected without rebuilding
or returning to the root-run unit. It does not preserve the old checkout,
dependencies or database schema, so a revision rollback still needs those
considered explicitly.
The procedure runs as one committed script. As root, from the tree root, run
bash scripts/deploy-revision.sh; it applies the source-tree guards described
below, plus its automated pending-migration and workflow-pinned CI gates: the
flock fence on /run/overflow-deploy.lock held on fd 9 for up to 900 seconds
and refusing with the serialization refusal when the lock is not acquired, the
.next anchor taken before the fetch and passed to
release:switch --expect-current, a git fetch origin main that moves only
refs, resolution of the deployed SHA from FETCH_HEAD, a refusal when HEAD
cannot fast-forward to that SHA (the tree is ahead of or diverged from main),
the tree-cleanliness gate that refuses the deploy when the working tree deviates
from HEAD — tracked modifications, staged changes and untracked non-ignored
files all survive a fast-forward, and a release is named for the commit it was
built from, so the tree must be that commit; it also refuses the deploy when the
tree holds an ignored untracked file outside the operational allowlist, since
git status never shows such a file (the .gitignore denies by default) and
the build would compile it into a release named for a commit that does not
contain it. The allowlist is exactly, and only at the tree root, the .next
anchor, the release directories and their .tsconfig.json sidecars,
.next-release-notes/, next-env.d.ts and node_modules/; anything else
ignored anywhere in the tree must be removed before re-running. .claude/ is on
that list for the opposite reason: it is this repository's own session machinery
— the autoloaded working rules and the priority board — which the checkout a
deploy runs from may carry and which nothing in the build reads, so an operator
must not have to delete it to deploy. The required-checks gate must then bless
the exact fetched SHA (below), and only after both gates pass the
git merge --ff-only that moves the tree to that SHA; then the redundant-deploy
skip that compares the resolved SHA against the serving release's REVISION
record and, on a match — the run that built the serving release migrated at that
same commit — exits without installing, migrating or building, the copy-import
install, the environment load (the file read as systemd's EnvironmentFile=
reads it, never run; below), db:migrate, a grammar-named release directory
created with a collision-aborting mkdir, generated-config preparation, the
build, whose clean step wipes the release directory (everything outside
cache|dev|lock|trace), the ownership reset excluding the serving cache, the
new cache handover to the service account, the conditional switch, the restart,
the is-active verification, the readiness endpoint and the sign-in smoke
(GET /api/auth/providers, which must answer 200), the webhook upgrade written
to a retained JSONL log with a nonzero upgrade exiting the script nonzero, after
which the exact source SHA is recorded in a REVISION file inside the release —
attesting a fully deployed release (built after the wipe, switched, verified),
so a redundant deploy may trust it — the retention listing, and the prune via
release:prune --keep 3.
The migration-safety analysis and every other guard below govern the script's
run exactly as they govern the manual block; read this whole section before
running either.
The script runs in two phases joined by one exec. Bash reads a script incrementally, and the fast-forward rewrites the script file mid-run, so the tail after the fast-forward would otherwise execute from the pre-merge copy. Phase 1 is everything through the fast-forward, run from wherever the deploy was started; it then re-executes, handing the resolved SHA and the pre-fetch serving anchor to the merged tree's own copy of the script, which runs the install→prune half. The deploy lock travels with the exec — the lock is held on the open file description, which the re-executed process inherits — so the fence still serializes the whole procedure. The handoff names are internal: the script sets both itself, production never seeds them, and a re-executed entry refuses fail-closed unless fd 9 is open on the deploy lock itself, the serving anchor is set, and the tree is already at the exact commit the handoff carries — the one the gates blessed. So the verification steps a deployed commit adds to the script run in their own deploy.
The script's prune is guarded in a way the manual path is not, separately from
the confirm-first rule below: before pruning, the script consults its own
retention listing and refuses to prune unless the recorded previous release is
inside the newest-3 retention set. When the previous release is older, the
script prints its name, prints the exact manual release:prune command with
--keep raised above 3, and runs nothing — the deploy still exits 0, because
the deploy itself succeeded and only the prune was withheld. The manual path
keeps a human as the guard through the confirm-first rule below; the script's
guard is additional automation, and the manual rule is what still applies when
pruning by hand.
The script deploys only a SHA that main's required checks have blessed. The
gate parses .github/required-checks.json at the deployed SHA with the tree's
scripts/required-checks-parse.jq, and when that parse fails it re-parses with
the target SHA's own copy of the same file, so a map-shape change cannot
deadlock a tree whose parser predates it. Right
after the fetch it resolves the exact SHA being deployed and, once the
tree-cleanliness gate passes, reads main's required checks from branch
protection and the branch-rules endpoint (repos/OWNER/REPO/rules/branches/main).
The gate uses their union, refusing when it is empty or either read fails for
a reason other than not-found (HTTP 404). Every required check's latest run on that SHA must conclude
success before the fast-forward, and so before install, migrations, build,
switch or restart. A failed, cancelled or otherwise non-successful conclusion refuses
immediately, and a check whose latest run is queued, in progress or has not
been created yet makes the script wait, polling every 60 seconds until
OVERFLOW_DEPLOY_CI_TIMEOUT (default 900) seconds elapse, then refusing with
the still-pending checks named, an absent run reported as <check> (absent).
Each poll lists check-runs and workflow runs with pagination. Jobs are read
to prove check-run attribution when their IDs are not yet known, or when a
run's attempt changes. Later polls reuse that job identity and read current
outcomes from check-runs; suite identity only narrows fallback reads and
cannot certify a check by itself. When all relevant job IDs and run attempts
are known and unchanged, a poll costs two requests if each listing fits on one page. Unattributed checks
still hold the gate, and any failed read refuses the deploy.
Every required check must therefore report on the exact main tip being
deployed: a check that runs only on pull requests never runs on the SHA a
rebase merge lands, so the gate would wait it out as <check> (absent) and
refuse. ratchet-guard normally reports on both — pull_request_target
for pull requests and push for the tip each push lands. The deploy gate
checks only the fetched tip of main.
If the gate waits on (absent) for every required check and
actions/runs?head_sha=<sha> lists no push runs, recover main's current
tip with workflow dispatches. A dispatch on main runs for main's tip, so it
cannot certify an older tip. For ratchet-guard, base must be the newest
main commit that already has a successful ratchet-guard run. The dispatch
judges the interval from that tip to main's tip as one endpoint change, just
as a push run does for a multi-commit push. If several consecutive pushes
were dropped, this is coarser than their separate push runs: a relaxation
later re-tightened past the base is not flagged.
base=$(gh api 'repos/Nitjsefnie/Overflow/actions/workflows/ratchet-guard.yml/runs?branch=main&status=success&per_page=50' --jq '[.workflow_runs[] | select(.event == "push" or .event == "workflow_dispatch")][0].head_sha')
gh workflow run ci.yml --ref main -f base="$base"
gh workflow run actionlint.yml --ref main
gh workflow run ratchet-guard.yml --ref main -f base="$base"After fetching main, check git rev-parse origin/main: if $base is empty or
null (no qualifying run among the 50 most recent successful ones) or equals
that SHA (the tip is already certified), do not dispatch ratchet-guard.
Dispatch ci without -f base in that case; it measures coverage. Find the
base by hand from older runs or stop before dispatching ratchet-guard.
The gate accepts these runs because it identifies each producer by its pinned
workflow path and job name, never by event. The newest run decides, so a
failed dispatch is superseded only by a newer successful one. An empty-commit
re-push is refused by branch protection. A dispatched ci run measures the
change against the supplied base, including each commit in the interval; a
dispatch without a base always measures coverage.
Each required check is resolved to the jobs of the workflow files
.github/required-checks.json pins it to — a pin may be one file or a list of
them, and every pinned context names both a repository-level file and a
pull-request-only file — and a same-named
check-run from any other producer holds the deploy as pending, so it is
refused at the deadline and never passed.
The required checks are pinned, in branch protection, to the Overflow Ledger
App (app id 5118623), and the gate's producer rule follows that pin: the gate
reads each required context's check-runs, a check-run posted by the App
decides the context, and the pinned workflow's job record decides only when
no App check-run exists — while any other check-run bearing a required name
keeps the check pending as unattributed. The App's check-runs are posted by
the relay (.github/workflows/ledger-relay.yml), which mirrors the pinned
producers' completed runs onto the App; the
Required checks relay subsection of OPERATING.md
carries its operations.
A refused gate leaves HEAD, the index and the working tree untouched, so the
tree stays on the commit it was on; only the refs the fetch wrote
(FETCH_HEAD, origin/main) have moved.
OVERFLOW_DEPLOY_CI_GATE=skip bypasses the entire gate with a loud warning
naming the skip and the SHA, and is reserved for rollback or recovery deploys
when main's CI is red; unset or empty enforces the gate, and any other value
refuses.
The script takes no arguments and reads exactly eight environment overrides
for the test harness; production sets none of them and runs on the defaults:
OVERFLOW_DEPLOY_TREE (default /srv/overflow),
OVERFLOW_DEPLOY_ENV_FILE (default /etc/overflow/overflow.env),
OVERFLOW_DEPLOY_LOCK (default /run/overflow-deploy.lock),
OVERFLOW_DEPLOY_UNIT (default overflow.service),
OVERFLOW_DEPLOY_URL (default http://127.0.0.1:3000/api/readiness),
OVERFLOW_DEPLOY_LOG_DIR (default /var/log/overflow),
OVERFLOW_DEPLOY_CI_TIMEOUT (default 900) and
OVERFLOW_DEPLOY_ENV_READ_TIMEOUT (default 5, seconds). The deployment URL must name the
readiness endpoint: the sign-in smoke derives its own URL from
OVERFLOW_DEPLOY_URL by replacing the trailing /api/readiness with
/api/auth/providers, so any other target derives a wrong smoke URL and
refuses the deploy — failing closed. One further override is
operator-facing, not a test knob: OVERFLOW_DEPLOY_CI_GATE, whose only
accepted non-default value is skip (the gate paragraph above).
Every revision deploy finishes by upgrading existing hooks after the new release
is serving and its readiness check succeeds. Migration 043 must precede the
scoped-credential receiver code. Legacy hooks cannot authenticate until this
upgrade configures their new callback UUID and independent secret. Keep the
callback base URLs and OVERFLOW_TOKEN_ENCRYPTION_KEY available; change the
encryption key only through section 11. Avoid concurrent manual hook-configuration edits. The command
preserves active state and unrelated subscriptions and verifies each persisted hook
at its current numeric-ID-resolved location. It requests full upstream repair only
when the subscribed events or callback URL changed, or the credential is finalized
for the first time.
Pending credential material is durable: retry a failed run without reminting it.
Missing sponsor credentials require relinking before that registration can upgrade.
After all registrations migrate, retire the old shared credential and assess any
reuse elsewhere. Rolling back to a shared-credential receiver reopens that trust boundary.
Historical missed deliveries have no known dirty subject; an ordinary incremental
queue pass cannot guarantee their repair. Startup recovery does not replace this upgrade.
Startup recovery is on by default. Before every deploy, review whether the
revision can affect fold-derived values. Changes to fold behavior must not
suppress the startup pass; leave OVERFLOW_SKIP_STARTUP_RECONCILIATION unset
in /etc/overflow/overflow.env. The same recovery-on default applies whenever
the impact is uncertain.
For a reviewed deploy that cannot affect fold-derived values (for example, a
copy-only change), an operator may deliberately add
OVERFLOW_SKIP_STARTUP_RECONCILIATION=1 to /etc/overflow/overflow.env before
the restart below. Only the exact value 1 suppresses the immediate startup
sweep; unset, blank, malformed and all other values run it. A shell export in
the deploy terminal does not configure the systemd service. This opt-out leaves
the recurring sweep, queued work, webhook-triggered reconciliation and manual
reconciliation running normally, but missed deliveries may remain unrecovered
until a later reconciliation.
scripts/deploy-revision.sh reads the environment file, not the deploy terminal,
and reads it the way systemd's EnvironmentFile= does, without running it:
single and double quotes, backslash escapes, line continuations and # or ;
comment lines, with no $ expansion and no commands, so the migration check,
the migrations, the build and both lines below see exactly the values the
service will see. Every name the file sets is exported to those steps over any
value the deploy terminal exported; a name the file does not set keeps the
terminal's value. If the file cannot be read at that first read (it is missing,
unreadable, or not read within the deadline), the deploy refuses before the
migration check. The script then says what the file holds. Before the restart
it prints one line: that the immediate startup sweep will be skipped when the
file sets exactly 1, or that it will run when the variable is unset or blank.
Any other value, such as true, is named in that line as not exactly 1, so an
override that does not skip is visible before the restart. Once the service is
active again, a file that still holds 1 is reported as still set, with a
reminder that every later restart will skip the sweep until the line is
removed. That reminder is the prompt for the removal below. If the file cannot
be read when either line is printed (it is missing or unreadable, or not read
within the deadline), that line says the value could not be read, rather than
reporting the variable as unset or saying nothing, and the deploy carries on.
After restart, verify the warning naming OVERFLOW_SKIP_STARTUP_RECONCILIATION
in the service journal. An authenticated moderator can also read
GET /api/moderation/rederivation: startupRecoverySkipped records whether
this process skipped its startup recovery pass, alongside the existing row
staleness counts. It remains true after later sweeps; it is startup history,
not a claim that every repository is still stale. Remove the override from
the environment file after the chosen restart so the next restart defaults to
recovery-on. Remove this temporary override with issue 196, when startup
reconciliation becomes incremental.
Install, migrate and build run as root inside the tree. Only the service runs as
overflow, and the ownership reset afterwards is what keeps it that way: a
build writes new files as root, and the new cache has to be handed back while
the serving cache stays writable. Build into a new release directory every time
so Next cannot rewrite the serving build's manifests, chunks and fallback error
page during the build. Run one deploy at a time; concurrent installs, config
generation, switches or prunes share the same tree.
On a host whose .next is still a real directory, use section 5's one-time
migration block at the switch step.
The block is serialized with an exclusive flock on /run/overflow-deploy.lock
(command -v flock: /usr/bin/flock, util-linux), held for the whole
procedure: pull, install, migrate, build, ownership reset, switch, restart,
verification and webhook upgrade all run under one lock, so two deploys
started together run one after the other instead of interleaving. A concurrent deploy waits up to 900 seconds for
the lock and then refuses; refusing is the fail-safe behavior, and a deploy
must never proceed without it. The lock is kernel-owned and disappears when the
holding process dies, so a crashed deploy cannot deadlock the next one.
The switch inside the block is conditional as the second defense, against
actors that skipped the lock: the block records the release .next resolves to
before touching anything and passes it as --expect-current, and
release:switch refuses unless .next still resolves there at switch time. An
off-procedure actor — an old copy of this document, a hand-run switch —
therefore cannot silently supersede an in-flight deploy: production never moves
backward and an already-verified release is never silently discarded. When the
switch reports the mismatch, re-run the whole procedure: the script from the
start, or the manual block below from its opening fence so it captures a fresh
serving anchor. A missing or
dangling .next at anchor time is a host that needs repair or the one-time
migration, not a routine deploy; the conditional switch refuses that state
rather than building on it.
Migrations apply before the build, and the release they accompany starts
serving only at the switch. Between those points the previous release serves
every request against the new schema, for the whole build duration. A migration
is safe to apply in that position exactly when the previous release's write
path cannot violate it: purely additive statements (a new table, a nullable
column, a plain index; a plain index built without CONCURRENTLY blocks the
previous release's writes for the scan's duration) are safe, and so is an
enforcing statement the previous release already satisfies on every write
path — settlements carries settlements_issue_unique (migration 003)
alongside the writer that maintains it.
After loading the environment and before pnpm db:migrate, the deploy script
runs node scripts/deploy-migration-status.ts and lists every migration not
yet recorded in schema_migrations. Each line is tab-separated:
<name>\t<verdict>[\t<shapes or safe note>]. Field 2 is - or review;
the optional third field names the matched shapes or the safe-marker note.
The deploy script still reads field 2 only.
Classification denies these shapes by default, ignoring SQL comments and matching case-insensitively after normalizing whitespace:
ALTER TABLE ... DROP [COLUMN] [IF EXISTS] <column>andDROP TABLE;- any
ALTER ... RENAMEform; ALTER [COLUMN] <column> [SET DATA] TYPE(theCOLUMNkeyword is optional);SET NOT NULL;ALTER TABLE ... ADD [COLUMN] [IF NOT EXISTS] <column> ... NOT NULLwithoutDEFAULTin the same top-level column definition.
Unicode-escaped U& identifiers and strings (including UESCAPE) are decoded
before matching. String constants continued across a newline are concatenated
before their contents are scanned. Lexical forms the classifier cannot decode
confidently require review, labelled unsupported lexical form; the safe
marker cannot override that failure.
The contents of every SQL string literal are classified too: a shape inside
a literal still requires review, even if that literal is data rather than
executed SQL. Statements inside literals (including DO or function bodies
and EXECUTE text) get no DEFAULT exemption: ADD COLUMN ... NOT NULL DEFAULT ... inside a literal is flagged too.
The literal substring overflow: mixed-version review anywhere
in the raw file, comments included, also forces review. A real SQL line
comment -- overflow: mixed-version safe: <reason> downgrades a shape hit to
- only with a non-empty reason explaining why retained releases are safe;
an empty reason does not count, and marker-looking text inside a block comment
never authorizes a migration. The review marker beats the safe marker when
both occur. Adding or dropping constraints and dropping a column default are
not classified; use the voluntary review marker for those risks too.
A pending review entry makes the script refuse before migration. Review
each entry against the still-serving release and every retained rollback
target, then rerun with the unchanged OVERFLOW_DEPLOY_MIGRATION_ACK=1 to
confirm that review. The acknowledgment has no effect when the status command
fails, and any value other than 1 is refused. The runner records the verdict
in schema_migrations.mixed_version_review in the same transaction as the
migration; already-applied legacy rows are not reclassified. Readiness renders
only ledger names matching the runner's migration filename grammar in the
journal. An invalid flagged name still refuses readiness, but the journal
replaces that name with <unrecognised migration name>. The anonymous 503
body contains only { "status": "unavailable" }; inspect the journal for the
reason. Only an actual boolean true flags a row; a legacy missing flag or a
drifted non-boolean flag is treated as unflagged.
Use expand/contract releases: drop or rename a column only one release after
the code stopped reading and writing it. Change a type or add NOT NULL only
after every retained release writes compatible values. Otherwise acknowledge
the migration knowingly: the still-serving release and every retained
rollback target older than it will fail against the changed schema.
During a flagged migration's build window, the still-serving old release
reports not-ready at /api/readiness, provided its readiness check reads the
review flag. On the container route, the image HEALTHCHECK and the
docker-compose.yml healthcheck mark the old container unhealthy; compose
does not restart it on that by itself. An older release whose readiness check
does not read the flag still reports 200 and leaves that window unprotected.
Acknowledge the migration knowingly. The deploy script probes readiness only
after the release switch and restart, so this old-release 503 does not block
the switch.
Before running the standing block below, read every migration the run will
apply for the first time — anything schema_migrations does not yet record —
and apply that test to each. New migrations that could break the previous
release's writes while both releases serve carry the marker in the file when
filed for review: overflow: mixed-version review. If one fails, stop: do not
run it. Land the
writer correction in this release and the enforcing statement in the next
deploy, or shape the constraint so the previous release cannot violate it
(for example, a partial constraint excluding the shape the old writer can
produce). Shipping the constraint and its writer correction in the same
release does not close the window: the previous release keeps serving with
the constraint applied for the whole build duration.
ADD CONSTRAINT ... NOT VALID does not rescue a failing constraint. A unique
constraint cannot be marked NOT VALID at
all, and where the mark exists (a CHECK or FOREIGN KEY constraint) it
defers only the scan of existing rows while still enforcing new-row writes
immediately, so it does not make a constraint safe to apply before the
corrected writer is serving.
Migrations are append-only: edits to applied migration files fail CI, and the
runner refuses a checksum mismatch with the value recorded in
schema_migrations.
The fenced blocks below are the manual fallback, for the case where the
script itself is what broke. They use the same fence, --expect-current
anchor and release grammar. They fetch origin main, resolve the fetched
commit, check ancestry, require a clean working tree and reject ignored files
outside the operational allowlist before git merge --ff-only. They skip a
redundant deploy when the serving release's REVISION matches that commit and
write the record after webhook upgrade succeeds.
The manual path does not carry the script's automated pending-migration gate.
After loading the environment, stop and run
node scripts/deploy-migration-status.ts by hand; if it fails, stop. Review
every listed migration and apply the mixed-version check above to each
review entry, then continue at pnpm db:migrate. OVERFLOW_DEPLOY_MIGRATION_ACK=1
only confirms review in the deploy script and does not add a manual gate.
The manual required-checks gate trusts check-run names; it does not use
.github/required-checks.json to attribute each name to its producing
workflow (a name may be pinned to several workflows, and the gate keeps
trusting the check-run name alone). It does not wait for CI: an
absent, non-completed or unsuccessful
required check refuses the deploy. Once CI completes, rerun the gates part of
the block. The script's OVERFLOW_DEPLOY_CI_GATE=skip setting has no manual
equivalent.
Run the standing block in parts, in one shell so the fd 9 fence and
expected_serving carry over. The ancestry, cleanliness, ignored-file and
required-checks gates run before the merge; if a gate refuses, the shell exits.
Diagnose and resolve the refusal, then start a new shell and re-run the block's
opening part from the fence. The only manual stop is after loading the
environment: run node scripts/deploy-migration-status.ts by hand, review every
listed migration as described above, stop if the command fails or a migration
is unsafe for the previous release's writes, and continue at pnpm db:migrate
only after that review.
Existing deployments: complete the ONE-TIME dependency migration below before running this standing procedure for the first time. Fresh installations using section 5's copy import do not need that migration.
set -e
cd /srv/overflow
exec 9>/run/overflow-deploy.lock
flock -w 900 9 || { echo "Could not acquire the deploy lock on /run/overflow-deploy.lock; refusing to deploy. Consult the deploy procedure's serialization notes before re-running." >&2; exit 1; }
expected_serving=$(readlink -f /srv/overflow/.next || printf absent)
git fetch origin main
full_sha=$(git rev-parse --verify 'FETCH_HEAD^{commit}')
ancestry_status=0
git merge-base --is-ancestor HEAD "${full_sha}" || ancestry_status=$?
if [ "${ancestry_status}" -eq 1 ]; then
printf 'HEAD in /srv/overflow is not an ancestor of the fetched main (%s), so it cannot fast-forward there; refusing to deploy. HEAD, the index and the working tree are untouched; only the fetched refs moved. Inspect git log %s..HEAD in the tree before re-running.\n' "${full_sha}" "${full_sha}" >&2
exit 1
elif [ "${ancestry_status}" -ne 0 ]; then
printf 'Could not determine whether HEAD in /srv/overflow is an ancestor of the fetched main (%s): git merge-base exited %s; refusing to deploy. HEAD, the index and the working tree are untouched; only the fetched refs moved. Investigate the repository state in the tree before re-running.\n' "${full_sha}" "${ancestry_status}" >&2
exit 1
fi
tree_status=$(git status --porcelain=v1 -uall) || {
printf 'Could not read the working-tree state in /srv/overflow; refusing to build a release whose source identity cannot be attested. Investigate git status in the tree before re-running.\n' >&2
exit 1
}
if [ -n "${tree_status}" ]; then
printf '%s\n' "${tree_status}"
printf 'The working tree in /srv/overflow deviates from HEAD; fast-forwarding it to %s would not make it that commit. A release is named for the commit it was built from; refusing to build one from a tree that is not that commit. Resolve every deviation above (git status), then re-run the deploy.\n' "${full_sha}" >&2
exit 1
fi
allowlist_re='^(\.next/?|\.claude/|\.next-release-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{7,40}/|\.next-release-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{7,40}\.tsconfig\.json|\.next-release-notes/|next-env\.d\.ts|node_modules/)$'
ignored_listing=$(mktemp)
ignored_status=0
git ls-files -z --others --ignored --exclude-standard --directory --no-empty-directory > "${ignored_listing}" || ignored_status=$?
mapfile -d '' -t ignored_entries < "${ignored_listing}"
rm -f "${ignored_listing}"
if [ "${ignored_status}" -ne 0 ]; then
printf 'Could not list the ignored untracked files in /srv/overflow (git ls-files exited %s); refusing to build a release whose source identity cannot be attested. HEAD, the index and the working tree are untouched; only the fetched refs moved. Investigate git ls-files in the tree before re-running.\n' "${ignored_status}" >&2
exit 1
fi
stray_ignored=()
for entry in "${ignored_entries[@]}"; do
[[ "${entry}" =~ ${allowlist_re} ]] || stray_ignored+=("${entry}")
done
if [ "${#stray_ignored[@]}" -gt 0 ]; then
printf ' %q\n' "${stray_ignored[@]}" >&2
printf 'The tree in /srv/overflow holds the ignored untracked files above, outside the operational allowlist (.next, .claude/, release directories and their .tsconfig.json sidecars, .next-release-notes/, next-env.d.ts and node_modules/, each at the tree root). These are ignored untracked files that git status does not show, and the build would compile them into a release named for %s, a commit that does not contain them; refusing to deploy. HEAD, the index and the working tree are untouched; only the fetched refs moved. Remove them, then re-run the deploy.\n' "${full_sha}" >&2
exit 1
fi
remote_url=$(git config --get remote.origin.url)
repo=
case "${remote_url}" in
git@github.com:*) repo="${remote_url#git@github.com:}" ;;
https://github.com/*) repo="${remote_url#https://github.com/}" ;;
esac
repo="${repo%.git}"
repo_valid=0
[[ "${repo}" =~ ^[^/]+/[^/]+$ ]] || repo_valid=$?
if [ "${repo_valid}" -ne 0 ]; then
printf 'Could not parse an OWNER/REPO GitHub slug from remote.origin.url (%s); refusing to deploy.\n' "${remote_url}" >&2
exit 1
fi
required_stderr=$(mktemp)
# Read HTTP status independently of diagnostic text; parse only successful bodies.
classic_required=
required_status=0
if required_response=$(gh api "repos/$repo/branches/main/protection" --include 2>"${required_stderr}"); then
# Drop the status line and headers through the first blank line (LF or CRLF).
required_body=${required_response#*$'\n'}
while [[ "${required_body}" == *$'\n'* ]]; do
required_header=${required_body%%$'\n'*}
required_body=${required_body#*$'\n'}
[ -n "${required_header%$'\r'}" ] || break
done
if required_body=$(jq -e 'if type == "object" then . else error("expected a protection object") end' <<<"${required_body}" 2>"${required_stderr}"); then
classic_required=$(jq -r '([.required_status_checks.contexts[]?] + [.required_status_checks.checks[]?.context]) | unique | .[]' <<<"${required_body}" 2>"${required_stderr}") || required_status=$?
else
required_status=$?
fi
elif [[ "${required_response%%$'\n'*}" =~ ^HTTP/[0-9]+[.][0-9]+[[:space:]]+404[[:space:]] ]]; then
classic_required=
else
required_status=1
fi
if [ "${required_status}" -ne 0 ]; then
rm -f "${required_stderr}"
printf 'could not determine required checks for main: branch-protection read failed; refusing to deploy\n' >&2
exit 1
fi
rules_required=
required_status=0
if required_response=$(gh api "repos/$repo/rules/branches/main" --include 2>"${required_stderr}"); then
# Drop the status line and headers through the first blank line (LF or CRLF).
required_body=${required_response#*$'\n'}
while [[ "${required_body}" == *$'\n'* ]]; do
required_header=${required_body%%$'\n'*}
required_body=${required_body#*$'\n'}
[ -n "${required_header%$'\r'}" ] || break
done
if required_body=$(jq -e 'if type == "array" then . else error("expected a rules array") end' <<<"${required_body}" 2>"${required_stderr}"); then
rules_required=$(jq -r '[.[] | select(.type == "required_status_checks") | .parameters.required_status_checks | if type == "array" then .[] else error("required_status_checks must be an array") end | (.context? // .)] | unique | .[]' <<<"${required_body}" 2>"${required_stderr}") || required_status=$?
else
required_status=$?
fi
elif [[ "${required_response%%$'\n'*}" =~ ^HTTP/[0-9]+[.][0-9]+[[:space:]]+404[[:space:]] ]]; then
rules_required=
else
required_status=1
fi
if [ "${required_status}" -ne 0 ]; then
rm -f "${required_stderr}"
printf 'could not determine required checks for main: rules read failed; refusing to deploy\n' >&2
exit 1
fi
rm -f "${required_stderr}"
required=
while IFS= read -r check; do
[ -n "${check}" ] || continue
if [[ $'\n'"${required}"$'\n' != *$'\n'"${check}"$'\n'* ]]; then
required+="${required:+$'\n'}${check}"
fi
done <<<"${classic_required}"$'\n'"${rules_required}"
if [ -z "${required}" ]; then
printf 'could not determine required checks for main: main publishes no required checks in either source; refusing to deploy\n' >&2
exit 1
fi
check_runs_status=0
check_runs=$(gh api "repos/${repo}/commits/${full_sha}/check-runs?filter=all&per_page=100" --paginate \
--jq '.check_runs[] | [.id, .name, .status, .conclusion] | @tsv') || check_runs_status=$?
if [ "${check_runs_status}" -ne 0 ]; then
printf 'Could not read check runs for %s on %s; refusing to deploy.\n' "${repo}" "${full_sha}" >&2
exit 1
fi
pending=()
while IFS= read -r check; do
[ -n "${check}" ] || continue
newest_id=0
newest_status=
newest_conclusion=
while IFS=$'\t' read -r id name status conclusion; do
[ "${name}" = "${check}" ] || continue
[[ "${id}" =~ ^[0-9]+$ ]] || continue
if [ "${id}" -gt "${newest_id}" ]; then
newest_id=$id
newest_status=$status
newest_conclusion=$conclusion
fi
done <<< "${check_runs}"
if [ "${newest_id}" -eq 0 ]; then
pending+=("${check} (absent)")
elif [ "${newest_status}" != completed ]; then
pending+=("${check} (${newest_status})")
elif [ "${newest_conclusion}" != success ]; then
pending+=("${check} (${newest_conclusion})")
fi
done <<< "${required}"
if [ "${#pending[@]}" -gt 0 ]; then
printf ' %s\n' "${pending[@]}" >&2
printf 'Required checks are absent, pending or unsuccessful on %s; refusing to deploy. Re-run the gates part of this block once CI completes.\n' "${full_sha}" >&2
exit 1
fi
git merge --ff-only "$full_sha"
serving_release=$(readlink -f /srv/overflow/.next || printf absent)
if [ "${serving_release}" != absent ] && [ -f "${serving_release}/REVISION" ]; then
if [ "$(cat "${serving_release}/REVISION")" = "${full_sha}" ]; then
printf 'Already serving %s (%s); the tree fast-forwarded to the serving commit, so install, migrate and build are skipped and the existing release stays.\n' "${serving_release}" "${full_sha}"
exit 0
fi
fi
npm_config_package_import_method=copy pnpm install --frozen-lockfile
. /srv/overflow/scripts/systemd-env-file.sh; export_env_file /etc/overflow/overflow.env
pnpm db:migrate
release=".next-release-$(date -u +%Y%m%dT%H%M%SZ)-$(git rev-parse --short=7 HEAD)"
mkdir "$release"
node scripts/release.ts prepare /srv/overflow "$release"
OVERFLOW_NEXT_DIST_DIR="$release" pnpm build
previous_release=$(readlink -f /srv/overflow/.next)
serving_cache="$previous_release/cache"
test -d "$serving_cache"
find /srv/overflow -path "$serving_cache" -prune -o \
-exec chown -h root:overflow {} +
find /srv/overflow -path "$serving_cache" -prune -o \
! -type l -exec chmod u=rwX,g=rX,o= {} +
mkdir -p "$release/cache"
chown -R overflow:overflow "$release/cache"
chmod -R u=rwX,g=rX,o= "$release/cache"
printf 'Previous build: %s\nNew build: %s\n' "$previous_release" "$release"
pnpm release:switch /srv/overflow "$release" --expect-current "$expected_serving"
systemctl restart overflow.service
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/auth/providers
install -d -m 0700 /var/log/overflow
upgrade_log="/var/log/overflow/webhook-upgrade-$release.jsonl"
upgrade_status=0
pnpm --silent webhooks:upgrade > "$upgrade_log" 2>&1 || upgrade_status=$?
cat "$upgrade_log"
printf 'Webhook upgrade log: %s\nWebhook upgrade exit status: %s\n' "$upgrade_log" "$upgrade_status"
test "$upgrade_status" -eq 0 || exit "$upgrade_status"
printf '%s\n' "$full_sha" > "$release/REVISION"
printf 'Source revision: %s\n' "$full_sha"ONE-TIME dependency migration for existing deployments
Before the first deploy using copy imports, remove the old node_modules and
reinstall it with the copy setting. The package manager otherwise reuses
unchanged packages, leaving their existing hardlinks intact. Run this once in a maintenance window:
removing dependencies can interrupt requests from the running service. After
the install succeeds, run the standing procedure above to build, restore tree
ownership and permissions, restart and verify the service. Do not include this
removal in routine deploys.
set -e
cd /srv/overflow
rm -rf -- node_modules
npm_config_package_import_method=copy pnpm install --frozen-lockfileThis breaks the deployment's links to the shared store; it does not repair ownership or modes already changed in the store or other checkouts.
The generated config belongs to $release and excludes serving and retained
release validators; Next adds the new release's types to that file. Omitting
preparation makes the build fail with the command needed to prepare this release.
The tracked config stays clean, and set -e stops a failed preparation or build
before the switch.
Retain the upgrade log and printed exit status with the deployment record.
REPOSITORY_GONE means the by-ID lookup answered 404 and does not fail the run.
It also covers a repository the sponsor's token cannot see, such as one made
private without that token's access. Rerun pnpm webhooks:upgrade for any
REPOSITORY_GONE line you doubt: a transient 404 during deployment can still
leave a REVISION record, which makes redeploying that SHA skip the upgrade.
A nonzero upgrade leaves the new release serving but the deployment incomplete;
inspect each sanitized failure code, restore sponsor credentials or hook admin
access as needed, and rerun the webhooks:upgrade package script with the same
environment.
An exception behind a failure also writes one stderr line to the same log,
{"repositoryId":…,"failure":…,"cause":…} (no repositoryId for
UPGRADE_FAILED): its class and message on one capped line, with connection
string credentials, password values and GitHub and Overflow token prefixes
redacted. Other token shapes in a message are not redacted, so keep the log private.
subscription: VERIFIED and queue: NOT_NEEDED means the subscription was
unchanged, the credential was already finalized and no earlier run left a repair
owed, so no repair was requested.
subscription: VERIFIED and queue: FAILED is partial completion; a rerun
queues the repair. For a credential not yet finalized the rerun retries the
finalization and its repair. Before updating an already-finalized hook the
upgrade records that the repository owes a repair and clears the record only
once the repair is queued (or the hook proved unchanged with nothing owed), so
the rerun queues it once even when the earlier run's queue write failed or the
forge's response to the update was lost (reported as SUBSCRIPTION_FAILED). A
hook update the forge refused leaves the record too: the next run queues one
full rederivation it may not have needed. Do not report the upgrade as
successful from a restart or startup sweep. Queue acceptance does not establish
that reconciliation has finished; check the worker's outcomes afterwards.
For section 5's one-time migration, run the upgrade/logging lines above after its switch/restart and section 10's HTTP verification. For a first installation or unit migration, run them after section 7's readiness verification with the environment from section 5 loaded. With no active registrations the command is a successful no-op.
The ownership reset keeps code root-owned and group-readable while preserving
the serving cache's Unix permissions. Resolve .next before resetting ownership:
find does not follow the symlink, so its exclusion must name the actual release's
cache. -prune skips that directory and everything inside it in both passes;
chown -h changes symlink ownership without following links, and the mode pass
skips symlinks. The old process can keep writing its cache throughout preparation,
even if preparation or switching stops before the restart. During the one-time
migration the resolved path is the real .next, so the same exclusion preserves
.next/cache. The copy imports described in section 5 keep these resets from
changing package files in the shared store or other checkouts.
The new cache lives at /srv/overflow/$release/cache, so create and hand over
that directory before the switch and restart. The unit still names
/srv/overflow/.next/cache and resolves it through the symlink. A running process
keeps its old writable mount across a swap, but that mount does not bypass Unix
ownership or mode checks; preserving the old cache permissions is required too.
The immediate restart picks up the new release's cache.
Expect active and HTTP 200, then exercise the application and inspect the
journal as in section 7. Only prune after those checks succeed. Prune in the
same shell that ran the deploy block above: the deploy lock is held until that
shell exits, and the prune fence below re-locks fd 9 before touching anything,
so pasting it into a fresh shell fails loudly by design instead of pruning
unserialized. Before pruning,
list the retained directories and confirm the recorded previous release is
among the three greatest names; if it is older, raise --keep enough to include
it or skip pruning. Failed build directories count too, and a rollback can
make the previously served release older than the normal retention window.
The one-time migration has no previous release directory to retain.
set -o pipefail
LC_ALL=C find /srv/overflow -regextype posix-extended -mindepth 1 -maxdepth 1 \
-type d -regex '.*/\.next-release-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{7,40}' \
-printf '%f\n' | LC_ALL=C sort -rflock -w 900 9 || { echo "Could not acquire the deploy lock on /run/overflow-deploy.lock; refusing to deploy. Consult the deploy procedure's serialization notes before re-running." >&2; exit 1; }
pnpm release:prune /srv/overflow --keep 3The release:prune package script takes <tree> [--keep N] and runs
node scripts/release.ts prune <tree> [--keep N]; pass the arguments directly,
without an extra -- separator, for both release package scripts. --keep must
be a positive integer and defaults to 3. Like the listing above, the script
enumerates only real directories directly inside the tree whose names match the
release grammar in section 5, ignoring files, symlinks and all other directory
names, and removes the <release>.tsconfig.json sidecar file beside each
directory it deletes. It then sweeps orphaned sidecars: regular files named
<release>.tsconfig.json at the tree root whose release directory is absent.
It checks no build markers when pruning, so even a failed build with a
matching name counts. It keeps the newest N directory names in descending
lexical order, not by modification time or build success. It also protects the
release .next resolves to and directories needed to resolve its symlink chain,
even outside that N. It prints each removed directory and sidecar, or a no-op
line if nothing was removed. A missing or dangling .next
is reported but protects no release and does not prevent deletion; do not prune
to recover from a failed switch. Pruning knows the symlink target, not which
build a still-running process has loaded, which is another reason to restart
and verify first.
Release-directory builds fix the build's writes under the serving output path;
they do not isolate git pull or dependency installation, which still change the
live checkout and node_modules. Testing on this host found that a same-lockfile
install and a live git checkout did not disturb a running server: all
application entry points are loaded at startup. That does not cover every lazy
internal dependency. With one compiled package removed, a cold /_next/image
request returned HTTP 500 with MODULE_NOT_FOUND and recovered once the package
was restored. These results do not guarantee that dependency changes during an
install are safe. This procedure also retains the restart interruption; it is
not a zero-downtime deployment scheme.
If the revision includes a change to overflow.service, repeat section 6 as well:
git pull updates the copy in the tree, not the one systemd reads. Nothing
here refreshes /root/overflow.service.pre-hardening or the checkout it starts
from, which is what the last paragraph of section 9 is about.
OVERFLOW_TOKEN_ENCRYPTION_KEY seals three kinds of stored credential: each user's
GitHub OAuth token (users.encrypted_oauth_token), each linked forge access
token (user_forge_identities.encrypted_token) and each registration's webhook
secret (registered_repositories.encrypted_webhook_secret). Every value written
in the current format names the key that sealed it and is bound to its own row;
values stored before that format carry neither until they are re-sealed (see
"Re-sealing without changing the key" below).
This section is the only sanctioned way to change OVERFLOW_TOKEN_ENCRYPTION_KEY.
Replacing the value in the environment file any other way leaves every stored
credential sealed under a key the service no longer holds, and each one fails
until it is minted again: sponsors sign in again, forge identities are linked
again, webhook secrets are replaced. For the same reason, never run
pnpm credentials:reencrypt against the production database outside this
procedure. The check-publishing App's private key is a different credential
with its own rotation, which section 14
carries; this section's steps do not extend to it and must not be applied
to it.
The rotation runs while the service keeps serving, and costs two restarts. It
relies on the service reading two keys: OVERFLOW_TOKEN_ENCRYPTION_KEY is the current
key, which seals every new write and opens what it sealed, and the optional
OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS only opens. While both are configured, every
row opens under whichever of the two sealed it.
Run every block as root from a fresh shell. Loading the environment file sets
variables and never unsets them, so a shell that loaded the file before an edit
keeps the values the edit removed. pnpm is on root's PATH and deliberately
not on the service's (section 3).
Do not deploy (section 10) between step 3's edit and step 6's restart. A deploy restarts the service and runs the webhook upgrade with whatever the environment file holds at that moment, or, from a shell loaded earlier, with keys the file no longer holds. Finish or roll back the rotation first.
Before each database command block in this section, run the configuration-loading block in the same shell, including after a key-file edit.
Take a fresh dump first, the way backup-restore.md's drill does. Once the old key is retired, it is the only way back should the new key be lost.
set -e
cd /srv/overflow
bash scripts/db-backup.shThe script prints the dump's path; record it with the rotation. The dump holds
the credentials sealed under the old key, so it restores usefully only together
with that key; step 3 keeps the old key in
/etc/overflow/token-encryption-key.old for that reason. The backup script
deletes overflow-*.dump files older than 14 days (section (d) of
backup-restore.md); copy this dump to a name outside that pattern, in the same
directory, if the rollback window has to outlast that.
The key goes straight into a private file owned by the operator and is never printed, so it stays
out of terminal scrollback. This block and the other guarded edits below run in
a subshell, ( … ), so a refusal ends the subshell with a message and leaves
your shell open:
(
set -e
test ! -e /etc/overflow/token-encryption-key.new || { echo "Refusing: token-encryption-key.new already exists." >&2; exit 1; }
umask 077
node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('base64url'))" > /etc/overflow/token-encryption-key.new
wc -c < /etc/overflow/token-encryption-key.new
)Expect 43: 32 random bytes as unpadded base64url, the only form the service
accepts. The refusal protects a key file left by an interrupted rotation; find
out whether that key was ever configured before removing it.
This block selects OVERFLOW_TOKEN_ENCRYPTION_KEY when that line is present,
otherwise TOKEN_ENCRYPTION_KEY. It keeps that old key in
/etc/overflow/token-encryption-key.old and writes the selected line as
OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS
without changing its value, appends the new key as OVERFLOW_TOKEN_ENCRYPTION_KEY,
confirms both lines read back as the two saved keys, and preserves the
environment file's mode. Run these file edits as its owner, with write access
to its directory; keep section 4's private permissions. The saved copy
inherits the environment file's mode without changing ownership. It refuses,
changing nothing, unless:
- step 2's file holds a 43-character key;
- the environment file ends with a newline, so the appended line cannot be glued onto the last one;
- no line names
TOKEN_ENCRYPTION_KEY_PREVIOUSwith either prefix, comments included. A file copied from.env.examplecarries its commented# OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=line and its explanation; delete both, and any emptyOVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=line, before running the block; - exactly one line names the selected key variable at all, comments included (the
pre-restart rollback counts the same way, so a file this block accepts is one
the rollback accepts), and that line holds the bare value and nothing else: no
quotes, spaces or comment. Rewrite a quoted line as
OVERFLOW_TOKEN_ENCRYPTION_KEY=<value>first; systemd and the shell read the bare form identically; - no
token-encryption-key.oldfrom an earlier rotation is in the way. Such a file belongs with that rotation's dump, so rename it rather than removing it.
(
set -e
key_name=OVERFLOW_TOKEN_ENCRYPTION_KEY
if ! grep -q '^OVERFLOW_TOKEN_ENCRYPTION_KEY=' /etc/overflow/overflow.env; then
key_name=TOKEN_ENCRYPTION_KEY
fi
grep -Eqx '[A-Za-z0-9_-]{43}' /etc/overflow/token-encryption-key.new || { echo "Refusing: token-encryption-key.new does not hold a 43-character key." >&2; exit 1; }
test -z "$(tail -c 1 /etc/overflow/overflow.env)" || { echo "Refusing: overflow.env does not end with a newline." >&2; exit 1; }
test "$(grep -c 'TOKEN_ENCRYPTION_KEY_PREVIOUS' /etc/overflow/overflow.env)" = 0 || { echo "Refusing: overflow.env already names OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS." >&2; exit 1; }
test "$(grep -c "$key_name" /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env must name $key_name on exactly one line." >&2; exit 1; }
grep -Eqx "$key_name=[A-Za-z0-9_-]{43}" /etc/overflow/overflow.env || { echo "Refusing: the $key_name value is not bare." >&2; exit 1; }
test ! -e /etc/overflow/token-encryption-key.old || { echo "Refusing: token-encryption-key.old already exists." >&2; exit 1; }
cp --preserve=mode /etc/overflow/overflow.env /etc/overflow/token-encryption-key.old
sed -n "s/^$key_name=//p" /etc/overflow/overflow.env > /etc/overflow/token-encryption-key.old
sed -i "s/^$key_name=/OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/" /etc/overflow/overflow.env
{ printf 'OVERFLOW_TOKEN_ENCRYPTION_KEY='; cat /etc/overflow/token-encryption-key.new; printf '\n'; } >> /etc/overflow/overflow.env
test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.new)" || { echo "The edited file does not read back as the saved keys; roll back as below." >&2; exit 1; }
test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.old)" || { echo "The edited file does not read back as the saved keys; roll back as below." >&2; exit 1; }
rm /etc/overflow/token-encryption-key.new
)Before restarting, confirm that both values are keys the service will accept. The re-encryption script's check mode reads the same file and writes nothing:
cd /srv/overflow
check_status=0
pnpm --silent credentials:reencrypt --check || check_status=$?
printf 'Check exit status: %s\n' "$check_status"Expect one JSON line for each of the three columns and, while any credential is
stored, exit status 1, because nothing is sealed under the new key yet. (Node's
experimental-feature warning on standard error accompanies every run of the
script.) {"failure":"KEYS_INVALID"} instead means one of the two values is not
a 32-byte base64url key: roll back as below, before the restart. The running
service still holds the old configuration in memory, so nothing has changed
yet.
Then restart, proving the switch the way section 7 does:
set -e
systemctl show overflow.service -p MainPID --value > /run/overflow-preswitch-mainpid
systemctl restart overflow.service
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
--retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness
printf 'MainPID before the switch: %s\nMainPID now: %s\n' \
"$(cat /run/overflow-preswitch-mainpid)" \
"$(systemctl show overflow.service -p MainPID --value)"Expect active, 200 and two different MainPID values. systemd reads the
environment file only when it starts the process, so a MainPID that has not
moved means the old process, holding only the old key, is still serving. From
this restart on, the service seals every new or refreshed credential under the
new key and still opens everything sealed under the old one.
cd /srv/overflow
reencrypt_status=0
pnpm --silent credentials:reencrypt || reencrypt_status=$?
check_status=0
pnpm --silent credentials:reencrypt --check || check_status=$?
printf 'Re-encryption exit status: %s\nCheck exit status: %s\n' "$reencrypt_status" "$check_status"Load the keys from the service's own environment file, as here, and never type
them on the command line. The script seals every row it rewrites under whatever
OVERFLOW_TOKEN_ENCRYPTION_KEY it is given, so a valid key the service does not hold
would re-seal those rows under a key the service cannot open.
The first command opens every stored credential not yet sealed under the
current key, with either key, and seals it again under the current key, bound
to its row. Rows whose credential is NULL are skipped. It prints one line per
column, {"table":…,"column":…,"reencrypted":N,"alreadyCurrent":N,"skipped":N,"failed":N},
plus one line for each row it could not open (step 5), and exits 0 only when no
row failed. skipped counts rows the service rewrote between the script's read
and its write, or whose GitHub user id, forge identity or webhook credential id
(what the seal is bound to) changed in that interval; the script leaves them as
they now are. The script is idempotent and safe to re-run: a second run
rewrites only what is still not current, so run the block again after any
failure or interruption. {"failure":"REENCRYPTION_FAILED"} means the database
could not be read or written; rows already re-sealed stay re-sealed. Its cause,
when present, is the error's code or class, never its message: CONNECT_TIMEOUT
when no database session was ready within five seconds of connecting, or a
PostgreSQL SQLSTATE such as 57P01 (the server terminated the connection). The output
names tables, columns, row ids, counts and that cause, never a credential or a key, so
retain it with the rotation record.
The second command writes nothing. It prints
{"table":…,"column":…,"current":N,"notCurrent":N} for each column and exits 0
only when every stored credential is sealed under the current key. It reads
which key sealed each row; it does not open them. Both statuses 0 means step 6
may follow; a re-encryption status of 1 with UNDECRYPTABLE lines means step 5.
{"table":"<table>","id":"<row id>","failure":"UNDECRYPTABLE"} names a row that
neither configured key opens for that row: it was sealed under some other key,
damaged, or copied from another row. The service could not read it before the
rotation either, so the rotation did not cause it, and the script leaves it
untouched. Each one is repaired by minting the credential again.
users. The user's next GitHub sign-in stores a fresh token. Without
waiting for that, clear the reported row's token:
row_id='REPLACE-WITH-REPORTED-ID'
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update users set encrypted_oauth_token = null where id = '$row_id'"Expect UPDATE 1; UPDATE 0 means no user has that id, so check it against
the reported line. A NULL token reads as "no token" everywhere the service
reads it, and the re-encryption skips it. Until the user signs in again, the
repositories they sponsor do not reconcile ("GitHub access token was not
available."), they cannot register a repository or read its labels, and the
webhook upgrade reports CREDENTIALS_FAILED for their registrations. Each of
those already failed while the token was unreadable. Signing in does not need
the stored token, and it writes a new one.
user_forge_identities. The user's next link of the same forge identity
stores a fresh token in the same row. Without waiting for that, clear the
reported row's token; never delete the row:
row_id='REPLACE-WITH-REPORTED-ID'
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update user_forge_identities set encrypted_token = null where id = '$row_id'"Expect UPDATE 1; UPDATE 0 means no identity has that id. The row keeps its
provider, instance and forge user id, which is what GitLab authorship resolves
through, so the user's GitLab work stays attributed to them; deleting the row
would lose that. A NULL token is never selected for use and the re-encryption
skips it, the same state account deletion leaves. Until the user links the
identity again, everything that needs their GitLab token on that instance
fails as it would with no identity linked: reconciling the GitLab repositories
they sponsor there, registering or unregistering a repository there, reading
its labels, and the webhook upgrade for those registrations. Each of those
already failed while the token was unreadable.
registered_repositories. The sponsor unregisters and registers the
repository again, or you clear the row's credential and let the webhook
upgrade mint and configure a new one. The upgrade mints only for a
registration holding no credential; one still holding an unreadable secret
fails there as well.
row_id='REPLACE-WITH-REPORTED-ID'
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update registered_repositories set webhook_credential_id = null, encrypted_webhook_secret = null, webhook_configured_at = null where id = '$row_id' and unregistered_at is null"Expect UPDATE 1, then run the webhooks:upgrade package script with the same
environment and retain its output, as section 10 describes. The hook rejects
deliveries until the upgrade configures its new secret, as it already did while
the old one was unreadable. UPDATE 0 means the id is not a registered
repository: either it is mistyped, or the repository is unregistered, in which
case the statement under "Webhook secrets of unregistered repositories" below
clears it.
Then run step 4 again. Do not retire the previous key while any row still
fails or while --check exits non-zero: a row it counts as not current may
be one that only the previous key opens.
Only after step 4's --check has exited 0, with the environment file unchanged
since. Remove the previous key, then repeat step 3's restart block with the same
expectations:
(
set -e
sed -i '/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/d' /etc/overflow/overflow.env
)Then, from a fresh shell:
cd /srv/overflow
check_status=0
pnpm --silent credentials:reencrypt --check || check_status=$?
printf 'Check exit status: %s\n' "$check_status"It must print status 0 with "notCurrent":0 on every line. Then exercise the
application and inspect its journal as in section 7. Keep
/etc/overflow/token-encryption-key.old for as long as you keep step 1's dump.
-
Before step 3's restart, the service still runs on the old configuration. If only step 2 ran,
rm /etc/overflow/token-encryption-key.newis all. If step 3's edit ran, drop the new key and give the old one its name back. The block removes the copy step 3 kept only after the file reads back as exactly that old key, on exactly one line that names the key. Otherwise it refuses and keeps the copy, the only clean one: repair the file by hand from/etc/overflow/token-encryption-key.oldbefore restarting anything.( set -e test "$(grep -c 'OVERFLOW_TOKEN_ENCRYPTION_KEY' /etc/overflow/overflow.env)" = 2 && test "$(grep -c '^OVERFLOW_TOKEN_ENCRYPTION_KEY=' /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env does not hold exactly the two key lines step 3 wrote." >&2; exit 1; } test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.old)" || { echo "Refusing: the previous key does not match token-encryption-key.old." >&2; exit 1; } sed -i -e '/^OVERFLOW_TOKEN_ENCRYPTION_KEY=/d' -e 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/OVERFLOW_TOKEN_ENCRYPTION_KEY=/' /etc/overflow/overflow.env test "$(grep -c 'OVERFLOW_TOKEN_ENCRYPTION_KEY' /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env does not name the key exactly once; token-encryption-key.old is kept." >&2; exit 1; } test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.old)" || { echo "Refusing: the key does not match token-encryption-key.old, which is kept." >&2; exit 1; } rm /etc/overflow/token-encryption-key.old )
-
While both keys are configured, from step 3's restart until step 6: swap them, so the old key is current and the new one previous, and restart with step 3's restart block. The block refuses unless the file holds exactly one current and one previous key line and names the key nowhere else; after step 6 there is no previous key to swap in, and swapping a lone key would leave the service with no current key:
( set -e test "$(grep -c 'OVERFLOW_TOKEN_ENCRYPTION_KEY' /etc/overflow/overflow.env)" = 2 && test "$(grep -c '^OVERFLOW_TOKEN_ENCRYPTION_KEY=' /etc/overflow/overflow.env)" = 1 && test "$(grep -c '^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=' /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env does not hold exactly one current and one previous key line." >&2; exit 1; } sed -i -e 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/OVERFLOW_TOKEN_ENCRYPTION_KEY=/' -e t -e 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY=/OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/' /etc/overflow/overflow.env )
Every row stays readable whichever key sealed it, and new writes go back under the old key. To finish, run steps 4 to 6 with the keys in that order; it is the same rotation, back to the old key.
token-encryption-key.oldthen holds the current key, so remove it after that step 6. -
After step 6, the rows are sealed under the new key alone. While the new key is still held, going back is another rotation by this section. If the new key is lost, the step 1 dump together with the old key is the only way back: restore it as section (e.2) of backup-restore.md describes, with
OVERFLOW_TOKEN_ENCRYPTION_KEYset to the old key and no previous key. Everything written since the dump is lost.
Section 9's release rollback is not a key rollback. A build older than the
key-identified credential format reads no OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS and
cannot open any credential written or re-sealed in that format, whichever key
sealed it.
Every credential stored before the release that introduced key identifiers
stays in the old format until something re-seals it. Such a value names no key
and is bound to no row, so it still opens if it is copied into another row, and
--check counts it as not current and exits 1 with no rotation under way. The
service reads both formats, so nothing is broken; the row binding simply does
not cover those rows yet. Nothing re-seals them on its own. Run this pass once,
after that release is deployed.
When: only once the release is settled and no section 9 rollback to a release older than it is intended. An older release accepts only the old format and refuses every re-sealed value. The same is already true, with or without this pass, of every value the new release writes on its own: a sign-in, a forge relink, a registration and a webhook upgrade all store the new format. So a rollback past this release leaves those rows unreadable too. The remedy then is to clear each such row's credential as step 5 does, by id, and have its user sign in again, relink the forge identity, or register the repository again.
How: with no key change and no restart, since the service already holds the key the pass seals under:
- Take step 1's backup. That dump holds the old format, which an older release can still read.
- Run step 4's block. The environment file names only
OVERFLOW_TOKEN_ENCRYPTION_KEY, so the script opens each old-format value with it and re-seals it under the same key, bound to its row. - Repair any row it reports undecryptable as step 5 describes, and run step 4's block again until both statuses are 0.
After that, --check exits 0 until a rotation starts. It reads only which key
sealed each row, so it cannot see a row sealed under the current key but bound
to another row; the service refuses such a value, and step 5's remedy for its
table applies.
Before either query below, run the configuration-loading block in the same shell.
Migration 048_unregistered_webhook_credentials.sql clears the webhook
credential of every repository unregistered before the release that clears it
at unregistration. Before deploying it, count the rows it will clear; the query
only reads:
psql "$OVERFLOW_DATABASE_URL" -tAc "select count(*) from registered_repositories where unregistered_at is not null and webhook_credential_id is not null"Its statement is idempotent. The previous release keeps serving from the
migration until the switch (section 10) and does not clear the credential when
it unregisters, so a repository unregistered in that window, or while a
rolled-back release serves, keeps its secret; pnpm db:migrate never applies
048 a second time. After such a deploy or rollback, run the statement again and
repeat the count, which must then print 0:
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update registered_repositories set webhook_credential_id = null, encrypted_webhook_secret = null, webhook_configured_at = null where unregistered_at is not null and webhook_credential_id is not null"When overflow.service or overflow-backup.service enters the failed state,
systemd starts overflow-alert@<failed unit>.service through that unit's
OnFailure=overflow-alert@%n.service. The template unit runs
/srv/overflow/scripts/overflow-alert.sh with the failed unit's name, and the
script mails the failed unit's current-boot journal tail — the last 200 lines
— to the address in /etc/overflow/alert-recipient.
Submitting the message is not the same as the message arriving, and the
script never treats the first as the second. The local exim daemon answers a
submission it has merely spooled with 250 OK id=<id>, and it writes
Completed in its mainlog whether it handed the message to the smarthost or
dropped it into a mailbox on this machine. Every signal below the SMTP
conversation is therefore green for a message that reached nobody, and a check
that stopped there would log an undelivered alert as delivered. So the
script does not stop at the daemon's answer: it takes the id off the final
250 OK id=, follows that id into /var/log/exim4/mainlog, and calls the
alert delivered only when exim Completed the id and the routing line for
that same id named a transport that leaves this host. A run that cannot show
both has failed, and it says which way in the alert unit's journal: the
submission never reached the daemon, the daemon took it but named no id to
follow, exim wrote a verdict against the id that ends the message for good
(rejected, bounce, blackhole, discarded or Failed), the budget closed
with no Completed behind it, or the message was routed to a transport that
stays on this host. A defer is deliberately not a failure on its own — that
is a temporary failure exim goes on to retry, so the script keeps waiting
through it and a greylist does not manufacture a dead alert. Neither is a
Failed to connect to line, the detail exim writes when the smarthost refuses
the connection (section 53.9): that message is queued for its retry too, and it
counts as a failure only where exim put a ** flag on the same line.
That definition is what the throttle keys on. Each failed unit is throttled to
one message per 30 minutes, and the record is written only after the alert
has actually left the host: a submission that failed, that could not be
followed, and a message exim routed to this machine's own mail spool all leave
no state behind, so the next failure mails again immediately instead of being
silenced for half an hour by a delivery that never happened. A suppressed
repeat is logged to the alert unit's journal and submits no mail, and the
throttle state lives under /run/overflow-alert, cleared at reboot. The alert
unit has no [Install] section and is never enabled: OnFailure= and a
manual systemctl start overflow-alert@<unit> are the only ways it runs.
The recipient file is host configuration, not part of this repository: root
only (root:root 0600), carrying exactly one line — the bare address. It is
never committed and never referenced by value in the repository, and the
script refuses anything but a single address on a single line (missing,
empty, no @, or more than one line each exit 2 without sending), so a
misconfigured file fails loudly in the alert unit's own journal instead of
mailing a broken message.
What that address resolves to is host configuration too, and it is the question this section exists to answer. An address exim can deliver to on this machine is an address nobody is told anything at, so the next subsection is not optional reading: it is where the routing gets established.
The local mail daemon must be running and able to relay: the script submits the message by SMTP to the local exim daemon, which relays it to the recipient. Check the daemon and the recipient file before installing anything:
systemctl is-active exim4
test -s /etc/overflow/alert-recipient && echo "recipient file present"The first must print active and the second must print its confirmation.
Neither says anything about whether the address leaves this host — a running
daemon with a recipient it delivers to its own mailbox satisfies both, and
that is the configuration the Verify step below exists to catch. Establish the
routing first.
If the file does not exist yet, create it with the address and nothing else:
install -d -o root -g root -m 0700 /etc/overflow
[ -e /etc/overflow/alert-recipient ] \
|| install -o root -g root -m 0600 /dev/null /etc/overflow/alert-recipient
printf '%s\n' '<address>' > /etc/overflow/alert-recipient
chown root:root /etc/overflow/alert-recipient
chmod 0600 /etc/overflow/alert-recipientThe script needs no install step — /srv/overflow is a checkout of this
repository at the deployed revision, so each deploy ships it. Install the
three unit files and reload:
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow-alert@.service /etc/systemd/system/
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow.service /etc/systemd/system/
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow-backup.service /etc/systemd/system/
systemctl daemon-reloadRe-copying the two watched units is what replaces the copies section 6 (or a previous deploy) installed; on a host whose installed units already carry local edits, diff before overwriting.
Nothing in this repository configures where the message goes. The routing
is exim's own host configuration under /etc/exim4/, and the change this
section documents does not apply it — what follows is a runbook for the
operator, not a record of something that has been done here. Run it after the
units are installed, and again whenever the address in
/etc/overflow/alert-recipient changes.
exim -bt runs the routers and the address verification and prints the
decision without queueing anything, so it answers this question without
sending a message. Read the address into the argument rather than typing it,
so it stays out of the shell history and out of anything pasted afterwards:
exim -bt "$(cat /etc/overflow/alert-recipient)"
Two shapes come back, and telling them apart is the whole job.
A local mailbox write resolves like this: R: mail4root for <address>, then
<address> -> /var/mail/mail, then transport = address_file. The message
ends in a file on this machine, nobody is told anything, and the script will
report the alert as a failure — correctly, because from the operator's side
that is what it is.
A smarthost resolves like this: R: smarthost for <address>, then
<address>, then router = smarthost, transport = remote_smtp_smarthost,
then host <smarthost> [...] port=587. That transport leaves the host, which
is what the script requires.
The discriminator is transport = on the resolution, not R: on the line
above it. R: names the router that made the decision, and mail4root is
a perfectly good name for a router that then hands the message to a local
delivery agent. Read the transport.
On this host, as measured on 2026-10-01, the alert recipient already leaves
the host. exim -bt above returned router = smarthost, transport = remote_smtp_smarthost with host smtp.gmail.com [...] port=587,
and the retained logs agree:
zcat -f /var/log/exim4/mainlog* | grep -F "$(cat /etc/overflow/alert-recipient)" | grep -oE 'T=[a-z_]+' | sort | uniq -c
zcat -f /var/log/exim4/mainlog* | grep 'T=address_file' | grep -cF "$(cat /etc/overflow/alert-recipient)"Five lines, every one T=remote_smtp_smarthost, and zero local writes to that
address. So the routing decision below is one to re-make, not one to make — run
the two commands again whenever the address changes, and treat a 0 on the
second as the thing to notice.
If the first shape is what comes back, the decision is yours, and it is not the repository's:
-
Route the address off the host. The configuration the Verify step below accepts, and the only one that can page a person.
-
Keep the local mailbox as a retained record. Defensible — a mailbox on the box is a durable record of every failure, and it is one of the reasons this host has an audit trail at all. It carries two obligations you have to accept deliberately rather than discover.
The first is that the script will treat every such alert as a failure and say so in the alert unit's journal, forever. That is the intended reading, not a defect to route around: the record is not the alert. The second follows from the first, and is easier to miss. Because a local write records no throttle state, the 30-minute throttle never engages on this configuration at all — there is no state file for it to read, so every
OnFailure=trigger for the unit mails, and a unit in a crash loop produces a message per failure rather than one per half hour. The service manager's start limit is the only thing bounding that, and it bounds a fast loop only: a unit that trips its start limit inside the interval is a unit that is not starting, and one that cycles more slowly than the window mails once per cycle for as long as it keeps cycling.The mailbox grows either way, and that obligation is not conditional on this decision.
/var/mail/mailon this host is root's own system mail regardless of where the alert recipient routes — every retainedT=address_fileline isR=mail4root ... => /var/mail/mail <root@vmi3458323.contaboserver.net>, and the alert address appears in none of them. Nologrotateentry on this host covers/var/mail/mail, so a retained record is an unbounded one unless you write the rule yourself, whether or not it ever holds an alert:ls -l /var/mail/mail # size grep -c '^From ' /var/mail/mail # message count zcat -f /var/log/exim4/mainlog* | grep -c 'T=address_file' # local writes, all mail
Those counts describe all mail this host has relayed, not the alert path's share of it. Measured 2026-10-01 they read 528 and 7, and they are a snapshot that only climbs as the log fills — which is exactly why the alert path's own routing is answered by the two commands above rather than by either number.
The deployment is not verified until an off-host delivery has been observed, not until one was attempted and the daemon accepted it. The Verify step below is what observes it, and a run that ends any other way is a run this section does not consider a pass.
Both watched units must name the alert template:
systemctl show overflow.service overflow-backup.service -p OnFailureEach line must name the alert template with that unit's own name as the
instance: OnFailure=overflow-alert@overflow.service.service for the web
unit and OnFailure=overflow-alert@overflow-backup.service.service for the
backup unit. systemd replaces the units' %n specifier when it loads them,
so the readback shows the expanded name — including the template's own
.service suffix, doubled next to the instance — not the raw
OnFailure=overflow-alert@%n.service line the repository's unit files
carry.
The alert template must also carry the group that lets the script read the exim mainlog it decides from:
systemctl show overflow-alert@test.service -p SupplementaryGroups --value | grep -qx adm && echo "the alert unit reads the exim log as a member of adm" || echo "the alert unit is NOT in group adm - every alert will wait out its budget, report that it did not leave this host, and record no throttle state"This is not tidiness: the script follows the accepted message id into
/var/log/exim4/mainlog, which is 0640 Debian-exim:adm, and the unit's
CapabilityBoundingSet= is empty — so root has neither CAP_DAC_OVERRIDE nor
CAP_DAC_READ_SEARCH and is refused the open() without this group. A unit
missing it mutes the path while making it look broken.
Then send a real message through the whole route with a throwaway instance —
the instance name need not be a unit that exists, and the message's subject
names it: [overflow] test failed on <host>. Mark where this run starts before
you start it — started=$(date -u '+%Y-%m-%d %H:%M:%S') — and keep it in the
same shell, because the id below is read from this run's lines only and an
earlier run's id is a worse mistake than no id at all.
systemctl start overflow-alert@test.service
journalctl -u overflow-alert@test.service --no-pager -n 20
tail -n 20 /var/log/exim4/mainlogThe script's own line is the verdict, and it is one of two shapes. A delivered
alert reads overflow-alert.sh: exim routed <id> to <transport> and Completed it; the alert left this host, and the alert unit exits 0. Anything else
reads overflow-alert.sh: <reason>; the send is NOT recorded, so the next failure for test.service alerts again, and the alert unit exits nonzero —
systemctl show overflow-alert@test.service -p ExecMainStatus reads 0 on the
first shape and something else on the second. Read that status in one
direction only: nonzero is a failure, and zero is not a delivery. A run the
throttle suppressed exits 0 having sent nothing, so a status of 0 alongside
a journal that never carried a verdict is a suppressed run, not a delivered
alert — the journal line is the verdict and the status only confirms it. The
reason names which failure it was: the submission never reached the daemon, the
daemon named no id to follow, a verdict exim recorded against the id, a local
transport the message was routed to, no Completed for the id inside the
60-second budget (with or without a verdict written against it), or a mainlog
that could not be read at all. A run that ends in the second shape has
failed, whatever the SMTP conversation looked like, and the second shape's own
wording is the finding — read it rather than re-running.
That tail is for seeing the shape of the traffic, not for finding your
delivery. Match on the id, never on the word Completed. The mainlog holds
every message this host has relayed, and on a host whose recipient resolved to
a local mailbox the alert path's own traffic fills it with Completed lines
that mean a file write — the last twenty lines will usually hold several, and
browsing them for one that looks like a delivery will find a file write and
call it a pass.
<id> is the handle onto this send and onto nothing else. Take it from this
run's journal lines rather than typing it —
journalctl -u overflow-alert@test.service --no-pager --since "$started" -n 20 | grep -oE '[0-9A-Za-z]{6}-[0-9A-Za-z]{6,}-[0-9A-Za-z]{4,}' | tail -n 1
— and substitute it for <id> in the three commands that follow. They are the
whole verification, and each one discriminates on your id and nothing else.
grep -F "<id> Completed" /var/log/exim4/mainlog prints the Completed line
for your message, or nothing.
grep -F "<id> =>" /var/log/exim4/mainlog prints your routing line verbatim,
R=<router> T=<transport>, or nothing.
awk -v id="<id>" '{at=index($0,id); if(at==0) next; a=index($0,"=>"); if(a==0||a<=at) next; r=substr($0,a+2); if(match(r,/T=[^[:space:]]+/)) t=substr(r,RSTART+2,RLENGTH-2)} END{if(t!="") print t; else print "(no routing line)"}' /var/log/exim4/mainlog
reduces that line to the single word the script judges on — the last routing
line for the id, which is the current decision on a message that deferred and
was re-routed. This is the same extraction the script itself performs.
If that prints nothing, stop there. There is no send to follow, and every one of
the three commands then runs against an empty <id> — the first grep matches
any Completed line in the log, the second any routing line, and the awk
reduces whichever of those carries a T=, so all three would "pass" on some
earlier, unrelated message. That is the exact false green this step exists to
end, and --since is what makes the empty answer trustworthy: it means this
run named no id, not that the window was too narrow. Two outcomes produce it,
and they want different responses. A refused submission reports a reason and
never names an id — nothing was sent, so there is nothing to follow and the
reason is the whole finding. A suppressed run logs one line, last alert for <unit> was <age> seconds ago, inside the 1800-second throttle window; suppressing, and that line carries no id either — which is the second start
described below, and it means this run deliberately sent nothing. Read which of
the two you are looking at before you conclude anything. If more than one id
comes back, the marker was captured inside an earlier run's second: mark again
and re-run.
All three, and only all three, is a pass: a Completed line for your id, a
routing line for your id, and a transport that leaves the host. The Completed
line on its own carries no T= at all and cannot say where the message went;
the routing line on its own is a decision exim had not yet carried out. The
third command's word is the verdict, and the list of transports the script
treats as staying on this machine is a maintenance obligation kept in the
script rather than restated here, so read it from the deployed copy with
sed -n '/^leaves_host()/,/^}/p' /srv/overflow/scripts/overflow-alert.sh.
Anything on that list — address_file is what a local mailbox write resolves
to — is a failed alert, and so is the (no routing line) case. When the
transport is remote_smtp_smarthost or another name not on the list, the
alert left the host.
A fourth check, and it is the one that catches a message the relay refused.
Exim writes Completed when the daemon is finished with a message, which
includes one it gave up on: a bounce, a rejection and a discard all end with the
same line a successful delivery does. So all three commands above can pass on a
message that was permanently refused, and the script — which reads a terminal
verdict before it reads Completed, for exactly that reason — will have called
it a failure. Check for one with grep -F "<id> **" /var/log/exim4/mainlog and
grep -F "<id> Failed" /var/log/exim4/mainlog. Either printing nothing is what
a pass requires. Anything they print is a verdict exim recorded against that
id, and it is the finding — rejected, bounce, blackhole, discarded or
Failed all end the message. There are two exceptions, and both mean the
message is queued and retrying rather than finished: ** defer, a greylist or
a temporary 4xx; and Failed to connect to, the detail line exim writes when a
smarthost refuses the connection (section 53.9 of the exim specification),
written before the deferral for the same id. The script keeps waiting through
both, and so should you — and the second counts as an exception only on a line
carrying no two-character flag. Where exim rejected the message and the
rejection reason happens to quote a refused connection
(** rejected: … 550 Failed to connect to …), the flag wins and the message is
finished.
Before you read an absent verdict as a broken relay, check there was a log to
read. /var/log/exim4/mainlog is rotated daily by
/etc/logrotate.d/exim4-base, and that rule carries nocreate: rotation
moves the live file to mainlog.1 and does not leave an empty one in its
place. The daemon puts it back the next time it has something to log, so on a
host whose mail goes quiet the file is simply absent — for a day, or for as
long as nothing is delivered — and every command above then reports nothing at
all. No Completed line, no routing line, (no routing line) from the awk:
an empty verdict that is evidence about nothing at all, including the relay.
This is not a rare race on this host. The previous rotation, at 00:08, was
followed by enough traffic to fill the file again by 23:38; the next one, at
00:38 the following morning, was not, and the file was still absent when this
section was written. A rotation with nothing after it leaves the verdict
unreadable until the next message, not until the next rotation.
The script fails closed on an unreadable log, which is the right direction to
fail: it takes a branch of its own and names the path in its verdict —
/var/log/exim4/mainlog is missing or unreadable, so no off-host acceptance of <id> could be observed within 60s — so the alert unit exits nonzero having
reported no verdict, which is a different report from a verdict of failure.
A failed alert unit is therefore not by itself evidence that the relay is
broken; on this host, at the time of writing, a failed alert unit with that
verdict is exactly the absent log and nothing else. A verdict that names the
log is telling you where to look, not what broke.
So check the file before drawing any conclusion from the absence of a verdict —
ls -l /var/log/exim4/mainlog — and if it has rotated out from under you, the
answer is in the rotation, which zcat -f /var/log/exim4/mainlog* reads across
the retained files. Substitute that in for /var/log/exim4/mainlog in the three
commands above and your id resolves against the same rules, because the same
rules are what the script applied. A run whose verdict you can only find in
mainlog.1 was a real run judged late, and the delivery or the failure recorded
there is the real one.
The throwaway instance leaves one file behind, and which file it leaves is
itself a check: an alert that left the host writes its throttle state under
/run/overflow-alert, and one that did not leaves no state at all. So
ls -l /run/overflow-alert showing a state file for test.service is
independent evidence of a delivery, and it is the one that catches a run whose
journal reads like a pass and whose mailbox never grew. The absence of one
after a run the journal called delivered is a contradiction — chase it rather
than tidy it away.
A second start of the same throwaway instance within half an hour mails
nothing — but only if the first one left throttle state behind, and that is
exactly what a failed delivery does not do. Suppression reads a state file
under /run/overflow-alert, and the delivered path is the only one that writes
one, so on a host whose recipient resolves to a local mailbox there is no file
to read and the second start is not suppressed at all: it runs the whole
verdict again, waits out its own 60-second budget, and appends a second verdict
line to the journal and a second write to /var/mail/mail. Two starts, two
verdicts, two writes — and the state file still absent, which is the check
above telling you the same thing twice.
When the first run did leave state, the second start is silent: the script logs
last alert for test.service was <age> seconds ago, inside the 1800-second throttle window; suppressing and submits no mail, and that line carries no
message id, so the id extraction above comes back empty and the guard there
applies. That is the only case in which a second start is expected to be
silent, and it is the one case where the state file's presence is not news.
Remove the instance's state file under /run/overflow-alert to send again
immediately.
What this step does not check, stated so it is not assumed: the trigger. It
proves the delivery half — that when the alert script runs, a message leaves
this host. It does not prove that anything makes it run. A unit that cannot
serve does not exit, so it never trips the OnFailure= condition the template
follows, and nothing else on this host observes that state either: a wedged
overflow.service is silent in exactly the same way a healthy one is until
something asks it a question. So a clean run through every check above is a
statement about the alert path only. The half this repository does not close is
noted in the issue, and until it is, treat "the alert path verified" as
distinct from "a failure would have been reported".
Revert the two OnFailure= lines — re-copy the units without them, or edit
the installed copies — remove the alert template, and reload:
rm /etc/systemd/system/overflow-alert@.service
systemctl daemon-reloadThe watched units work unchanged without the wiring: nothing else references
the alert unit, and a failure of overflow.service or
overflow-backup.service is visible in the journal.
Everything above is built around one route leaving the host:
overflow-alert.sh submits to the local exim daemon, exim relays to the
smarthost, the smarthost delivers to one mailbox — and the subsection above is
what establishes that it does. Nothing on that route reports its own failure,
so a route that stopped delivering looks exactly like a host with nothing to
report. When enabled, overflow-canary.timer starts overflow-canary.service
daily at 03:20 UTC to check that route.
On the documented deployment the timer is disabled by maintainer decision, because the daily mail was noise; the maintainer accepted the consequence. The scheduled check does not run, so a silent mail-path failure is not checked automatically by the canary. The unit files still ship, and the install and verification instructions below remain available for an operator who chooses to restore that check.
With the canary installed and its host files configured, run one check by hand and inspect its verdict:
systemctl start overflow-canary.service
journalctl -u overflow-canary.service --no-pager -n 20To resume the daily check, run systemctl enable --now overflow-canary.timer,
then systemctl list-timers overflow-canary.timer --no-pager and
systemctl is-enabled overflow-canary.timer. The persistent timer can also
catch up a missed check when activated. The install block below enables the
timer too; running it on the documented deployment ends the suspension.
The canary runs the same script and the same route, then reads the verdict
from somewhere the route cannot fake: it submits with curl -v, takes the
exim message id off the final 250 OK id= line, and waits for a line reading
<id> Completed in /var/log/exim4/mainlog. A relay that defers, rejects,
bounces or discards the message logs defer, rejected, bounce,
blackhole or discarded against that id instead, and the script names that
verdict in the report. A Failed to connect to line is not one of those: it is
the detail a refused connection leaves behind (section 53.9), the message is
still queued, and the script waits it out rather than calling it dead — unless
exim put a ** on that same line, which makes it a verdict again.
The absence of Completed is the failure, and it
is the only observation on this host that can see the smarthost leg at all.
It reads that log as a member of adm, and cannot read it at all without
that membership. /var/log/exim4/mainlog is 0640 Debian-exim:adm in a
2750 Debian-exim:adm directory, and the canary unit runs with
CapabilityBoundingSet= empty — the same hardening the alert template beside
it uses, and for the same reason it is not merely safe there. An empty bounding
set strips CAP_DAC_OVERRIDE, which is the only reason a root process could
open a file whose group it did not hold; ProtectSystem=strict is not what
decides this, because it governs the mount namespace and not file access. The
alert script reads that same mainlog to decide whether an alert left the host,
so the alert template carries SupplementaryGroups=adm too, pinned to adm in
tests/deploy/alert-units.test.ts for the reason given below. Both units
therefore carry the directive: the ordinary group mechanism, no capability, and
the group this host's own log files are already owned by. It is wider than one
file — group adm also owns the nginx, postgresql, fail2ban, cloud-init, tor,
privoxy and tinyproxy logs here — so it is pinned to adm in both unit
tests rather than admitted by name with its value left open.
If that membership is ever wrong, the canary says so instead of blaming the
relay. Before it submits anything the script checks that it can read the
log, and if it cannot, it refuses: exit 2, no message sent, and a report whose
header is "the canary on <host> cannot run" — never the outage header
"the failure-alert path is not delivering", which would be a false claim
about a relay that may be perfectly healthy. The same check runs again at the
deadline, so a log that goes unreadable during the wait is refused the same
way rather than reported as a relay that timed out.
The refusal has its own dedup state, /run/overflow-canary/canary-fault,
separate from the dead-streak marker. That separation is deliberate in both
directions: the fault marker keeps a canary broken for a week to one report
rather than seven, and the dead-streak marker is never written by a
refusal, because it means "a real outage of the alert path was reported" and
a run that never reached a verdict is not one — writing it there silences the
next real outage, which is the failure this whole feature exists to prevent.
Nothing on this host reads the canary's exit status, which is why a refusal
posts at all: a canary that quietly stopped checking would be, one level up,
the same silent route this section exists to end. A healthy run retires the
fault marker, and so does a run that reports a real outage after a stretch of
blindness.
That distinction is the whole point: without it, a canary lacking the
membership and capability to open the log spends its full 60-second budget
proving it and then reports that the relay never completed a message exim had
logged as Completed, a false page, on the one channel the maintainer
trusts, about a healthy host.
What Completed means, precisely — and what it does not. Where the
message was routed off the host, it means the smarthost accepted the
message. That covers the classes worth naming: a stopped daemon (curl fails
outright), revoked SMTP credentials, a daily limit, greylisting, and any 5xx
at RCPT or DATA — all of which leave a defer or rejected line and never a
Completed. What it does not cover is the most likely way a mail alert
path quietly dies: a recipient address that is wrong, deleted or converted to
a route that silently accepts.
That case is not detectable from here, and pretending otherwise would be
worse than saying so. Measured against the real smarthost and an RFC 2606
reserved domain, so that delivery is impossible by construction, the relay
answers 250 2.0.0 OK … - gsmtp, exim logs Completed, and the canary
reports the path healthy, which is correct for what it can see. Once the
message has left the queue there is no observation available through the
local exim that distinguishes "the smarthost accepted it" from "the remote
mailbox exists" — exim -Mvb inspects queued messages, and such a
message is long gone by then.
So the detector for a dead mailbox is the heartbeat itself stopping arriving, and it is the operator's job, not the script's. This canary proves the pipe is open; it cannot prove there is a person at the other end. Keep the recipient address somewhere you will notice it is stale, and treat a run of missing canary mail as a finding in its own right — the same weight as a failed backup.
One limitation belongs in this subsection rather than in a changelog, because
this is where a reader decides what the canary is worth. Completed says
the daemon finished with the message and carries no transport, so a canary
addressed to a local mailbox reports a healthy path for a message that never
left this host. scripts/overflow-canary.sh does not read the routing line
and does not distinguish a local write; scripts/overflow-alert.sh does, and
reports the alert as a failure instead. The alert path proves the message
left the host. The canary does not. Nothing in this change alters the canary,
and a green canary run is not evidence about where its own message went — the
routing proof for the canary is the exim -bt check above, run against
/etc/overflow/canary-recipient.
A dead path is reported out of band, to the Discord webhook in
/etc/overflow/canary-discord-webhook — a channel that shares no leg with the
route being checked — and the oneshot exits nonzero, so the unit also lands in
the failed state. One report per dead streak: the marker under
/run/overflow-canary records an outage that has been reported, so a path dead
for a week posts once rather than seven times, and a run that completes clears
the marker so the next failure posts again. The marker is written only once a
report has actually been delivered, so an unreachable webhook leaves the next
run free to try.
Both host files are configuration, like alert-recipient, and are never
committed or referenced by value here. The recipient file is validated exactly
as alert-recipient is — one line, one address, no CR — and the webhook file
must be a single nonempty line. A file that fails either check exits 2 and
sends nothing, so a misconfiguration is loud in the canary's own journal
rather than a canary quietly reporting health through a channel it cannot
reach. The exim mainlog is checked before the submission for the same reason,
and a log this process cannot read yields no verdict — which is reported as
the canary being broken, never as the relay failing.
systemctl is-active exim4
test -s /etc/overflow/canary-recipient && echo "recipient file present"
test -s /etc/overflow/canary-discord-webhook && echo "webhook file present"
systemctl show overflow-canary.service -p SupplementaryGroups --value | grep -qx adm && echo "canary reads the exim log as a member of adm" || echo "the canary is NOT in group adm - every run will refuse with the log named as unreadable"The second, third and fourth commands must each print their confirmation. The
fourth is an exact match on purpose: --value prints the bare adm on
this host and prints an empty line with exit 0 when the directive is unset,
so grep -qx adm fails closed in both cases, and it also fails on a drifted
installed copy that has picked up a second group — which -w would have
passed. If the fourth prints its warning, the installed unit is not the one in
this repository: reinstall it from
/srv/overflow/deploy/overflow-canary.service rather than editing the
installed copy, because the reviewed set in
tests/deploy/canary-units.test.ts is what keeps the next directive beside
this one from arriving unnoticed. Create the files with the address and the
webhook URL and nothing else:
install -d -o root -g root -m 0700 /etc/overflow
install -o root -g root -m 0600 /dev/null /etc/overflow/canary-recipient
printf '%s\n' '<address>' > /etc/overflow/canary-recipient
chown root:root /etc/overflow/canary-recipient
chmod 0600 /etc/overflow/canary-recipient
install -o root -g root -m 0600 /dev/null /etc/overflow/canary-discord-webhook
printf '%s\n' '<webhook-url>' > /etc/overflow/canary-discord-webhook
chown root:root /etc/overflow/canary-discord-webhook
chmod 0600 /etc/overflow/canary-discord-webhookThe script needs no install step — /srv/overflow is a checkout of this
repository at the deployed revision, so each deploy ships it. Note that
scripts/deploy-revision.sh does not reinstall units, so a host that has
already been deployed needs these two copies run by hand. Install the two
files, reload, and enable the timer:
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow-canary.service /etc/systemd/system/
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow-canary.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now overflow-canary.timer
systemctl list-timers overflow-canary.timer --no-pagerFirst, prove the real webhook is live. Every other check in this section either never posts to it or posts to a loopback stand-in, so without this one nothing in the procedure establishes that the configured Discord webhook still exists and still accepts posts — and a canary reporting into a deleted webhook fails in silence, which is the failure mode the whole section is about.
The URL is read from the host file into a shell variable and is never echoed,
printed or pasted; only the HTTP status leaves this block. --fail makes a
404 Unknown Webhook or a revoked token a nonzero exit, which is the point:
without it curl would call a refusal a success.
webhook=$(cat /etc/overflow/canary-discord-webhook)
printf '%s' '{"content":"[overflow-canary] section 12 verification post - the failure-alert canary is being installed and this webhook is live."}' | curl -sS --fail --max-time 15 --connect-timeout 5 -H 'Content-Type: application/json' --data-binary @- "$webhook" -o /dev/null -w 'webhook answered %{http_code}\n'Expect webhook answered 204, and a message saying the canary is being
installed must appear in that channel — that is the confirmation, and it is
what a delivered report looks like.
Scope this honestly, because it is narrower than it looks: it proves the
webhook URL is live and accepting posts, not that the alert recipient
mailbox exists — see what Completed does and does not mean above. And it
sends one real message, so run it once, not on a loop.
Then prove both directions of the mail path, healthy first and then broken.
Never send a burst at the address in /etc/overflow/alert-recipient to do
this.
A healthy run must exit zero and leave no marker:
systemctl start overflow-canary.service
journalctl -u overflow-canary.service --no-pager -n 20
tail -n 5 /var/log/exim4/mainlogThe journal must name the relay's Completed line for the message id it
submitted, and the exim mainlog must carry a Completed line for that same
id. Both are about the id, and the id is what makes them evidence — a
Completed line found by browsing the log belongs to some other message. This
says nothing about whether the message left the host; the limitation above is
the whole of that, and the exim -bt check there is what covers it. Confirm
the unit did not record an outage, and that the timer is armed for the next
03:20 UTC:
test ! -e /run/overflow-canary/dead && echo "no outage recorded"
systemctl list-timers overflow-canary.timer --no-pagerIf a scheduled run ever reports the path dead and the mail route looks
fine, check for a log rotation before you believe it — and read the journal
line before you conclude anything from it, because one rotation can produce
two different reports. The verdict is a single-file read of
/var/log/exim4/mainlog, and this host rotates it with nocreate (see
/etc/logrotate.d/exim4-base), so the new file is created by exim itself when
it next reopens the path. Which of the two you get depends on when exim gets
to it:
- The path is recreated before the wait closes — the common case, and the
one that reads as an outage. The
Completedline is sitting inmainlog.1, the newmainlogis empty but perfectly readable, so the run finds noCompleted, exits 1 and posts the not delivering report. That is a false page, and it fails closed: a rotation can produce a false dead verdict but never a false healthy one. - The path is still absent when the 60 seconds run out — a narrow window
between the rename and exim's reopen. The canary cannot read a file that
does not exist, so it refuses instead: exit 2 and the "the canary on
<host>cannot run" report, never a claim about the relay.
So the journal line is what tells you which you are looking at, and a line naming the log as unreadable is a rotation or a membership that has gone — not an outage to act on — while a line naming a relay verdict is a real one, or a rotation to rule out first. It is cheap, it is rare, and the alternative is chasing a relay that is working:
ls -l --time-style=full-iso /var/log/exim4/mainlog /var/log/exim4/mainlog.1Then prove the dead direction, which is the one the canary exists for. Take a
throwaway copy of the script, point its OVERFLOW_CANARY_SMTP_URL at a
closed port, and run it. The port below is 127.0.0.1:1, which nothing
listens on; check yours with ss -ltn first, because a port that answers
proves nothing. The run must post to the webhook and exit nonzero:
cp /srv/overflow/scripts/overflow-canary.sh /tmp/canary-probe.sh
chmod +x /tmp/canary-probe.sh
OVERFLOW_CANARY_SMTP_URL=smtp://127.0.0.1:1 \
OVERFLOW_CANARY_STATE_DIR=/tmp/canary-probe-state \
/bin/sh /tmp/canary-probe.shA nonzero exit is the pass condition here, and the report must have arrived in the channel the webhook file names. A second identical run posts nothing, because the probe's marker records the outage:
OVERFLOW_CANARY_SMTP_URL=smtp://127.0.0.1:1 \
OVERFLOW_CANARY_STATE_DIR=/tmp/canary-probe-state \
/bin/sh /tmp/canary-probe.shRemove the probe and its state, then confirm the real unit is untouched:
rm /tmp/canary-probe.sh
rm -rf /tmp/canary-probe-state
systemctl show overflow-canary.service -p ExecStart -p EnvironmentThe two steps above each run one thing: the first drives the real unit, the
second runs a copy of the script with nothing between it and the host. The
only one that exercises the webhook POST from inside
overflow-canary.service's sandbox is neither of them. That matters more
here than it does for the alert template, because the alert unit only ever
talks to 127.0.0.1 — every socket it opens is loopback — whereas the
canary has to resolve discord.com and complete a TLS connection under
ProtectSystem=strict, PrivateTmp, PrivateDevices,
RestrictAddressFamilies and SystemCallFilter=@system-service.
systemd-analyze verify is a parser, not a runtime, and it will happily
accept a unit whose seccomp filter or address-family mask blocks the egress.
So prove it at runtime. PrivateTmp gives the unit a private /tmp and
/var/tmp but does not isolate the loopback interface, so a listener started
outside the sandbox on 127.0.0.1 is reachable from inside it. The step
below forces the dead path (SMTP to a closed port) and points the webhook at
that listener, so the unit posts out of band and records the outage only if
the sandbox permits the connection. This proves the network leg of the
sandbox; it does not send a real Discord message, and it says nothing about
whether the webhook URL in /etc/overflow/canary-discord-webhook is still
valid, nor about DNS or TLS to discord.com — a loopback listener is
neither of those. The step that covers the real channel is the first one
under Verify, which posts to the configured webhook itself.
The verdict below is the payload the listener actually received. A
listener that has not finished binding when the unit posts makes the report
fail for a reason that has nothing to do with the sandbox, and a step that
reports its finding when the finding is absent is worse than no step: it
teaches the operator to distrust the one check that would have caught a real
regression. The bounded wait below is what keeps the two apart — and nothing
in this step reads /run/overflow-canary/dead, so there is no second signal
to fall back on: the recorded payload is the whole verdict, which is why the
wait has to be trustworthy rather than merely present.
Check the port is free, then install a drop-in that redirects the run. It overrides the two paths, so the host's own webhook file is never read, written or exposed:
ss -ltn | grep 18099 && echo "port 18099 is BUSY - stop, pick another port and re-run this step" || echo "port 18099 is free"
install -d -o root -g root -m 0755 /etc/systemd/system/overflow-canary.service.d
printf '%s\n' 'http://127.0.0.1:18099/probe' > /etc/overflow/canary-sandbox-probe-webhook
rm -f /etc/overflow/canary-sandbox-probe-received
printf '%s\n' '[Service]' 'Environment=OVERFLOW_CANARY_SMTP_URL=smtp://127.0.0.1:1' 'Environment=OVERFLOW_CANARY_WEBHOOK_FILE=/etc/overflow/canary-sandbox-probe-webhook' > /etc/systemd/system/overflow-canary.service.d/sandbox-probe.conf
systemctl daemon-reloadThe first line is a precondition, not a report. If it prints BUSY, stop
here: the listener below would die on EADDRINUSE without saying so, and
the unit's report would be posted to whatever stranger holds that port.
Start the listener, then wait for it to be listening before starting the
unit. The wait is bounded, and deliberately its own block: an unbounded
wait cannot tell "not bound yet" from "died a minute ago", and a step that
leaves the operator watching a silent spin with a systemctl start queued
behind it fails in the same unreadable way this whole section exists to
prevent. Fifteen attempts is roughly sixty times the bind time measured on
this host, so it cannot reintroduce the race a bare start would have:
python3 -c "import http.server as h;H=type('H',(h.BaseHTTPRequestHandler,),{'do_POST':lambda s:(open('/etc/overflow/canary-sandbox-probe-received','ab').write(s.rfile.read(int(s.headers['Content-Length']))),s.send_response(200),s.end_headers()),'log_message':lambda *a:None});h.HTTPServer(('127.0.0.1',18099),H).serve_forever()" &for _ in $(seq 1 15) ; do sleep 1 ; ss -ltn | grep -q 18099 && break ; done
ss -ltn | grep -q 18099 || { echo "the listener did not bind 127.0.0.1:18099 after 15 attempts (about 16s) - stop and read the python error above" ; false ; }The loop sleeps first and checks second, so its last act is a check rather
than a sleep and it stops the moment the port answers. The bound it
enforces is fifteen one-second waits, which measures 15.8-15.9 s here — the
message names that, not a bare "15s", because sleep 1 is a floor and the
ss in each iteration costs a few milliseconds on top of it.
If that block prints its message, or exits nonzero, stop there and do not run the next block — the listener is not up, and the run that follows would report a sandbox failure that does not exist. Read the error the listener printed on the terminal that started it: a traceback there names a missing module, a bad one-liner or a permission problem on the write path, none of which the port check above can see. Then start the unit and read the verdict:
systemctl start overflow-canary.service
journalctl -u overflow-canary.service --no-pager -n 20
grep -q failure-alert /etc/overflow/canary-sandbox-probe-received && echo "sandbox reached the out-of-band channel" || echo "the listener received no report - see the journal above"failure-alert path is the canary's own text inside the report body, so this
line is true only when the unit's own payload arrived. No such line means one
of two things, and the journal above distinguishes them: a report that could
not be delivered at all is the egress being blocked — the finding this step
exists to surface — while a report that was delivered and a listener that
never received it means the probe itself was misrun.
Remove the drop-in, the probe files and the (now empty) drop-in directory, reload, and confirm the unit is back to running its own defaults:
rm /etc/systemd/system/overflow-canary.service.d/sandbox-probe.conf
rmdir /etc/systemd/system/overflow-canary.service.d
rm /etc/overflow/canary-sandbox-probe-webhook
rm -f /etc/overflow/canary-sandbox-probe-received
systemctl daemon-reload
systemctl show overflow-canary.service -p Environmentrmdir refuses to remove a directory that still has anything in it, so it
cannot take a drop-in another procedure added; if it reports the directory
as not empty, look before removing it by hand.
Stop the listener from the shell that started it, with the job's own control —
jobs to list it, kill %1 to end it — once the journal line above has been
read. If the unit recorded an outage, or refused because it could not read
the exim log, clear both state files so the first scheduled run is judged on
its own merits. canary-fault matters as much as dead here: a canary that
was refusing leaves it behind, and a run that finds it will refuse again and
post nothing — the silence this section exists to end, arriving immediately
after a procedure whose stated purpose is to un-silence it.
rm -f /run/overflow-canary/dead /run/overflow-canary/canary-faultDisable the timer and remove both files. The alert path itself is unchanged by them, so nothing else needs reverting:
systemctl disable --now overflow-canary.timer
rm /etc/systemd/system/overflow-canary.timer
rm /etc/systemd/system/overflow-canary.service
systemctl daemon-reloadThe two host files may stay: nothing reads them once the canary unit is gone,
and /etc/overflow/canary-recipient is the address a future canary would
use.
Everything above watches the alert route from the sending side: the alert
template mails a journal tail when a watched unit fails, and the canary proves
once a day that the relay accepts a message. What nothing on that side can see
is the route's own mail coming back. When a message on the alert route cannot
be delivered, the relay's mailer daemon mails a delivery-failure notification
back to the envelope sender — overflow-canary@<fqdn> or
overflow-alert@<fqdn> — and until those addresses are aliased they are
unrouteable, so exim refuses the notification outright and every delivery
failure on the alert route vanishes without a trace (verified on the host with
exim4 -bt). overflow-bounce.service, started every 15 minutes by
overflow-bounce.timer, is the reporting leg for that mail: a root-run oneshot
that tails root's mail spool from a persisted byte offset and forwards each new
bounce to the out-of-band Discord webhook the canary already posts to.
With the canary plus this watcher the route distinguishes all three outcomes it
can have. Off-host accepted: silence — the canary's Completed line and no
watcher report. Local write: the watcher's Class B report, the class
measured at 183 local writes against 5 off-host — a non-DSN message whose
From header local part is overflow-alert or overflow-canary is an alert
that resolved to this host, and a successful local write produces no bounce, so
this class produces no other signal anywhere on the host. Remote failure:
the watcher's Class A report — a delivery-failure notification that references
an overflow address.
The acceptance leg is host configuration, and it comes first: /etc/aliases
must gain overflow-canary: root and overflow-alert: root, then
newaliases. Without it exim refuses the bounces at RCPT and the watcher has
nothing to read. The obvious cheaper design — piping those two aliases
straight at the webhook — is rejected on host evidence: Debian's
address_pipe transport sets no user=, so an alias pipe runs as
Debian-exim, which cannot read the root-only webhook file. The
spool-watcher shape keeps the webhook root-only: root reads the spool, root
posts the report, and no file's permissions move.
An IMAP or POP probe of the recipient mailbox is the definitive remote-mailbox
check — the only option that observes the mailbox itself — and it needs a Gmail
app-password credential that does not exist on this host, so its cost is one new
third-party credential plus one poll per canary interval; it is deliberately not
implemented. Piping the aliases at the webhook is rejected above, on the
address_pipe evidence. Doing nothing is rejected because both of its detectors
already fail: the heartbeat-stops detector is human-in-the-loop (the canary's own
closing paragraphs), and the local-write class produces no heartbeat gap at all.
The daemon and the webhook file are the canary's own prerequisites; the two aliases must not be present yet:
systemctl is-active exim4
test -s /etc/overflow/canary-discord-webhook && echo "webhook file present"
grep -q '^overflow-canary:' /etc/aliases && echo "overflow-canary alias already present" || echo "overflow-canary alias absent"
grep -q '^overflow-alert:' /etc/aliases && echo "overflow-alert alias already present" || echo "overflow-alert alias absent"The first must answer active, the second must print its confirmation, and
each alias line must print absent. An alias already present is a decision,
not a step: an existing entry wins at alias resolution, so appending the line
this section installs would change nothing and the verification below would
fail in a way that looks like a broken watcher. Resolve the existing entry
before installing.
Unlike the canary unit, the bounce unit carries one capability. The spool it
tails is delivery state owned by the MTA — mail:mail, mode 0600 on this
host — and systemd's empty capability bounding set strips CAP_DAC_OVERRIDE,
so root's plain open of the spool is denied under the unit's sandbox and every
run exits 2 at the script's spool readability check. That is measured, not
assumed. The unit therefore pins CapabilityBoundingSet=CAP_DAC_OVERRIDE and
AmbientCapabilities=CAP_DAC_OVERRIDE and nothing beyond that one bit. Two
cheaper-looking shapes are tested and rejected: SupplementaryGroups=mail
still cannot open the file, because a 0600 spool carries no group bits for
the supplementary group to exercise; and re-owning the spool to root would
rewrite MTA-owned delivery state, which is not host configuration this
repository prescribes.
The script needs no install step — /srv/overflow is a checkout of this
repository at the deployed revision, so each deploy ships it.
scripts/deploy-revision.sh does not install units and does not edit
/etc/aliases, so every line below is run by hand. Add the two aliases and
rebuild the alias database, install the two unit files, reload, and enable
the timer:
grep -q '^overflow-canary:' /etc/aliases || printf '%s\n' 'overflow-canary: root' >> /etc/aliases
grep -q '^overflow-alert:' /etc/aliases || printf '%s\n' 'overflow-alert: root' >> /etc/aliases
newaliases
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow-bounce.service /etc/systemd/system/
install -o root -g root -m 0644 \
/srv/overflow/deploy/overflow-bounce.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now overflow-bounce.timer
systemctl list-timers overflow-bounce.timer --no-pagerFirst the acceptance leg. Both envelope-sender addresses must route to root's mailbox:
exim4 -bt overflow-canary@"$(hostname -f)"
exim4 -bt overflow-alert@"$(hostname -f)"Each must print an R: system_aliases for <address> line — the alias is what
accepts the mail — and end at the spool delivery, <address> -> /var/mail/mail
with transport = address_file, which is how root's own address resolves on
this host. Then the issue-848 lesson, and the check that fails most often in
the wrong direction: the alert recipient must still resolve off-host.
exim4 -bt "$(cat /etc/overflow/alert-recipient)"This one must show R: smarthost for <recipient> and
router = smarthost, transport = remote_smtp_smarthost. A
transport = address_file resolution here is the local-write class — the alert
is being filed into a local mailbox and never leaves the host — and
it means the alert path is not verified; fix the resolution before
continuing, because the watcher's Class B report is the only other thing that
would ever name it.
Then the reporting leg, which posts one real report to the channel — run it once, and read this paragraph before running it. The first run initializes the offset at the spool's current size and posts nothing: the backlog is cron mail that predates the watcher, and it is never re-read (see what this verification does not prove, below). Start the unit once for that:
systemctl start overflow-bounce.service
journalctl -u overflow-bounce.service --no-pager -n 20The journal must show a clean exit and no report. Then deliver one synthetic
delivery-failure notification through the local daemon — a sendmail one-shot
message addressed to root only, whose body carries the DSN marker line and the
string overflow-canary@. Nothing is sent to any remote address; the only
off-host traffic in this step is the webhook post itself:
printf '%s\n' \
'Subject: [overflow] synthetic delivery-failure notification (verification)' \
'' \
'This message was created automatically by mail delivery software.' \
'' \
'A delivery-failure notification arrived for overflow-canary@ - verification marker' \
| sendmail root
tail -n 20 /var/log/exim4/mainlogThe mainlog must show the local delivery into /var/mail/mail. Then run the
watcher once and read the verdict out of band:
systemctl start overflow-bounce.service
journalctl -u overflow-bounce.service --no-pager -n 20The journal must show a clean exit again, and the Discord channel the webhook
file names must show one message reading
[overflow] a delivery-failure notification arrived for an overflow alert or canary message on <fqdn>: [overflow] synthetic delivery-failure notification (verification) — subject-only, because the synthetic body carries no exim
address-failure marker; that is the same shape a Gmail-format DSN reports
(known limitations below). That post is the confirmation, and it is what a
delivered bounce report looks like.
What this verification does not prove, so it is not claimed: the watcher never re-reads the pre-install backlog, so bounces that arrived before the offset was initialized are invisible to it by design; and a partial tail — a message caught mid-append — parks the offset at the last complete message and costs one timer period, never a missed message.
Disable the timer, remove both unit files, undo the aliases, and remove the state directory:
systemctl disable --now overflow-bounce.timer
rm /etc/systemd/system/overflow-bounce.timer
rm /etc/systemd/system/overflow-bounce.service
systemctl daemon-reload
sed -i -e '/^overflow-canary:/d' -e '/^overflow-alert:/d' /etc/aliases
newaliases
rm -rf /var/lib/overflow-bounceThe alert path itself is unchanged by any of it: removing the aliases returns the route to refusing bounces for the two sender addresses at RCPT, and the alerts and the canary keep working unchanged.
Known limitations. A Gmail-format DSN names its failed recipient
differently from exim's, so those reports render subject-only — the shape the
synthetic verification above deliberately shows. A unit that hangs without
exiting never enters the failed state, so OnFailure= never fires for it —
issue 848's second finding, still open there; the watcher reports the mail the
path does deliver, never a path that produces none. The watcher reads the
spool as mbox with exim's own From-escaping assumption, the same assumption
every mbox reader makes, and a message misread across a boundary is bounded to
one misclassification. The definitive remote-mailbox check remains the IMAP
probe priced above, which needs the app-password credential recorded as
externally blocked on issue 842.
Every successful privileged action — a moderator grant or revoke, an audit
decision, a recalibration closure, a ban reversal, a credit adjustment or its
reversal, a rederivation request, a settlement-override decision, a sanction
contest decision — writes one Privileged action line into the service
journal, naming the acting account, the credential reference, and the client
address the request claimed. That address is read from X-Real-IP, which the
production vhost sets. The application listens on loopback only, but loopback
is not trust: any local process can reach the listener and set that header
itself, so an unmarked address in the journal is a claim, not a fact.
To keep the journal honest about the difference, the application marks an
address verified only when the request also carries the
x-privileged-proxy-secret header with exactly the value of the
OVERFLOW_PRIVILEGED_PROXY_SECRET environment variable. Only nginx — and the operator
who configured both — hold that secret, so a header that carries it cannot
come from a forger on loopback.
The application change is safe to deploy before any host step: until the
variable is set, every entry is written with the claimed address recorded and
clientAddressVerified: false, so each entry is marked unverified. The steps
below close that gap.
-
Generate the secret once per host:
openssl rand -hex 32
-
Put
OVERFLOW_PRIVILEGED_PROXY_SECRET=<value>in/etc/overflow/overflow.env(root:root0600, like the rest of that file), and restart the service so the running process picks the variable up:systemctl restart overflow.service
-
Put the echo header in its own include file rather than the vhost: vhost files are typically world-readable, which would hand the secret to any local process — the exact adversary this section defends against — while nginx's master process reads its configuration as root, so a root-only include works:
install -o root -g root -m 0600 /dev/null /etc/nginx/overflow-privileged-proxy.conf
Into that file, with the literal replaced by the value generated in step 1:
proxy_set_header X-Privileged-Proxy-Secret "<value from step 1>";Then include it in every proxying
locationblock in the Overflow vhost, alongside that block'sX-Real-IPsetting. On the documented host these are/api/overrides,/api/moderation, and/. A location that sets anyproxy_set_headerinherits none of those directives from an outer level, so putting the attestation header only at server level or only in/leaves the other locations without it:include /etc/nginx/overflow-privileged-proxy.conf;The literal is generated per host: it is the same value
OVERFLOW_PRIVILEGED_PROXY_SECRETcarries in/etc/overflow/overflow.env. -
Validate and reload nginx:
nginx -t && systemctl reload nginx
Steps 2 and 3 may land in either order: a header the application does not have the secret for, or a secret with no header echoing it, leaves entries marked unverified for the gap — it never marks an unverified address verified.
Verify the end state with one privileged action (a moderator grant through the moderation page, say) and the journal:
journalctl -u overflow.service --no-pager -e | grep "Privileged action"The newest line must carry clientAddressVerified: true beside the address
nginx recorded. A later vhost edit that drops the header, or a value changed
on one side only, shows up the same way: entries keep arriving, marked
unverified.
The check-publishing App is the one identity whose key rotation this section
carries. Section 11 owns OVERFLOW_TOKEN_ENCRYPTION_KEY; the procedure below
does not apply to that key.
The Overflow Ledger App publishes the required checks. Its private key is
stored as the LEDGER_APP_KEY secret in the repository's overflow-ledger
GitHub environment. Both
ledger-relay.yml and
event-policy.yml read that secret.
This App has no credential on the deployment host. Perform this procedure
from the operator's machine with gh authenticated as a repository admin;
the downloaded PEM files below are operator-side staging and rollback
material, never deployment-host files.
On the check-publishing App's settings page (Settings → Developer settings
→ GitHub Apps → the Overflow Ledger App → Credentials → Key pairs), record
the current key's fingerprint. Keep that key's previously downloaded PEM
available on the operator's machine for rollback: GitHub does not return
an Actions secret's value or fingerprint, so the old PEM cannot be
recovered from LEDGER_APP_KEY.
Click New key on this App's key-pairs page and retain the downloaded
PEM locally as /local/path/to/the-downloaded.pem in the command below.
Record the new key's fingerprint from the key-pairs page, distinct from the
old one. While both keys remain registered, either key's JWT verifies.
Set LEDGER_APP_KEY from the downloaded key through stdin, never an inline
value, so key material appears nowhere on a command line:
(
set -e
gh secret set LEDGER_APP_KEY --repo Nitjsefnie/Overflow --env overflow-ledger < /local/path/to/the-downloaded.pem
gh secret list --repo Nitjsefnie/Overflow --env overflow-ledger
)Expect the list to name LEDGER_APP_KEY with an Updated at of the current
time. The value cannot be read back. The timestamp and the verification
below are the evidence that the secret holds the replacement key. Keep
the downloaded key and the previous PEM until the rotation is complete.
The relay probe mints an installation token before sweeping. A dispatch with no run id is a sweep-only start and exits nonzero if that mint fails:
gh workflow run ledger-relay.yml --repo Nitjsefnie/Overflow
run_id=$(gh run list --repo Nitjsefnie/Overflow --workflow ledger-relay.yml --limit 1 --json databaseId --jq '.[0].databaseId')
printf 'relay probe run: %s\n' "$run_id"
gh run watch "$run_id" --repo Nitjsefnie/Overflow --exit-statusIf the list prints nothing, the dispatched run is not registered yet; wait
a moment and run the lookup again. Expect watch to exit 0 on a green run.
A red run naming the installation-token mint means the secret does not
hold a working replacement key: repeat CI step 2, then verify again.
The next merge to main verifies check publication. Read the required
contexts back, filtered to the App's identity:
gh api 'repos/Nitjsefnie/Overflow/commits/MERGE-COMMIT-SHA/check-runs?filter=all&per_page=100' --paginate \
--jq '.check_runs[] | select(.app.id == 5118623) | {name: .name, status: .status, conclusion: .conclusion}'Expect one line per pinned context, each completed and success. The
pin map (.github/required-checks.json)
names the contexts. Diagnose a missing or failed line before retiring the
old key.
After the probe and check publication are green, delete the old key in the check-publishing App's settings using the fingerprint recorded in CI step 1. GitHub cannot re-add a deleted key, so this closes the rollback window. Confirm the next merge's required checks with the same readback from CI step 3, then dispose of the operator-side staged PEM files.
While the old key is still registered, restore the environment secret from the previous key's retained PEM on the operator's machine:
gh secret set LEDGER_APP_KEY --repo Nitjsefnie/Overflow --env overflow-ledger < /local/path/to/the-previous.pemRerun CI step 3 under the old key before deleting the replacement key in the App settings. If the previous PEM is unavailable, the secret cannot be restored from GitHub; repair forward with a working replacement key and repeat CI step 3. After the old key is deleted, repair is forward-only.