Skip to content

Latest commit

 

History

History
3193 lines (2766 loc) · 175 KB

README.md

File metadata and controls

3193 lines (2766 loc) · 175 KB

Deploying Overflow as an unprivileged service

overflow.service in this directory is the systemd unit the production deployment runs. It starts Next as the dedicated overflow system account, with the filesystem read-only apart from one cache directory, no capabilities, and no path under /root. tests/deploy/unit-file.test.ts fails if any of that is removed from the unit — and also if a [Service] directive is added to it or given a value other than the reviewed one, because the reviewed set there is closed on both, and if the file uses a shape of systemd's grammar the guard does not model rather than guessing at it.

This file is the procedure that makes a host match what the unit expects. The commands are run as root. Everything except the running service is root-owned: the service account can read the code it executes and cannot write it, so code execution inside the web process cannot rewrite what the next restart runs.

The optional container route — an alternative to section 10's host procedure, not used in production on this host — lives in container.md.

Backing the production database up and restoring it — the least-privilege backup role, the daily overflow-backup.timer, the exact restore steps and the drill procedure — lives in backup-restore.md.

Containing and scoping an incident, preserving evidence and recording the maintainer's decisions — incident-response.md.

Screening the contributor scoring for data-protection impact — the activities, the human-review routes and the dated conclusion — dpia-screening.md.

Values used throughout: deployment tree /srv/overflow, service account overflow:overflow, Node 24.17.0 at /usr/local/lib/nodejs/node-v24.17.0, secrets in /etc/overflow/overflow.env, listener 127.0.0.1:3000 behind nginx.

The unit assumes the database is local. It carries Requires=postgresql.service alongside After=postgresql.service, as the unit it replaces did. Requires= propagates stops, so taking Postgres down for maintenance takes Overflow down with it, and Overflow does not come back on its own when Postgres returns — restart it. On a host whose database is remote there is no postgresql.service to require and the unit refuses to start at all: drop both postgresql.service references from [Unit] and keep After=network-online.target.

1. Keep the unit you are replacing

On a host that already runs Overflow, save the current unit before touching anything. This copy is the rollback in section 9, and it is the only one: the unit lives only in /etc/systemd/system, so systemctl revert — which exists to drop overrides of a vendor-supplied unit under /usr/lib/systemd/system — has nothing to revert to here.

cp -a /etc/systemd/system/overflow.service /root/overflow.service.pre-hardening

Leave the previous checkout and Node installation in place until the hardened service has been running long enough to trust. Rollback needs them.

2. Create the service account

A system account with no login shell and no password. Its home is the deployment tree, which is why the tree lives under /srv: the unit sets ProtectHome=yes, which makes /home, /root and /run/user unreachable to the service, and a home directory under /home would be hidden from the process that owns it.

groupadd --system overflow
useradd --system --gid overflow --home-dir /srv/overflow \
  --shell /usr/sbin/nologin --no-create-home overflow

3. Install the Node runtime outside /root

/root is mode 0700, so an unprivileged account cannot reach a runtime installed under it — including an nvm installation in root's home. Install the pinned version system-wide and symlink the binary the unit names.

cd /tmp
curl -fsSLO https://nodejs.org/dist/v24.17.0/node-v24.17.0-linux-x64.tar.xz
curl -fsSLO https://nodejs.org/dist/v24.17.0/SHASUMS256.txt
grep node-v24.17.0-linux-x64.tar.xz SHASUMS256.txt | sha256sum --check
mkdir -p /usr/local/lib/nodejs
tar -xJf node-v24.17.0-linux-x64.tar.xz -C /usr/local/lib/nodejs
mv /usr/local/lib/nodejs/node-v24.17.0-linux-x64 /usr/local/lib/nodejs/node-v24.17.0
ln -sfn /usr/local/lib/nodejs/node-v24.17.0/bin/node /usr/local/bin/node
/usr/local/bin/node --version

The last command must print v24.17.0.

The package manager is needed for installs, migrations and builds, all of which run as root. The service never runs it, so the corepack shims go in /usr/local/sbin, which the unit's PATH of /usr/local/bin:/usr/bin:/bin does not reach and root's default PATH does.

ln -sfn /usr/local/lib/nodejs/node-v24.17.0/bin/corepack /usr/local/sbin/corepack
corepack enable --install-directory /usr/local/sbin
corepack prepare pnpm@10.33.0 --activate
pnpm --version

The last command must print 10.33.0, the version package.json pins.

4. Create the environment file

The environment file holds OVERFLOW_DATABASE_URL, AUTH_SECRET, the OAuth credentials, OVERFLOW_TOKEN_ENCRYPTION_KEY (with OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS beside it only while section 11's key rotation is under way), and the public callback URLs in OVERFLOW_GITHUB_WEBHOOK_URL and OVERFLOW_GITLAB_WEBHOOK_URL. Registrations using repository webhooks get their own secret, generated at registration time; App-based registration uses the separate App secret described below. The Environment reference in OPERATING.md describes each variable. This section is only about where the file lives and who may read it.

The reverse proxy must allow request bodies of at least 25 MiB on /api/github/webhooks, /api/github/app-webhook and /api/gitlab/webhooks, matching the application's delivery limit. The reverse-proxy requirements below cover body caps and the address, origin and resource bounds that must accompany them.

systemd reads EnvironmentFile= as PID 1, before it drops to User=overflow, so the service account does not need to read overflow.env directly. Keep that file 0600 root:root: the process receives its values in the environment, but cannot reread the file. This restriction applies to overflow.env.

install -d -o root -g root -m 0700 /etc/overflow
[ -e /etc/overflow/overflow.env ] \
  || install -o root -g root -m 0600 /dev/null /etc/overflow/overflow.env
chown root:root /etc/overflow/overflow.env
chmod 0600 /etc/overflow/overflow.env

The [ -e ] guard is what makes this safe on a host that already runs Overflow: install would otherwise truncate the secrets that are already there. Populate the file before section 5 — its migration step reads OVERFLOW_DATABASE_URL out of it.

A host that configured the read-only reconciliation App retires it: remove GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_PATH and OVERFLOW_GITHUB_APP_SLUG from /etc/overflow/overflow.env, delete /etc/overflow/github-reconcile-app/, return /etc/overflow to the mode this section sets above, and delete the App in GitHub's developer settings (Settings → Developer settings → GitHub Apps → the app → Delete). The fold reconciles every repository with its sponsor's OAuth token and reads no App credential.

App-based registration uses a GitHub App with three optional variables:

Variable Value
GITHUB_REGISTRATION_APP_ID The registration App's id.
GITHUB_REGISTRATION_APP_PRIVATE_KEY_PATH Path to its PEM RSA private key, readable by the overflow service account.
OVERFLOW_GITHUB_REGISTRATION_APP_SLUG Optional public slug of the registration App; the landing and registration pages link its install page. Unset or invalid shows no link.
OVERFLOW_GITHUB_REGISTRATION_APP_WEBHOOK_SECRET The webhook secret configured on that App, used to authenticate its deliveries.

Leave these unset to keep the feature off: App-based registration is unavailable, and App webhook deliveries receive HTTP 503 (fail-closed). Either the App id or key path unset or empty disables App-based registration; an unset or empty webhook secret makes the receiver answer 503. With the id and key path configured, an unreadable, unparseable or non-RSA key throws at wiring time.

Configure the registration App's webhook URL as https://<public-host>/api/github/app-webhook (receiver: POST /api/github/app-webhook). Subscribe to exactly issues, issue_comment, pull_request and pull_request_review, the events exported as githubWebhookEvents in webhook-schema.ts. Grant repository permissions metadata: read, issues: read and pull_requests: read. Install the App on repositories that will register through it; those registrations use App deliveries instead of creating a repository webhook.

Loading configuration for administrative commands

Before a recovery or credential-maintenance block that uses the database, paste this block into the same shell. Repeat it after editing the environment file. It exports the canonical database URL for direct clients and derived restore URLs: a set canonical value wins, including an empty string; only an unset canonical variable falls back to the legacy value. The file is read the way the service's EnvironmentFile= reads it, by the deploy script's own reader, so a value holding $ or quotes loads exactly as the service sees it. The file itself is unchanged.

. /srv/overflow/scripts/systemd-env-file.sh; export_env_file /etc/overflow/overflow.env
export OVERFLOW_DATABASE_URL="${OVERFLOW_DATABASE_URL-${DATABASE_URL-}}"

Reverse-proxy requirements

The unit binds Next to 127.0.0.1:3000. The documented deployment puts nginx behind Cloudflare; these controls belong to the proxy, not to the readiness probe. Preserve them when constructing or changing a vhost:

  • Client-address trust. Set real_ip_header CF-Connecting-IP and restrict set_real_ip_from to Cloudflare's IPv4 and IPv6 edge ranges; keep that list current. Never trust this header from arbitrary peers. In every proxying location, overwrite X-Real-IP with $remote_addr after that real-ip step, rather than forwarding the caller's header. The privileged-action logger reads only X-Real-IP, requires one valid IP, and never uses the appendable X-Forwarded-For chain. The host's range list is /etc/nginx/snippets/cloudflare-realip.conf.
  • Origin lock and HTTPS. Require authenticated origin pulls: the host configures ssl_client_certificate with the Cloudflare origin-pull CA and ssl_verify_client on. Enable the corresponding authenticated-origin-pull setting at Cloudflare too; do not remove certificate verification to make a direct origin request succeed. The HTTP vhost redirects to HTTPS, and the HTTPS vhost emits Strict-Transport-Security: max-age=31536000; includeSubDomains with always. These are deployment controls; the application does not supply the origin lock or this HSTS policy.
  • Forwarded origin. Each proxying location sets Host and X-Forwarded-Host to $host, X-Forwarded-Proto to $scheme, and X-Forwarded-Port to $server_port. Keep OVERFLOW_APP_URL set to the public HTTPS origin; the application's cookie-authenticated mutation guard compares the request's Origin to that configured origin.
  • Body caps. The required proxy body caps are 25 MiB (client_max_body_size 25m) on /api/github/webhooks, /api/github/app-webhook and /api/gitlab/webhooks, and 64 KiB (64k) on /api/moderation and /api/overrides, including their subpaths, matching the host configuration. The application's limits on moderation and overrides are 8 or 32 KiB. Other routes retain their own application streaming body limits: 4 KiB on /api/account and /api/forge-identities, 128 KiB on /api/repositories, and 1 MiB on /api/mcp. The server-level 25m cap is acceptable for these other routes because the application enforces those smaller limits. A smaller proxy cap there is optional defence-in-depth and must not fall below the corresponding application limit.
  • Proxy timeouts. All three proxying locations on the host use proxy_connect_timeout 5s, proxy_read_timeout 60s, and proxy_send_timeout 60s. Preserve these bounds. Read/send timeouts bound inactivity between operations, not the entire request duration; do not treat them as a 60-second reconciliation deadline. Webhooks enqueue durable work, and the worker folds it separately from the delivery request.
  • Origin rate limits. Keep a rate limit at nginx, keyed by the trusted real client address after the real-ip step. Application limits on expensive routes run after authentication and use account/token identities; they do not replace the origin's request bound. The host's /etc/nginx/conf.d/overflow-ratelimit.conf defines expensive writes and account export at 12 requests/minute (burst 20), MCP at 60/minute (burst 30), and general traffic at 20/second (burst 200), all with nodelay and rejection status 429. MCP and expensive requests also encounter the general bound. Loopback and the host's own addresses are exempt; webhook receivers are exempt from the general class and retain the application's shared receiver buckets. Review exemptions for the deployment's own callers rather than copying another host's trusted addresses.
  • Address attestation in every location. Follow section 13 to set OVERFLOW_PRIVILEGED_PROXY_SECRET and put the matching X-Privileged-Proxy-Secret directive in a root-only include. Include it in every proxying location, including /api/overrides, /api/moderation and / on the host. nginx's header inheritance rule means that a location setting any proxy_set_header inherits none from an outer level. A missing or mismatched attestation leaves logged addresses marked clientAddressVerified: false; it does not stop the privileged action.

The host's vhost is /etc/nginx/sites-enabled/overflow.nitjsefni.eu. Review the vhost and its nonsecret includes against this list; keep the attestation include root-only and do not copy its value into deployment records. Section 13 provides the validation/reload and privileged-journal check after a change. Use the operational state map to distinguish a proxy failure from a database or reconciliation failure.

5. Build the deployment tree

Clone, install, migrate and build as root, with the production settings loaded from the environment file so the db:migrate script reaches the right database. Run these blocks in Bash and stop on a failed command; set -e makes a pasted block stop too, before a failed build can be switched into service.

A first installation has no serving release: nothing serves while this section runs, and the release built here first serves after its switch, when section 6 starts the service. The migrate-then-build window that section 10's migration check guards does not exist on this path — there is no previous release to serve against the new schema — so the check is not part of this flow. A host being hardened by sections 1 and 6 is the exception: its old deployment still serves from the previous checkout while this section runs, so section 10's migration check applies on that path too. Carry the check out before this section's migrate step, treat whatever is serving at migrate time as the previous release for the test, and stop to split the change across deploys instead of running the block below if a migration fails it.

set -e
git clone https://github.com/Nitjsefnie/Overflow.git /srv/overflow
cd /srv/overflow
npm_config_package_import_method=copy pnpm install --frozen-lockfile
. /srv/overflow/scripts/systemd-env-file.sh; export_env_file /etc/overflow/overflow.env
pnpm db:migrate
release=".next-release-$(date -u +%Y%m%dT%H%M%SZ)-$(git rev-parse --short=7 HEAD)"
mkdir "$release"
node scripts/release.ts prepare /srv/overflow "$release"
OVERFLOW_NEXT_DIST_DIR="$release" pnpm build

Each build gets a new directory at the tree root, alongside .next, named .next-release-<YYYYMMDDTHHMMSSZ>-<7 to 40 lowercase hex characters>. The UTC timestamp makes the names sort in deployment order, and the SHA identifies the source revision. The explicit --short=7 requests at least seven hexadecimal characters even when core.abbrev is shorter. prune deletes directories and the <release>.tsconfig.json sidecars beside them, so it only touches names matching this grammar to be certain they are ones the deployment procedure created. A directory named .next-release-notes is safe beside the releases: prune ignores it entirely, including when counting retention slots. Reserve matching names for releases; the name check does not prove a build succeeded. mkdir deliberately has no -p: a collision must stop the deploy, not reuse an existing build. Never build into .next on a serving host, or reuse a release directory, even to retry a failed build.

The release directory must be one path segment deep. The tracked tsconfig.json includes .next/types/**/*.ts, and Next generates <distDir>/types/validator.ts with relative imports back to the project's src directory. TypeScript resolves those imports lexically through the .next include, without following the symlink. A nested layout such as .next-releases/<id> generates ../../../src/... imports, which land one directory above the project when read as .next/types/validator.ts. Keeping the release beside .next gives both paths the same depth. Release builds also isolate their generated types as described below, so a validator from the serving release cannot prevent a route from being removed.

next.config.ts reads OVERFLOW_NEXT_DIST_DIR for this build only. It trims the value and rejects either path separator (/ or \), absolute paths, . (zero depth), .. and an existing symlink at the output path. Use a single directory name inside the tree, as above, with no ./ prefix or trailing slash. With the variable unset or blank, the local build script still uses .next. Do not export OVERFLOW_NEXT_DIST_DIR for the service or add it to /etc/overflow/overflow.env: next start uses .next at runtime, with the variable unset.

Before each release build, run node scripts/release.ts prepare <tree> <releaseDir> with the same release name used for OVERFLOW_NEXT_DIST_DIR. It generates the ignored <releaseDir>.tsconfig.json beside the release directory, preserving the tracked tsconfig.json compiler settings and source includes while removing all .next* includes. Each release has its own config; for example, .next-release-20260907T101500Z-abc1234 uses .next-release-20260907T101500Z-abc1234.tsconfig.json at the tree root.

Skipping preparation now fails loudly. With OVERFLOW_NEXT_DIST_DIR set, next.config.ts refuses a missing, stale or invalid generated config before Next can create defaults or consume it for type checking. The error prints the preparation command to run. A config left by a previous release cannot satisfy this check. The generated file must be a regular file, name the intended release, and match fingerprints of both the tracked input and the prepared configuration. Changing tracked compiler options after preparation or editing the generated settings requires running preparation again.

Next uses the checked file through typescript.tsconfigPath, appends the current release's type entries, and type-checks those validators. The fingerprint allows those specific additions. Preparation accepts TypeScript config comments and trailing commas using the installed TypeScript parser. It requires an explicit include array of strings and rejects extends and references with a diagnostic naming the unsupported property. Keep the tracked config self-contained; the pinned Next 16.3.8 updates the generated type includes in the prepared config. releaseConfig is reserved for the generated preparation metadata.

Next also regenerates the ignored next-env.d.ts. Release builds leave the tracked tsconfig.json untouched, including on failure, so no restore step is needed. With OVERFLOW_NEXT_DIST_DIR unset or blank, the preparation check does no filesystem work; local development uses the tracked config and the running service continues to use .next. Start a deploy with a clean tracked tree.

Then set the ownership the unit assumes. The tree is root-owned and readable by the overflow group; nothing in it is group-writable.

chown -R root:overflow /srv/overflow
chmod -R u=rwX,g=rX,o= /srv/overflow

The install commands here and in section 10 set npm_config_package_import_method=copy for each invocation, importing package files into private inodes instead of hardlinking the shared store. Ownership and mode resets therefore leave the store and unrelated checkouts' package files untouched. Existing deployments must first complete section 10's one-time dependency migration: an unchanged install does not replace old hardlinks. The service's permissions inside the deployment tree stay as they are.

next start writes inside .next/cache — the image optimizer's output and the .previewinfo and .rscinfo files — and that directory is the only one the unit makes writable. The cache lives inside the new release. Create it and hand it to overflow before switching and starting the service: ReadWritePaths naming a missing path is a start failure, and a root-owned cache is not writable by the service account.

mkdir -p "$release/cache"
chown -R overflow:overflow "$release/cache"
chmod -R u=rwX,g=rX,o= "$release/cache"
pnpm release:switch /srv/overflow "$release" --expect-current absent

The release:switch package script takes <tree> <releaseDir> and runs node scripts/release.ts switch <tree> <releaseDir>; a relative release argument is relative to the tree. Passing --expect-current <absent|path> makes the switch conditional: it refuses unless .next still resolves to <path> at switch time, or to nothing at all when the value is absent; the deploy procedure always passes it. The resolved directory must be a direct child of the canonical tree, whether the argument is relative, absolute or a symlink alias. A symlink to a nested or outside build is refused; an alias resolving to a valid direct child is accepted. switch also refuses a resolved directory name that does not match the release grammar above. Inventing a name therefore produces a clear failure instead of selecting a release that silently accumulates because prune cannot manage it. The script requires a real BUILD_ID file and a real cache directory, refuses symlinks for those two markers, then renames a temporary relative symlink over .next and prints the resolved release path. That rename keeps an existing .next symlink resolvable throughout the swap. The marker checks do not validate every manifest or prove that a build succeeded; the successful build and the service verification remain required.

node scripts/release.ts check <tree> <releaseDir> runs the same candidate checks as switch: resolved depth, release name, BUILD_ID file and cache directory, without following symlinks for either marker. It prints the resolved path and changes nothing on disk. It can check a candidate while .next is still a real directory; switch continues to require the one-time migration below before replacing that directory.

The unit stays unchanged for this layout. Every path it names still reads /srv/overflow/..., including ReadWritePaths=/srv/overflow/.next/cache. On this host, next start served routes, static assets and RSC requests with HTTP 200 and no missing-manifest errors through the .next symlink, with OVERFLOW_NEXT_DIST_DIR unset at runtime. systemd resolved the writable cache path through the symlink too: the sandboxed process could write in that cache and got EROFS everywhere else. No service environment change or unit edit is needed.

One-time migration from a real .next directory

An existing host has a real /srv/overflow/.next directory. A symlink cannot be renamed over a real directory; scripts/release.ts switch refuses it with a one-time migration message rather than deleting the serving build silently.

Unlike a first installation, this host has a serving release throughout: section 10's block migrates and builds while the current release keeps serving, and the release built here starts serving only at this subsection's switch. Section 10's migration check therefore applies on this path too — carry it out before running section 10's block, and stop to split the change across deploys instead of running the block if a migration fails it.

Use the same Bash shell for section 10's preparation and this migration block. Follow section 10 through config preparation, the build, ownership reset, new cache handover and previous/new build printout, stopping before its switch and restart lines. The reset excludes the serving cache, which is still in the real .next directory on this host; leave that directory in place while preparing the new release. Replace section 10's switch and restart lines with the block below, pasted into that same shell so $release still names the completed new build. In a separate shell, ${release:?} stops with parameter null or not set before removal and leaves the old .next intact:

set -e
cd /srv/overflow
test -d /srv/overflow/.next
test ! -L /srv/overflow/.next
node scripts/release.ts check /srv/overflow "${release:?}"
rm -rf -- /srv/overflow/.next
pnpm release:switch /srv/overflow "$release" --expect-current absent
systemctl restart overflow.service

The candidate check must succeed before removal, so a malformed name, wrong depth or invalid build marker leaves the serving directory in place. This one deploy still hits the old missing-build window: removing the real .next can break requests until the symlink is installed and the service restarted. It is the last deploy that needs that removal; later builds leave the serving release in place. Build and prepare the new release first to keep this window to the removal, switch and restart, rather than the whole build. The removed directory provides no release-level rollback. If switching fails after removal, keep the completed new release, correct the reported failure and rerun the switch and restart; do not start another build into .next. A refused switch can leave .next a symlink, so re-running this block aborts at test ! -L /srv/overflow/.next; that abort is the signal to recover through the standing section 10 procedure, which expects the symlink layout. Continue with section 10's verification before pruning.

6. Install the unit and switch onto it

systemctl show overflow.service -p MainPID --value > /run/overflow-preswitch-mainpid
install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow.service /etc/systemd/system/overflow.service
systemd-analyze verify /etc/systemd/system/overflow.service
systemctl daemon-reload
systemctl enable overflow.service
systemctl restart overflow.service

systemd-analyze verify prints nothing and exits 0 for a well-formed unit; it names any directive systemd does not recognise.

The restart is the switchover, and it is separate from enable on purpose. --now on enable means start — man systemctl: "also start/stop/try-restart the units after the specified unit file operations succeed" — and start on a unit that is already active is a no-op with no job and no message. On the host section 1 is written for, the old root process would keep serving while systemd held the new unit file loaded and unapplied, and section 7 would report the new unit's User= beside a ps line owned by root. The recorded MainPID is what section 7 compares against; on a host that has never run Overflow it is 0, which is the same evidence read the same way.

7. Verify

Wait for the HTTP readiness check to succeed before inspecting the process owner. Keep that order when pasting the commands separately too.

set -e
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/auth/providers
printf 'MainPID before the switch: %s\nMainPID now:               %s\n' \
  "$(cat /run/overflow-preswitch-mainpid)" \
  "$(systemctl show overflow.service -p MainPID --value)"
systemctl show overflow.service \
  -p MainPID -p User -p Group -p NoNewPrivileges -p ProtectSystem
ps -o user=,pid=,args= -p "$(systemctl show overflow.service -p MainPID --value)"

Expected: active; 200 from curl — the readiness endpoint answers 200 only when PostgreSQL is reachable and every migration this build bundles is recorded applied in schema_migrations (a bounded probe: at most a few seconds, ~3 s worst case), so a 200 here is a real dependency check, not a bare port probe. An ahead schema answers 200 only when its extra migrations are not flagged for mixed-version review. A missing bundled migration or an extra migration flagged for that review answers 503; inspect Readiness probe failed: in the journal before continuing a deployment or rollback. The second 200 is the sign-in smoke: GET /api/auth/providers runs the Auth.js configuration, and answers 500 ([auth][error] UntrustedHost in the journal) when the deployment's environment cannot sign in — a failure class the readiness endpoint cannot see. The two MainPID values differ and the current one is not 0; User=overflow, Group=overflow, NoNewPrivileges=yes, ProtectSystem=strict; one ps line, owned by overflow and never root.

The two MainPID values are the check that distinguishes a switch from a reload. systemctl show reports the merged effective configuration, so it answers User=overflow from the moment daemon-reload runs, whether or not anything restarted; only a new PID says the process serving requests is the one the hardened unit started. A pair that has not moved means the old process is still serving and section 6's restart did not run.

A host drop-in under /etc/systemd/system/overflow.service.d/ can override the installed unit. Section 6 installs only overflow.service; the repository guard does not inspect host drop-ins. The systemctl show check above reveals overrides to the properties it lists because it reports the merged configuration, including drop-ins, rather than just the unit file.

Two details in those commands are the point of them. Ask systemctl for the PID instead of asking ps for Node processes: ps -C node matches on comm, and Next rewrites its process title to next-server (v…, so ps -o user= -C node never sees this service at all and answers root from whatever unrelated Node processes the host happens to run — the alarming answer whether the hardening worked or not. And Type=simple has no readiness barrier, so systemctl start returns as soon as the process is forked, before the drop to overflow and before Next binds the port. An early ps can therefore print root for a process still starting. The curl retry absorbs that readiness race; only inspect the PID and owner after it succeeds, so a connection refusal or the transient startup owner is not mistaken for a failed deploy.

systemd-analyze security overflow.service reports the remaining exposure and is worth reading after any change to the unit.

Restrictions only bite on the code paths that use them, so exercise the application through the browser before calling the switch done: sign in with GitHub, open the dashboard, and register a repository. That is what makes DNS resolution, an outbound HTTPS call to GitHub and a database write happen inside the sandboxed process; the reconcile package script does not, because it runs as root outside the unit and so is subject to none of these restrictions. The unit sets SystemCallErrorNumber=EPERM, so a syscall the filter blocks returns an error to the process instead of killing it. That buys resilience, not visibility: a seccomp errno action logs nothing of its own, and it is the default kill action — the one EPERM replaces — that would have reached the journal, as status=31/SYS. So what you are looking for is the application's own report of an operation that failed:

journalctl -u overflow.service -n 100 --no-pager

8. Test the rollback before you need it

Do this once, immediately, while the previous unit and checkout are still on the host. Restore the saved copy, confirm the service comes back on it, then reinstall the hardened unit and confirm again with section 7.

The rollback has three preconditions and nothing on the host keeps them alive — a cleanup of /root voids the rollback silently, months later. Check them first, here and in section 9; if any fails, there is no rollback and a broken hardened unit has to be fixed forward instead.

missing=0
for path in /root/overflow.service.pre-hardening /root/overflow \
            /root/.nvm/versions/node/v24.17.0/bin/pnpm; do
  if [ -e "$path" ]; then
    echo "present: $path"
  else
    echo "MISSING: $path" >&2
    missing=1
  fi
done
[ "$missing" = 0 ] \
  || echo "No rollback is available. Fix the hardened unit forward instead." >&2

Every path prints, present or missing, because test prints nothing either way: three blank results is what an operator sees whether the rollback is intact or gone, and a check that cannot be read has replaced the silent expiry it exists to catch.

systemctl show overflow.service -p MainPID --value > /run/overflow-preswitch-mainpid
systemctl stop overflow.service
cp -a /root/overflow.service.pre-hardening /etc/systemd/system/overflow.service
systemctl daemon-reload
systemctl start overflow.service
systemctl is-active overflow.service
printf 'MainPID before the rollback: %s\nMainPID now:                 %s\n' \
  "$(cat /run/overflow-preswitch-mainpid)" \
  "$(systemctl show overflow.service -p MainPID --value)"
ps -o user=,pid=,args= -p "$(systemctl show overflow.service -p MainPID --value)"
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/

active, a MainPID that moved, a ps line owned by root again and 200 mean the rollback path works — the root is the point of it, since that is the state the saved unit runs in. Then repeat section 6 and section 7 to get back to the hardened unit; section 6's restart is what makes that return leg real, and its MainPID pair is what proves it happened. A rollback that has never been run is an assumption. This block keeps the landing-page URL on purpose: the restored old checkout predates the readiness endpoint, and what the block verifies is unit-rollback serving, not dependency reachability.

9. Rolling back

If a deploy fails, capture why before restoring anything:

systemctl status overflow.service --no-pager
journalctl -u overflow.service -n 100 --no-pager

The failures this configuration produces:

  • ReadWritePaths names a path that does not exist — /srv/overflow/.next/cache now resolves through the .next symlink into the selected release. A missing or dangling link, or a missing cache inside that release, prevents startup. The cache must also be owned by overflow for runtime writes to work.
  • /etc/overflow/overflow.env does not exist — section 4 was skipped. Its mode is not a start failure: systemd reads the file as root, before the drop to User=overflow, so 0600 root:root is correct and a service that cannot read the file itself is the design, not a fault.
  • postgresql.service does not exist on this host, so the unit's Requires=postgresql.service refuses the start outright. That is the remote-database case; see the note at the top of this file.
  • /usr/local/bin/node is missing or is a dangling symlink into /root.
  • The tree is unreadable to the group, so next cannot load its own build.

For a failed release, switch back to the retained previous build and restart the same hardened unit. Section 10 prints the previous release path before switching; record it with the deploy. Replace the value below with that recorded path, and confirm the directory still exists. The ownership reset on a later deploy also resets retained caches other than the serving cache, so hand the previous cache back before the restart, even if it was writable when that release last ran. The block takes the same deploy lock as section 10, so a rollback cannot interleave with a deploy, and its switch is conditional on .next still resolving to the release the rollback started from. Paste the block into a fresh shell, never one that already holds the deploy lock: its exec 9> re-opens fd 9, which would release the lock that deploy still holds.

set -e
cd /srv/overflow
exec 9>/run/overflow-deploy.lock
flock -w 900 9 || { echo "Could not acquire the deploy lock on /run/overflow-deploy.lock; refusing to deploy. Consult the deploy procedure's serialization notes before re-running." >&2; exit 1; }
expected_serving=$(readlink -f /srv/overflow/.next || printf absent)
previous_release='.next-release-REPLACE-WITH-RECORDED-ID'
test -f "$previous_release/BUILD_ID"
test -d "$previous_release/cache"
chown -R overflow:overflow "$previous_release/cache"
chmod -R u=rwX,g=rX,o= "$previous_release/cache"
pnpm release:switch /srv/overflow "$previous_release" --expect-current "$expected_serving"
systemctl restart overflow.service
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness

Expect active and HTTP 200, then exercise the application and inspect its journal as in section 7. Restart immediately after switching: a process running across the swap retains its old writable cache mount, while the fresh start picks up the selected release's cache. A symlink switch alone is not a deploy or a rollback.

A retained release built before the readiness endpoint existed answers 404 there. curl -f treats 404 as a failure, so the verification fails — readiness unknown rather than confirmed — and that is the intended fail-safe: never report a rollback healthy on a signal that cannot see the database. The caveat goes moot once every retained release postdates the endpoint.

The endpoint refuses a database missing a migration the release bundles with 503. A rollback never undoes migrations, so the database can also be ahead of the retained release. A release that checks the review flag answers 503 when an applied migration recorded as review is absent from its bundled list; extra unflagged migrations still allow 200. An older release that does not check the flag still answers 200 against an ahead schema, even after a flagged migration, so its readiness signal cannot establish compatibility. The classifier's limits below also mean that a 200 does not replace review of the retained release against the schema as it stands.

This rolls back the build, not the revision. The checkout and node_modules are shared with the current revision, and database migrations are not undone. The retained build must work with those dependencies, configuration and schema; the switch script does not check compatibility. If those need reverting too, restore the intended revision and its dependencies and build it into a new release using section 10's build, ownership, switch and verification steps; assess the database schema separately.

If the hardened unit itself is broken and needs the old unit restored, check section 8's three preconditions, then restore with the same commands. That restored state runs the application as root again, which is the state this procedure exists to leave behind, so fix the failure rather than forgetting it. systemctl is-active reporting active and the HTTP check returning 200 are the evidence the rollback worked; a journal that shows the process starting as root:root is what tells you the old unit — not the new one — is the one now running.

The old-unit rollback decays with every deploy, and neither of those two checks reports it. Section 10 migrates the production database and does not touch /root/overflow, so from the first revision deploy onwards the saved unit runs older code against a newer schema. Once the hardened service is trusted, stop treating the old checkout as the rollback. Retaining previous releases partially restores the path: a compatible previous build can be selected without rebuilding or returning to the root-run unit. It does not preserve the old checkout, dependencies or database schema, so a revision rollback still needs those considered explicitly.

10. Deploying a new revision

The procedure runs as one committed script. As root, from the tree root, run bash scripts/deploy-revision.sh; it applies the source-tree guards described below, plus its automated pending-migration and workflow-pinned CI gates: the flock fence on /run/overflow-deploy.lock held on fd 9 for up to 900 seconds and refusing with the serialization refusal when the lock is not acquired, the .next anchor taken before the fetch and passed to release:switch --expect-current, a git fetch origin main that moves only refs, resolution of the deployed SHA from FETCH_HEAD, a refusal when HEAD cannot fast-forward to that SHA (the tree is ahead of or diverged from main), the tree-cleanliness gate that refuses the deploy when the working tree deviates from HEAD — tracked modifications, staged changes and untracked non-ignored files all survive a fast-forward, and a release is named for the commit it was built from, so the tree must be that commit; it also refuses the deploy when the tree holds an ignored untracked file outside the operational allowlist, since git status never shows such a file (the .gitignore denies by default) and the build would compile it into a release named for a commit that does not contain it. The allowlist is exactly, and only at the tree root, the .next anchor, the release directories and their .tsconfig.json sidecars, .next-release-notes/, next-env.d.ts and node_modules/; anything else ignored anywhere in the tree must be removed before re-running. .claude/ is on that list for the opposite reason: it is this repository's own session machinery — the autoloaded working rules and the priority board — which the checkout a deploy runs from may carry and which nothing in the build reads, so an operator must not have to delete it to deploy. The required-checks gate must then bless the exact fetched SHA (below), and only after both gates pass the git merge --ff-only that moves the tree to that SHA; then the redundant-deploy skip that compares the resolved SHA against the serving release's REVISION record and, on a match — the run that built the serving release migrated at that same commit — exits without installing, migrating or building, the copy-import install, the environment load (the file read as systemd's EnvironmentFile= reads it, never run; below), db:migrate, a grammar-named release directory created with a collision-aborting mkdir, generated-config preparation, the build, whose clean step wipes the release directory (everything outside cache|dev|lock|trace), the ownership reset excluding the serving cache, the new cache handover to the service account, the conditional switch, the restart, the is-active verification, the readiness endpoint and the sign-in smoke (GET /api/auth/providers, which must answer 200), the webhook upgrade written to a retained JSONL log with a nonzero upgrade exiting the script nonzero, after which the exact source SHA is recorded in a REVISION file inside the release — attesting a fully deployed release (built after the wipe, switched, verified), so a redundant deploy may trust it — the retention listing, and the prune via release:prune --keep 3. The migration-safety analysis and every other guard below govern the script's run exactly as they govern the manual block; read this whole section before running either.

The script runs in two phases joined by one exec. Bash reads a script incrementally, and the fast-forward rewrites the script file mid-run, so the tail after the fast-forward would otherwise execute from the pre-merge copy. Phase 1 is everything through the fast-forward, run from wherever the deploy was started; it then re-executes, handing the resolved SHA and the pre-fetch serving anchor to the merged tree's own copy of the script, which runs the install→prune half. The deploy lock travels with the exec — the lock is held on the open file description, which the re-executed process inherits — so the fence still serializes the whole procedure. The handoff names are internal: the script sets both itself, production never seeds them, and a re-executed entry refuses fail-closed unless fd 9 is open on the deploy lock itself, the serving anchor is set, and the tree is already at the exact commit the handoff carries — the one the gates blessed. So the verification steps a deployed commit adds to the script run in their own deploy.

The script's prune is guarded in a way the manual path is not, separately from the confirm-first rule below: before pruning, the script consults its own retention listing and refuses to prune unless the recorded previous release is inside the newest-3 retention set. When the previous release is older, the script prints its name, prints the exact manual release:prune command with --keep raised above 3, and runs nothing — the deploy still exits 0, because the deploy itself succeeded and only the prune was withheld. The manual path keeps a human as the guard through the confirm-first rule below; the script's guard is additional automation, and the manual rule is what still applies when pruning by hand.

The script deploys only a SHA that main's required checks have blessed. The gate parses .github/required-checks.json at the deployed SHA with the tree's scripts/required-checks-parse.jq, and when that parse fails it re-parses with the target SHA's own copy of the same file, so a map-shape change cannot deadlock a tree whose parser predates it. Right after the fetch it resolves the exact SHA being deployed and, once the tree-cleanliness gate passes, reads main's required checks from branch protection and the branch-rules endpoint (repos/OWNER/REPO/rules/branches/main). The gate uses their union, refusing when it is empty or either read fails for a reason other than not-found (HTTP 404). Every required check's latest run on that SHA must conclude success before the fast-forward, and so before install, migrations, build, switch or restart. A failed, cancelled or otherwise non-successful conclusion refuses immediately, and a check whose latest run is queued, in progress or has not been created yet makes the script wait, polling every 60 seconds until OVERFLOW_DEPLOY_CI_TIMEOUT (default 900) seconds elapse, then refusing with the still-pending checks named, an absent run reported as <check> (absent). Each poll lists check-runs and workflow runs with pagination. Jobs are read to prove check-run attribution when their IDs are not yet known, or when a run's attempt changes. Later polls reuse that job identity and read current outcomes from check-runs; suite identity only narrows fallback reads and cannot certify a check by itself. When all relevant job IDs and run attempts are known and unchanged, a poll costs two requests if each listing fits on one page. Unattributed checks still hold the gate, and any failed read refuses the deploy. Every required check must therefore report on the exact main tip being deployed: a check that runs only on pull requests never runs on the SHA a rebase merge lands, so the gate would wait it out as <check> (absent) and refuse. ratchet-guard normally reports on both — pull_request_target for pull requests and push for the tip each push lands. The deploy gate checks only the fetched tip of main.

Recovering a main tip whose push launched no runs

If the gate waits on (absent) for every required check and actions/runs?head_sha=<sha> lists no push runs, recover main's current tip with workflow dispatches. A dispatch on main runs for main's tip, so it cannot certify an older tip. For ratchet-guard, base must be the newest main commit that already has a successful ratchet-guard run. The dispatch judges the interval from that tip to main's tip as one endpoint change, just as a push run does for a multi-commit push. If several consecutive pushes were dropped, this is coarser than their separate push runs: a relaxation later re-tightened past the base is not flagged.

base=$(gh api 'repos/Nitjsefnie/Overflow/actions/workflows/ratchet-guard.yml/runs?branch=main&status=success&per_page=50' --jq '[.workflow_runs[] | select(.event == "push" or .event == "workflow_dispatch")][0].head_sha')
gh workflow run ci.yml --ref main -f base="$base"
gh workflow run actionlint.yml --ref main
gh workflow run ratchet-guard.yml --ref main -f base="$base"

After fetching main, check git rev-parse origin/main: if $base is empty or null (no qualifying run among the 50 most recent successful ones) or equals that SHA (the tip is already certified), do not dispatch ratchet-guard. Dispatch ci without -f base in that case; it measures coverage. Find the base by hand from older runs or stop before dispatching ratchet-guard.

The gate accepts these runs because it identifies each producer by its pinned workflow path and job name, never by event. The newest run decides, so a failed dispatch is superseded only by a newer successful one. An empty-commit re-push is refused by branch protection. A dispatched ci run measures the change against the supplied base, including each commit in the interval; a dispatch without a base always measures coverage.

Each required check is resolved to the jobs of the workflow files .github/required-checks.json pins it to — a pin may be one file or a list of them, and every pinned context names both a repository-level file and a pull-request-only file — and a same-named check-run from any other producer holds the deploy as pending, so it is refused at the deadline and never passed.

The required checks are pinned, in branch protection, to the Overflow Ledger App (app id 5118623), and the gate's producer rule follows that pin: the gate reads each required context's check-runs, a check-run posted by the App decides the context, and the pinned workflow's job record decides only when no App check-run exists — while any other check-run bearing a required name keeps the check pending as unattributed. The App's check-runs are posted by the relay (.github/workflows/ledger-relay.yml), which mirrors the pinned producers' completed runs onto the App; the Required checks relay subsection of OPERATING.md carries its operations. A refused gate leaves HEAD, the index and the working tree untouched, so the tree stays on the commit it was on; only the refs the fetch wrote (FETCH_HEAD, origin/main) have moved. OVERFLOW_DEPLOY_CI_GATE=skip bypasses the entire gate with a loud warning naming the skip and the SHA, and is reserved for rollback or recovery deploys when main's CI is red; unset or empty enforces the gate, and any other value refuses.

The script takes no arguments and reads exactly eight environment overrides for the test harness; production sets none of them and runs on the defaults: OVERFLOW_DEPLOY_TREE (default /srv/overflow), OVERFLOW_DEPLOY_ENV_FILE (default /etc/overflow/overflow.env), OVERFLOW_DEPLOY_LOCK (default /run/overflow-deploy.lock), OVERFLOW_DEPLOY_UNIT (default overflow.service), OVERFLOW_DEPLOY_URL (default http://127.0.0.1:3000/api/readiness), OVERFLOW_DEPLOY_LOG_DIR (default /var/log/overflow), OVERFLOW_DEPLOY_CI_TIMEOUT (default 900) and OVERFLOW_DEPLOY_ENV_READ_TIMEOUT (default 5, seconds). The deployment URL must name the readiness endpoint: the sign-in smoke derives its own URL from OVERFLOW_DEPLOY_URL by replacing the trailing /api/readiness with /api/auth/providers, so any other target derives a wrong smoke URL and refuses the deploy — failing closed. One further override is operator-facing, not a test knob: OVERFLOW_DEPLOY_CI_GATE, whose only accepted non-default value is skip (the gate paragraph above).

Every revision deploy finishes by upgrading existing hooks after the new release is serving and its readiness check succeeds. Migration 043 must precede the scoped-credential receiver code. Legacy hooks cannot authenticate until this upgrade configures their new callback UUID and independent secret. Keep the callback base URLs and OVERFLOW_TOKEN_ENCRYPTION_KEY available; change the encryption key only through section 11. Avoid concurrent manual hook-configuration edits. The command preserves active state and unrelated subscriptions and verifies each persisted hook at its current numeric-ID-resolved location. It requests full upstream repair only when the subscribed events or callback URL changed, or the credential is finalized for the first time. Pending credential material is durable: retry a failed run without reminting it. Missing sponsor credentials require relinking before that registration can upgrade. After all registrations migrate, retire the old shared credential and assess any reuse elsewhere. Rolling back to a shared-credential receiver reopens that trust boundary. Historical missed deliveries have no known dirty subject; an ordinary incremental queue pass cannot guarantee their repair. Startup recovery does not replace this upgrade.

Startup recovery is on by default. Before every deploy, review whether the revision can affect fold-derived values. Changes to fold behavior must not suppress the startup pass; leave OVERFLOW_SKIP_STARTUP_RECONCILIATION unset in /etc/overflow/overflow.env. The same recovery-on default applies whenever the impact is uncertain.

For a reviewed deploy that cannot affect fold-derived values (for example, a copy-only change), an operator may deliberately add OVERFLOW_SKIP_STARTUP_RECONCILIATION=1 to /etc/overflow/overflow.env before the restart below. Only the exact value 1 suppresses the immediate startup sweep; unset, blank, malformed and all other values run it. A shell export in the deploy terminal does not configure the systemd service. This opt-out leaves the recurring sweep, queued work, webhook-triggered reconciliation and manual reconciliation running normally, but missed deliveries may remain unrecovered until a later reconciliation.

scripts/deploy-revision.sh reads the environment file, not the deploy terminal, and reads it the way systemd's EnvironmentFile= does, without running it: single and double quotes, backslash escapes, line continuations and # or ; comment lines, with no $ expansion and no commands, so the migration check, the migrations, the build and both lines below see exactly the values the service will see. Every name the file sets is exported to those steps over any value the deploy terminal exported; a name the file does not set keeps the terminal's value. If the file cannot be read at that first read (it is missing, unreadable, or not read within the deadline), the deploy refuses before the migration check. The script then says what the file holds. Before the restart it prints one line: that the immediate startup sweep will be skipped when the file sets exactly 1, or that it will run when the variable is unset or blank. Any other value, such as true, is named in that line as not exactly 1, so an override that does not skip is visible before the restart. Once the service is active again, a file that still holds 1 is reported as still set, with a reminder that every later restart will skip the sweep until the line is removed. That reminder is the prompt for the removal below. If the file cannot be read when either line is printed (it is missing or unreadable, or not read within the deadline), that line says the value could not be read, rather than reporting the variable as unset or saying nothing, and the deploy carries on.

After restart, verify the warning naming OVERFLOW_SKIP_STARTUP_RECONCILIATION in the service journal. An authenticated moderator can also read GET /api/moderation/rederivation: startupRecoverySkipped records whether this process skipped its startup recovery pass, alongside the existing row staleness counts. It remains true after later sweeps; it is startup history, not a claim that every repository is still stale. Remove the override from the environment file after the chosen restart so the next restart defaults to recovery-on. Remove this temporary override with issue 196, when startup reconciliation becomes incremental.

Install, migrate and build run as root inside the tree. Only the service runs as overflow, and the ownership reset afterwards is what keeps it that way: a build writes new files as root, and the new cache has to be handed back while the serving cache stays writable. Build into a new release directory every time so Next cannot rewrite the serving build's manifests, chunks and fallback error page during the build. Run one deploy at a time; concurrent installs, config generation, switches or prunes share the same tree. On a host whose .next is still a real directory, use section 5's one-time migration block at the switch step.

The block is serialized with an exclusive flock on /run/overflow-deploy.lock (command -v flock: /usr/bin/flock, util-linux), held for the whole procedure: pull, install, migrate, build, ownership reset, switch, restart, verification and webhook upgrade all run under one lock, so two deploys started together run one after the other instead of interleaving. A concurrent deploy waits up to 900 seconds for the lock and then refuses; refusing is the fail-safe behavior, and a deploy must never proceed without it. The lock is kernel-owned and disappears when the holding process dies, so a crashed deploy cannot deadlock the next one.

The switch inside the block is conditional as the second defense, against actors that skipped the lock: the block records the release .next resolves to before touching anything and passes it as --expect-current, and release:switch refuses unless .next still resolves there at switch time. An off-procedure actor — an old copy of this document, a hand-run switch — therefore cannot silently supersede an in-flight deploy: production never moves backward and an already-verified release is never silently discarded. When the switch reports the mismatch, re-run the whole procedure: the script from the start, or the manual block below from its opening fence so it captures a fresh serving anchor. A missing or dangling .next at anchor time is a host that needs repair or the one-time migration, not a routine deploy; the conditional switch refuses that state rather than building on it.

Migrations apply before the build, and the release they accompany starts serving only at the switch. Between those points the previous release serves every request against the new schema, for the whole build duration. A migration is safe to apply in that position exactly when the previous release's write path cannot violate it: purely additive statements (a new table, a nullable column, a plain index; a plain index built without CONCURRENTLY blocks the previous release's writes for the scan's duration) are safe, and so is an enforcing statement the previous release already satisfies on every write path — settlements carries settlements_issue_unique (migration 003) alongside the writer that maintains it.

After loading the environment and before pnpm db:migrate, the deploy script runs node scripts/deploy-migration-status.ts and lists every migration not yet recorded in schema_migrations. Each line is tab-separated: <name>\t<verdict>[\t<shapes or safe note>]. Field 2 is - or review; the optional third field names the matched shapes or the safe-marker note. The deploy script still reads field 2 only.

Classification denies these shapes by default, ignoring SQL comments and matching case-insensitively after normalizing whitespace:

  • ALTER TABLE ... DROP [COLUMN] [IF EXISTS] <column> and DROP TABLE;
  • any ALTER ... RENAME form;
  • ALTER [COLUMN] <column> [SET DATA] TYPE (the COLUMN keyword is optional);
  • SET NOT NULL;
  • ALTER TABLE ... ADD [COLUMN] [IF NOT EXISTS] <column> ... NOT NULL without DEFAULT in the same top-level column definition.

Unicode-escaped U& identifiers and strings (including UESCAPE) are decoded before matching. String constants continued across a newline are concatenated before their contents are scanned. Lexical forms the classifier cannot decode confidently require review, labelled unsupported lexical form; the safe marker cannot override that failure.

The contents of every SQL string literal are classified too: a shape inside a literal still requires review, even if that literal is data rather than executed SQL. Statements inside literals (including DO or function bodies and EXECUTE text) get no DEFAULT exemption: ADD COLUMN ... NOT NULL DEFAULT ... inside a literal is flagged too. The literal substring overflow: mixed-version review anywhere in the raw file, comments included, also forces review. A real SQL line comment -- overflow: mixed-version safe: <reason> downgrades a shape hit to - only with a non-empty reason explaining why retained releases are safe; an empty reason does not count, and marker-looking text inside a block comment never authorizes a migration. The review marker beats the safe marker when both occur. Adding or dropping constraints and dropping a column default are not classified; use the voluntary review marker for those risks too.

A pending review entry makes the script refuse before migration. Review each entry against the still-serving release and every retained rollback target, then rerun with the unchanged OVERFLOW_DEPLOY_MIGRATION_ACK=1 to confirm that review. The acknowledgment has no effect when the status command fails, and any value other than 1 is refused. The runner records the verdict in schema_migrations.mixed_version_review in the same transaction as the migration; already-applied legacy rows are not reclassified. Readiness renders only ledger names matching the runner's migration filename grammar in the journal. An invalid flagged name still refuses readiness, but the journal replaces that name with <unrecognised migration name>. The anonymous 503 body contains only { "status": "unavailable" }; inspect the journal for the reason. Only an actual boolean true flags a row; a legacy missing flag or a drifted non-boolean flag is treated as unflagged.

Use expand/contract releases: drop or rename a column only one release after the code stopped reading and writing it. Change a type or add NOT NULL only after every retained release writes compatible values. Otherwise acknowledge the migration knowingly: the still-serving release and every retained rollback target older than it will fail against the changed schema.

During a flagged migration's build window, the still-serving old release reports not-ready at /api/readiness, provided its readiness check reads the review flag. On the container route, the image HEALTHCHECK and the docker-compose.yml healthcheck mark the old container unhealthy; compose does not restart it on that by itself. An older release whose readiness check does not read the flag still reports 200 and leaves that window unprotected. Acknowledge the migration knowingly. The deploy script probes readiness only after the release switch and restart, so this old-release 503 does not block the switch.

Before running the standing block below, read every migration the run will apply for the first time — anything schema_migrations does not yet record — and apply that test to each. New migrations that could break the previous release's writes while both releases serve carry the marker in the file when filed for review: overflow: mixed-version review. If one fails, stop: do not run it. Land the writer correction in this release and the enforcing statement in the next deploy, or shape the constraint so the previous release cannot violate it (for example, a partial constraint excluding the shape the old writer can produce). Shipping the constraint and its writer correction in the same release does not close the window: the previous release keeps serving with the constraint applied for the whole build duration. ADD CONSTRAINT ... NOT VALID does not rescue a failing constraint. A unique constraint cannot be marked NOT VALID at all, and where the mark exists (a CHECK or FOREIGN KEY constraint) it defers only the scan of existing rows while still enforcing new-row writes immediately, so it does not make a constraint safe to apply before the corrected writer is serving. Migrations are append-only: edits to applied migration files fail CI, and the runner refuses a checksum mismatch with the value recorded in schema_migrations.

The fenced blocks below are the manual fallback, for the case where the script itself is what broke. They use the same fence, --expect-current anchor and release grammar. They fetch origin main, resolve the fetched commit, check ancestry, require a clean working tree and reject ignored files outside the operational allowlist before git merge --ff-only. They skip a redundant deploy when the serving release's REVISION matches that commit and write the record after webhook upgrade succeeds.

The manual path does not carry the script's automated pending-migration gate. After loading the environment, stop and run node scripts/deploy-migration-status.ts by hand; if it fails, stop. Review every listed migration and apply the mixed-version check above to each review entry, then continue at pnpm db:migrate. OVERFLOW_DEPLOY_MIGRATION_ACK=1 only confirms review in the deploy script and does not add a manual gate.

The manual required-checks gate trusts check-run names; it does not use .github/required-checks.json to attribute each name to its producing workflow (a name may be pinned to several workflows, and the gate keeps trusting the check-run name alone). It does not wait for CI: an absent, non-completed or unsuccessful required check refuses the deploy. Once CI completes, rerun the gates part of the block. The script's OVERFLOW_DEPLOY_CI_GATE=skip setting has no manual equivalent.

Run the standing block in parts, in one shell so the fd 9 fence and expected_serving carry over. The ancestry, cleanliness, ignored-file and required-checks gates run before the merge; if a gate refuses, the shell exits. Diagnose and resolve the refusal, then start a new shell and re-run the block's opening part from the fence. The only manual stop is after loading the environment: run node scripts/deploy-migration-status.ts by hand, review every listed migration as described above, stop if the command fails or a migration is unsafe for the previous release's writes, and continue at pnpm db:migrate only after that review.

Existing deployments: complete the ONE-TIME dependency migration below before running this standing procedure for the first time. Fresh installations using section 5's copy import do not need that migration.

set -e
cd /srv/overflow
exec 9>/run/overflow-deploy.lock
flock -w 900 9 || { echo "Could not acquire the deploy lock on /run/overflow-deploy.lock; refusing to deploy. Consult the deploy procedure's serialization notes before re-running." >&2; exit 1; }
expected_serving=$(readlink -f /srv/overflow/.next || printf absent)
git fetch origin main
full_sha=$(git rev-parse --verify 'FETCH_HEAD^{commit}')
ancestry_status=0
git merge-base --is-ancestor HEAD "${full_sha}" || ancestry_status=$?
if [ "${ancestry_status}" -eq 1 ]; then
  printf 'HEAD in /srv/overflow is not an ancestor of the fetched main (%s), so it cannot fast-forward there; refusing to deploy. HEAD, the index and the working tree are untouched; only the fetched refs moved. Inspect git log %s..HEAD in the tree before re-running.\n' "${full_sha}" "${full_sha}" >&2
  exit 1
elif [ "${ancestry_status}" -ne 0 ]; then
  printf 'Could not determine whether HEAD in /srv/overflow is an ancestor of the fetched main (%s): git merge-base exited %s; refusing to deploy. HEAD, the index and the working tree are untouched; only the fetched refs moved. Investigate the repository state in the tree before re-running.\n' "${full_sha}" "${ancestry_status}" >&2
  exit 1
fi
tree_status=$(git status --porcelain=v1 -uall) || {
  printf 'Could not read the working-tree state in /srv/overflow; refusing to build a release whose source identity cannot be attested. Investigate git status in the tree before re-running.\n' >&2
  exit 1
}
if [ -n "${tree_status}" ]; then
  printf '%s\n' "${tree_status}"
  printf 'The working tree in /srv/overflow deviates from HEAD; fast-forwarding it to %s would not make it that commit. A release is named for the commit it was built from; refusing to build one from a tree that is not that commit. Resolve every deviation above (git status), then re-run the deploy.\n' "${full_sha}" >&2
  exit 1
fi
allowlist_re='^(\.next/?|\.claude/|\.next-release-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{7,40}/|\.next-release-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{7,40}\.tsconfig\.json|\.next-release-notes/|next-env\.d\.ts|node_modules/)$'
ignored_listing=$(mktemp)
ignored_status=0
git ls-files -z --others --ignored --exclude-standard --directory --no-empty-directory > "${ignored_listing}" || ignored_status=$?
mapfile -d '' -t ignored_entries < "${ignored_listing}"
rm -f "${ignored_listing}"
if [ "${ignored_status}" -ne 0 ]; then
  printf 'Could not list the ignored untracked files in /srv/overflow (git ls-files exited %s); refusing to build a release whose source identity cannot be attested. HEAD, the index and the working tree are untouched; only the fetched refs moved. Investigate git ls-files in the tree before re-running.\n' "${ignored_status}" >&2
  exit 1
fi
stray_ignored=()
for entry in "${ignored_entries[@]}"; do
  [[ "${entry}" =~ ${allowlist_re} ]] || stray_ignored+=("${entry}")
done
if [ "${#stray_ignored[@]}" -gt 0 ]; then
  printf '  %q\n' "${stray_ignored[@]}" >&2
  printf 'The tree in /srv/overflow holds the ignored untracked files above, outside the operational allowlist (.next, .claude/, release directories and their .tsconfig.json sidecars, .next-release-notes/, next-env.d.ts and node_modules/, each at the tree root). These are ignored untracked files that git status does not show, and the build would compile them into a release named for %s, a commit that does not contain them; refusing to deploy. HEAD, the index and the working tree are untouched; only the fetched refs moved. Remove them, then re-run the deploy.\n' "${full_sha}" >&2
  exit 1
fi
remote_url=$(git config --get remote.origin.url)
repo=
case "${remote_url}" in
  git@github.com:*) repo="${remote_url#git@github.com:}" ;;
  https://github.com/*) repo="${remote_url#https://github.com/}" ;;
esac
repo="${repo%.git}"
repo_valid=0
[[ "${repo}" =~ ^[^/]+/[^/]+$ ]] || repo_valid=$?
if [ "${repo_valid}" -ne 0 ]; then
  printf 'Could not parse an OWNER/REPO GitHub slug from remote.origin.url (%s); refusing to deploy.\n' "${remote_url}" >&2
  exit 1
fi
required_stderr=$(mktemp)
# Read HTTP status independently of diagnostic text; parse only successful bodies.
classic_required=
required_status=0
if required_response=$(gh api "repos/$repo/branches/main/protection" --include 2>"${required_stderr}"); then
  # Drop the status line and headers through the first blank line (LF or CRLF).
  required_body=${required_response#*$'\n'}
  while [[ "${required_body}" == *$'\n'* ]]; do
    required_header=${required_body%%$'\n'*}
    required_body=${required_body#*$'\n'}
    [ -n "${required_header%$'\r'}" ] || break
  done
  if required_body=$(jq -e 'if type == "object" then . else error("expected a protection object") end' <<<"${required_body}" 2>"${required_stderr}"); then
    classic_required=$(jq -r '([.required_status_checks.contexts[]?] + [.required_status_checks.checks[]?.context]) | unique | .[]' <<<"${required_body}" 2>"${required_stderr}") || required_status=$?
  else
    required_status=$?
  fi
elif [[ "${required_response%%$'\n'*}" =~ ^HTTP/[0-9]+[.][0-9]+[[:space:]]+404[[:space:]] ]]; then
  classic_required=
else
  required_status=1
fi
if [ "${required_status}" -ne 0 ]; then
  rm -f "${required_stderr}"
  printf 'could not determine required checks for main: branch-protection read failed; refusing to deploy\n' >&2
  exit 1
fi
rules_required=
required_status=0
if required_response=$(gh api "repos/$repo/rules/branches/main" --include 2>"${required_stderr}"); then
  # Drop the status line and headers through the first blank line (LF or CRLF).
  required_body=${required_response#*$'\n'}
  while [[ "${required_body}" == *$'\n'* ]]; do
    required_header=${required_body%%$'\n'*}
    required_body=${required_body#*$'\n'}
    [ -n "${required_header%$'\r'}" ] || break
  done
  if required_body=$(jq -e 'if type == "array" then . else error("expected a rules array") end' <<<"${required_body}" 2>"${required_stderr}"); then
    rules_required=$(jq -r '[.[] | select(.type == "required_status_checks") | .parameters.required_status_checks | if type == "array" then .[] else error("required_status_checks must be an array") end | (.context? // .)] | unique | .[]' <<<"${required_body}" 2>"${required_stderr}") || required_status=$?
  else
    required_status=$?
  fi
elif [[ "${required_response%%$'\n'*}" =~ ^HTTP/[0-9]+[.][0-9]+[[:space:]]+404[[:space:]] ]]; then
  rules_required=
else
  required_status=1
fi
if [ "${required_status}" -ne 0 ]; then
  rm -f "${required_stderr}"
  printf 'could not determine required checks for main: rules read failed; refusing to deploy\n' >&2
  exit 1
fi
rm -f "${required_stderr}"
required=
while IFS= read -r check; do
  [ -n "${check}" ] || continue
  if [[ $'\n'"${required}"$'\n' != *$'\n'"${check}"$'\n'* ]]; then
    required+="${required:+$'\n'}${check}"
  fi
done <<<"${classic_required}"$'\n'"${rules_required}"
if [ -z "${required}" ]; then
  printf 'could not determine required checks for main: main publishes no required checks in either source; refusing to deploy\n' >&2
  exit 1
fi
check_runs_status=0
check_runs=$(gh api "repos/${repo}/commits/${full_sha}/check-runs?filter=all&per_page=100" --paginate \
    --jq '.check_runs[] | [.id, .name, .status, .conclusion] | @tsv') || check_runs_status=$?
if [ "${check_runs_status}" -ne 0 ]; then
  printf 'Could not read check runs for %s on %s; refusing to deploy.\n' "${repo}" "${full_sha}" >&2
  exit 1
fi
pending=()
while IFS= read -r check; do
  [ -n "${check}" ] || continue
  newest_id=0
  newest_status=
  newest_conclusion=
  while IFS=$'\t' read -r id name status conclusion; do
    [ "${name}" = "${check}" ] || continue
    [[ "${id}" =~ ^[0-9]+$ ]] || continue
    if [ "${id}" -gt "${newest_id}" ]; then
      newest_id=$id
      newest_status=$status
      newest_conclusion=$conclusion
    fi
  done <<< "${check_runs}"
  if [ "${newest_id}" -eq 0 ]; then
    pending+=("${check} (absent)")
  elif [ "${newest_status}" != completed ]; then
    pending+=("${check} (${newest_status})")
  elif [ "${newest_conclusion}" != success ]; then
    pending+=("${check} (${newest_conclusion})")
  fi
done <<< "${required}"
if [ "${#pending[@]}" -gt 0 ]; then
  printf '  %s\n' "${pending[@]}" >&2
  printf 'Required checks are absent, pending or unsuccessful on %s; refusing to deploy. Re-run the gates part of this block once CI completes.\n' "${full_sha}" >&2
  exit 1
fi
git merge --ff-only "$full_sha"
serving_release=$(readlink -f /srv/overflow/.next || printf absent)
if [ "${serving_release}" != absent ] && [ -f "${serving_release}/REVISION" ]; then
  if [ "$(cat "${serving_release}/REVISION")" = "${full_sha}" ]; then
    printf 'Already serving %s (%s); the tree fast-forwarded to the serving commit, so install, migrate and build are skipped and the existing release stays.\n' "${serving_release}" "${full_sha}"
    exit 0
  fi
fi
npm_config_package_import_method=copy pnpm install --frozen-lockfile
. /srv/overflow/scripts/systemd-env-file.sh; export_env_file /etc/overflow/overflow.env
pnpm db:migrate
release=".next-release-$(date -u +%Y%m%dT%H%M%SZ)-$(git rev-parse --short=7 HEAD)"
mkdir "$release"
node scripts/release.ts prepare /srv/overflow "$release"
OVERFLOW_NEXT_DIST_DIR="$release" pnpm build
previous_release=$(readlink -f /srv/overflow/.next)
serving_cache="$previous_release/cache"
test -d "$serving_cache"
find /srv/overflow -path "$serving_cache" -prune -o \
  -exec chown -h root:overflow {} +
find /srv/overflow -path "$serving_cache" -prune -o \
  ! -type l -exec chmod u=rwX,g=rX,o= {} +
mkdir -p "$release/cache"
chown -R overflow:overflow "$release/cache"
chmod -R u=rwX,g=rX,o= "$release/cache"
printf 'Previous build: %s\nNew build: %s\n' "$previous_release" "$release"
pnpm release:switch /srv/overflow "$release" --expect-current "$expected_serving"
systemctl restart overflow.service
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/auth/providers
install -d -m 0700 /var/log/overflow
upgrade_log="/var/log/overflow/webhook-upgrade-$release.jsonl"
upgrade_status=0
pnpm --silent webhooks:upgrade > "$upgrade_log" 2>&1 || upgrade_status=$?
cat "$upgrade_log"
printf 'Webhook upgrade log: %s\nWebhook upgrade exit status: %s\n' "$upgrade_log" "$upgrade_status"
test "$upgrade_status" -eq 0 || exit "$upgrade_status"
printf '%s\n' "$full_sha" > "$release/REVISION"
printf 'Source revision: %s\n' "$full_sha"

ONE-TIME dependency migration for existing deployments

Before the first deploy using copy imports, remove the old node_modules and reinstall it with the copy setting. The package manager otherwise reuses unchanged packages, leaving their existing hardlinks intact. Run this once in a maintenance window: removing dependencies can interrupt requests from the running service. After the install succeeds, run the standing procedure above to build, restore tree ownership and permissions, restart and verify the service. Do not include this removal in routine deploys.

set -e
cd /srv/overflow
rm -rf -- node_modules
npm_config_package_import_method=copy pnpm install --frozen-lockfile

This breaks the deployment's links to the shared store; it does not repair ownership or modes already changed in the store or other checkouts.

The generated config belongs to $release and excludes serving and retained release validators; Next adds the new release's types to that file. Omitting preparation makes the build fail with the command needed to prepare this release. The tracked config stays clean, and set -e stops a failed preparation or build before the switch.

Retain the upgrade log and printed exit status with the deployment record. REPOSITORY_GONE means the by-ID lookup answered 404 and does not fail the run. It also covers a repository the sponsor's token cannot see, such as one made private without that token's access. Rerun pnpm webhooks:upgrade for any REPOSITORY_GONE line you doubt: a transient 404 during deployment can still leave a REVISION record, which makes redeploying that SHA skip the upgrade. A nonzero upgrade leaves the new release serving but the deployment incomplete; inspect each sanitized failure code, restore sponsor credentials or hook admin access as needed, and rerun the webhooks:upgrade package script with the same environment. An exception behind a failure also writes one stderr line to the same log, {"repositoryId":…,"failure":…,"cause":…} (no repositoryId for UPGRADE_FAILED): its class and message on one capped line, with connection string credentials, password values and GitHub and Overflow token prefixes redacted. Other token shapes in a message are not redacted, so keep the log private. subscription: VERIFIED and queue: NOT_NEEDED means the subscription was unchanged, the credential was already finalized and no earlier run left a repair owed, so no repair was requested. subscription: VERIFIED and queue: FAILED is partial completion; a rerun queues the repair. For a credential not yet finalized the rerun retries the finalization and its repair. Before updating an already-finalized hook the upgrade records that the repository owes a repair and clears the record only once the repair is queued (or the hook proved unchanged with nothing owed), so the rerun queues it once even when the earlier run's queue write failed or the forge's response to the update was lost (reported as SUBSCRIPTION_FAILED). A hook update the forge refused leaves the record too: the next run queues one full rederivation it may not have needed. Do not report the upgrade as successful from a restart or startup sweep. Queue acceptance does not establish that reconciliation has finished; check the worker's outcomes afterwards.

For section 5's one-time migration, run the upgrade/logging lines above after its switch/restart and section 10's HTTP verification. For a first installation or unit migration, run them after section 7's readiness verification with the environment from section 5 loaded. With no active registrations the command is a successful no-op.

The ownership reset keeps code root-owned and group-readable while preserving the serving cache's Unix permissions. Resolve .next before resetting ownership: find does not follow the symlink, so its exclusion must name the actual release's cache. -prune skips that directory and everything inside it in both passes; chown -h changes symlink ownership without following links, and the mode pass skips symlinks. The old process can keep writing its cache throughout preparation, even if preparation or switching stops before the restart. During the one-time migration the resolved path is the real .next, so the same exclusion preserves .next/cache. The copy imports described in section 5 keep these resets from changing package files in the shared store or other checkouts.

The new cache lives at /srv/overflow/$release/cache, so create and hand over that directory before the switch and restart. The unit still names /srv/overflow/.next/cache and resolves it through the symlink. A running process keeps its old writable mount across a swap, but that mount does not bypass Unix ownership or mode checks; preserving the old cache permissions is required too. The immediate restart picks up the new release's cache.

Expect active and HTTP 200, then exercise the application and inspect the journal as in section 7. Only prune after those checks succeed. Prune in the same shell that ran the deploy block above: the deploy lock is held until that shell exits, and the prune fence below re-locks fd 9 before touching anything, so pasting it into a fresh shell fails loudly by design instead of pruning unserialized. Before pruning, list the retained directories and confirm the recorded previous release is among the three greatest names; if it is older, raise --keep enough to include it or skip pruning. Failed build directories count too, and a rollback can make the previously served release older than the normal retention window. The one-time migration has no previous release directory to retain.

set -o pipefail
LC_ALL=C find /srv/overflow -regextype posix-extended -mindepth 1 -maxdepth 1 \
  -type d -regex '.*/\.next-release-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{7,40}' \
  -printf '%f\n' | LC_ALL=C sort -r
flock -w 900 9 || { echo "Could not acquire the deploy lock on /run/overflow-deploy.lock; refusing to deploy. Consult the deploy procedure's serialization notes before re-running." >&2; exit 1; }
pnpm release:prune /srv/overflow --keep 3

The release:prune package script takes <tree> [--keep N] and runs node scripts/release.ts prune <tree> [--keep N]; pass the arguments directly, without an extra -- separator, for both release package scripts. --keep must be a positive integer and defaults to 3. Like the listing above, the script enumerates only real directories directly inside the tree whose names match the release grammar in section 5, ignoring files, symlinks and all other directory names, and removes the <release>.tsconfig.json sidecar file beside each directory it deletes. It then sweeps orphaned sidecars: regular files named <release>.tsconfig.json at the tree root whose release directory is absent. It checks no build markers when pruning, so even a failed build with a matching name counts. It keeps the newest N directory names in descending lexical order, not by modification time or build success. It also protects the release .next resolves to and directories needed to resolve its symlink chain, even outside that N. It prints each removed directory and sidecar, or a no-op line if nothing was removed. A missing or dangling .next is reported but protects no release and does not prevent deletion; do not prune to recover from a failed switch. Pruning knows the symlink target, not which build a still-running process has loaded, which is another reason to restart and verify first.

Release-directory builds fix the build's writes under the serving output path; they do not isolate git pull or dependency installation, which still change the live checkout and node_modules. Testing on this host found that a same-lockfile install and a live git checkout did not disturb a running server: all application entry points are loaded at startup. That does not cover every lazy internal dependency. With one compiled package removed, a cold /_next/image request returned HTTP 500 with MODULE_NOT_FOUND and recovered once the package was restored. These results do not guarantee that dependency changes during an install are safe. This procedure also retains the restart interruption; it is not a zero-downtime deployment scheme.

If the revision includes a change to overflow.service, repeat section 6 as well: git pull updates the copy in the tree, not the one systemd reads. Nothing here refreshes /root/overflow.service.pre-hardening or the checkout it starts from, which is what the last paragraph of section 9 is about.

11. Rotating the credential encryption key

OVERFLOW_TOKEN_ENCRYPTION_KEY seals three kinds of stored credential: each user's GitHub OAuth token (users.encrypted_oauth_token), each linked forge access token (user_forge_identities.encrypted_token) and each registration's webhook secret (registered_repositories.encrypted_webhook_secret). Every value written in the current format names the key that sealed it and is bound to its own row; values stored before that format carry neither until they are re-sealed (see "Re-sealing without changing the key" below).

This section is the only sanctioned way to change OVERFLOW_TOKEN_ENCRYPTION_KEY. Replacing the value in the environment file any other way leaves every stored credential sealed under a key the service no longer holds, and each one fails until it is minted again: sponsors sign in again, forge identities are linked again, webhook secrets are replaced. For the same reason, never run pnpm credentials:reencrypt against the production database outside this procedure. The check-publishing App's private key is a different credential with its own rotation, which section 14 carries; this section's steps do not extend to it and must not be applied to it.

The rotation runs while the service keeps serving, and costs two restarts. It relies on the service reading two keys: OVERFLOW_TOKEN_ENCRYPTION_KEY is the current key, which seals every new write and opens what it sealed, and the optional OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS only opens. While both are configured, every row opens under whichever of the two sealed it.

Run every block as root from a fresh shell. Loading the environment file sets variables and never unsets them, so a shell that loaded the file before an edit keeps the values the edit removed. pnpm is on root's PATH and deliberately not on the service's (section 3).

Do not deploy (section 10) between step 3's edit and step 6's restart. A deploy restarts the service and runs the webhook upgrade with whatever the environment file holds at that moment, or, from a shell loaded earlier, with keys the file no longer holds. Finish or roll back the rotation first.

Before each database command block in this section, run the configuration-loading block in the same shell, including after a key-file edit.

Step 1: back up the database

Take a fresh dump first, the way backup-restore.md's drill does. Once the old key is retired, it is the only way back should the new key be lost.

set -e
cd /srv/overflow
bash scripts/db-backup.sh

The script prints the dump's path; record it with the rotation. The dump holds the credentials sealed under the old key, so it restores usefully only together with that key; step 3 keeps the old key in /etc/overflow/token-encryption-key.old for that reason. The backup script deletes overflow-*.dump files older than 14 days (section (d) of backup-restore.md); copy this dump to a name outside that pattern, in the same directory, if the rollback window has to outlast that.

Step 2: generate the new key

The key goes straight into a private file owned by the operator and is never printed, so it stays out of terminal scrollback. This block and the other guarded edits below run in a subshell, ( … ), so a refusal ends the subshell with a message and leaves your shell open:

(
set -e
test ! -e /etc/overflow/token-encryption-key.new || { echo "Refusing: token-encryption-key.new already exists." >&2; exit 1; }
umask 077
node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('base64url'))" > /etc/overflow/token-encryption-key.new
wc -c < /etc/overflow/token-encryption-key.new
)

Expect 43: 32 random bytes as unpadded base64url, the only form the service accepts. The refusal protects a key file left by an interrupted rotation; find out whether that key was ever configured before removing it.

Step 3: configure both keys and restart

This block selects OVERFLOW_TOKEN_ENCRYPTION_KEY when that line is present, otherwise TOKEN_ENCRYPTION_KEY. It keeps that old key in /etc/overflow/token-encryption-key.old and writes the selected line as OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS without changing its value, appends the new key as OVERFLOW_TOKEN_ENCRYPTION_KEY, confirms both lines read back as the two saved keys, and preserves the environment file's mode. Run these file edits as its owner, with write access to its directory; keep section 4's private permissions. The saved copy inherits the environment file's mode without changing ownership. It refuses, changing nothing, unless:

  • step 2's file holds a 43-character key;
  • the environment file ends with a newline, so the appended line cannot be glued onto the last one;
  • no line names TOKEN_ENCRYPTION_KEY_PREVIOUS with either prefix, comments included. A file copied from .env.example carries its commented # OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS= line and its explanation; delete both, and any empty OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS= line, before running the block;
  • exactly one line names the selected key variable at all, comments included (the pre-restart rollback counts the same way, so a file this block accepts is one the rollback accepts), and that line holds the bare value and nothing else: no quotes, spaces or comment. Rewrite a quoted line as OVERFLOW_TOKEN_ENCRYPTION_KEY=<value> first; systemd and the shell read the bare form identically;
  • no token-encryption-key.old from an earlier rotation is in the way. Such a file belongs with that rotation's dump, so rename it rather than removing it.
(
set -e
key_name=OVERFLOW_TOKEN_ENCRYPTION_KEY
if ! grep -q '^OVERFLOW_TOKEN_ENCRYPTION_KEY=' /etc/overflow/overflow.env; then
  key_name=TOKEN_ENCRYPTION_KEY
fi
grep -Eqx '[A-Za-z0-9_-]{43}' /etc/overflow/token-encryption-key.new || { echo "Refusing: token-encryption-key.new does not hold a 43-character key." >&2; exit 1; }
test -z "$(tail -c 1 /etc/overflow/overflow.env)" || { echo "Refusing: overflow.env does not end with a newline." >&2; exit 1; }
test "$(grep -c 'TOKEN_ENCRYPTION_KEY_PREVIOUS' /etc/overflow/overflow.env)" = 0 || { echo "Refusing: overflow.env already names OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS." >&2; exit 1; }
test "$(grep -c "$key_name" /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env must name $key_name on exactly one line." >&2; exit 1; }
grep -Eqx "$key_name=[A-Za-z0-9_-]{43}" /etc/overflow/overflow.env || { echo "Refusing: the $key_name value is not bare." >&2; exit 1; }
test ! -e /etc/overflow/token-encryption-key.old || { echo "Refusing: token-encryption-key.old already exists." >&2; exit 1; }
cp --preserve=mode /etc/overflow/overflow.env /etc/overflow/token-encryption-key.old
sed -n "s/^$key_name=//p" /etc/overflow/overflow.env > /etc/overflow/token-encryption-key.old
sed -i "s/^$key_name=/OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/" /etc/overflow/overflow.env
{ printf 'OVERFLOW_TOKEN_ENCRYPTION_KEY='; cat /etc/overflow/token-encryption-key.new; printf '\n'; } >> /etc/overflow/overflow.env
test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.new)" || { echo "The edited file does not read back as the saved keys; roll back as below." >&2; exit 1; }
test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.old)" || { echo "The edited file does not read back as the saved keys; roll back as below." >&2; exit 1; }
rm /etc/overflow/token-encryption-key.new
)

Before restarting, confirm that both values are keys the service will accept. The re-encryption script's check mode reads the same file and writes nothing:

cd /srv/overflow
check_status=0
pnpm --silent credentials:reencrypt --check || check_status=$?
printf 'Check exit status: %s\n' "$check_status"

Expect one JSON line for each of the three columns and, while any credential is stored, exit status 1, because nothing is sealed under the new key yet. (Node's experimental-feature warning on standard error accompanies every run of the script.) {"failure":"KEYS_INVALID"} instead means one of the two values is not a 32-byte base64url key: roll back as below, before the restart. The running service still holds the old configuration in memory, so nothing has changed yet.

Then restart, proving the switch the way section 7 does:

set -e
systemctl show overflow.service -p MainPID --value > /run/overflow-preswitch-mainpid
systemctl restart overflow.service
systemctl is-active overflow.service
curl --connect-timeout 5 --max-time 30 --retry 30 --retry-delay 1 \
  --retry-connrefused -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/readiness
printf 'MainPID before the switch: %s\nMainPID now:               %s\n' \
  "$(cat /run/overflow-preswitch-mainpid)" \
  "$(systemctl show overflow.service -p MainPID --value)"

Expect active, 200 and two different MainPID values. systemd reads the environment file only when it starts the process, so a MainPID that has not moved means the old process, holding only the old key, is still serving. From this restart on, the service seals every new or refreshed credential under the new key and still opens everything sealed under the old one.

Step 4: re-encrypt the stored credentials

cd /srv/overflow
reencrypt_status=0
pnpm --silent credentials:reencrypt || reencrypt_status=$?
check_status=0
pnpm --silent credentials:reencrypt --check || check_status=$?
printf 'Re-encryption exit status: %s\nCheck exit status: %s\n' "$reencrypt_status" "$check_status"

Load the keys from the service's own environment file, as here, and never type them on the command line. The script seals every row it rewrites under whatever OVERFLOW_TOKEN_ENCRYPTION_KEY it is given, so a valid key the service does not hold would re-seal those rows under a key the service cannot open.

The first command opens every stored credential not yet sealed under the current key, with either key, and seals it again under the current key, bound to its row. Rows whose credential is NULL are skipped. It prints one line per column, {"table":…,"column":…,"reencrypted":N,"alreadyCurrent":N,"skipped":N,"failed":N}, plus one line for each row it could not open (step 5), and exits 0 only when no row failed. skipped counts rows the service rewrote between the script's read and its write, or whose GitHub user id, forge identity or webhook credential id (what the seal is bound to) changed in that interval; the script leaves them as they now are. The script is idempotent and safe to re-run: a second run rewrites only what is still not current, so run the block again after any failure or interruption. {"failure":"REENCRYPTION_FAILED"} means the database could not be read or written; rows already re-sealed stay re-sealed. Its cause, when present, is the error's code or class, never its message: CONNECT_TIMEOUT when no database session was ready within five seconds of connecting, or a PostgreSQL SQLSTATE such as 57P01 (the server terminated the connection). The output names tables, columns, row ids, counts and that cause, never a credential or a key, so retain it with the rotation record.

The second command writes nothing. It prints {"table":…,"column":…,"current":N,"notCurrent":N} for each column and exits 0 only when every stored credential is sealed under the current key. It reads which key sealed each row; it does not open them. Both statuses 0 means step 6 may follow; a re-encryption status of 1 with UNDECRYPTABLE lines means step 5.

Step 5: rows reported undecryptable

{"table":"<table>","id":"<row id>","failure":"UNDECRYPTABLE"} names a row that neither configured key opens for that row: it was sealed under some other key, damaged, or copied from another row. The service could not read it before the rotation either, so the rotation did not cause it, and the script leaves it untouched. Each one is repaired by minting the credential again.

users. The user's next GitHub sign-in stores a fresh token. Without waiting for that, clear the reported row's token:

row_id='REPLACE-WITH-REPORTED-ID'
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update users set encrypted_oauth_token = null where id = '$row_id'"

Expect UPDATE 1; UPDATE 0 means no user has that id, so check it against the reported line. A NULL token reads as "no token" everywhere the service reads it, and the re-encryption skips it. Until the user signs in again, the repositories they sponsor do not reconcile ("GitHub access token was not available."), they cannot register a repository or read its labels, and the webhook upgrade reports CREDENTIALS_FAILED for their registrations. Each of those already failed while the token was unreadable. Signing in does not need the stored token, and it writes a new one.

user_forge_identities. The user's next link of the same forge identity stores a fresh token in the same row. Without waiting for that, clear the reported row's token; never delete the row:

row_id='REPLACE-WITH-REPORTED-ID'
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update user_forge_identities set encrypted_token = null where id = '$row_id'"

Expect UPDATE 1; UPDATE 0 means no identity has that id. The row keeps its provider, instance and forge user id, which is what GitLab authorship resolves through, so the user's GitLab work stays attributed to them; deleting the row would lose that. A NULL token is never selected for use and the re-encryption skips it, the same state account deletion leaves. Until the user links the identity again, everything that needs their GitLab token on that instance fails as it would with no identity linked: reconciling the GitLab repositories they sponsor there, registering or unregistering a repository there, reading its labels, and the webhook upgrade for those registrations. Each of those already failed while the token was unreadable.

registered_repositories. The sponsor unregisters and registers the repository again, or you clear the row's credential and let the webhook upgrade mint and configure a new one. The upgrade mints only for a registration holding no credential; one still holding an unreadable secret fails there as well.

row_id='REPLACE-WITH-REPORTED-ID'
psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update registered_repositories set webhook_credential_id = null, encrypted_webhook_secret = null, webhook_configured_at = null where id = '$row_id' and unregistered_at is null"

Expect UPDATE 1, then run the webhooks:upgrade package script with the same environment and retain its output, as section 10 describes. The hook rejects deliveries until the upgrade configures its new secret, as it already did while the old one was unreadable. UPDATE 0 means the id is not a registered repository: either it is mistyped, or the repository is unregistered, in which case the statement under "Webhook secrets of unregistered repositories" below clears it.

Then run step 4 again. Do not retire the previous key while any row still fails or while --check exits non-zero: a row it counts as not current may be one that only the previous key opens.

Step 6: retire the previous key

Only after step 4's --check has exited 0, with the environment file unchanged since. Remove the previous key, then repeat step 3's restart block with the same expectations:

(
set -e
sed -i '/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/d' /etc/overflow/overflow.env
)

Then, from a fresh shell:

cd /srv/overflow
check_status=0
pnpm --silent credentials:reencrypt --check || check_status=$?
printf 'Check exit status: %s\n' "$check_status"

It must print status 0 with "notCurrent":0 on every line. Then exercise the application and inspect its journal as in section 7. Keep /etc/overflow/token-encryption-key.old for as long as you keep step 1's dump.

Rolling back a rotation

  • Before step 3's restart, the service still runs on the old configuration. If only step 2 ran, rm /etc/overflow/token-encryption-key.new is all. If step 3's edit ran, drop the new key and give the old one its name back. The block removes the copy step 3 kept only after the file reads back as exactly that old key, on exactly one line that names the key. Otherwise it refuses and keeps the copy, the only clean one: repair the file by hand from /etc/overflow/token-encryption-key.old before restarting anything.

    (
    set -e
    test "$(grep -c 'OVERFLOW_TOKEN_ENCRYPTION_KEY' /etc/overflow/overflow.env)" = 2 && test "$(grep -c '^OVERFLOW_TOKEN_ENCRYPTION_KEY=' /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env does not hold exactly the two key lines step 3 wrote." >&2; exit 1; }
    test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.old)" || { echo "Refusing: the previous key does not match token-encryption-key.old." >&2; exit 1; }
    sed -i -e '/^OVERFLOW_TOKEN_ENCRYPTION_KEY=/d' -e 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/OVERFLOW_TOKEN_ENCRYPTION_KEY=/' /etc/overflow/overflow.env
    test "$(grep -c 'OVERFLOW_TOKEN_ENCRYPTION_KEY' /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env does not name the key exactly once; token-encryption-key.old is kept." >&2; exit 1; }
    test "$(sed -n 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY=//p' /etc/overflow/overflow.env)" = "$(cat /etc/overflow/token-encryption-key.old)" || { echo "Refusing: the key does not match token-encryption-key.old, which is kept." >&2; exit 1; }
    rm /etc/overflow/token-encryption-key.old
    )
  • While both keys are configured, from step 3's restart until step 6: swap them, so the old key is current and the new one previous, and restart with step 3's restart block. The block refuses unless the file holds exactly one current and one previous key line and names the key nowhere else; after step 6 there is no previous key to swap in, and swapping a lone key would leave the service with no current key:

    (
    set -e
    test "$(grep -c 'OVERFLOW_TOKEN_ENCRYPTION_KEY' /etc/overflow/overflow.env)" = 2 && test "$(grep -c '^OVERFLOW_TOKEN_ENCRYPTION_KEY=' /etc/overflow/overflow.env)" = 1 && test "$(grep -c '^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=' /etc/overflow/overflow.env)" = 1 || { echo "Refusing: overflow.env does not hold exactly one current and one previous key line." >&2; exit 1; }
    sed -i -e 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/OVERFLOW_TOKEN_ENCRYPTION_KEY=/' -e t -e 's/^OVERFLOW_TOKEN_ENCRYPTION_KEY=/OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS=/' /etc/overflow/overflow.env
    )

    Every row stays readable whichever key sealed it, and new writes go back under the old key. To finish, run steps 4 to 6 with the keys in that order; it is the same rotation, back to the old key. token-encryption-key.old then holds the current key, so remove it after that step 6.

  • After step 6, the rows are sealed under the new key alone. While the new key is still held, going back is another rotation by this section. If the new key is lost, the step 1 dump together with the old key is the only way back: restore it as section (e.2) of backup-restore.md describes, with OVERFLOW_TOKEN_ENCRYPTION_KEY set to the old key and no previous key. Everything written since the dump is lost.

Section 9's release rollback is not a key rollback. A build older than the key-identified credential format reads no OVERFLOW_TOKEN_ENCRYPTION_KEY_PREVIOUS and cannot open any credential written or re-sealed in that format, whichever key sealed it.

Re-sealing without changing the key

Every credential stored before the release that introduced key identifiers stays in the old format until something re-seals it. Such a value names no key and is bound to no row, so it still opens if it is copied into another row, and --check counts it as not current and exits 1 with no rotation under way. The service reads both formats, so nothing is broken; the row binding simply does not cover those rows yet. Nothing re-seals them on its own. Run this pass once, after that release is deployed.

When: only once the release is settled and no section 9 rollback to a release older than it is intended. An older release accepts only the old format and refuses every re-sealed value. The same is already true, with or without this pass, of every value the new release writes on its own: a sign-in, a forge relink, a registration and a webhook upgrade all store the new format. So a rollback past this release leaves those rows unreadable too. The remedy then is to clear each such row's credential as step 5 does, by id, and have its user sign in again, relink the forge identity, or register the repository again.

How: with no key change and no restart, since the service already holds the key the pass seals under:

  1. Take step 1's backup. That dump holds the old format, which an older release can still read.
  2. Run step 4's block. The environment file names only OVERFLOW_TOKEN_ENCRYPTION_KEY, so the script opens each old-format value with it and re-seals it under the same key, bound to its row.
  3. Repair any row it reports undecryptable as step 5 describes, and run step 4's block again until both statuses are 0.

After that, --check exits 0 until a rotation starts. It reads only which key sealed each row, so it cannot see a row sealed under the current key but bound to another row; the service refuses such a value, and step 5's remedy for its table applies.

Webhook secrets of unregistered repositories

Before either query below, run the configuration-loading block in the same shell.

Migration 048_unregistered_webhook_credentials.sql clears the webhook credential of every repository unregistered before the release that clears it at unregistration. Before deploying it, count the rows it will clear; the query only reads:

psql "$OVERFLOW_DATABASE_URL" -tAc "select count(*) from registered_repositories where unregistered_at is not null and webhook_credential_id is not null"

Its statement is idempotent. The previous release keeps serving from the migration until the switch (section 10) and does not clear the credential when it unregisters, so a repository unregistered in that window, or while a rolled-back release serves, keeps its secret; pnpm db:migrate never applies 048 a second time. After such a deploy or rollback, run the statement again and repeat the count, which must then print 0:

psql "$OVERFLOW_DATABASE_URL" -v ON_ERROR_STOP=1 -c "update registered_repositories set webhook_credential_id = null, encrypted_webhook_secret = null, webhook_configured_at = null where unregistered_at is not null and webhook_credential_id is not null"

12. Failure alerts

When overflow.service or overflow-backup.service enters the failed state, systemd starts overflow-alert@<failed unit>.service through that unit's OnFailure=overflow-alert@%n.service. The template unit runs /srv/overflow/scripts/overflow-alert.sh with the failed unit's name, and the script mails the failed unit's current-boot journal tail — the last 200 lines — to the address in /etc/overflow/alert-recipient.

Submitting the message is not the same as the message arriving, and the script never treats the first as the second. The local exim daemon answers a submission it has merely spooled with 250 OK id=<id>, and it writes Completed in its mainlog whether it handed the message to the smarthost or dropped it into a mailbox on this machine. Every signal below the SMTP conversation is therefore green for a message that reached nobody, and a check that stopped there would log an undelivered alert as delivered. So the script does not stop at the daemon's answer: it takes the id off the final 250 OK id=, follows that id into /var/log/exim4/mainlog, and calls the alert delivered only when exim Completed the id and the routing line for that same id named a transport that leaves this host. A run that cannot show both has failed, and it says which way in the alert unit's journal: the submission never reached the daemon, the daemon took it but named no id to follow, exim wrote a verdict against the id that ends the message for good (rejected, bounce, blackhole, discarded or Failed), the budget closed with no Completed behind it, or the message was routed to a transport that stays on this host. A defer is deliberately not a failure on its own — that is a temporary failure exim goes on to retry, so the script keeps waiting through it and a greylist does not manufacture a dead alert. Neither is a Failed to connect to line, the detail exim writes when the smarthost refuses the connection (section 53.9): that message is queued for its retry too, and it counts as a failure only where exim put a ** flag on the same line.

That definition is what the throttle keys on. Each failed unit is throttled to one message per 30 minutes, and the record is written only after the alert has actually left the host: a submission that failed, that could not be followed, and a message exim routed to this machine's own mail spool all leave no state behind, so the next failure mails again immediately instead of being silenced for half an hour by a delivery that never happened. A suppressed repeat is logged to the alert unit's journal and submits no mail, and the throttle state lives under /run/overflow-alert, cleared at reboot. The alert unit has no [Install] section and is never enabled: OnFailure= and a manual systemctl start overflow-alert@<unit> are the only ways it runs.

The recipient file is host configuration, not part of this repository: root only (root:root 0600), carrying exactly one line — the bare address. It is never committed and never referenced by value in the repository, and the script refuses anything but a single address on a single line (missing, empty, no @, or more than one line each exit 2 without sending), so a misconfigured file fails loudly in the alert unit's own journal instead of mailing a broken message.

What that address resolves to is host configuration too, and it is the question this section exists to answer. An address exim can deliver to on this machine is an address nobody is told anything at, so the next subsection is not optional reading: it is where the routing gets established.

Prerequisites

The local mail daemon must be running and able to relay: the script submits the message by SMTP to the local exim daemon, which relays it to the recipient. Check the daemon and the recipient file before installing anything:

systemctl is-active exim4
test -s /etc/overflow/alert-recipient && echo "recipient file present"

The first must print active and the second must print its confirmation. Neither says anything about whether the address leaves this host — a running daemon with a recipient it delivers to its own mailbox satisfies both, and that is the configuration the Verify step below exists to catch. Establish the routing first.

If the file does not exist yet, create it with the address and nothing else:

install -d -o root -g root -m 0700 /etc/overflow
[ -e /etc/overflow/alert-recipient ] \
  || install -o root -g root -m 0600 /dev/null /etc/overflow/alert-recipient
printf '%s\n' '<address>' > /etc/overflow/alert-recipient
chown root:root /etc/overflow/alert-recipient
chmod 0600 /etc/overflow/alert-recipient

Install

The script needs no install step — /srv/overflow is a checkout of this repository at the deployed revision, so each deploy ships it. Install the three unit files and reload:

install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow-alert@.service /etc/systemd/system/
install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow.service /etc/systemd/system/
install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow-backup.service /etc/systemd/system/
systemctl daemon-reload

Re-copying the two watched units is what replaces the copies section 6 (or a previous deploy) installed; on a host whose installed units already carry local edits, diff before overwriting.

Prove the recipient leaves the host

Nothing in this repository configures where the message goes. The routing is exim's own host configuration under /etc/exim4/, and the change this section documents does not apply it — what follows is a runbook for the operator, not a record of something that has been done here. Run it after the units are installed, and again whenever the address in /etc/overflow/alert-recipient changes.

exim -bt runs the routers and the address verification and prints the decision without queueing anything, so it answers this question without sending a message. Read the address into the argument rather than typing it, so it stays out of the shell history and out of anything pasted afterwards:

exim -bt "$(cat /etc/overflow/alert-recipient)"

Two shapes come back, and telling them apart is the whole job.

A local mailbox write resolves like this: R: mail4root for <address>, then <address> -> /var/mail/mail, then transport = address_file. The message ends in a file on this machine, nobody is told anything, and the script will report the alert as a failure — correctly, because from the operator's side that is what it is.

A smarthost resolves like this: R: smarthost for <address>, then <address>, then router = smarthost, transport = remote_smtp_smarthost, then host <smarthost> [...] port=587. That transport leaves the host, which is what the script requires.

The discriminator is transport = on the resolution, not R: on the line above it. R: names the router that made the decision, and mail4root is a perfectly good name for a router that then hands the message to a local delivery agent. Read the transport.

On this host, as measured on 2026-10-01, the alert recipient already leaves the host. exim -bt above returned router = smarthost, transport = remote_smtp_smarthost with host smtp.gmail.com [...] port=587, and the retained logs agree:

zcat -f /var/log/exim4/mainlog* | grep -F "$(cat /etc/overflow/alert-recipient)" | grep -oE 'T=[a-z_]+' | sort | uniq -c
zcat -f /var/log/exim4/mainlog* | grep 'T=address_file' | grep -cF "$(cat /etc/overflow/alert-recipient)"

Five lines, every one T=remote_smtp_smarthost, and zero local writes to that address. So the routing decision below is one to re-make, not one to make — run the two commands again whenever the address changes, and treat a 0 on the second as the thing to notice.

If the first shape is what comes back, the decision is yours, and it is not the repository's:

  • Route the address off the host. The configuration the Verify step below accepts, and the only one that can page a person.

  • Keep the local mailbox as a retained record. Defensible — a mailbox on the box is a durable record of every failure, and it is one of the reasons this host has an audit trail at all. It carries two obligations you have to accept deliberately rather than discover.

    The first is that the script will treat every such alert as a failure and say so in the alert unit's journal, forever. That is the intended reading, not a defect to route around: the record is not the alert. The second follows from the first, and is easier to miss. Because a local write records no throttle state, the 30-minute throttle never engages on this configuration at all — there is no state file for it to read, so every OnFailure= trigger for the unit mails, and a unit in a crash loop produces a message per failure rather than one per half hour. The service manager's start limit is the only thing bounding that, and it bounds a fast loop only: a unit that trips its start limit inside the interval is a unit that is not starting, and one that cycles more slowly than the window mails once per cycle for as long as it keeps cycling.

    The mailbox grows either way, and that obligation is not conditional on this decision. /var/mail/mail on this host is root's own system mail regardless of where the alert recipient routes — every retained T=address_file line is R=mail4root ... => /var/mail/mail <root@vmi3458323.contaboserver.net>, and the alert address appears in none of them. No logrotate entry on this host covers /var/mail/mail, so a retained record is an unbounded one unless you write the rule yourself, whether or not it ever holds an alert:

    ls -l /var/mail/mail                                   # size
    grep -c '^From ' /var/mail/mail                       # message count
    zcat -f /var/log/exim4/mainlog* | grep -c 'T=address_file'   # local writes, all mail

    Those counts describe all mail this host has relayed, not the alert path's share of it. Measured 2026-10-01 they read 528 and 7, and they are a snapshot that only climbs as the log fills — which is exactly why the alert path's own routing is answered by the two commands above rather than by either number.

The deployment is not verified until an off-host delivery has been observed, not until one was attempted and the daemon accepted it. The Verify step below is what observes it, and a run that ends any other way is a run this section does not consider a pass.

Verify

Both watched units must name the alert template:

systemctl show overflow.service overflow-backup.service -p OnFailure

Each line must name the alert template with that unit's own name as the instance: OnFailure=overflow-alert@overflow.service.service for the web unit and OnFailure=overflow-alert@overflow-backup.service.service for the backup unit. systemd replaces the units' %n specifier when it loads them, so the readback shows the expanded name — including the template's own .service suffix, doubled next to the instance — not the raw OnFailure=overflow-alert@%n.service line the repository's unit files carry.

The alert template must also carry the group that lets the script read the exim mainlog it decides from:

systemctl show overflow-alert@test.service -p SupplementaryGroups --value | grep -qx adm && echo "the alert unit reads the exim log as a member of adm" || echo "the alert unit is NOT in group adm - every alert will wait out its budget, report that it did not leave this host, and record no throttle state"

This is not tidiness: the script follows the accepted message id into /var/log/exim4/mainlog, which is 0640 Debian-exim:adm, and the unit's CapabilityBoundingSet= is empty — so root has neither CAP_DAC_OVERRIDE nor CAP_DAC_READ_SEARCH and is refused the open() without this group. A unit missing it mutes the path while making it look broken.

Then send a real message through the whole route with a throwaway instance — the instance name need not be a unit that exists, and the message's subject names it: [overflow] test failed on <host>. Mark where this run starts before you start it — started=$(date -u '+%Y-%m-%d %H:%M:%S') — and keep it in the same shell, because the id below is read from this run's lines only and an earlier run's id is a worse mistake than no id at all.

systemctl start overflow-alert@test.service
journalctl -u overflow-alert@test.service --no-pager -n 20
tail -n 20 /var/log/exim4/mainlog

The script's own line is the verdict, and it is one of two shapes. A delivered alert reads overflow-alert.sh: exim routed <id> to <transport> and Completed it; the alert left this host, and the alert unit exits 0. Anything else reads overflow-alert.sh: <reason>; the send is NOT recorded, so the next failure for test.service alerts again, and the alert unit exits nonzero — systemctl show overflow-alert@test.service -p ExecMainStatus reads 0 on the first shape and something else on the second. Read that status in one direction only: nonzero is a failure, and zero is not a delivery. A run the throttle suppressed exits 0 having sent nothing, so a status of 0 alongside a journal that never carried a verdict is a suppressed run, not a delivered alert — the journal line is the verdict and the status only confirms it. The reason names which failure it was: the submission never reached the daemon, the daemon named no id to follow, a verdict exim recorded against the id, a local transport the message was routed to, no Completed for the id inside the 60-second budget (with or without a verdict written against it), or a mainlog that could not be read at all. A run that ends in the second shape has failed, whatever the SMTP conversation looked like, and the second shape's own wording is the finding — read it rather than re-running.

That tail is for seeing the shape of the traffic, not for finding your delivery. Match on the id, never on the word Completed. The mainlog holds every message this host has relayed, and on a host whose recipient resolved to a local mailbox the alert path's own traffic fills it with Completed lines that mean a file write — the last twenty lines will usually hold several, and browsing them for one that looks like a delivery will find a file write and call it a pass.

<id> is the handle onto this send and onto nothing else. Take it from this run's journal lines rather than typing it — journalctl -u overflow-alert@test.service --no-pager --since "$started" -n 20 | grep -oE '[0-9A-Za-z]{6}-[0-9A-Za-z]{6,}-[0-9A-Za-z]{4,}' | tail -n 1 — and substitute it for <id> in the three commands that follow. They are the whole verification, and each one discriminates on your id and nothing else.

grep -F "<id> Completed" /var/log/exim4/mainlog prints the Completed line for your message, or nothing.

grep -F "<id> =>" /var/log/exim4/mainlog prints your routing line verbatim, R=<router> T=<transport>, or nothing.

awk -v id="<id>" '{at=index($0,id); if(at==0) next; a=index($0,"=>"); if(a==0||a<=at) next; r=substr($0,a+2); if(match(r,/T=[^[:space:]]+/)) t=substr(r,RSTART+2,RLENGTH-2)} END{if(t!="") print t; else print "(no routing line)"}' /var/log/exim4/mainlog reduces that line to the single word the script judges on — the last routing line for the id, which is the current decision on a message that deferred and was re-routed. This is the same extraction the script itself performs.

If that prints nothing, stop there. There is no send to follow, and every one of the three commands then runs against an empty <id> — the first grep matches any Completed line in the log, the second any routing line, and the awk reduces whichever of those carries a T=, so all three would "pass" on some earlier, unrelated message. That is the exact false green this step exists to end, and --since is what makes the empty answer trustworthy: it means this run named no id, not that the window was too narrow. Two outcomes produce it, and they want different responses. A refused submission reports a reason and never names an id — nothing was sent, so there is nothing to follow and the reason is the whole finding. A suppressed run logs one line, last alert for <unit> was <age> seconds ago, inside the 1800-second throttle window; suppressing, and that line carries no id either — which is the second start described below, and it means this run deliberately sent nothing. Read which of the two you are looking at before you conclude anything. If more than one id comes back, the marker was captured inside an earlier run's second: mark again and re-run.

All three, and only all three, is a pass: a Completed line for your id, a routing line for your id, and a transport that leaves the host. The Completed line on its own carries no T= at all and cannot say where the message went; the routing line on its own is a decision exim had not yet carried out. The third command's word is the verdict, and the list of transports the script treats as staying on this machine is a maintenance obligation kept in the script rather than restated here, so read it from the deployed copy with sed -n '/^leaves_host()/,/^}/p' /srv/overflow/scripts/overflow-alert.sh.

Anything on that list — address_file is what a local mailbox write resolves to — is a failed alert, and so is the (no routing line) case. When the transport is remote_smtp_smarthost or another name not on the list, the alert left the host.

A fourth check, and it is the one that catches a message the relay refused. Exim writes Completed when the daemon is finished with a message, which includes one it gave up on: a bounce, a rejection and a discard all end with the same line a successful delivery does. So all three commands above can pass on a message that was permanently refused, and the script — which reads a terminal verdict before it reads Completed, for exactly that reason — will have called it a failure. Check for one with grep -F "<id> **" /var/log/exim4/mainlog and grep -F "<id> Failed" /var/log/exim4/mainlog. Either printing nothing is what a pass requires. Anything they print is a verdict exim recorded against that id, and it is the finding — rejected, bounce, blackhole, discarded or Failed all end the message. There are two exceptions, and both mean the message is queued and retrying rather than finished: ** defer, a greylist or a temporary 4xx; and Failed to connect to, the detail line exim writes when a smarthost refuses the connection (section 53.9 of the exim specification), written before the deferral for the same id. The script keeps waiting through both, and so should you — and the second counts as an exception only on a line carrying no two-character flag. Where exim rejected the message and the rejection reason happens to quote a refused connection (** rejected: … 550 Failed to connect to …), the flag wins and the message is finished.

Before you read an absent verdict as a broken relay, check there was a log to read. /var/log/exim4/mainlog is rotated daily by /etc/logrotate.d/exim4-base, and that rule carries nocreate: rotation moves the live file to mainlog.1 and does not leave an empty one in its place. The daemon puts it back the next time it has something to log, so on a host whose mail goes quiet the file is simply absent — for a day, or for as long as nothing is delivered — and every command above then reports nothing at all. No Completed line, no routing line, (no routing line) from the awk: an empty verdict that is evidence about nothing at all, including the relay. This is not a rare race on this host. The previous rotation, at 00:08, was followed by enough traffic to fill the file again by 23:38; the next one, at 00:38 the following morning, was not, and the file was still absent when this section was written. A rotation with nothing after it leaves the verdict unreadable until the next message, not until the next rotation.

The script fails closed on an unreadable log, which is the right direction to fail: it takes a branch of its own and names the path in its verdict — /var/log/exim4/mainlog is missing or unreadable, so no off-host acceptance of <id> could be observed within 60s — so the alert unit exits nonzero having reported no verdict, which is a different report from a verdict of failure. A failed alert unit is therefore not by itself evidence that the relay is broken; on this host, at the time of writing, a failed alert unit with that verdict is exactly the absent log and nothing else. A verdict that names the log is telling you where to look, not what broke.

So check the file before drawing any conclusion from the absence of a verdict — ls -l /var/log/exim4/mainlog — and if it has rotated out from under you, the answer is in the rotation, which zcat -f /var/log/exim4/mainlog* reads across the retained files. Substitute that in for /var/log/exim4/mainlog in the three commands above and your id resolves against the same rules, because the same rules are what the script applied. A run whose verdict you can only find in mainlog.1 was a real run judged late, and the delivery or the failure recorded there is the real one.

The throwaway instance leaves one file behind, and which file it leaves is itself a check: an alert that left the host writes its throttle state under /run/overflow-alert, and one that did not leaves no state at all. So ls -l /run/overflow-alert showing a state file for test.service is independent evidence of a delivery, and it is the one that catches a run whose journal reads like a pass and whose mailbox never grew. The absence of one after a run the journal called delivered is a contradiction — chase it rather than tidy it away.

A second start of the same throwaway instance within half an hour mails nothing — but only if the first one left throttle state behind, and that is exactly what a failed delivery does not do. Suppression reads a state file under /run/overflow-alert, and the delivered path is the only one that writes one, so on a host whose recipient resolves to a local mailbox there is no file to read and the second start is not suppressed at all: it runs the whole verdict again, waits out its own 60-second budget, and appends a second verdict line to the journal and a second write to /var/mail/mail. Two starts, two verdicts, two writes — and the state file still absent, which is the check above telling you the same thing twice.

When the first run did leave state, the second start is silent: the script logs last alert for test.service was <age> seconds ago, inside the 1800-second throttle window; suppressing and submits no mail, and that line carries no message id, so the id extraction above comes back empty and the guard there applies. That is the only case in which a second start is expected to be silent, and it is the one case where the state file's presence is not news. Remove the instance's state file under /run/overflow-alert to send again immediately.

What this step does not check, stated so it is not assumed: the trigger. It proves the delivery half — that when the alert script runs, a message leaves this host. It does not prove that anything makes it run. A unit that cannot serve does not exit, so it never trips the OnFailure= condition the template follows, and nothing else on this host observes that state either: a wedged overflow.service is silent in exactly the same way a healthy one is until something asks it a question. So a clean run through every check above is a statement about the alert path only. The half this repository does not close is noted in the issue, and until it is, treat "the alert path verified" as distinct from "a failure would have been reported".

Rollback

Revert the two OnFailure= lines — re-copy the units without them, or edit the installed copies — remove the alert template, and reload:

rm /etc/systemd/system/overflow-alert@.service
systemctl daemon-reload

The watched units work unchanged without the wiring: nothing else references the alert unit, and a failure of overflow.service or overflow-backup.service is visible in the journal.

The alert path canary

Everything above is built around one route leaving the host: overflow-alert.sh submits to the local exim daemon, exim relays to the smarthost, the smarthost delivers to one mailbox — and the subsection above is what establishes that it does. Nothing on that route reports its own failure, so a route that stopped delivering looks exactly like a host with nothing to report. When enabled, overflow-canary.timer starts overflow-canary.service daily at 03:20 UTC to check that route.

On the documented deployment the timer is disabled by maintainer decision, because the daily mail was noise; the maintainer accepted the consequence. The scheduled check does not run, so a silent mail-path failure is not checked automatically by the canary. The unit files still ship, and the install and verification instructions below remain available for an operator who chooses to restore that check.

With the canary installed and its host files configured, run one check by hand and inspect its verdict:

systemctl start overflow-canary.service
journalctl -u overflow-canary.service --no-pager -n 20

To resume the daily check, run systemctl enable --now overflow-canary.timer, then systemctl list-timers overflow-canary.timer --no-pager and systemctl is-enabled overflow-canary.timer. The persistent timer can also catch up a missed check when activated. The install block below enables the timer too; running it on the documented deployment ends the suspension.

The canary runs the same script and the same route, then reads the verdict from somewhere the route cannot fake: it submits with curl -v, takes the exim message id off the final 250 OK id= line, and waits for a line reading <id> Completed in /var/log/exim4/mainlog. A relay that defers, rejects, bounces or discards the message logs defer, rejected, bounce, blackhole or discarded against that id instead, and the script names that verdict in the report. A Failed to connect to line is not one of those: it is the detail a refused connection leaves behind (section 53.9), the message is still queued, and the script waits it out rather than calling it dead — unless exim put a ** on that same line, which makes it a verdict again. The absence of Completed is the failure, and it is the only observation on this host that can see the smarthost leg at all.

It reads that log as a member of adm, and cannot read it at all without that membership. /var/log/exim4/mainlog is 0640 Debian-exim:adm in a 2750 Debian-exim:adm directory, and the canary unit runs with CapabilityBoundingSet= empty — the same hardening the alert template beside it uses, and for the same reason it is not merely safe there. An empty bounding set strips CAP_DAC_OVERRIDE, which is the only reason a root process could open a file whose group it did not hold; ProtectSystem=strict is not what decides this, because it governs the mount namespace and not file access. The alert script reads that same mainlog to decide whether an alert left the host, so the alert template carries SupplementaryGroups=adm too, pinned to adm in tests/deploy/alert-units.test.ts for the reason given below. Both units therefore carry the directive: the ordinary group mechanism, no capability, and the group this host's own log files are already owned by. It is wider than one file — group adm also owns the nginx, postgresql, fail2ban, cloud-init, tor, privoxy and tinyproxy logs here — so it is pinned to adm in both unit tests rather than admitted by name with its value left open.

If that membership is ever wrong, the canary says so instead of blaming the relay. Before it submits anything the script checks that it can read the log, and if it cannot, it refuses: exit 2, no message sent, and a report whose header is "the canary on <host> cannot run" — never the outage header "the failure-alert path is not delivering", which would be a false claim about a relay that may be perfectly healthy. The same check runs again at the deadline, so a log that goes unreadable during the wait is refused the same way rather than reported as a relay that timed out.

The refusal has its own dedup state, /run/overflow-canary/canary-fault, separate from the dead-streak marker. That separation is deliberate in both directions: the fault marker keeps a canary broken for a week to one report rather than seven, and the dead-streak marker is never written by a refusal, because it means "a real outage of the alert path was reported" and a run that never reached a verdict is not one — writing it there silences the next real outage, which is the failure this whole feature exists to prevent. Nothing on this host reads the canary's exit status, which is why a refusal posts at all: a canary that quietly stopped checking would be, one level up, the same silent route this section exists to end. A healthy run retires the fault marker, and so does a run that reports a real outage after a stretch of blindness.

That distinction is the whole point: without it, a canary lacking the membership and capability to open the log spends its full 60-second budget proving it and then reports that the relay never completed a message exim had logged as Completed, a false page, on the one channel the maintainer trusts, about a healthy host.

What Completed means, precisely — and what it does not. Where the message was routed off the host, it means the smarthost accepted the message. That covers the classes worth naming: a stopped daemon (curl fails outright), revoked SMTP credentials, a daily limit, greylisting, and any 5xx at RCPT or DATA — all of which leave a defer or rejected line and never a Completed. What it does not cover is the most likely way a mail alert path quietly dies: a recipient address that is wrong, deleted or converted to a route that silently accepts.

That case is not detectable from here, and pretending otherwise would be worse than saying so. Measured against the real smarthost and an RFC 2606 reserved domain, so that delivery is impossible by construction, the relay answers 250 2.0.0 OK … - gsmtp, exim logs Completed, and the canary reports the path healthy, which is correct for what it can see. Once the message has left the queue there is no observation available through the local exim that distinguishes "the smarthost accepted it" from "the remote mailbox exists" — exim -Mvb inspects queued messages, and such a message is long gone by then.

So the detector for a dead mailbox is the heartbeat itself stopping arriving, and it is the operator's job, not the script's. This canary proves the pipe is open; it cannot prove there is a person at the other end. Keep the recipient address somewhere you will notice it is stale, and treat a run of missing canary mail as a finding in its own right — the same weight as a failed backup.

One limitation belongs in this subsection rather than in a changelog, because this is where a reader decides what the canary is worth. Completed says the daemon finished with the message and carries no transport, so a canary addressed to a local mailbox reports a healthy path for a message that never left this host. scripts/overflow-canary.sh does not read the routing line and does not distinguish a local write; scripts/overflow-alert.sh does, and reports the alert as a failure instead. The alert path proves the message left the host. The canary does not. Nothing in this change alters the canary, and a green canary run is not evidence about where its own message went — the routing proof for the canary is the exim -bt check above, run against /etc/overflow/canary-recipient.

A dead path is reported out of band, to the Discord webhook in /etc/overflow/canary-discord-webhook — a channel that shares no leg with the route being checked — and the oneshot exits nonzero, so the unit also lands in the failed state. One report per dead streak: the marker under /run/overflow-canary records an outage that has been reported, so a path dead for a week posts once rather than seven times, and a run that completes clears the marker so the next failure posts again. The marker is written only once a report has actually been delivered, so an unreachable webhook leaves the next run free to try.

Both host files are configuration, like alert-recipient, and are never committed or referenced by value here. The recipient file is validated exactly as alert-recipient is — one line, one address, no CR — and the webhook file must be a single nonempty line. A file that fails either check exits 2 and sends nothing, so a misconfiguration is loud in the canary's own journal rather than a canary quietly reporting health through a channel it cannot reach. The exim mainlog is checked before the submission for the same reason, and a log this process cannot read yields no verdict — which is reported as the canary being broken, never as the relay failing.

Prerequisites

systemctl is-active exim4
test -s /etc/overflow/canary-recipient && echo "recipient file present"
test -s /etc/overflow/canary-discord-webhook && echo "webhook file present"
systemctl show overflow-canary.service -p SupplementaryGroups --value | grep -qx adm && echo "canary reads the exim log as a member of adm" || echo "the canary is NOT in group adm - every run will refuse with the log named as unreadable"

The second, third and fourth commands must each print their confirmation. The fourth is an exact match on purpose: --value prints the bare adm on this host and prints an empty line with exit 0 when the directive is unset, so grep -qx adm fails closed in both cases, and it also fails on a drifted installed copy that has picked up a second group — which -w would have passed. If the fourth prints its warning, the installed unit is not the one in this repository: reinstall it from /srv/overflow/deploy/overflow-canary.service rather than editing the installed copy, because the reviewed set in tests/deploy/canary-units.test.ts is what keeps the next directive beside this one from arriving unnoticed. Create the files with the address and the webhook URL and nothing else:

install -d -o root -g root -m 0700 /etc/overflow
install -o root -g root -m 0600 /dev/null /etc/overflow/canary-recipient
printf '%s\n' '<address>' > /etc/overflow/canary-recipient
chown root:root /etc/overflow/canary-recipient
chmod 0600 /etc/overflow/canary-recipient
install -o root -g root -m 0600 /dev/null /etc/overflow/canary-discord-webhook
printf '%s\n' '<webhook-url>' > /etc/overflow/canary-discord-webhook
chown root:root /etc/overflow/canary-discord-webhook
chmod 0600 /etc/overflow/canary-discord-webhook

Install

The script needs no install step — /srv/overflow is a checkout of this repository at the deployed revision, so each deploy ships it. Note that scripts/deploy-revision.sh does not reinstall units, so a host that has already been deployed needs these two copies run by hand. Install the two files, reload, and enable the timer:

install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow-canary.service /etc/systemd/system/
install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow-canary.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now overflow-canary.timer
systemctl list-timers overflow-canary.timer --no-pager

Verify

First, prove the real webhook is live. Every other check in this section either never posts to it or posts to a loopback stand-in, so without this one nothing in the procedure establishes that the configured Discord webhook still exists and still accepts posts — and a canary reporting into a deleted webhook fails in silence, which is the failure mode the whole section is about.

The URL is read from the host file into a shell variable and is never echoed, printed or pasted; only the HTTP status leaves this block. --fail makes a 404 Unknown Webhook or a revoked token a nonzero exit, which is the point: without it curl would call a refusal a success.

webhook=$(cat /etc/overflow/canary-discord-webhook)
printf '%s' '{"content":"[overflow-canary] section 12 verification post - the failure-alert canary is being installed and this webhook is live."}' | curl -sS --fail --max-time 15 --connect-timeout 5 -H 'Content-Type: application/json' --data-binary @- "$webhook" -o /dev/null -w 'webhook answered %{http_code}\n'

Expect webhook answered 204, and a message saying the canary is being installed must appear in that channel — that is the confirmation, and it is what a delivered report looks like.

Scope this honestly, because it is narrower than it looks: it proves the webhook URL is live and accepting posts, not that the alert recipient mailbox exists — see what Completed does and does not mean above. And it sends one real message, so run it once, not on a loop.

Then prove both directions of the mail path, healthy first and then broken. Never send a burst at the address in /etc/overflow/alert-recipient to do this.

A healthy run must exit zero and leave no marker:

systemctl start overflow-canary.service
journalctl -u overflow-canary.service --no-pager -n 20
tail -n 5 /var/log/exim4/mainlog

The journal must name the relay's Completed line for the message id it submitted, and the exim mainlog must carry a Completed line for that same id. Both are about the id, and the id is what makes them evidence — a Completed line found by browsing the log belongs to some other message. This says nothing about whether the message left the host; the limitation above is the whole of that, and the exim -bt check there is what covers it. Confirm the unit did not record an outage, and that the timer is armed for the next 03:20 UTC:

test ! -e /run/overflow-canary/dead && echo "no outage recorded"
systemctl list-timers overflow-canary.timer --no-pager

If a scheduled run ever reports the path dead and the mail route looks fine, check for a log rotation before you believe it — and read the journal line before you conclude anything from it, because one rotation can produce two different reports. The verdict is a single-file read of /var/log/exim4/mainlog, and this host rotates it with nocreate (see /etc/logrotate.d/exim4-base), so the new file is created by exim itself when it next reopens the path. Which of the two you get depends on when exim gets to it:

  • The path is recreated before the wait closes — the common case, and the one that reads as an outage. The Completed line is sitting in mainlog.1, the new mainlog is empty but perfectly readable, so the run finds no Completed, exits 1 and posts the not delivering report. That is a false page, and it fails closed: a rotation can produce a false dead verdict but never a false healthy one.
  • The path is still absent when the 60 seconds run out — a narrow window between the rename and exim's reopen. The canary cannot read a file that does not exist, so it refuses instead: exit 2 and the "the canary on <host> cannot run" report, never a claim about the relay.

So the journal line is what tells you which you are looking at, and a line naming the log as unreadable is a rotation or a membership that has gone — not an outage to act on — while a line naming a relay verdict is a real one, or a rotation to rule out first. It is cheap, it is rare, and the alternative is chasing a relay that is working:

ls -l --time-style=full-iso /var/log/exim4/mainlog /var/log/exim4/mainlog.1

Then prove the dead direction, which is the one the canary exists for. Take a throwaway copy of the script, point its OVERFLOW_CANARY_SMTP_URL at a closed port, and run it. The port below is 127.0.0.1:1, which nothing listens on; check yours with ss -ltn first, because a port that answers proves nothing. The run must post to the webhook and exit nonzero:

cp /srv/overflow/scripts/overflow-canary.sh /tmp/canary-probe.sh
chmod +x /tmp/canary-probe.sh
OVERFLOW_CANARY_SMTP_URL=smtp://127.0.0.1:1 \
  OVERFLOW_CANARY_STATE_DIR=/tmp/canary-probe-state \
  /bin/sh /tmp/canary-probe.sh

A nonzero exit is the pass condition here, and the report must have arrived in the channel the webhook file names. A second identical run posts nothing, because the probe's marker records the outage:

OVERFLOW_CANARY_SMTP_URL=smtp://127.0.0.1:1 \
  OVERFLOW_CANARY_STATE_DIR=/tmp/canary-probe-state \
  /bin/sh /tmp/canary-probe.sh

Remove the probe and its state, then confirm the real unit is untouched:

rm /tmp/canary-probe.sh
rm -rf /tmp/canary-probe-state
systemctl show overflow-canary.service -p ExecStart -p Environment

Proving the sandbox can reach the out-of-band channel

The two steps above each run one thing: the first drives the real unit, the second runs a copy of the script with nothing between it and the host. The only one that exercises the webhook POST from inside overflow-canary.service's sandbox is neither of them. That matters more here than it does for the alert template, because the alert unit only ever talks to 127.0.0.1 — every socket it opens is loopback — whereas the canary has to resolve discord.com and complete a TLS connection under ProtectSystem=strict, PrivateTmp, PrivateDevices, RestrictAddressFamilies and SystemCallFilter=@system-service. systemd-analyze verify is a parser, not a runtime, and it will happily accept a unit whose seccomp filter or address-family mask blocks the egress.

So prove it at runtime. PrivateTmp gives the unit a private /tmp and /var/tmp but does not isolate the loopback interface, so a listener started outside the sandbox on 127.0.0.1 is reachable from inside it. The step below forces the dead path (SMTP to a closed port) and points the webhook at that listener, so the unit posts out of band and records the outage only if the sandbox permits the connection. This proves the network leg of the sandbox; it does not send a real Discord message, and it says nothing about whether the webhook URL in /etc/overflow/canary-discord-webhook is still valid, nor about DNS or TLS to discord.com — a loopback listener is neither of those. The step that covers the real channel is the first one under Verify, which posts to the configured webhook itself.

The verdict below is the payload the listener actually received. A listener that has not finished binding when the unit posts makes the report fail for a reason that has nothing to do with the sandbox, and a step that reports its finding when the finding is absent is worse than no step: it teaches the operator to distrust the one check that would have caught a real regression. The bounded wait below is what keeps the two apart — and nothing in this step reads /run/overflow-canary/dead, so there is no second signal to fall back on: the recorded payload is the whole verdict, which is why the wait has to be trustworthy rather than merely present.

Check the port is free, then install a drop-in that redirects the run. It overrides the two paths, so the host's own webhook file is never read, written or exposed:

ss -ltn | grep 18099 && echo "port 18099 is BUSY - stop, pick another port and re-run this step" || echo "port 18099 is free"
install -d -o root -g root -m 0755 /etc/systemd/system/overflow-canary.service.d
printf '%s\n' 'http://127.0.0.1:18099/probe' > /etc/overflow/canary-sandbox-probe-webhook
rm -f /etc/overflow/canary-sandbox-probe-received
printf '%s\n' '[Service]' 'Environment=OVERFLOW_CANARY_SMTP_URL=smtp://127.0.0.1:1' 'Environment=OVERFLOW_CANARY_WEBHOOK_FILE=/etc/overflow/canary-sandbox-probe-webhook' > /etc/systemd/system/overflow-canary.service.d/sandbox-probe.conf
systemctl daemon-reload

The first line is a precondition, not a report. If it prints BUSY, stop here: the listener below would die on EADDRINUSE without saying so, and the unit's report would be posted to whatever stranger holds that port.

Start the listener, then wait for it to be listening before starting the unit. The wait is bounded, and deliberately its own block: an unbounded wait cannot tell "not bound yet" from "died a minute ago", and a step that leaves the operator watching a silent spin with a systemctl start queued behind it fails in the same unreadable way this whole section exists to prevent. Fifteen attempts is roughly sixty times the bind time measured on this host, so it cannot reintroduce the race a bare start would have:

python3 -c "import http.server as h;H=type('H',(h.BaseHTTPRequestHandler,),{'do_POST':lambda s:(open('/etc/overflow/canary-sandbox-probe-received','ab').write(s.rfile.read(int(s.headers['Content-Length']))),s.send_response(200),s.end_headers()),'log_message':lambda *a:None});h.HTTPServer(('127.0.0.1',18099),H).serve_forever()" &
for _ in $(seq 1 15) ; do sleep 1 ; ss -ltn | grep -q 18099 && break ; done
ss -ltn | grep -q 18099 || { echo "the listener did not bind 127.0.0.1:18099 after 15 attempts (about 16s) - stop and read the python error above" ; false ; }

The loop sleeps first and checks second, so its last act is a check rather than a sleep and it stops the moment the port answers. The bound it enforces is fifteen one-second waits, which measures 15.8-15.9 s here — the message names that, not a bare "15s", because sleep 1 is a floor and the ss in each iteration costs a few milliseconds on top of it.

If that block prints its message, or exits nonzero, stop there and do not run the next block — the listener is not up, and the run that follows would report a sandbox failure that does not exist. Read the error the listener printed on the terminal that started it: a traceback there names a missing module, a bad one-liner or a permission problem on the write path, none of which the port check above can see. Then start the unit and read the verdict:

systemctl start overflow-canary.service
journalctl -u overflow-canary.service --no-pager -n 20
grep -q failure-alert /etc/overflow/canary-sandbox-probe-received && echo "sandbox reached the out-of-band channel" || echo "the listener received no report - see the journal above"

failure-alert path is the canary's own text inside the report body, so this line is true only when the unit's own payload arrived. No such line means one of two things, and the journal above distinguishes them: a report that could not be delivered at all is the egress being blocked — the finding this step exists to surface — while a report that was delivered and a listener that never received it means the probe itself was misrun.

Remove the drop-in, the probe files and the (now empty) drop-in directory, reload, and confirm the unit is back to running its own defaults:

rm /etc/systemd/system/overflow-canary.service.d/sandbox-probe.conf
rmdir /etc/systemd/system/overflow-canary.service.d
rm /etc/overflow/canary-sandbox-probe-webhook
rm -f /etc/overflow/canary-sandbox-probe-received
systemctl daemon-reload
systemctl show overflow-canary.service -p Environment

rmdir refuses to remove a directory that still has anything in it, so it cannot take a drop-in another procedure added; if it reports the directory as not empty, look before removing it by hand.

Stop the listener from the shell that started it, with the job's own control — jobs to list it, kill %1 to end it — once the journal line above has been read. If the unit recorded an outage, or refused because it could not read the exim log, clear both state files so the first scheduled run is judged on its own merits. canary-fault matters as much as dead here: a canary that was refusing leaves it behind, and a run that finds it will refuse again and post nothing — the silence this section exists to end, arriving immediately after a procedure whose stated purpose is to un-silence it.

rm -f /run/overflow-canary/dead /run/overflow-canary/canary-fault

Rollback

Disable the timer and remove both files. The alert path itself is unchanged by them, so nothing else needs reverting:

systemctl disable --now overflow-canary.timer
rm /etc/systemd/system/overflow-canary.timer
rm /etc/systemd/system/overflow-canary.service
systemctl daemon-reload

The two host files may stay: nothing reads them once the canary unit is gone, and /etc/overflow/canary-recipient is the address a future canary would use.

The bounce watcher

Everything above watches the alert route from the sending side: the alert template mails a journal tail when a watched unit fails, and the canary proves once a day that the relay accepts a message. What nothing on that side can see is the route's own mail coming back. When a message on the alert route cannot be delivered, the relay's mailer daemon mails a delivery-failure notification back to the envelope sender — overflow-canary@<fqdn> or overflow-alert@<fqdn> — and until those addresses are aliased they are unrouteable, so exim refuses the notification outright and every delivery failure on the alert route vanishes without a trace (verified on the host with exim4 -bt). overflow-bounce.service, started every 15 minutes by overflow-bounce.timer, is the reporting leg for that mail: a root-run oneshot that tails root's mail spool from a persisted byte offset and forwards each new bounce to the out-of-band Discord webhook the canary already posts to.

With the canary plus this watcher the route distinguishes all three outcomes it can have. Off-host accepted: silence — the canary's Completed line and no watcher report. Local write: the watcher's Class B report, the class measured at 183 local writes against 5 off-host — a non-DSN message whose From header local part is overflow-alert or overflow-canary is an alert that resolved to this host, and a successful local write produces no bounce, so this class produces no other signal anywhere on the host. Remote failure: the watcher's Class A report — a delivery-failure notification that references an overflow address.

The acceptance leg is host configuration, and it comes first: /etc/aliases must gain overflow-canary: root and overflow-alert: root, then newaliases. Without it exim refuses the bounces at RCPT and the watcher has nothing to read. The obvious cheaper design — piping those two aliases straight at the webhook — is rejected on host evidence: Debian's address_pipe transport sets no user=, so an alias pipe runs as Debian-exim, which cannot read the root-only webhook file. The spool-watcher shape keeps the webhook root-only: root reads the spool, root posts the report, and no file's permissions move.

An IMAP or POP probe of the recipient mailbox is the definitive remote-mailbox check — the only option that observes the mailbox itself — and it needs a Gmail app-password credential that does not exist on this host, so its cost is one new third-party credential plus one poll per canary interval; it is deliberately not implemented. Piping the aliases at the webhook is rejected above, on the address_pipe evidence. Doing nothing is rejected because both of its detectors already fail: the heartbeat-stops detector is human-in-the-loop (the canary's own closing paragraphs), and the local-write class produces no heartbeat gap at all.

Prerequisites

The daemon and the webhook file are the canary's own prerequisites; the two aliases must not be present yet:

systemctl is-active exim4
test -s /etc/overflow/canary-discord-webhook && echo "webhook file present"
grep -q '^overflow-canary:' /etc/aliases && echo "overflow-canary alias already present" || echo "overflow-canary alias absent"
grep -q '^overflow-alert:' /etc/aliases && echo "overflow-alert alias already present" || echo "overflow-alert alias absent"

The first must answer active, the second must print its confirmation, and each alias line must print absent. An alias already present is a decision, not a step: an existing entry wins at alias resolution, so appending the line this section installs would change nothing and the verification below would fail in a way that looks like a broken watcher. Resolve the existing entry before installing.

Unlike the canary unit, the bounce unit carries one capability. The spool it tails is delivery state owned by the MTA — mail:mail, mode 0600 on this host — and systemd's empty capability bounding set strips CAP_DAC_OVERRIDE, so root's plain open of the spool is denied under the unit's sandbox and every run exits 2 at the script's spool readability check. That is measured, not assumed. The unit therefore pins CapabilityBoundingSet=CAP_DAC_OVERRIDE and AmbientCapabilities=CAP_DAC_OVERRIDE and nothing beyond that one bit. Two cheaper-looking shapes are tested and rejected: SupplementaryGroups=mail still cannot open the file, because a 0600 spool carries no group bits for the supplementary group to exercise; and re-owning the spool to root would rewrite MTA-owned delivery state, which is not host configuration this repository prescribes.

Install

The script needs no install step — /srv/overflow is a checkout of this repository at the deployed revision, so each deploy ships it. scripts/deploy-revision.sh does not install units and does not edit /etc/aliases, so every line below is run by hand. Add the two aliases and rebuild the alias database, install the two unit files, reload, and enable the timer:

grep -q '^overflow-canary:' /etc/aliases || printf '%s\n' 'overflow-canary: root' >> /etc/aliases
grep -q '^overflow-alert:' /etc/aliases || printf '%s\n' 'overflow-alert: root' >> /etc/aliases
newaliases
install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow-bounce.service /etc/systemd/system/
install -o root -g root -m 0644 \
  /srv/overflow/deploy/overflow-bounce.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now overflow-bounce.timer
systemctl list-timers overflow-bounce.timer --no-pager

Verify

First the acceptance leg. Both envelope-sender addresses must route to root's mailbox:

exim4 -bt overflow-canary@"$(hostname -f)"
exim4 -bt overflow-alert@"$(hostname -f)"

Each must print an R: system_aliases for <address> line — the alias is what accepts the mail — and end at the spool delivery, <address> -> /var/mail/mail with transport = address_file, which is how root's own address resolves on this host. Then the issue-848 lesson, and the check that fails most often in the wrong direction: the alert recipient must still resolve off-host.

exim4 -bt "$(cat /etc/overflow/alert-recipient)"

This one must show R: smarthost for <recipient> and router = smarthost, transport = remote_smtp_smarthost. A transport = address_file resolution here is the local-write class — the alert is being filed into a local mailbox and never leaves the host — and it means the alert path is not verified; fix the resolution before continuing, because the watcher's Class B report is the only other thing that would ever name it.

Then the reporting leg, which posts one real report to the channel — run it once, and read this paragraph before running it. The first run initializes the offset at the spool's current size and posts nothing: the backlog is cron mail that predates the watcher, and it is never re-read (see what this verification does not prove, below). Start the unit once for that:

systemctl start overflow-bounce.service
journalctl -u overflow-bounce.service --no-pager -n 20

The journal must show a clean exit and no report. Then deliver one synthetic delivery-failure notification through the local daemon — a sendmail one-shot message addressed to root only, whose body carries the DSN marker line and the string overflow-canary@. Nothing is sent to any remote address; the only off-host traffic in this step is the webhook post itself:

printf '%s\n' \
  'Subject: [overflow] synthetic delivery-failure notification (verification)' \
  '' \
  'This message was created automatically by mail delivery software.' \
  '' \
  'A delivery-failure notification arrived for overflow-canary@ - verification marker' \
  | sendmail root
tail -n 20 /var/log/exim4/mainlog

The mainlog must show the local delivery into /var/mail/mail. Then run the watcher once and read the verdict out of band:

systemctl start overflow-bounce.service
journalctl -u overflow-bounce.service --no-pager -n 20

The journal must show a clean exit again, and the Discord channel the webhook file names must show one message reading [overflow] a delivery-failure notification arrived for an overflow alert or canary message on <fqdn>: [overflow] synthetic delivery-failure notification (verification) — subject-only, because the synthetic body carries no exim address-failure marker; that is the same shape a Gmail-format DSN reports (known limitations below). That post is the confirmation, and it is what a delivered bounce report looks like.

What this verification does not prove, so it is not claimed: the watcher never re-reads the pre-install backlog, so bounces that arrived before the offset was initialized are invisible to it by design; and a partial tail — a message caught mid-append — parks the offset at the last complete message and costs one timer period, never a missed message.

Rollback

Disable the timer, remove both unit files, undo the aliases, and remove the state directory:

systemctl disable --now overflow-bounce.timer
rm /etc/systemd/system/overflow-bounce.timer
rm /etc/systemd/system/overflow-bounce.service
systemctl daemon-reload
sed -i -e '/^overflow-canary:/d' -e '/^overflow-alert:/d' /etc/aliases
newaliases
rm -rf /var/lib/overflow-bounce

The alert path itself is unchanged by any of it: removing the aliases returns the route to refusing bounces for the two sender addresses at RCPT, and the alerts and the canary keep working unchanged.

Known limitations. A Gmail-format DSN names its failed recipient differently from exim's, so those reports render subject-only — the shape the synthetic verification above deliberately shows. A unit that hangs without exiting never enters the failed state, so OnFailure= never fires for it — issue 848's second finding, still open there; the watcher reports the mail the path does deliver, never a path that produces none. The watcher reads the spool as mbox with exim's own From-escaping assumption, the same assumption every mbox reader makes, and a message misread across a boundary is bounded to one misclassification. The definitive remote-mailbox check remains the IMAP probe priced above, which needs the app-password credential recorded as externally blocked on issue 842.

13. Verifying the privileged-action journal's client addresses

Every successful privileged action — a moderator grant or revoke, an audit decision, a recalibration closure, a ban reversal, a credit adjustment or its reversal, a rederivation request, a settlement-override decision, a sanction contest decision — writes one Privileged action line into the service journal, naming the acting account, the credential reference, and the client address the request claimed. That address is read from X-Real-IP, which the production vhost sets. The application listens on loopback only, but loopback is not trust: any local process can reach the listener and set that header itself, so an unmarked address in the journal is a claim, not a fact.

To keep the journal honest about the difference, the application marks an address verified only when the request also carries the x-privileged-proxy-secret header with exactly the value of the OVERFLOW_PRIVILEGED_PROXY_SECRET environment variable. Only nginx — and the operator who configured both — hold that secret, so a header that carries it cannot come from a forger on loopback.

The application change is safe to deploy before any host step: until the variable is set, every entry is written with the claimed address recorded and clientAddressVerified: false, so each entry is marked unverified. The steps below close that gap.

  1. Generate the secret once per host:

    openssl rand -hex 32
  2. Put OVERFLOW_PRIVILEGED_PROXY_SECRET=<value> in /etc/overflow/overflow.env (root:root 0600, like the rest of that file), and restart the service so the running process picks the variable up:

    systemctl restart overflow.service
  3. Put the echo header in its own include file rather than the vhost: vhost files are typically world-readable, which would hand the secret to any local process — the exact adversary this section defends against — while nginx's master process reads its configuration as root, so a root-only include works:

    install -o root -g root -m 0600 /dev/null /etc/nginx/overflow-privileged-proxy.conf

    Into that file, with the literal replaced by the value generated in step 1:

    proxy_set_header X-Privileged-Proxy-Secret "<value from step 1>";

    Then include it in every proxying location block in the Overflow vhost, alongside that block's X-Real-IP setting. On the documented host these are /api/overrides, /api/moderation, and /. A location that sets any proxy_set_header inherits none of those directives from an outer level, so putting the attestation header only at server level or only in / leaves the other locations without it:

    include /etc/nginx/overflow-privileged-proxy.conf;

    The literal is generated per host: it is the same value OVERFLOW_PRIVILEGED_PROXY_SECRET carries in /etc/overflow/overflow.env.

  4. Validate and reload nginx:

    nginx -t && systemctl reload nginx

Steps 2 and 3 may land in either order: a header the application does not have the secret for, or a secret with no header echoing it, leaves entries marked unverified for the gap — it never marks an unverified address verified.

Verify the end state with one privileged action (a moderator grant through the moderation page, say) and the journal:

journalctl -u overflow.service --no-pager -e | grep "Privileged action"

The newest line must carry clientAddressVerified: true beside the address nginx recorded. A later vhost edit that drops the header, or a value changed on one side only, shows up the same way: entries keep arriving, marked unverified.

14. Rotating GitHub App private keys

The check-publishing App is the one identity whose key rotation this section carries. Section 11 owns OVERFLOW_TOKEN_ENCRYPTION_KEY; the procedure below does not apply to that key.

Check-publishing App: CI-secret-only

The Overflow Ledger App publishes the required checks. Its private key is stored as the LEDGER_APP_KEY secret in the repository's overflow-ledger GitHub environment. Both ledger-relay.yml and event-policy.yml read that secret. This App has no credential on the deployment host. Perform this procedure from the operator's machine with gh authenticated as a repository admin; the downloaded PEM files below are operator-side staging and rollback material, never deployment-host files.

CI step 1: record the old key and generate the replacement

On the check-publishing App's settings page (Settings → Developer settings → GitHub Apps → the Overflow Ledger App → Credentials → Key pairs), record the current key's fingerprint. Keep that key's previously downloaded PEM available on the operator's machine for rollback: GitHub does not return an Actions secret's value or fingerprint, so the old PEM cannot be recovered from LEDGER_APP_KEY.

Click New key on this App's key-pairs page and retain the downloaded PEM locally as /local/path/to/the-downloaded.pem in the command below. Record the new key's fingerprint from the key-pairs page, distinct from the old one. While both keys remain registered, either key's JWT verifies.

CI step 2: replace the environment secret

Set LEDGER_APP_KEY from the downloaded key through stdin, never an inline value, so key material appears nowhere on a command line:

(
set -e
gh secret set LEDGER_APP_KEY --repo Nitjsefnie/Overflow --env overflow-ledger < /local/path/to/the-downloaded.pem
gh secret list --repo Nitjsefnie/Overflow --env overflow-ledger
)

Expect the list to name LEDGER_APP_KEY with an Updated at of the current time. The value cannot be read back. The timestamp and the verification below are the evidence that the secret holds the replacement key. Keep the downloaded key and the previous PEM until the rotation is complete.

CI step 3: verify before retiring the old key

The relay probe mints an installation token before sweeping. A dispatch with no run id is a sweep-only start and exits nonzero if that mint fails:

gh workflow run ledger-relay.yml --repo Nitjsefnie/Overflow
run_id=$(gh run list --repo Nitjsefnie/Overflow --workflow ledger-relay.yml --limit 1 --json databaseId --jq '.[0].databaseId')
printf 'relay probe run: %s\n' "$run_id"
gh run watch "$run_id" --repo Nitjsefnie/Overflow --exit-status

If the list prints nothing, the dispatched run is not registered yet; wait a moment and run the lookup again. Expect watch to exit 0 on a green run. A red run naming the installation-token mint means the secret does not hold a working replacement key: repeat CI step 2, then verify again.

The next merge to main verifies check publication. Read the required contexts back, filtered to the App's identity:

gh api 'repos/Nitjsefnie/Overflow/commits/MERGE-COMMIT-SHA/check-runs?filter=all&per_page=100' --paginate \
  --jq '.check_runs[] | select(.app.id == 5118623) | {name: .name, status: .status, conclusion: .conclusion}'

Expect one line per pinned context, each completed and success. The pin map (.github/required-checks.json) names the contexts. Diagnose a missing or failed line before retiring the old key.

CI step 4: retire the old key

After the probe and check publication are green, delete the old key in the check-publishing App's settings using the fingerprint recorded in CI step 1. GitHub cannot re-add a deleted key, so this closes the rollback window. Confirm the next merge's required checks with the same readback from CI step 3, then dispose of the operator-side staged PEM files.

CI rollback

While the old key is still registered, restore the environment secret from the previous key's retained PEM on the operator's machine:

gh secret set LEDGER_APP_KEY --repo Nitjsefnie/Overflow --env overflow-ledger < /local/path/to/the-previous.pem

Rerun CI step 3 under the old key before deleting the replacement key in the App settings. If the previous PEM is unavailable, the secret cannot be restored from GitHub; repair forward with a working replacement key and repeat CI step 3. After the old key is deleted, repair is forward-only.