feat(ci): nightly pg_dump backup workflow + restore runbook - #173
feat(ci): nightly pg_dump backup workflow + restore runbook#173Otfrugger wants to merge 1 commit into
Conversation
Adds a scheduled GitHub Actions workflow that pg_dumps each network's Postgres database independently (DATABASE_URL_TESTNET/_MAINNET), gzips the result, and uploads it as a 14-day artifact. Skips a network gracefully if its backup secret isn't configured yet. Also documents the restore procedure and marks Miracle656#165 done in the dual-network roadmap. closes Miracle656#165
|
@Otfrugger Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Miracle656
left a comment
There was a problem hiding this comment.
The workflow engineering is good — per-network jobs, a clean skip with a warning when the secret isn't configured rather than a red run, if-no-files-found: error so a silent empty backup can't masquerade as success, bounded timeout-minutes, and a runbook alongside it. That's the shape #165 asked for.
One blocker, and it's serious enough that I'd rather be explicit than tactful.
This publishes every webhook signing secret
Miracle656/wraith is a public repository. GitHub Actions artifacts on a public repo are downloadable by anyone — no write access needed, no org membership. And the dump is unfiltered and unencrypted:
- name: Upload backup artifact
with:
name: wraith-db-backup-testnet-${{ github.run_id }}
path: backups/*.dump.gz
retention-days: 14prisma/schema.prisma:176:
model WebhookSubscription {
url String
secret String // "The secret is used to HMAC-sign payloads"Plaintext. So the artifact contains every subscriber's HMAC signing secret, and the concrete consequence is that anyone can forge webhook deliveries that pass signature verification — for every subscriber, for fourteen days, on a nightly schedule. The signature stops being evidence of anything.
It also ships every subscriber url, which is a customer list.
The mainnet job makes it worse, not better: that's where real subscribers will be.
This is the same issue I raised on lens#122, which is still open for the same reason. Worth coordinating so both land with the same approach.
Three ways to fix it, in the order I'd pick
- Don't put the dump in an artifact at all. Push to object storage the repo doesn't expose — S3/R2/B2 with credentials in Actions secrets. This is what the job is really for; artifacts are a convenient default that happens to be public here.
- Encrypt before upload.
ageorgpg --symmetricwith the passphrase in a secret. Simple, keeps the artifact flow, and a leaked artifact is then inert. Make sure the restore runbook covers decryption, or you've built a backup nobody can use under pressure. - Exclude the sensitive tables —
pg_dump --exclude-table=wraith.\"WebhookSubscription\" --exclude-table=wraith.\"WebhookDelivery\". Cheapest, but it makes the backup incomplete, and a restore that silently loses subscriptions is its own incident. Only reasonable combined with (1) or (2).
I'd take 1 or 2. Whichever you choose, please say in the runbook why — the next person will otherwise reintroduce a plain artifact upload as a simplification.
Smaller notes
- Verify the restore. The runbook describes restoring; nothing proves the dump is restorable. A step that restores into a scratch database and asserts a row count would turn this from a backup into a tested backup. That distinction usually gets discovered at the worst moment.
- Retention vs. schedule. 14 days of nightlies is 14 copies live at once. Fine once encrypted; worth stating deliberately in the runbook either way.
docs/DUAL_NETWORK.mdalready warns that Render's free Postgres is deleted after 90 days — worth linking from the runbook, since that's the scenario this exists for.
Everything except the artifact exposure is ready. Fix that and I'll take it.
closes #165
Summary
.github/workflows/db-backup.yml: a scheduled (nightly, 03:00 UTC) +workflow_dispatchGitHub Actions workflow that runspg_dump --format=custom --no-owner --no-privilegesagainst each network's database independently, gzips the dump, and uploads it as a 14-day-retention artifact per network (wraith-db-backup-<network>-<run_id>).ops/backup/dump.sh, the dump+gzip script the workflow calls, matching the style of the existingops/canary/*.shscripts.docs/backup-restore.md: the restore runbook — required secrets, manual-dispatch instructions, and step-by-steppg_restoreprocedure into a fresh instance (e.g. Neon), including a warning against cross-restoring testnet/mainnet dumps.docs/DUAL_NETWORK.mdto mark Nightlypg_dumpbackup workflow + restore runbook #165 done and point at the new runbook + required secrets.Design notes
DATABASE_URL_TESTNET/DATABASE_URL_MAINNET, separate from the runtimeDATABASE_URLeach deployment uses — mirroring the_TESTNET/_MAINNETsuffix convention already used forSOROBAN_RPC_URL_MAINNETetc. indocs/DUAL_NETWORK.md. This lets one workflow back up both network databases regardless of which network any given deployment is running.::warning::and skips cleanly rather than failing the whole workflow — so this can land before both databases exist.Verification
pg_dump/dockeraren't available in the environment I developed this in, so I could not run an actual end-to-end dump/restore — acceptance criterion "Restore runbook verified end-to-end once" still needs a maintainer to run it once against a real (or scratch) Postgres instance with real secrets, as noted at the bottom ofdocs/backup-restore.md.yaml.safe_load),ops/backup/dump.shpassesbash -nsyntax check.tsc/build/tests are unaffected.