Problem
The existing migration checks (Alembic policy/drift, PostgreSQL upgrade-from-empty, lock serialization) all run against empty or tiny databases, so a revision that is instant in CI can block startup for many minutes at production data volumes. 20260722_000000_backfill_request_log_useragent_families passed every gate and then ran for ~10 minutes over a ~3.2M-row request_logs on a production deployment, during which the app serves nothing (see #1470 for the runtime-side proposal).
Proposal
A nightly (or migrations-path-triggered) CI job that:
- Seeds a PostgreSQL service container with a synthetic production-scale fixture (e.g. 3–5M
request_logs rows with realistic column distributions; generation script checked in).
- Runs
python -m app.db.migrate upgrade across the revisions added since the last stable tag and records per-revision wall time.
- Fails (or requires a
long-migration label plus a release-notes callout) when any single revision exceeds a threshold (suggest 30s), so "this release needs a maintenance window" is visible at merge/release time instead of during the rollout.
Complements #1470: that issue removes the startup-blocking property for data backfills; this gate catches whatever remains blocking (schema rewrites, index builds without CONCURRENTLY, unexpected table scans).
Notes
- Threshold and fixture size should be config in one place so they can track production growth.
- Per-revision timing output belongs in the job summary for quick triage.
Problem
The existing migration checks (Alembic policy/drift, PostgreSQL upgrade-from-empty, lock serialization) all run against empty or tiny databases, so a revision that is instant in CI can block startup for many minutes at production data volumes.
20260722_000000_backfill_request_log_useragent_familiespassed every gate and then ran for ~10 minutes over a ~3.2M-rowrequest_logson a production deployment, during which the app serves nothing (see #1470 for the runtime-side proposal).Proposal
A nightly (or migrations-path-triggered) CI job that:
request_logsrows with realistic column distributions; generation script checked in).python -m app.db.migrate upgradeacross the revisions added since the last stable tag and records per-revision wall time.long-migrationlabel plus a release-notes callout) when any single revision exceeds a threshold (suggest 30s), so "this release needs a maintenance window" is visible at merge/release time instead of during the rollout.Complements #1470: that issue removes the startup-blocking property for data backfills; this gate catches whatever remains blocking (schema rewrites, index builds without
CONCURRENTLY, unexpected table scans).Notes