Skip to content

ci(db): gate long-running migrations with a production-scale duration check #1471

Description

@Soju06

Problem

The existing migration checks (Alembic policy/drift, PostgreSQL upgrade-from-empty, lock serialization) all run against empty or tiny databases, so a revision that is instant in CI can block startup for many minutes at production data volumes. 20260722_000000_backfill_request_log_useragent_families passed every gate and then ran for ~10 minutes over a ~3.2M-row request_logs on a production deployment, during which the app serves nothing (see #1470 for the runtime-side proposal).

Proposal

A nightly (or migrations-path-triggered) CI job that:

  1. Seeds a PostgreSQL service container with a synthetic production-scale fixture (e.g. 3–5M request_logs rows with realistic column distributions; generation script checked in).
  2. Runs python -m app.db.migrate upgrade across the revisions added since the last stable tag and records per-revision wall time.
  3. Fails (or requires a long-migration label plus a release-notes callout) when any single revision exceeds a threshold (suggest 30s), so "this release needs a maintenance window" is visible at merge/release time instead of during the rollout.

Complements #1470: that issue removes the startup-blocking property for data backfills; this gate catches whatever remains blocking (schema rewrites, index builds without CONCURRENTLY, unexpected table scans).

Notes

  • Threshold and fixture size should be config in one place so they can track production growth.
  • Per-revision timing output belongs in the job summary for quick triage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions