Skip to content

infra(api): migrations should not run from every pod on startup #125

Description

@rudingma

Problem

hearthly-api runs Drizzle migrations from every pod's startup hook
(apps/hearthly-api/src/database/database.service.ts:32-40).

With replicas: 1 (current state: apps/hearthly-api/deploy/chart/values.yaml)
there is no race. The moment the API is scaled to ≥ 2 replicas — which the
broader cluster-resilience work will require so that node drains can succeed —
two pods race to apply the same migration set against the same database on
every deploy.

Drizzle's migrator does not coordinate across processes by default. Concurrent
migrators against Postgres tend to either:

  • both succeed silently with one becoming a no-op (best case), or
  • conflict on DDL with one returning a transaction error and crashlooping the
    pod, or in rare cases corrupt the __drizzle_migrations ledger.

This is a latent issue surfaced during the 2026-05 cluster incident analysis
(Codex review thread 019e6ebc-2d5e-7792-85bf-05263912df6a).

Why it matters

Today: no failures, because only one pod runs.
After we adopt multi-replica stateless workloads (part of the cluster
resilience epic): every deploy is a coin flip.

Recommendation

One of:

  1. Extract migrations into a one-shot Job invoked as an Argo PreSync hook or
    Helm pre-install,pre-upgrade hook. The API Deployment then no longer
    contains migration logic; the API pod just starts and serves.
  2. Wrap the existing migrator in a Postgres advisory lock
    (pg_advisory_lock(<constant>) … pg_advisory_unlock(<constant>)) around the
    drizzle-orm/postgres-js/migrator call so concurrent callers serialize.

Option 1 is cleaner (separates concern + makes migration failures fail the
deploy, not the pod startup).

Acceptance criteria

  • Migrations run exactly once per deploy regardless of replica count.
  • API pod startup no longer depends on schema state — readiness probe can
    assume schema is current.
  • Local dev experience (bun run flow) still applies migrations on first run.

Files

  • apps/hearthly-api/src/database/database.service.ts
  • apps/hearthly-api/deploy/chart/templates/deployment.yaml
  • apps/hearthly-api/deploy/chart/values.yaml
  • apps/hearthly-api/CLAUDE.md (update conventions if pattern changes)

Metadata

Metadata

Assignees

No one assigned

    Labels

    infraInfrastructure/DevOps

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions