Problem
hearthly-api runs Drizzle migrations from every pod's startup hook
(apps/hearthly-api/src/database/database.service.ts:32-40).
With replicas: 1 (current state: apps/hearthly-api/deploy/chart/values.yaml)
there is no race. The moment the API is scaled to ≥ 2 replicas — which the
broader cluster-resilience work will require so that node drains can succeed —
two pods race to apply the same migration set against the same database on
every deploy.
Drizzle's migrator does not coordinate across processes by default. Concurrent
migrators against Postgres tend to either:
- both succeed silently with one becoming a no-op (best case), or
- conflict on DDL with one returning a transaction error and crashlooping the
pod, or in rare cases corrupt the __drizzle_migrations ledger.
This is a latent issue surfaced during the 2026-05 cluster incident analysis
(Codex review thread 019e6ebc-2d5e-7792-85bf-05263912df6a).
Why it matters
Today: no failures, because only one pod runs.
After we adopt multi-replica stateless workloads (part of the cluster
resilience epic): every deploy is a coin flip.
Recommendation
One of:
- Extract migrations into a one-shot
Job invoked as an Argo PreSync hook or
Helm pre-install,pre-upgrade hook. The API Deployment then no longer
contains migration logic; the API pod just starts and serves.
- Wrap the existing migrator in a Postgres advisory lock
(pg_advisory_lock(<constant>) … pg_advisory_unlock(<constant>)) around the
drizzle-orm/postgres-js/migrator call so concurrent callers serialize.
Option 1 is cleaner (separates concern + makes migration failures fail the
deploy, not the pod startup).
Acceptance criteria
- Migrations run exactly once per deploy regardless of replica count.
- API pod startup no longer depends on schema state — readiness probe can
assume schema is current.
- Local dev experience (
bun run flow) still applies migrations on first run.
Files
apps/hearthly-api/src/database/database.service.ts
apps/hearthly-api/deploy/chart/templates/deployment.yaml
apps/hearthly-api/deploy/chart/values.yaml
apps/hearthly-api/CLAUDE.md (update conventions if pattern changes)
Problem
hearthly-apiruns Drizzle migrations from every pod's startup hook(
apps/hearthly-api/src/database/database.service.ts:32-40).With
replicas: 1(current state:apps/hearthly-api/deploy/chart/values.yaml)there is no race. The moment the API is scaled to ≥ 2 replicas — which the
broader cluster-resilience work will require so that node drains can succeed —
two pods race to apply the same migration set against the same database on
every deploy.
Drizzle's migrator does not coordinate across processes by default. Concurrent
migrators against Postgres tend to either:
pod, or in rare cases corrupt the
__drizzle_migrationsledger.This is a latent issue surfaced during the 2026-05 cluster incident analysis
(Codex review thread
019e6ebc-2d5e-7792-85bf-05263912df6a).Why it matters
Today: no failures, because only one pod runs.
After we adopt multi-replica stateless workloads (part of the cluster
resilience epic): every deploy is a coin flip.
Recommendation
One of:
Jobinvoked as an Argo PreSync hook orHelm
pre-install,pre-upgradehook. The API Deployment then no longercontains migration logic; the API pod just starts and serves.
(
pg_advisory_lock(<constant>) … pg_advisory_unlock(<constant>)) around thedrizzle-orm/postgres-js/migratorcall so concurrent callers serialize.Option 1 is cleaner (separates concern + makes migration failures fail the
deploy, not the pod startup).
Acceptance criteria
assume schema is current.
bun runflow) still applies migrations on first run.Files
apps/hearthly-api/src/database/database.service.tsapps/hearthly-api/deploy/chart/templates/deployment.yamlapps/hearthly-api/deploy/chart/values.yamlapps/hearthly-api/CLAUDE.md(update conventions if pattern changes)