fix(db): add postgres shm_size and raise default pool headroom - #1791
Conversation
Production single-replica PostgreSQL hit two capacity ceilings:
- Docker's default 64MB /dev/shm makes parallel hash joins fail with
"could not resize shared memory segment ... No space left on device"
(asyncpg DiskFullError). Pin shm_size: 1gb on the Compose postgres
service. The Helm bundled Bitnami sub-chart already mounts a
memory-backed /dev/shm by default.
- Default pool 15/10 with the fixed 30s checkout timeout exhausts under
slow-query pile-ups ("QueuePool limit of size 15 overflow 10
reached"). Raise defaults to 25/15 so one replica's two pooled
engines cap at (25+15)*2 = 80 connections, preserving the documented
20 raw-slot reserve on PostgreSQL's default max_connections=100.
Helm deployments are unaffected: the chart injects its own pool values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 7 minutes Limit details: You’ve used all 3 included reviews currently available under your plan. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
Comment |
Why
Two independent capacity ceilings observed on a production single-replica PostgreSQL deployment (36h log window):
/dev/shmexhaustion. The Compose postgres container runs with Docker's default 64MB/dev/shm(shm_sizeunset; verifieddocker inspect→ShmSize: 67108864). Withwork_mem=32MBandmax_parallel_workers_per_gather=2, parallel hash joins spill through dynamic shared memory under/dev/shmand abort withcould not resize shared memory segment ... No space left on device, surfaced by asyncpg asDiskFullErroron the request path — multiple occurrences in the last 36h.pool_size=15,max_overflow=10, fixed 30s checkout timeout) exhausts under slow-query pile-ups (observed account deletion 313s, admin search 207s, listings 2.1s holding checkouts): 73 occurrences ofQueuePool limit of size 15 overflow 10 reached, connection timed outin 36h, failing healthy requests after a 30s wait.What Changes
docker-compose.ymlpostgres service setsshm_size: 1gb.database_pool_size15 → 25 anddatabase_max_overflow10 → 15 (app/core/config/settings.py). Both stay operator-configurable viaCODEX_LB_DATABASE_POOL_SIZE/CODEX_LB_DATABASE_MAX_OVERFLOW; no new settings are added.docs/reference/settings.md; updated default assertions; added a compose policy test pinningshm_size.expand-postgres-shm-and-pool-headroom(deltas ondeployment-installationanddatabase-backends), strict validation green.Connection budget
Per-replica worst case stays inside PostgreSQL's default
max_connections=100using the reserve rule the Helm capacity guidance already mandates:(25 + 15) * 2 pooled engines * 1 worker = 80application connections, leaving >= 20 raw server slots (3 superuser-reserved, 2 migration-path peak, operations). Verified against production:max_connections=100, observed peaknumbackends=36.Helm deployments are unaffected — the chart always injects its own pool values (
databasePoolSize/databaseMaxOverflow, default 3/1, prod overlay 1/1), and the bundled Bitnami PostgreSQL sub-chart already mounts a memory-backed/dev/shmby default (shmVolume.enabled=true), so no chart change is needed.Ops note
Applying
shm_sizerequires the postgres container to be recreated (docker compose up -d postgresrecreates it; a few seconds of downtime). Schedule it for the next deploy/maintenance window rather than a livedocker compose restart, which does not apply HostConfig changes.Validation
uv run pytest tests/unit— 6101 passed, 3 skippeduv run ruff check ./uv run ruff format --check .— cleanopenspec validate expand-postgres-shm-and-pool-headroom --strict— validcodex review --base origin/main× 2 rounds — no findings🤖 Generated with Claude Code