Skip to content

fix(db): add postgres shm_size and raise default pool headroom - #1791

Merged
Soju06 merged 1 commit into
mainfrom
fix/pg-shm-and-pool-headroom
Aug 17, 2026
Merged

fix(db): add postgres shm_size and raise default pool headroom#1791
Soju06 merged 1 commit into
mainfrom
fix/pg-shm-and-pool-headroom

Conversation

@Soju06

@Soju06 Soju06 commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Why

Two independent capacity ceilings observed on a production single-replica PostgreSQL deployment (36h log window):

  • /dev/shm exhaustion. The Compose postgres container runs with Docker's default 64MB /dev/shm (shm_size unset; verified docker inspectShmSize: 67108864). With work_mem=32MB and max_parallel_workers_per_gather=2, parallel hash joins spill through dynamic shared memory under /dev/shm and abort with could not resize shared memory segment ... No space left on device, surfaced by asyncpg as DiskFullError on the request path — multiple occurrences in the last 36h.
  • Pool exhaustion. The default SQLAlchemy QueuePool (pool_size=15, max_overflow=10, fixed 30s checkout timeout) exhausts under slow-query pile-ups (observed account deletion 313s, admin search 207s, listings 2.1s holding checkouts): 73 occurrences of QueuePool limit of size 15 overflow 10 reached, connection timed out in 36h, failing healthy requests after a 30s wait.

What Changes

  • docker-compose.yml postgres service sets shm_size: 1gb.
  • Default database_pool_size 15 → 25 and database_max_overflow 10 → 15 (app/core/config/settings.py). Both stay operator-configurable via CODEX_LB_DATABASE_POOL_SIZE / CODEX_LB_DATABASE_MAX_OVERFLOW; no new settings are added.
  • Regenerated docs/reference/settings.md; updated default assertions; added a compose policy test pinning shm_size.
  • OpenSpec change expand-postgres-shm-and-pool-headroom (deltas on deployment-installation and database-backends), strict validation green.

Connection budget

Per-replica worst case stays inside PostgreSQL's default max_connections=100 using the reserve rule the Helm capacity guidance already mandates: (25 + 15) * 2 pooled engines * 1 worker = 80 application connections, leaving >= 20 raw server slots (3 superuser-reserved, 2 migration-path peak, operations). Verified against production: max_connections=100, observed peak numbackends=36.

Helm deployments are unaffected — the chart always injects its own pool values (databasePoolSize/databaseMaxOverflow, default 3/1, prod overlay 1/1), and the bundled Bitnami PostgreSQL sub-chart already mounts a memory-backed /dev/shm by default (shmVolume.enabled=true), so no chart change is needed.

Ops note

Applying shm_size requires the postgres container to be recreated (docker compose up -d postgres recreates it; a few seconds of downtime). Schedule it for the next deploy/maintenance window rather than a live docker compose restart, which does not apply HostConfig changes.

Validation

  • uv run pytest tests/unit — 6101 passed, 3 skipped
  • uv run ruff check . / uv run ruff format --check . — clean
  • openspec validate expand-postgres-shm-and-pool-headroom --strict — valid
  • codex review --base origin/main × 2 rounds — no findings

🤖 Generated with Claude Code

Production single-replica PostgreSQL hit two capacity ceilings:

- Docker's default 64MB /dev/shm makes parallel hash joins fail with
  "could not resize shared memory segment ... No space left on device"
  (asyncpg DiskFullError). Pin shm_size: 1gb on the Compose postgres
  service. The Helm bundled Bitnami sub-chart already mounts a
  memory-backed /dev/shm by default.
- Default pool 15/10 with the fixed 30s checkout timeout exhausts under
  slow-query pile-ups ("QueuePool limit of size 15 overflow 10
  reached"). Raise defaults to 25/15 so one replica's two pooled
  engines cap at (25+15)*2 = 80 connections, preserving the documented
  20 raw-slot reserve on PostgreSQL's default max_connections=100.

Helm deployments are unaffected: the chart injects its own pool values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Aug 16, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Soju06, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 7 minutes

Limit details: You’ve used all 3 included reviews currently available under your plan.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6c7810bd-3346-4fdb-be39-d8bfd12aff5e

📥 Commits

Reviewing files that changed from the base of the PR and between 0c8d921 and 6dcb846.

📒 Files selected for processing (10)
  • app/core/config/settings.py
  • docker-compose.yml
  • docs/reference/settings.md
  • openspec/changes/expand-postgres-shm-and-pool-headroom/proposal.md
  • openspec/changes/expand-postgres-shm-and-pool-headroom/specs/database-backends/spec.md
  • openspec/changes/expand-postgres-shm-and-pool-headroom/specs/deployment-installation/spec.md
  • openspec/changes/expand-postgres-shm-and-pool-headroom/tasks.md
  • tests/unit/test_db_session.py
  • tests/unit/test_docker_compose_postgres.py
  • tests/unit/test_settings_trace_and_removed.py

Comment @coderabbitai help to get the list of available commands.

@Soju06
Soju06 merged commit 539cf93 into main Aug 17, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant