mailbridge

IMAP sync engine that mirrors email metadata from any IMAP server to Supabase/Postgres. Designed for reliability with advisory locks, UIDVALIDITY handling, and rate limiting.

Spec: rackspace-email-sync-engine-spec.md - the authoritative technical specification. Deploy: docs/DEPLOYMENT.md - step-by-step Render deployment guide. Using the API: docs/USING_THE_API.md - API reference and database query guide for building apps.

Sync Freshness Expectations

• New mail: Shows up fast. INBOX/Sent: ~1-2 minutes. Other folders: a few minutes depending on how many non-priority folders you have (incremental sync runs every 60s; 5 non-priority folders per cycle).
• Read/unread or star changes: INBOX/Sent: usually within ~2 minutes. Other folders: about every 6 hours (flag scan schedule; max 2 flag scans per cycle).
• Deletes from your mail client: Disappear here within a few hours. Deletes are only seen during reconcile (~6 hours + up to 15 minutes jitter, one reconcile per cycle; priority folders first, then round-robin), so slower than new mail.

Reference constants live in packages/shared/src/config.ts.

Setup

pnpm install
cp .env.example .env  # DATABASE_URL, IMAP_ENCRYPTION_KEY

Run migrations: psql -f supabase/migrations/00000000000000_baseline.sql

Add Your First Account

pnpm add-account

This will prompt for your IMAP credentials, test the connection, and save encrypted to the database. The worker will automatically pick up the account on the next poll cycle (60s).

Database Connection

Supabase offers two connection methods:

Environment	Connection Type	Why
Production (Render)	Direct Connection	Lower latency, Render has IPv6
Local development	Session Pooler	Works on any WiFi (IPv4 compatible)

# Direct Connection (IPv6 only) - use in production
DATABASE_URL=postgresql://postgres:[PASSWORD]@db.xxx.supabase.co:5432/postgres

# Session Pooler (IPv4/IPv6) - use for local dev
DATABASE_URL=postgresql://postgres.xxx:[PASSWORD]@aws-0-us-west-2.pooler.supabase.com:5432/postgres

Both work identically with advisory locks. Get connection strings from Supabase Dashboard -> Connect.

Development

pnpm dev             # Pre-flight checks -> start API + Worker

This runs a pre-flight check that validates:

• Environment variables (DATABASE_URL, IMAP_ENCRYPTION_KEY)
• Database connectivity to Supabase
• Schema exists (migrations applied)
• Account count within limits

On success, both services start with clear status indicators:

[api]    ✓ Database connected
[api]    ✓ Lock self-test passed
[api]    ✓ Ready -> http://localhost:3000

[worker] ✓ Database connected
[worker] ✓ Lock self-test passed
[worker] ✓ Accounts: 1/20
[worker] ✓ Ready -> Poll loop running (60s interval)

Account Management

pnpm add-account     # Add new IMAP account (interactive)
pnpm list-accounts   # Show all accounts with sync status

Other Commands

pnpm dev:preflight   # Run only the pre-flight checks
pnpm dev:start       # Skip pre-flight, start immediately
pnpm dev:api         # Run only the API
pnpm dev:worker      # Run only the Worker
pnpm build           # Build all packages

Note: If you modify packages/shared/, restart dev to pick up the changes.

Production

pnpm build           # Compile TypeScript to dist/
pnpm start           # Pre-flight checks -> start API + Worker (production)

Production preflight validates:

• Environment variables set
• Database connectivity
• Schema exists (migrations applied)
• At least one account configured (required for production)
• Compiled dist/ folders exist

Individual service commands:

pnpm start:api       # Run only the API (production)
pnpm start:worker    # Run only the Worker (production)

Testing

Prerequisites

• Docker: Required for the local Postgres test container

Quick Start

# Set test database URL (uses local Supabase on port 54322)
export DATABASE_URL_TEST="postgresql://postgres:postgres@localhost:54322/postgres"

# Run all tests (starts Supabase if needed)
pnpm test:all

Test Commands

Command	Description
pnpm test:all	Run env guard -> start Supabase -> worker tests -> API tests
pnpm test:db-up	Start the Supabase test container (idempotent)
pnpm -C packages/worker test	Run worker tests only
pnpm -C packages/api test	Run API tests only

DATABASE_URL_TEST Requirements

The test environment enforces these safety rails:

• Required: DATABASE_URL_TEST must be set (refuses DATABASE_URL)
• No prod: Host cannot contain "prod"
• No poolers: Blocks pooler, pgbouncer hosts and port 6543
• Test isolation: DB name must end with _test OR URL must include search_path=test
• Session-safe: Port 5432/54322 (not 6543), direct Postgres connections only

Architecture

packages/
├── api/      # HTTP (account CRUD, health, body fetch)
├── worker/   # Background sync (poll loop, IMAP)
└── shared/   # DB, types, config, locks, throttle, metrics

Runtime Architecture

flowchart LR
  Client[Clients / Tools] -->|HTTPS| API[API Service]
  Worker[Worker Service]

  DB[(Postgres)]
  IMAP[(IMAP Provider)]
  Logs[(Logs and Metrics)]

  subgraph Shared[Shared Library]
    S[packages/shared<br/>DB helpers, locks, config, crypto, throttle, metrics]
  end

  API -->|read write| DB
  Worker -->|read write| DB

  Worker -->|IMAP metadata sync| IMAP
  API -->|IMAP body fetch| IMAP

  API -.->|advisory lock per account| DB
  Worker -.->|advisory lock per account| DB

  API -.->|imports| S
  Worker -.->|imports| S

  API -->|logs| Logs
  Worker -->|logs metrics| Logs

Loading

Key invariant: All IMAP operations for a given account (worker sync and API body fetch) are serialized by the same per-account Postgres advisory lock. If the worker holds the lock, the API body fetch can return a retryable "busy" response.

Worker Sync Flow

flowchart TB
  start[Poll tick] --> select[Select due accounts]
  select --> any{Any accounts}
  any -->|no| sleep[Sleep]
  any -->|yes| loop[Process accounts sequentially]

  loop --> mark[Mark syncing and heartbeat]
  mark --> lock{Acquire advisory lock}

  lock -->|busy| recover{Stale lock}
  recover -->|yes| clear[Clear orphaned lock]
  clear --> lock
  recover -->|no| skip[Skip account]

  lock -->|acquired| connect[Connect IMAP]
  connect --> discovery{Discovery due}
  discovery -->|yes| discover[Folder discovery and exclusions]
  discovery -->|no| plan[Select folders]

  discover --> plan
  plan --> folder{Per folder}

  folder --> uidv[Check UIDVALIDITY]
  uidv --> mode{Initial complete}

  mode -->|no| initial[Initial sync metadata]
  mode -->|yes| incr[Incremental sync metadata]

  initial --> post[Update heartbeat]
  incr --> post

  post --> flags{Flag scan due}
  flags -->|yes| flagscan[Flag scan]
  flags -->|no| recon{Reconcile due}

  flagscan --> recon
  recon -->|yes| reconcile[Reconcile]
  recon -->|no| update[Update state]

  reconcile --> update
  update --> more{More folders}
  more -->|yes| folder
  more -->|no| finish[Evaluate health and release lock]

  finish --> loop
  skip --> loop
  sleep --> start

Loading

Status: 100% Spec Complete

Feature	Spec	Status
Initial sync (newest-first, resumable)	§10.4	✅
Incremental sync (all-or-nothing + timeout)	§10.5	✅
Reconciliation (safety net)	§10.7	✅
Flag tracking	§10.6	✅
Health state machine	§12	✅
Body fetch endpoint	§14	✅
Command throttling (200/min)	§9.3	✅
Metrics emission	§16	✅
Advisory locks + self-test	§8	✅
Folder discovery + exclusions	§6	✅
UIDVALIDITY reset handling + limit	§11	✅
Retention/expiry job	§15	✅
Graceful shutdown	§17	✅
Integration tests	§18	✅
Account limit (API + worker)	§0	✅

Debugging

# Check account sync state
pnpm tsx scripts/check-account.ts

# Reset stuck account (if currently_syncing stuck at true)
pnpm tsx scripts/reset-account.ts

Optional Extensions

This engine syncs IMAP metadata to Postgres. You can extend it with:

• Threading: Build conversation threads using in_reply_to, references_header, and provider_thread_id columns
• Identity resolution: Match email addresses to contacts using the from_email, to_emails, cc_emails fields
• Full-text search: Add Postgres full-text search on subject and fetched body content

License

MIT