Skip to content

feat(db): Optional PostgreSQL support for the main app database#4809

Open
bentennison94-gif wants to merge 1 commit into
pewdiepie-archdaemon:devfrom
bentennison94-gif:feat/postgres-support
Open

feat(db): Optional PostgreSQL support for the main app database#4809
bentennison94-gif wants to merge 1 commit into
pewdiepie-archdaemon:devfrom
bentennison94-gif:feat/postgres-support

Conversation

@bentennison94-gif

Copy link
Copy Markdown

Summary

Odysseus stores all primary state in a single SQLite file (data/app.db). SQLite is
perfect as the zero-config single-host default, but it's a single-file, single-writer
database — multiple Odysseus instances/endpoints can't point at the same data and read
and edit it concurrently. This PR adds optional, opt-in PostgreSQL support for the
main app database
, engaged only when the operator sets
DATABASE_URL=postgresql://…, so user data can be hosted in the cloud and shared
across instances. SQLite stays the zero-config default, byte-for-byte unchanged. It is
the DB-only subset of the closed #3763 (which had working Postgres code but was closed
for bundling i18n/UI), kept deliberately single-purpose.

What changed

  • core/database.py — add one IS_SQLITE flag (engine.dialect.name == "sqlite")
    and gate init_db(): the ~45 legacy SQLite-only _migrate_*() upgraders (raw
    sqlite3/PRAGMA/ALTER) run only on SQLite — a fresh Postgres DB never had an old
    schema to patch. create_all() (which builds the full portable schema) and the three
    encryption-at-rest migrations run on both dialects.
  • mcp_servers/email_server.py — read email accounts via the app engine/ORM so the
    one main-app.db raw read follows DATABASE_URL instead of opening SQLite directly.
  • requirements-optional.txt — add opt-in psycopg2-binary (never mandatory).
  • tests/test_postgres_compat.py (new) — dual-dialect round-trip test.
  • tests/test_mcp_email_decode_header_spaces.py — update the account fixture to the
    new engine-based read path (it was tied to the old raw-sqlite3 path).
  • .github/workflows/ci.yml — add an informational python-tests-postgres job.
  • .env.example, docs/setup.md — document the opt-in and its v1 limits.

Deliberate v1 behaviour (called out so it isn't read as a bug)

  • Fresh-Postgres-only. No SQLite→Postgres data-copy tool yet; switching starts from
    an empty database. Legacy backfill/seed migrations are skipped on Postgres (a fresh DB
    has no legacy rows to patch). A dedicated migration tool is the natural follow-up.
  • The email side-DBs (scheduled_emails.db / email_cache.db) stay local SQLite.
  • Chat full-text search falls back from FTS5 to LIKE on Postgres.
  • No Alembic — create_all() covers fresh installs; versioned migrations can come later.

Target branch

  • This PR targets dev, not main.

Linked Issue

Fixes #4807

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)
  • New feature (non-breaking — adds new behaviour)
  • Breaking change (changes or removes existing behaviour)
  • Refactor / cleanup (behaviour unchanged)
  • Documentation only
  • CI / tooling / configuration

Checklist

How to Test

Verified end-to-end on real PostgreSQL (managed cloud Postgres, over the network):

  • tests/test_postgres_compat.py against the live DB → 2 passed (SQLite + Postgres
    legs). Asserts that JSON, Boolean, EncryptedText (encrypt→decrypt), and
    Integer-autoincrement values all round-trip correctly on Postgres.
  • Real app, full restart: booted with DATABASE_URL=postgresql://…, created a note
    via POST /api/notes, killed the process and booted a fresh one against the same
    Postgres, and GET /api/notes returned the same note (also confirmed the row directly
    in Postgres via SQL). Proves data genuinely persists in Postgres across restarts.
  • Clean boot: no PRAGMA/sqlite3 warnings or errors on the Postgres boot —
    confirms the IS_SQLITE gate correctly skips the SQLite-only migrations.

SQLite (default, unchanged):

  • Ran the affected DB + MCP-email test files → 59 passed, 1 skipped (the Postgres leg
    skips automatically without TEST_DATABASE_URL). Includes the new test's SQLite leg
    and the updated test_mcp_email_decode_header_spaces.py.

Reproduce the Postgres leg yourself (also runs in the new CI job):

  1. docker run -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:16
  2. pip install -r requirements-optional.txt
  3. TEST_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/odysseus_test python -m pytest -q tests/test_postgres_compat.py

Why:
Odysseus keeps all primary state in a single SQLite file (data/app.db).
SQLite is ideal as the zero-config single-host default, but it is a
single-file, single-writer store, so multiple Odysseus instances/endpoints
cannot share one live database. To host user data in the cloud and let
several endpoints read and edit the same data concurrently, the main DB
needs a networked backend. This adds optional PostgreSQL support, engaged
only via DATABASE_URL=postgresql://...; SQLite stays the default, unchanged.

What:
- core/database.py: add IS_SQLITE (authoritative engine.dialect.name) and
  gate init_db() so the legacy raw-sqlite3/PRAGMA _migrate_* upgraders run
  only on SQLite (a fresh Postgres DB gets the full schema from create_all
  and never had an old one); create_all() and the three encryption-at-rest
  migrations run on both dialects.
- mcp_servers/email_server.py: read email accounts via the app engine/ORM
  so it follows DATABASE_URL instead of opening app.db with raw sqlite3.
- requirements-optional.txt: add opt-in psycopg2-binary (never mandatory).
- tests/test_postgres_compat.py: dual-dialect round-trip (SQLite always;
  Postgres when TEST_DATABASE_URL is set) asserting JSON/Boolean/
  EncryptedText/autoincrement values survive on both.
- tests/test_mcp_email_decode_header_spaces.py: update the account fixture
  to the new engine-based read path (was tied to the old raw-sqlite3 path).
- .github/workflows/ci.yml: add informational python-tests-postgres job.
- .env.example, docs/setup.md: document the opt-in and v1 limits.

Deliberate v1 behaviours (not bugs): fresh-Postgres-only (no SQLite->PG
copy; legacy backfills skipped on PG); email side-DBs stay SQLite; chat
search falls back from FTS5 to LIKE on Postgres.

DB-only subset of pewdiepie-archdaemon#3763, kept single-purpose.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added the ready for review Description complete — ready for maintainer review label Jun 24, 2026
@bentennison94-gif bentennison94-gif changed the title Optional PostgreSQL support for the main app database feat(db): Optional PostgreSQL support for the main app database Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optional PostgreSQL backend for the main app database (enables cloud-hosted, multi-instance shared data)

1 participant