feat(db): Optional PostgreSQL support for the main app database#4809
Open
bentennison94-gif wants to merge 1 commit into
Open
feat(db): Optional PostgreSQL support for the main app database#4809bentennison94-gif wants to merge 1 commit into
bentennison94-gif wants to merge 1 commit into
Conversation
Why: Odysseus keeps all primary state in a single SQLite file (data/app.db). SQLite is ideal as the zero-config single-host default, but it is a single-file, single-writer store, so multiple Odysseus instances/endpoints cannot share one live database. To host user data in the cloud and let several endpoints read and edit the same data concurrently, the main DB needs a networked backend. This adds optional PostgreSQL support, engaged only via DATABASE_URL=postgresql://...; SQLite stays the default, unchanged. What: - core/database.py: add IS_SQLITE (authoritative engine.dialect.name) and gate init_db() so the legacy raw-sqlite3/PRAGMA _migrate_* upgraders run only on SQLite (a fresh Postgres DB gets the full schema from create_all and never had an old one); create_all() and the three encryption-at-rest migrations run on both dialects. - mcp_servers/email_server.py: read email accounts via the app engine/ORM so it follows DATABASE_URL instead of opening app.db with raw sqlite3. - requirements-optional.txt: add opt-in psycopg2-binary (never mandatory). - tests/test_postgres_compat.py: dual-dialect round-trip (SQLite always; Postgres when TEST_DATABASE_URL is set) asserting JSON/Boolean/ EncryptedText/autoincrement values survive on both. - tests/test_mcp_email_decode_header_spaces.py: update the account fixture to the new engine-based read path (was tied to the old raw-sqlite3 path). - .github/workflows/ci.yml: add informational python-tests-postgres job. - .env.example, docs/setup.md: document the opt-in and v1 limits. Deliberate v1 behaviours (not bugs): fresh-Postgres-only (no SQLite->PG copy; legacy backfills skipped on PG); email side-DBs stay SQLite; chat search falls back from FTS5 to LIKE on Postgres. DB-only subset of pewdiepie-archdaemon#3763, kept single-purpose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Odysseus stores all primary state in a single SQLite file (
data/app.db). SQLite isperfect as the zero-config single-host default, but it's a single-file, single-writer
database — multiple Odysseus instances/endpoints can't point at the same data and read
and edit it concurrently. This PR adds optional, opt-in PostgreSQL support for the
main app database, engaged only when the operator sets
DATABASE_URL=postgresql://…, so user data can be hosted in the cloud and sharedacross instances. SQLite stays the zero-config default, byte-for-byte unchanged. It is
the DB-only subset of the closed #3763 (which had working Postgres code but was closed
for bundling i18n/UI), kept deliberately single-purpose.
What changed
core/database.py— add oneIS_SQLITEflag (engine.dialect.name == "sqlite")and gate
init_db(): the ~45 legacy SQLite-only_migrate_*()upgraders (rawsqlite3/PRAGMA/ALTER) run only on SQLite — a fresh Postgres DB never had an oldschema to patch.
create_all()(which builds the full portable schema) and the threeencryption-at-rest migrations run on both dialects.
mcp_servers/email_server.py— read email accounts via the app engine/ORM so theone main-
app.dbraw read followsDATABASE_URLinstead of opening SQLite directly.requirements-optional.txt— add opt-inpsycopg2-binary(never mandatory).tests/test_postgres_compat.py(new) — dual-dialect round-trip test.tests/test_mcp_email_decode_header_spaces.py— update the account fixture to thenew engine-based read path (it was tied to the old raw-sqlite3 path).
.github/workflows/ci.yml— add an informationalpython-tests-postgresjob..env.example,docs/setup.md— document the opt-in and its v1 limits.Deliberate v1 behaviour (called out so it isn't read as a bug)
an empty database. Legacy backfill/seed migrations are skipped on Postgres (a fresh DB
has no legacy rows to patch). A dedicated migration tool is the natural follow-up.
scheduled_emails.db/email_cache.db) stay local SQLite.LIKEon Postgres.create_all()covers fresh installs; versioned migrations can come later.Target branch
dev, notmain.Linked Issue
Fixes #4807
Type of Change
Checklist
dev.uvicorn app:app) and verified the change works end-to-end against a real Postgres database — see "How to Test".How to Test
Verified end-to-end on real PostgreSQL (managed cloud Postgres, over the network):
tests/test_postgres_compat.pyagainst the live DB → 2 passed (SQLite + Postgreslegs). Asserts that JSON, Boolean,
EncryptedText(encrypt→decrypt), andInteger-autoincrement values all round-trip correctly on Postgres.
DATABASE_URL=postgresql://…, created a notevia
POST /api/notes, killed the process and booted a fresh one against the samePostgres, and
GET /api/notesreturned the same note (also confirmed the row directlyin Postgres via SQL). Proves data genuinely persists in Postgres across restarts.
PRAGMA/sqlite3warnings or errors on the Postgres boot —confirms the
IS_SQLITEgate correctly skips the SQLite-only migrations.SQLite (default, unchanged):
skips automatically without
TEST_DATABASE_URL). Includes the new test's SQLite legand the updated
test_mcp_email_decode_header_spaces.py.Reproduce the Postgres leg yourself (also runs in the new CI job):
docker run -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:16pip install -r requirements-optional.txtTEST_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/odysseus_test python -m pytest -q tests/test_postgres_compat.py