Repository navigation
Delphi: comment embeddings as one row per comment (migration) - #3035
Merged
Merged
Conversation
…draft) Adds delphi_embedding_models, delphi_comment_embeddings (real[] per conversation, model and comment, write-once) and delphi_embedding_sets (a run's closed set: every row up to a sequence mark), with fenced pd_embedding_put / pd_embedding_close and the read functions pd_embedding_set, pd_embedding_served and pd_embedding_rows. Additive: nothing existing is redefined and no current code calls the new objects. A reversal script refuses while any embedding is stored. Declared additive in compat.txt, listed in release.txt and docs/migrations.md; migration report regenerated; forward.py counts it among the tolerated additive migrations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The reversal of 000038 checked for stored embeddings without a table lock, so a put still in flight could commit after the check and be dropped with the tables. It now takes ACCESS EXCLUSIVE on both embedding tables before the check: an in-flight put either commits first and the script refuses, or waits and fails once the tables are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…d prune path Embedding rows outlive the run that first wrote them, so batch_id is now a plain provenance column with no foreign key: rows no longer keep a result batch from being pruned. Sets keep their foreign key to their batch, so a batch prune must remove the batch's sets first and fails loudly otherwise. Rows and sets still refuse UPDATE, and refuse DELETE except inside the new owner-only pd_embedding_prune, which: - superseded: per model, keeps the sets the served pointers resolve to, any newer set, the newest older one for rollback by pointer, and any set an unfinished job still consumes; deletes other sets, then rows of runs that never closed a set when no job for the conversation is unfinished; - conversation: deletes every row and set of a conversation; - model: deletes every row and set of a retired model. It takes the same per-conversation, per-model lock as put and close, and refuses the conversation and model modes while a job is unfinished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… retired models only pd_embedding_prune no longer has a mode that deletes superseded sets of a conversation that stays. Deciding which sets a later pointer, a rollback or an unfinished graph can still resolve to needs a rule per report scope over fully finished graphs; that is left for a later migration. A run's set is a small metadata row, since the embedding rows themselves are stored once per comment and shared. The function keeps two modes, conversation (optionally one model) and model, both refused while a job of an affected conversation is unfinished. Its signature drops the keep argument; the reversal script follows it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRAFT — SCHEMA CHANGE, out of the merge queue. Not for merge until the schema discussion has happened; merging it means the migration applies at the next deploy.
Observable changes
000038_create_delphi_comment_embeddings.sqladds:delphi_embedding_models(seeded with one row:sentence-transformers/all-MiniLM-L6-v2, 384 dims),delphi_comment_embeddingsanddelphi_embedding_sets;pd_embedding_put,pd_embedding_close,pd_embedding_set,pd_embedding_served,pd_embedding_rows), the owner-onlypd_embedding_prune(the only path that deletes rows or sets: every row and set of one conversation, optionally of one model, or of a retired model across the environment; refused while a job of an affected conversation is unfinished), and three internal ones. Embedding rows keep the run that first wrote them as provenance only (no foreign key), so they never keep a result batch from being pruned.It changes no existing table, function, view or grant. It is declared
additiveincompat.txt, so a release without it runs beside it.conversationsanddelphi_result_batchesuntil commit. Writes to those two tables wait for milliseconds; reads do not. An open writer transaction makes the migration waitlock_timeout(5 s), fail with no history row, and the deploy retries.What it is
Comment embeddings are stored one row per (environment, conversation, model, comment) as
real[], instead of one codec text value per run. Today that text value is bounded at 64 MiB, which about 5,950 comments of 384 floats already exceed.pd_embedding_rowsreturns a set in comment order, forCOPY (SELECT …) TO STDOUT (FORMAT binary).real[]rather than pgvectorvector(384). The dev, CI and fresh-database images (postgres:17-alpine,postgres:17.11) have no vector extension. The same bytes convert later withALTER TABLE … ALTER COLUMN embedding TYPE vector USING embedding::vector.down/000038_drop_delphi_comment_embeddings.sqldrops everything 000038 added. It locks both tables, then refuses while any embedding is stored; rolling back what is served is a pointer change, not a drop.Also in this PR:
docs/migrations.mdgains the declaration row.build-migration-report.pyandreport/first-deploy.sqlare regenerated.forward.pycounts 000038 among the tolerated additive migrations.Evidence (run locally on a work box, Postgres 17)
build-migration-report.py --check;prove.py(25),indexes.py(6),adoption.py(3),selection.py(5),forward.py(11),lexer.py(10);fresh-image.sh. All pass.pd_embedding_prune, and that permission ends when the call returns;Not in this PR: the Rust daemon, Python writer and importer changes that would use these objects, behind a flag that defaults to off.
🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s