Repository navigation
Delphi: real-model end-to-end for the comment-embeddings table - #3049
Merged
Merged
Conversation
Add a job-graph proof that runs the queue daemon with DELPHI_EMBEDDINGS_TABLE on and the real all-MiniLM-L6-v2 model against Postgres with every migration applied, wired into CI the way the existing graph proofs run (scripts/test-embeddings-table.sh + .github/workflows/embeddings-table.yml, reusing the worker image that bakes the model weights at build time so the stage loads them offline). With the flag on the proof shows: - a first run writes every comment's embedding to the table and publishes the set with the run's results; the read-back is bit-exact (float32) to what the model produced; - a second run after comments are added embeds only the new comments (checked against the spool the child actually wrote) and serves a set containing all of them, reusing the earlier rows unchanged; - a single flag-on run and a single flag-off run of the same input serve the same results, with the embeddings bit-identical whether they live in the table or the codec family. Also cap read_spool's pre-allocation in the queue embedding module by the remaining byte length, so a crafted header row count cannot drive a huge reservation; covered by a new unit test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An automated end-to-end proof for the comment-embeddings table (
DELPHI_EMBEDDINGS_TABLE, default off), plus one hardening fix in the queue embedding module.The embeddings-table read/write code and migration 000038 are merged, but there was no automated end-to-end exercising the daemon glue (
stage_embeddings,hydrate_set/hydrate_cluster/hydrate_embed_prior) together with the Python flag-ongraph_embed/graph_clusterpath and the realsentence-transformers/all-MiniLM-L6-v2model. The flag should not be turned on in production until that passes. This adds it.The proof
scripts/test-embeddings-table.sh(new workflowEmbeddings table) runs a real Delphi job graph (graph_embed → graph_cluster → graph_topics → graph_narrative) through thepolis-jobsdaemon with the flag on and the real MiniLM model, against Postgres with every migration applied. It is wired into CI the same way the existing graph proofs run (a script invoked by a workflow), reusing the worker image that bakes the model weights at build time so the stage loads them offline (local_files_only).It asserts, with the flag on:
delphi_comment_embeddingsand publishes the set with the run's results; the read-back is bit-exact (float32) to what the model produced, and the embeddings are served from the table, not theDelphi_CommentEmbeddingscodec family.Hardening
read_spoolin the queue embedding module caps its pre-allocation by the remaining byte length, so a crafted header row count cannot drive a huge reservation before any row is read. Covered by a new unit test with a crafted header.Scope
The flag stays default-off everywhere except this test. No migration, schema, importer, model-choice or server changes.
Notes for review
🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s