Repository navigation
Promote edge to stable: owner checks on invite routes, connection budget, embeddings tables - #3052
Merged
Merged
Conversation
…3043) * CI: build the test-stack images once with layer cache; Cypress in parallel shards The E2E, Delphi and server integration checks each rebuilt every test-stack image from nothing in every job (about 7 minutes per job, three times over for the Delphi shards), and Cypress ran every spec one after another in one job. Images, once per workflow run - ci/local/test-images.sh builds the images docker-compose.test.yml builds, with `docker buildx bake` over the compose file as `docker compose config` renders it with test.env (the contexts and build arguments `compose build` uses). It takes an optional BuildKit cache (POLIS_TEST_IMAGES_CACHE_FROM/TO), can build a subset (--only), save the images as archives and load them, and writes a compose override that runs services from them without building or pulling. - .github/workflows/test-images.yml (reusable) runs it once per workflow run with the GitHub Actions layer cache and hands the images to the run's other jobs as a one-day artifact. Only pushes write the cache, and only from E2E Tests (it runs on every push), so two builds never write one scope at once. Pull requests, forks included, read it and pass images the same way. Cypress in shards - ci/local/e2e.sh is the E2E check as one local entry point (certificates, stack, migrations, readiness, auth setup check, Cypress) with --shard I/N and a combine step. Specs are split deterministically by recorded duration (ci/local/e2e-spec-durations.json; a new spec gets a default weight, so it is never left out). Cypress runs through its Module API (ci/local/e2e_run.cjs) so each spec's counts come from Cypress itself. ci/local/e2e_shards.py combine proves that the shards' selections partition the spec list, that each shard ran exactly the specs it selected, that shards 1..N are all present, and adds up the counts per suite. - E2E Tests: images once, six Cypress shards, `cypress-run` combines them. Delphi - Delphi Python Tests: postgres and delphi images once, six shards (was three, each building every image), `test` combines them as before. - ci/local/delphi-python.sh starts only what the suite uses (Postgres, DynamoDB, MinIO, the math poller and the delphi container), waits for the one-shot init containers instead of a fixed 20-second sleep, and skips the serial pass in a shard whose manifest lists no serial-marked tests (that pass would only collect the suite again and select nothing). - The deploy-hook check (scripts/test-deploy-hooks.sh) runs its tests in one shard; the others copy its inputs only (DEPLOY_HOOKS_COPY_ONLY=1). It ran once per shard before. - A shard unit made of a few long tests cannot finish before its longest test, however many workers share it: the durations record now holds [parallel, serial, longest] per unit and the plan weighs a unit as max(parallel / workers, longest) + serial (two-value entries still read). ci/local/delphi-test-durations.json is refreshed from a hosted six-shard run. Server integration - Its three images come from the same layer cache; the fixed 30-second sleep is replaced by a readiness wait after the dependency install, which now overlaps service start-up. No test is removed, skipped or loosened; per-suite counts are unchanged. The postgres image is built from the layer cache like the others (the old `--no-cache` comment assumed stale migration files could survive; BuildKit keys COPY layers on file content, and ci/test-migrations.sh applies and checks every migration). The "free disk space" steps are dropped from jobs that no longer build (runners have about 86 GB free before it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * CI: shard every spec Cypress's default pattern matches The planner globbed only .cy.js; Cypress's default specPattern also takes .cy.jsx, .cy.ts and .cy.tsx, so a future spec with one of those extensions would have run in no shard and been missing from the combine's spec list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Delphi: comment embeddings as one row per comment (migration 000038, draft) Adds delphi_embedding_models, delphi_comment_embeddings (real[] per conversation, model and comment, write-once) and delphi_embedding_sets (a run's closed set: every row up to a sequence mark), with fenced pd_embedding_put / pd_embedding_close and the read functions pd_embedding_set, pd_embedding_served and pd_embedding_rows. Additive: nothing existing is redefined and no current code calls the new objects. A reversal script refuses while any embedding is stored. Declared additive in compat.txt, listed in release.txt and docs/migrations.md; migration report regenerated; forward.py counts it among the tolerated additive migrations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * Delphi embeddings migration: the reversal script locks before it checks The reversal of 000038 checked for stored embeddings without a table lock, so a put still in flight could commit after the check and be dropped with the tables. It now takes ACCESS EXCLUSIVE on both embedding tables before the check: an in-flight put either commits first and the script refuses, or waits and fails once the tables are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * Delphi embeddings migration: no batch foreign key on rows; one guarded prune path Embedding rows outlive the run that first wrote them, so batch_id is now a plain provenance column with no foreign key: rows no longer keep a result batch from being pruned. Sets keep their foreign key to their batch, so a batch prune must remove the batch's sets first and fails loudly otherwise. Rows and sets still refuse UPDATE, and refuse DELETE except inside the new owner-only pd_embedding_prune, which: - superseded: per model, keeps the sets the served pointers resolve to, any newer set, the newest older one for rollback by pointer, and any set an unfinished job still consumes; deletes other sets, then rows of runs that never closed a set when no job for the conversation is unfinished; - conversation: deletes every row and set of a conversation; - model: deletes every row and set of a retired model. It takes the same per-conversation, per-model lock as put and close, and refuses the conversation and model modes while a job is unfinished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * Delphi embeddings migration: the prune removes whole conversations or retired models only pd_embedding_prune no longer has a mode that deletes superseded sets of a conversation that stays. Deciding which sets a later pointer, a rollback or an unfinished graph can still resolve to needs a rule per report scope over fully finished graphs; that is left for a later migration. A run's set is a small metadata row, since the embedding rows themselves are stored once per comment and shared. The function keeps two modes, conversation (optionally one model) and model, both refused while a job of an affected conversation is unfinished. Its signature drops the keep argument; the reversal script follows it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…efault off) (#3046) * Delphi: comment-embeddings spool format and config flag (default off) Introduce the per-row comment-embeddings path from migration 000038 behind one switch, DELPHI_EMBEDDINGS_TABLE, read by the job daemon and shared with the Delphi Python side. Default off is today's codec-family behaviour, byte for byte. - A single bounded spool layout moves embeddings between the daemon and the Delphi children, implemented identically in Rust (jobs::embedding) and Python (delphi_storage/embedding_spool.py); a pinned cross-language digest guards the byte layout. The Rust side also decodes a binary COPY stream and reproduces the set receipt's content digest from the exact array bytes. - The queue daemon's transport layer gains typed integer[]/bytea[] binding and registers the new read RPCs. - The flag is documented in docs/configuration.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * Delphi: daemon writes and reads the comment-embeddings table behind the flag With the flag on, the job daemon routes a run's embeddings into delphi_comment_embeddings instead of the codec family and reads them back for the cluster stage. - Writer: a two-dimensional real[] matrix binds through a typed path (the postgres crate cannot express it as Vec<Vec<f32>>), never as JSON; the daemon chunks pd_embedding_put and calls pd_embedding_close before pd_result_seal, and drops the embeddings family from the family spool. - Reader: the cluster stage's set is resolved by receipt, streamed out by binary COPY to a spool file whose digest is checked against the receipt; the embed stage is handed the currently served set so it re-embeds only missing comments. The frame carries the spool path and sha, not the bytes. - The writer manifest may carry one optional embeddings-spool descriptor (validated in Rust and in the JSON schema); absent, the manifest is exactly today's. - A new integration test applies the whole release migration chain (through 000038) to a throwaway database, scaffolds a fenced graph job through the real admission RPCs, then proves: write, read back with a matching digest and matrix round-trip, a second run adding only new comments, a closed set never changing, close-before-seal ordering, put-after-close refusal, and a tampered stream refused. With the flag off (the default) every path above is skipped and behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * Delphi: graph stages produce and consume the embeddings spool behind the flag With the flag on (carried in the job frame, never the child's environment): - graph_embed embeds only the comments the current set is missing and writes the embeddings spool for the daemon instead of the codec family; graph_cluster reads that spool rather than the hydrated family. - The graph child driver writes the bounded spool file and records its descriptor on the artifact. - The writer manifest validator accepts the optional embeddings-spool descriptor, mirroring the daemon's Rust validator. - Tests cover the spool round trip, the pinned cross-language digest, the flag reader and the manifest descriptor. With the flag off (the default) the codec-family path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…re read-capacity budget (#3047) A conversation or report created while the one-shot census runs is accepted and its already-scanned rows are rebound, but a conversation whose result rows are written after the scan has passed their page would be accepted with nothing to bind and otherwise missed silently. Record every mid-run addition's identity in follow-up.json inside the run's attempt directory on the private import mount (never on a relayed channel, never on stdout/stderr), and add a counts-only follow_up to the importer's stdout result, so a follow-up run re-imports anything this run could not see; re-importing a conversation already imported in the same run is idempotent. Removed conversations and re-pointed reports still refuse. Add an integration measurement against local DynamoDB proving a census held to a configured read-unit ceiling consumes read units no faster than the ceiling with every page metered (no fallback), alongside the existing fake-clock unit tests for pacing, jitter backoff and resume. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ation path (#3045) * server: run reports, exports and heavy admin reads on a small dedicated pool Reports, the data-export CSV routes, the comment-clusters export, conversationStats and the moderator participant query now use a small, timeout-bounded connection pool, kept separate from the pools the participant-facing path uses (vote and comment writes and next-statement selection on the read-write pool; the math blob read, GET /votes and participation reads on the read pool). A flood of slow reports or exports can no longer take the connections votes, comments and the next statement need: an export that cannot get one of the pool's connections within the acquire timeout is rejected rather than waiting on, or spilling into, a participant-facing connection, and each statement on the pool is capped by a statement timeout. The pool size, acquire timeout and statement timeout are configurable (REPORT_EXPORT_POOL_MAX, REPORT_EXPORT_ACQUIRE_TIMEOUT_MS, REPORT_EXPORT_STATEMENT_TIMEOUT_MS) and documented in docs/configuration.md, alongside the per-workload connection budget. Also hardens the shared pooled-query helper: a failed or timed-out connection acquire previously threw a TypeError that masked the real error, because it called a release callback the pool does not provide on that path. A load guard (participation-pg/export-budget.pgtest.ts) saturates the export pool and proves a vote, a comment and the next-statement call stay prompt while a further export is rejected. It runs in the existing Postgres result-reads CI job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: fix export-pool unit mocks, move participation stats to the export pool, free export connections on client disconnect - Update the export and empty-conversation unit mocks to the dedicated report/export query methods, so the senders' existing assertions hold and prove the senders read on the report/export pool. - Run the owner-gated GET /participation whole-conversation group-by scans on the report/export pool, the same class as conversationStats; getXids stays on the read pool (shared helper, bounded paginated join). - Release the report/export connection when the HTTP client disconnects mid-stream: close the cursor and stop fetching so an aborted download cannot hold one of the few export connections. Success-path output is unchanged. Add a guard proving a disconnected export frees its connection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: update the remaining stale export-sign unit mock to the report/export methods The vote-sign export byte test mocks pg-query and exercises the report senders, which now read on the dedicated report/export pool. Point its mock at queryP_reportExport / stream_queryP_reportExport so its assertions hold, the same fix as the other two export unit suites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: run reports, exports and heavy admin reads one at a time by default REPORT_EXPORT_POOL_MAX now defaults to 1, so further exports wait for the acquire timeout and the primary keeps more room for participation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add a job-graph proof that runs the queue daemon with DELPHI_EMBEDDINGS_TABLE on and the real all-MiniLM-L6-v2 model against Postgres with every migration applied, wired into CI the way the existing graph proofs run (scripts/test-embeddings-table.sh + .github/workflows/embeddings-table.yml, reusing the worker image that bakes the model weights at build time so the stage loads them offline). With the flag on the proof shows: - a first run writes every comment's embedding to the table and publishes the set with the run's results; the read-back is bit-exact (float32) to what the model produced; - a second run after comments are added embeds only the new comments (checked against the spool the child actually wrote) and serves a set containing all of them, reusing the earlier rows unchanged; - a single flag-on run and a single flag-off run of the same input serve the same results, with the embeddings bit-identical whether they live in the table or the codec family. Also cap read_spool's pre-allocation in the queue embedding module by the remaining byte length, so a crafted header row count cannot drive a huge reservation; covered by a new unit test. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…the per-row table behind one flag (#3050) Behind DELPHI_EMBEDDINGS_TABLE (default off everywhere; byte-identical to today when off), comment embeddings move from the Delphi_CommentEmbeddings codec family (bounded at 64 MiB, which the two largest production conversations already exceed) to the per-row delphi_comment_embeddings table from migration 000038, through the daemon's existing fenced pd_embedding_put / pd_embedding_close path. No schema or migration change; the flag is not enabled anywhere. - Importer (production_import, import_dynamo_production, report_dynamo_import, import_report): with the flag on, materialize writes each embedded comment to a bounded binary spool -- the legacy model name mapped to the one seeded model, every value checked float32-exact, the comment text's SHA-256 read once from the comments table -- and leaves the codec family empty, so a conversation whose embeddings exceed 64 MiB imports. The detached child re-emits the spool and the daemon routes it; after import the embedding-set receipt digest is verified against the spool, proving the float32 vectors round-tripped through Postgres exactly. The dry-run report and the apply result gain an embedding_models field (per-model-string counts). A comment whose text is gone is skipped and counted. - Single-process pipeline (run_pipeline, the writer resource, run_delphi, the datamap reader, storage): with the flag on, a run's embeddings go to the per-row table via the writer spool (each comment's text SHA-256 as provenance) instead of the codec family, which is excluded from the writer base and stages empty; the datamap reader reads the served set's comment ids from the table. Off, every path is unchanged. - Tests: importer boundary tests; writer-resource unit tests for the embedding spool and the base exclusion; a daemon-backed proof in the shared CI/local entry point (scripts/test-production-import.sh), sized by IMPORT_PROOF_EMBEDDINGS_COMMENTS (default CI-fast). At a large size the embeddings codec family exceeds 64 MiB: the flag-off apply refuses it, the flag-on apply imports every comment into the per-row table with the codec family empty and the set receipt verified. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Conversation invite and uuid routes now require the conversation owner or a moderator; GET zinvites no longer returns the uuid; a data migration clears existing uuids, which owners re-mint on demand.
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owner checks on conversation invite and uuid routes; existing conversation download keys are cleared (migration 000039), so owners re-copy saved XID download links once. Reports, exports and heavy admin reads run one at a time on their own pool. New comment-embeddings tables (migration 000038) and their code, behind a switch that stays off. Importer follow-up record; faster CI. Schema: migrations 000038 and 000039 apply at deploy.