Repository navigation
Promote edge to stable (2026-10-08) - #2994
Merged
Merged
Conversation
* queue-rs: polis-jobs crate skeleton, configuration and transports (off unless POLIS_JOBS_ENABLED=1) A second binary in queue-rs. Configuration comes from the environment and is validated before anything connects: heartbeat below a third of the lease, 000019's lease bounds, a closed stage list, the delphi worker class. The DSN goes through one of three transports: tls (pinned CA, exact host allowlist, the coordinator-rs rules), local (Unix socket, only when chosen explicitly) or loopback (literal loopback IPs, the queue-rs rule). The /2 RPCs are typed by signature_v2 and bound by the same helper as the /1 adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: polis-jobs main loop, claim, heartbeat, fencing and reaper Checks polis_queue_install.contract_version (exit 3 when it is not polis-queue/2), then LISTENs on polis_queue_wakeup_v1 with a poll fallback and claims with the six-argument pq_claim and worker class (owned or none). An uncertain claim is renewed by heartbeat, never reissued. Each job heartbeats on its own connection; fenced (or a whole lease without a successful heartbeat) kills the child's process group. The paged /2 pq_reap runs every lease and raises an alarm line for a job parked exit_unconfirmed too long. Readiness and transition lines follow the math poller's shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: polis-jobs child protocol, manifest finalize and provider intents The unchanged Delphi scripts run in a new session with the job frame on stdin and in DELPHI_FRAME, the DELPHI_* identity in the environment and no queue DSN. Every attempt ends only after the process group is empty: the manifest's exact bytes as a manifest log row, confirm_exit, then the seven-argument pq_finalize; failures, parks and releases use the /2 forms with exit proof; a fenced or cancelled attempt still confirms its exit with its original identity. Provider intents are recorded with pd_provider_intent and the acknowledgement file is written only after that commit; several intents per attempt are taken in sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: polis-jobs logs, shutdown and restart journal Child output is batched into polis_queue_logs (50 lines or 2 s), capped per attempt with one truncated marker, lines over 1 MiB split, replays absorbed by ON CONFLICT DO NOTHING. SIGTERM stops claiming, kills and reaps each child's group and ends the attempt with pq_fail(..., 'interrupted_by_shutdown', true). Before each spawn an fsynced journal entry records the attempt identity; on start, entries whose process is provably gone are confirmed with pq_fail(..., 'daemon_restarted', true). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: polis-jobs unit and PostgreSQL 17 integration tests Integration tests (feature jobs-integration) apply the repository's 000000-000022 chain and the draft 000023 (vendored as a test fixture) to a template database and run real daemon processes with a generated fixture child: claim/heartbeat/finalize, finalize refused without exit proof or the manifest row, cancel, a stalled daemon fenced after a park, the restart journal within and after the lease, poison, SIGTERM, start refusals and exit 3 without the contract, the provider intent committed before its ack and scope release only after completion, two daemons running one job once, a killed daemon blocking the other until its restart confirms, and the offline Unix-socket profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: polis-jobs review fixes: no immortal lease, journal kept on transient errors, one daemon per journal - A malformed recheck_after is manifest_invalid instead of a panic, and the heartbeat stops on drop, so a panicked job thread can no longer renew a lease for ever. - Lock/statement timeouts, failover and lost connections are retried, and a restart-journal entry is kept until its /2 call succeeds. - An exclusive flock on the journal directory: a second daemon on the same directory exits 2. - A journal entry without a pgid is unproven on the same host; if the pgid cannot be journaled the child is stopped and the attempt ended at once. - DSN logging shows only host, port, database and user. An uncertain claim whose renewal fails is ended as claim_uncertain. A manifest row the database did not take is retryable, not poison. Attempt work directories are removed after the final call. The readiness line carries emitted_ms and seq. - Tests for the child timeout, exit 6, the malformed recheck_after and the shared journal; the stalled-daemon test no longer depends on which daemon takes the retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * ci: build, lint and test queue-rs, with a PostgreSQL 17 service for the polis-jobs integration tests Runs on changes to queue-rs/, the server migrations or this workflow: clippy -D warnings with and without the integration feature, the unit and protocol tests, and the integration tests against a throwaway postgres:17 service container. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: polis-jobs validates the output manifest against the whole schema Required root keys, the closed key sets of inputs, models, cost, outputs and provider batches, and the nested types (integer ticks and token counts, hex comment digest, string models, non-empty partitions) are all checked before anything is recorded. Explicit nulls stay allowed where the schema permits unknowns, and recheck_after is set exactly when the outcome is parked. Any failure ends the attempt as manifest_invalid and never finalizes. The five schema-invalid manifests the second review drove to succeeded through the daemon and the S1 SQL, plus the wrong-identity control, are integration tests; the unit tests add further refusals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * queue-rs: workspace with a polis-api member; the shared transport takes its parts directly Connector::from_parts applies the same TLS, local and loopback rules for a second binary with its own configuration names and application_name. Connector::new is unchanged in behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: serve GET /api/v3/math/pca2 with Node's bytes One Rust process for one route. It reproduces the Express middleware chain, res.send/res.json, freshness, the compression middleware (with Node's own zlib and its stream chunking), the pca2 handler's labelled entity tag and 304 rules, getPca's caches and reshaping, and presentPca's comment backfill. Three typed read-only queries over the workspace's shared transport. Not routed anywhere by this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: byte-for-byte replay against Node and an nginx routing check conformance/run.sh starts the sealed characterization stack with polis-api beside the Node server, replays every recorded pca2 request from the pinned public-fixture archive to both as raw bytes, plus generated extra requests, and compares the raw responses with nothing normalised. nginx-routing.sh checks the RUST_API_ROUTES switch with stub upstreams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * nginx-proxy: route /api/v3/math/pca2 to polis-api only when RUST_API_ROUTES=pca2 Unset, both new include files are empty and the rendered config routes every request to Node as before. Set, a map sends the exact pca2 path to polis-api with Node as the backup upstream (down, timeout or 502). An unknown route name stops the container. Compose gains a polis-api service under the rust-api profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * ci: build, lint and test polis-api and check the nginx routing switch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: drop an agent name from a test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: a database failure is a 502 so nginx asks Node Connect failures, a pool wait over 2 s, statement timeouts (3 s) and query errors are typed as unavailable and answered 502, as is any path other than the exact route. Connections are retired after a fixed lifetime. Reads come from READ_ONLY_DATABASE_URL when set, else DATABASE_URL, as Node's read pool does. The image defaults NODE_ENV to production like the server's prod image. One http_request log line per request (method, path, status, duration, cache outcome). Empty API_DEV_HOSTNAME/API_PROD_HOSTNAME fall back as in Node, and trimming uses JavaScript's whitespace. Postgres-backed tests of the three queries, the read-only session, the timeout and connection retirement run with --features db-tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * nginx-proxy: polis-api never takes the proxy down; Node answers whenever it cannot The Rust upstream is resolved per request through Docker's DNS, so nginx starts and serves with the polis-api container absent, restarting or gone. A refused connection, 2 s connect or 5 s read timeout, or a 502 from polis-api goes to Node through a named location. An unknown route name logs an error and leaves every route on Node instead of stopping the container. The routing check covers off, absent at start, up, 502, dead and an unknown name. Compose passes NODE_ENV to polis-api as the server gets it and lets the database follow the read-pool rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * ci, docs: polis-api queries against PostgreSQL 17 and the image build in CI; enable-time caveats in the README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * ci: allowlist polis-api's math_tick DEFAULT -1 test-schema line (not a vote) The vote-sign lint flagged queue-rs/polis-api/src/db.rs because the last_vote_timestamp column name sits within its window. The -1 is math_main.math_tick's default, copied from 000000_initial.sql. Entry added with its reason; header sites 591 -> 592 (sum of counts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: nginx sends it only bodyless GET/HEAD, and no Rust setting can stop nginx With RUST_API_ROUTES unset nothing here is rendered: both include files stay empty and every request goes to Node as before. With it set: a route named twice is rendered once (pca2,pca2 used to stop nginx with a conflicting map parameter); RUST_API_UPSTREAM must be host:port and RUST_API_RESOLVER a list of addresses; and a rendered configuration that nginx -t refuses is emptied again and logged, so nginx always starts with every route on Node. Only GET and HEAD without a body reach polis-api; every other method and any request with a body go to Node before nginx reads it, so Node's body parsing and non-GET answers stay its own. The location comment states the fallback's real boundary: before polis-api's status line. nginx-routing.sh: check 6 reads the whole log before matching (docker logs | grep -q could fail under pipefail when grep exits early), and checks 7-10 cover the repeated name, the method/body gate, bad upstream/resolver settings, and the truncated-reply limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: a pool slot belongs to the database work, not to its waiter The permit was held by the awaiting future, outside spawn_blocking, so a cancelled caller freed its slot while its query or connect kept running; a pool of one could run two operations and keep two idle clients. The owned permit now moves into the blocking closure and is released after the client is returned or dropped, and idle clients are capped at the pool size. Witnesses: a stalled-connect test (no database) and a cancelled pg_sleep in the Postgres-backed test; both fail when the permit is released early. Inert unless polis-api runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-api: HTTP refusal without Vary; bodies it does not model go to Node Node's plain-HTTP 400 for non-GET requests is sent before the compression middleware, so it carries no Vary: Accept-Encoding; polis-api now sends it the same way (HEAD, OPTIONS, POST cases added). A request body with a Content-Encoding, a non-UTF-8 JSON/urlencoded charset, or a multipart type is answered 502, which nginx turns into Node's answer, instead of a different status (nginx no longer sends bodies here at all). README narrows the fallback claim to failures before the response starts and documents the new limits. Inert unless polis-api runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ews (#2959) Records, for twelve generated states built from the recorded Delphi route responses, the rendered HTML, request order and hook output of CommentsReport, TopicPage (including a link to another topic), TopicStats, TopicsVizReport, TopicDataProvider with TopicSectionsBuilder, and the participation client's Topic Agenda and useTopicData hook. The paid collective-statement POST that TopicPage sends on view is recorded with its endpoint, body and token argument; no provider is called. A guard refuses a change to view code together with the golden or its rules. Intended changes go through named expected-difference entries using the vote-path ruling syntax; pending entries fail. A mutation runner breaks twelve sites in the views and requires a failing recorded-value assertion for each. The ordinary client-report test run excludes these suites; a dedicated job runs them through `npm run test:jobviews` in client-report. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ates (#2958) Adds pure helpers that read which Delphi job produced a topic from the job id prefix already present in each topic_key, in TypeScript for the participation client and generated JavaScript for the report client (a drift check keeps the two identical). Nothing imports them yet, so no shipped bundle changes. The helpers are replayed against the nine recorded Delphi route states plus generated fixture cases: malformed keys, timezone-less timestamps, two jobs' topics mixed in one day's run, and a newer unfinished job whose topics are already present. A mutation runner breaks the rule at seven sites and requires a failing assertion for each. CI calls one local entry point: server/characterization/jobviews/check-provenance.sh. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…migration count (#2969) The collective-statement recordings' index pinned "migrations": 22, and the golden guard forbids re-recording in a PR that touches server/postgres/, so every new migration made the replay red with all 51 cases still matching. The index now pins the shape of the relations the fixture writes and the routes read (columns, constraints, rules, triggers of users, oidc_user_mappings, conversations, zinvites, reports, participants, comments, votes, votes_latest_unique) and the definition of the one function the vote writer asks for, read after the migrations at record time (schema.cjs). Replay fails only when a recorded case differs or that shape drifts, naming the relation and the change; a migration touching anything else passes. Pinning migration-file digests was rejected: the harness would have to parse SQL to decide which files touch these relations, and a comment edit to 000000_initial.sql would still turn the replay red. schema.test.cjs covers the rule against a catalog stub and, with the replay job's Postgres, against the real migrations: one more unrelated migration (new table, column on an unread table, index on a read one) passes; a column added to comments fails. The CI rules step runs it with that Postgres. index.json was regenerated by `main.cjs record` on edge's server; the 51 case recordings are byte for byte unchanged. The guard's rules for pull requests are unchanged. The other suites do not pin a migration count: the Delphi routes pin a fixture digest, routing lifts one function from 000000_initial.sql, the vote-path manifests carry the migration digest as provenance only, and the job views have no Postgres. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
`npx cdk synth`, `diff` and `deploy` failed on any machine without a container runtime: the database-backup function was built by the PythonFunction construct of `@aws-cdk/aws-lambda-python-alpha`, which runs pip inside a Docker container and dies with `spawnSync docker ENOENT` otherwise. The function is now a plain `lambda.Function` whose code is `Code.fromAsset` over `lambda/handler` with the construct's local bundling: pip on the synthesizing machine installs the wheel pinned by version and sha256 in `requirements.txt` for the Lambda's own platform (CPython 3.12, x86_64, manylinux2014) and the handler source is copied beside it, the layout the container build produced. The pin is the version the deployed function carries (psycopg2-binary 2.9.10, read from the live function's code package); it was unpinned before. Nothing is vendored into the repository, and a failed pip run fails the synth with its own message instead of falling back to Docker. The alpha package is removed from the cdk dependencies. The deployed function is unchanged in handler, runtime, role, VPC placement, timeout, memory, ephemeral storage, environment, layers and schedule: the synthesized template differs from edge's only in the function's asset key (and the CDKMetadata analytics string), and a `cdk diff` against the live stack shows only that key changing. The package itself matches the live function's code file for file apart from the bytecode caches pip no longer writes and the pinned requirements file. Tests: a new `test/dbBackupLambda.test.ts` pins the function's shape with bundling skipped and runs the pip packaging once for real; the large-class snapshot moves by the one asset-key line; two test comments no longer mention Docker. README: how the package is built and how to produce it by hand. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…B keeps running every job) (#2968) * tests: the Delphi job table is expected in the repository migration chain (failing first) The polis-jobs daemon (#2947) runs on migration 000023, contract polis-queue/2, which so far existed only as a copy vendored under queue-rs/tests/fixtures. These tests expect it where every other migration lives: - queue-rs/tests/foundation_migration.rs (no database): the up and down scripts are at server/postgres/migrations and match their seal; the up script admits exactly the daemon's stages beside noop; with POLIS_JOBS_ENABLED unset the daemon exits 0 before reading any DSN. - queue-rs/tests/jobs_integration.rs: the chain is 000000-000023; jobs_base stops at 000022 (the "contract missing" shape), jobs_v2 applies the repository's 000023 rather than the fixture copy. - server/__tests__/integration/delphi-job-table.test.ts: on the test database the seven job tables exist, are owned by polis_queue_owner and are empty; polis_queue_install reads polis-queue/2; only the executor may call the /2 RPCs; with the flag off the server's queue helper refuses before it opens a transaction. - server/__tests__/integration/queue-substrate.test.ts: once 000023 is in place, a 000019 replay against the test database is refused by 000019's own catalog fingerprint (nothing silently reverts to /1); the scratch database checks are unchanged. On mm2 at this commit: foundation_migration 1 passed, 2 failed (the migration is missing), as intended. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the Delphi job table (migration 000023, polis-queue/2; flag off; DynamoDB keeps running every job) Observable changes: none at runtime. The migration is additive and creates empty tables; no route, flag or worker reads them unless switched on. Moves the polis-jobs daemon's foundation migration from its test-fixture copy into the repository chain, byte for byte in its DDL: server/postgres/migrations/000023_create_delphi_foundation.sql server/postgres/migrations/down/000023_drop_delphi_foundation.sql server/postgres/migrations/down/000023-files.sha256 (seal) server/postgres/migrations/down/test_000023_down.sh (the proof) What it creates, all owned by polis_queue_owner: delphi_jobs (id, env, conversation, kind, parent for sub-jobs, run, status kept in step with the queue row by trigger, digests, cost, timestamps, error), delphi_job_aliases, delphi_job_inputs, delphi_current (empty; nothing in /2 moves it), delphi_job_guards, delphi_provider_requests (a paid batch is recorded before it is submitted), polis_queue_logs, delphi_foundation_install. On 000019's tables: contract_version, the two Delphi stages beside noop, worker_class, process_exit_confirmed_at, binding_expires_at. No result table. Proven on mm2 by test_000023_down.sh against the real chain (seal, a-h): forward; a second apply refused by the fingerprint guard with the catalog unchanged; a 000019 replay over /2 refused ("queue catalog drift", nothing reverts); the noop /1 path still working on /2 with delphi_jobs empty; a refused down with a /2 row present; the down restoring a byte-identical schema dump and role set; apply -> down -> apply -> down reproducing both states; the down failing cleanly where 000023 was never applied. queue-rs reads the repository file (README, CI comment); docs/queue-substrate.md gains the /2 section, including that applying it to production, 000019 first, is the owner's separate, explicit step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the queue migrations name their parent lock and apply through a checked wrapper with budgets and a preflight Both 000019 and 000023 create foreign keys to public.conversations and so hold ShareRowExclusiveLock on that table from that statement until COMMIT, which blocks every INSERT, UPDATE and DELETE on conversations meanwhile and makes the apply wait behind any open writer. The second reviewer's pre-merge map asked for that to be named, bounded and witnessed before #2968 merges. - server/postgres/bin/apply-migration.sh: the one way to apply 000019 or 000023 to an existing database. Preflight (seal, PostgreSQL 17, rights, 000019 installed for 000023, queue tables empty, no other transaction older than --max-xact-age, operator-measured free disk above the floor), then the budgets as session settings ahead of the file: lock_timeout 5s, statement_timeout 60s, transaction_timeout 120s, idle_in_transaction_session_timeout 30s; a post-check after. - Both migration headers, docs/queue-substrate.md and the queue-rs README name the lock, require an idle or controlled writer window, and point at the wrapper. The headers are comment-only changes; 000023 is resealed in down/000023-files.sha256 and 000019's pins (QUEUE_SQL_SHA256 in server/src/queue/protocol.ts and delphi/polismath/queue/executor.py, and the coordinator schema census entry) carry its new sha256. - test_000023_down.sh check (i): the contended-parent witness through the wrapper, for both files: an uncommitted UPDATE on conversations makes the apply wait for ShareRowExclusiveLock, fail on lock_timeout and change nothing; an older writer is refused by the preflight; the same command applies once the writer is gone. - Wording: delphi_job_guards is a guard held until safe explicit release (pd_release_scope), not "one active job per scope"; attempt logs are direct INSERTs with a per-row limit and no retention yet. No CASCADE, no automatic release, no change to any queue function. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…4, polis-queue/3; nothing enqueues yet) (#2971) * tests: the Delphi job table is expected in the repository migration chain (failing first) The polis-jobs daemon (#2947) runs on migration 000023, contract polis-queue/2, which so far existed only as a copy vendored under queue-rs/tests/fixtures. These tests expect it where every other migration lives: - queue-rs/tests/foundation_migration.rs (no database): the up and down scripts are at server/postgres/migrations and match their seal; the up script admits exactly the daemon's stages beside noop; with POLIS_JOBS_ENABLED unset the daemon exits 0 before reading any DSN. - queue-rs/tests/jobs_integration.rs: the chain is 000000-000023; jobs_base stops at 000022 (the "contract missing" shape), jobs_v2 applies the repository's 000023 rather than the fixture copy. - server/__tests__/integration/delphi-job-table.test.ts: on the test database the seven job tables exist, are owned by polis_queue_owner and are empty; polis_queue_install reads polis-queue/2; only the executor may call the /2 RPCs; with the flag off the server's queue helper refuses before it opens a transaction. - server/__tests__/integration/queue-substrate.test.ts: once 000023 is in place, a 000019 replay against the test database is refused by 000019's own catalog fingerprint (nothing silently reverts to /1); the scratch database checks are unchanged. On mm2 at this commit: foundation_migration 1 passed, 2 failed (the migration is missing), as intended. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the Delphi job table (migration 000023, polis-queue/2; flag off; DynamoDB keeps running every job) Observable changes: none at runtime. The migration is additive and creates empty tables; no route, flag or worker reads them unless switched on. Moves the polis-jobs daemon's foundation migration from its test-fixture copy into the repository chain, byte for byte in its DDL: server/postgres/migrations/000023_create_delphi_foundation.sql server/postgres/migrations/down/000023_drop_delphi_foundation.sql server/postgres/migrations/down/000023-files.sha256 (seal) server/postgres/migrations/down/test_000023_down.sh (the proof) What it creates, all owned by polis_queue_owner: delphi_jobs (id, env, conversation, kind, parent for sub-jobs, run, status kept in step with the queue row by trigger, digests, cost, timestamps, error), delphi_job_aliases, delphi_job_inputs, delphi_current (empty; nothing in /2 moves it), delphi_job_guards, delphi_provider_requests (a paid batch is recorded before it is submitted), polis_queue_logs, delphi_foundation_install. On 000019's tables: contract_version, the two Delphi stages beside noop, worker_class, process_exit_confirmed_at, binding_expires_at. No result table. Proven on mm2 by test_000023_down.sh against the real chain (seal, a-h): forward; a second apply refused by the fingerprint guard with the catalog unchanged; a 000019 replay over /2 refused ("queue catalog drift", nothing reverts); the noop /1 path still working on /2 with delphi_jobs empty; a refused down with a /2 row present; the down restoring a byte-identical schema dump and role set; apply -> down -> apply -> down reproducing both states; the down failing cleanly where 000023 was never applied. queue-rs reads the repository file (README, CI comment); docs/queue-substrate.md gains the /2 section, including that applying it to production, 000019 first, is the owner's separate, explicit step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * tests: the large worker class (polis-queue/3, migration 000024) is expected in the repository chain (failing first) The crate's migration pins expect 000024 sealed beside 000023 and admitting class large with stage math_rebuild; the integration chain is 000000-000024. The server's catalog pins expect the install row to read polis-queue/3, the stage and class CHECKs to admit math_rebuild and large (bound to each other), 29 pq_ functions and pq_class_depth granted to the executor. At this commit the migration does not exist, so these fail. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the large worker class (migration 000024, polis-queue/3; nothing enqueues yet) On 000023's Delphi job table: worker_class admits large, stage admits math_rebuild, the two are bound to each other (pq_stage_large), delphi_jobs.kind admits math_rebuild so the one-active-job-per-scope guard covers a rebuild unchanged, runs admit contract polis-queue/3, and the install row reads polis-queue/3. pq_claim and pq_reap take the class as given (delphi or large) and see only jobs of that class; pd_enqueue admits math_rebuild (no report id; scope math:<label>:<zid>) under the guards it already applies; pd_queue_binding binds the rebuild row to its kind, class and contract; pq_result says polis-queue/3 for a rebuild. One new read, pq_class_depth(env, class): queued, leased, parked, dead and the oldest unresolved created_at, for the small poller's capacity line. No new job table; every job table stays empty; the /1 noop and /2 Delphi paths are unchanged. Sealed with its down script (down/000024-files.sha256); the down restores the /2 catalog from the recorded baseline and refuses with any /3 row present. Proven on a work box against the real chain by down/test_000024_down.sh (seal, a-i): forward, refused replays of 000019/000023/000024, the rebuild admitted, guarded, claimed by its class only, finalized and released, refused down with /3 data, byte-identical dump after the down, re-apply, clean failure where 000024 or 000023 is absent, and 000023's own down unwinding after it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: polis-queue/3 gains the poison latch and the job view's scope key (000024 resealed) Two composed-contract defects from the operating-surface map, fixed where the contract lives: - pd_enqueue: when a scope's last three jobs all died under the code image being admitted now, the reply is outcome `poisoned` naming the latest dead job and no job is made; a new image, or a succeeded or cancelled job among the last three, admits again. Without it an automated producer re-admits a permanent failure forever, one attempt budget per pass. - pd_job_view gains `scope_key` (the guard the job's root holds, null once released), so the polis-jobs daemon can release the scope through pd_release_scope after a terminal attempt whose exit it proved. The safe release function is unchanged: terminal status alone never releases. test_000024_down.sh (h) witnesses both: the job view's scope before and after a release; three deaths, a poisoned fourth admission (no job, no guard), a new image admitting again. Both migration files resealed in down/000024-files.sha256. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…loyment runs one yet) (#2972) * tests: the Delphi job table is expected in the repository migration chain (failing first) The polis-jobs daemon (#2947) runs on migration 000023, contract polis-queue/2, which so far existed only as a copy vendored under queue-rs/tests/fixtures. These tests expect it where every other migration lives: - queue-rs/tests/foundation_migration.rs (no database): the up and down scripts are at server/postgres/migrations and match their seal; the up script admits exactly the daemon's stages beside noop; with POLIS_JOBS_ENABLED unset the daemon exits 0 before reading any DSN. - queue-rs/tests/jobs_integration.rs: the chain is 000000-000023; jobs_base stops at 000022 (the "contract missing" shape), jobs_v2 applies the repository's 000023 rather than the fixture copy. - server/__tests__/integration/delphi-job-table.test.ts: on the test database the seven job tables exist, are owned by polis_queue_owner and are empty; polis_queue_install reads polis-queue/2; only the executor may call the /2 RPCs; with the flag off the server's queue helper refuses before it opens a transaction. - server/__tests__/integration/queue-substrate.test.ts: once 000023 is in place, a 000019 replay against the test database is refused by 000019's own catalog fingerprint (nothing silently reverts to /1); the scratch database checks are unchanged. On mm2 at this commit: foundation_migration 1 passed, 2 failed (the migration is missing), as intended. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the Delphi job table (migration 000023, polis-queue/2; flag off; DynamoDB keeps running every job) Observable changes: none at runtime. The migration is additive and creates empty tables; no route, flag or worker reads them unless switched on. Moves the polis-jobs daemon's foundation migration from its test-fixture copy into the repository chain, byte for byte in its DDL: server/postgres/migrations/000023_create_delphi_foundation.sql server/postgres/migrations/down/000023_drop_delphi_foundation.sql server/postgres/migrations/down/000023-files.sha256 (seal) server/postgres/migrations/down/test_000023_down.sh (the proof) What it creates, all owned by polis_queue_owner: delphi_jobs (id, env, conversation, kind, parent for sub-jobs, run, status kept in step with the queue row by trigger, digests, cost, timestamps, error), delphi_job_aliases, delphi_job_inputs, delphi_current (empty; nothing in /2 moves it), delphi_job_guards, delphi_provider_requests (a paid batch is recorded before it is submitted), polis_queue_logs, delphi_foundation_install. On 000019's tables: contract_version, the two Delphi stages beside noop, worker_class, process_exit_confirmed_at, binding_expires_at. No result table. Proven on mm2 by test_000023_down.sh against the real chain (seal, a-h): forward; a second apply refused by the fingerprint guard with the catalog unchanged; a 000019 replay over /2 refused ("queue catalog drift", nothing reverts); the noop /1 path still working on /2 with delphi_jobs empty; a refused down with a /2 row present; the down restoring a byte-identical schema dump and role set; apply -> down -> apply -> down reproducing both states; the down failing cleanly where 000023 was never applied. queue-rs reads the repository file (README, CI comment); docs/queue-substrate.md gains the /2 section, including that applying it to production, 000019 first, is the owner's separate, explicit step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * tests: the large worker class (polis-queue/3, migration 000024) is expected in the repository chain (failing first) The crate's migration pins expect 000024 sealed beside 000023 and admitting class large with stage math_rebuild; the integration chain is 000000-000024. The server's catalog pins expect the install row to read polis-queue/3, the stage and class CHECKs to admit math_rebuild and large (bound to each other), 29 pq_ functions and pq_class_depth granted to the executor. At this commit the migration does not exist, so these fail. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the large worker class (migration 000024, polis-queue/3; nothing enqueues yet) On 000023's Delphi job table: worker_class admits large, stage admits math_rebuild, the two are bound to each other (pq_stage_large), delphi_jobs.kind admits math_rebuild so the one-active-job-per-scope guard covers a rebuild unchanged, runs admit contract polis-queue/3, and the install row reads polis-queue/3. pq_claim and pq_reap take the class as given (delphi or large) and see only jobs of that class; pd_enqueue admits math_rebuild (no report id; scope math:<label>:<zid>) under the guards it already applies; pd_queue_binding binds the rebuild row to its kind, class and contract; pq_result says polis-queue/3 for a rebuild. One new read, pq_class_depth(env, class): queued, leased, parked, dead and the oldest unresolved created_at, for the small poller's capacity line. No new job table; every job table stays empty; the /1 noop and /2 Delphi paths are unchanged. Sealed with its down script (down/000024-files.sha256); the down restores the /2 catalog from the recorded baseline and refuses with any /3 row present. Proven on a work box against the real chain by down/test_000024_down.sh (seal, a-i): forward, refused replays of 000019/000023/000024, the rebuild admitted, guarded, claimed by its class only, finalized and released, refused down with /3 data, byte-identical dump after the down, re-apply, clean failure where 000024 or 000023 is absent, and 000023's own down unwinding after it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * tests: polis-jobs as a worker of class large runs the math rebuild and nothing else (failing first) Against the repository's 000024 (polis-queue/3, template jobs_v3): a class-large daemon claims the math_rebuild job, runs the math poller's job entry with `--job` and the zid in the frame, finalizes a manifest with no DynamoDB outputs, and never claims a Delphi job; a class-delphi daemon on the same database runs the Delphi job and never the rebuild; both report the installed contract. On /2 a class-large daemon exits 3 (contract missing); a Delphi stage in a large worker's stage list exits 2. The fixture child gains the math poller's job entry. At this commit the daemon admits only class delphi, so these fail. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-jobs: the large worker class and the math rebuild stage (no deployment runs one yet) POLIS_JOBS_WORKER_CLASS admits large beside delphi, as a closed type with its own stage list and the contracts it may start on: delphi runs the two Delphi stages on polis-queue/2 or /3, large runs math_rebuild on /3 only. A worker claims only jobs of its class (the /3 claim takes the class as given) and admits only its class's stages. Stage math_rebuild maps to the child command scripts/math_poller.py --job, the math poller's job entry, with the zid in the frame; the frame and output-manifest schemas gain the stage; a rebuild's manifest lists no DynamoDB outputs. A class-large worker on a /2 database exits 3 naming the contract it needs; a stage of another class exits 2 before any connection. The readiness line reports the installed contract; the typed RPC table gains pq_class_depth. The math poller's job entry itself is a later change; the fixture child stands in for it here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: polis-queue/3 gains the poison latch and the job view's scope key (000024 resealed) Two composed-contract defects from the operating-surface map, fixed where the contract lives: - pd_enqueue: when a scope's last three jobs all died under the code image being admitted now, the reply is outcome `poisoned` naming the latest dead job and no job is made; a new image, or a succeeded or cancelled job among the last three, admits again. Without it an automated producer re-admits a permanent failure forever, one attempt budget per pass. - pd_job_view gains `scope_key` (the guard the job's root holds, null once released), so the polis-jobs daemon can release the scope through pd_release_scope after a terminal attempt whose exit it proved. The safe release function is unchanged: terminal status alone never releases. test_000024_down.sh (h) witnesses both: the job view's scope before and after a release; three deaths, a poisoned fourth admission (no job, no guard), a new image admitting again. Both migration files resealed in down/000024-files.sha256. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-jobs: the typed math config crosses the child boundary; a terminal job frees its scope Two composed-contract defects from the operating-surface map (findings 2 and 3): - The frame of a math_rebuild job carried only the Delphi config keys, so the child could not check the source commit, the target label or the declared need. The admission's config is now a typed MathConfig (staged_label, target_label, need_bytes, input_through_ms, binding, source_commit), checked whole by the daemon before any child spawns (an admission without it is math_config_invalid, permanent, exit proof honest) and carried whole in the frame (schemas/job-frame-v1.json). The fixture child asserts the six typed keys arrived through the real frame. - Nobody called pd_release_scope, so the first terminal job occupied its conversation forever. The daemon now releases the scope after a terminal reply (succeeded, dead, cancelled) for an attempt whose exit it proved, on the task, journal-recovery and reaper paths, reading the scope from pd_job_view (scope_key, polis-queue/3). The SQL re-checks every condition and refuses otherwise; one transition says scope_released or scope_held with the reason; the readiness line counts released_total. Terminal status alone releases nothing. Integration (real daemons, real chain to 000024): the rebuild's typed config reaches the child and its scope is released by the daemon, a Delphi job on /3 too; an untyped rebuild is dead before any child and the scope is free; cancel -> re-enqueue succeeds; dead frees the scope; three deaths under one image make the fourth admission poisoned (no job, no guard) and a new image admits again. 39 unit, 23 integration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-jobs: module list in rustfmt order (cargo fmt --check) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…e queue; manifest remnants removed (#2974) Behind the existing -c enableLargeClass flag; with the flag off the template is byte-identical to edge's (pinned by a snapshot test). cdk/workerClasses.ts replaces cdk/largeClass.ts. A table of queue worker classes becomes one Auto Scaling group per class, each with its own launch template, instance role and queue-depth alarms: math-large r7i.2xlarge, 0..1, cgroup 52g, POLIS_JOBS_WORKER_CLASS=large, service type delphi-large (the existing AsgDelphiLarge ids) delphi r7i.2xlarge, 0..3 (context delphiClassMaxCapacity), cgroup 52g, POLIS_JOBS_WORKER_CLASS=delphi, service type delphi-worker The math-serve group (AsgDelphiSmall) is untouched. - Metric filters read the small poller's capacity line (large_demand, large_leased, large_parked, large_dead, large_poisoned, oldest_queued_age_ms, and the queue-wide keys). Scale-out on queued >= 1 (exact capacity); scale-in only when queued + leased + parked = 0 with every term reported (a missing term is never read as 0). - The queue alarm table as CloudWatch alarms on metric filters: Parked, Dead, OldestQueued, MissingWorker, WorkerDisk per class; Unreachable, Bytes, SweepLag, Full once. No Lambda, no custom resource. - Worker user data writes the class, memory limit, image and the NAME of the restricted queue login's secret, and starts the daemon with its class as a systemd unit that reads the password by secret name at each start. - The delphi group is not in the CodeDeploy deployment group until the deploy hooks have a delphi-worker branch. - Removed: the manifest bucket/key context, CapacityManifestRead, grantManifestReadWrite and the shared InstanceRole policy change. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…P-073 r2) (#2973) * tests: the Delphi job table is expected in the repository migration chain (failing first) The polis-jobs daemon (#2947) runs on migration 000023, contract polis-queue/2, which so far existed only as a copy vendored under queue-rs/tests/fixtures. These tests expect it where every other migration lives: - queue-rs/tests/foundation_migration.rs (no database): the up and down scripts are at server/postgres/migrations and match their seal; the up script admits exactly the daemon's stages beside noop; with POLIS_JOBS_ENABLED unset the daemon exits 0 before reading any DSN. - queue-rs/tests/jobs_integration.rs: the chain is 000000-000023; jobs_base stops at 000022 (the "contract missing" shape), jobs_v2 applies the repository's 000023 rather than the fixture copy. - server/__tests__/integration/delphi-job-table.test.ts: on the test database the seven job tables exist, are owned by polis_queue_owner and are empty; polis_queue_install reads polis-queue/2; only the executor may call the /2 RPCs; with the flag off the server's queue helper refuses before it opens a transaction. - server/__tests__/integration/queue-substrate.test.ts: once 000023 is in place, a 000019 replay against the test database is refused by 000019's own catalog fingerprint (nothing silently reverts to /1); the scratch database checks are unchanged. On mm2 at this commit: foundation_migration 1 passed, 2 failed (the migration is missing), as intended. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the Delphi job table (migration 000023, polis-queue/2; flag off; DynamoDB keeps running every job) Observable changes: none at runtime. The migration is additive and creates empty tables; no route, flag or worker reads them unless switched on. Moves the polis-jobs daemon's foundation migration from its test-fixture copy into the repository chain, byte for byte in its DDL: server/postgres/migrations/000023_create_delphi_foundation.sql server/postgres/migrations/down/000023_drop_delphi_foundation.sql server/postgres/migrations/down/000023-files.sha256 (seal) server/postgres/migrations/down/test_000023_down.sh (the proof) What it creates, all owned by polis_queue_owner: delphi_jobs (id, env, conversation, kind, parent for sub-jobs, run, status kept in step with the queue row by trigger, digests, cost, timestamps, error), delphi_job_aliases, delphi_job_inputs, delphi_current (empty; nothing in /2 moves it), delphi_job_guards, delphi_provider_requests (a paid batch is recorded before it is submitted), polis_queue_logs, delphi_foundation_install. On 000019's tables: contract_version, the two Delphi stages beside noop, worker_class, process_exit_confirmed_at, binding_expires_at. No result table. Proven on mm2 by test_000023_down.sh against the real chain (seal, a-h): forward; a second apply refused by the fingerprint guard with the catalog unchanged; a 000019 replay over /2 refused ("queue catalog drift", nothing reverts); the noop /1 path still working on /2 with delphi_jobs empty; a refused down with a /2 row present; the down restoring a byte-identical schema dump and role set; apply -> down -> apply -> down reproducing both states; the down failing cleanly where 000023 was never applied. queue-rs reads the repository file (README, CI comment); docs/queue-substrate.md gains the /2 section, including that applying it to production, 000019 first, is the owner's separate, explicit step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * tests: the large worker class (polis-queue/3, migration 000024) is expected in the repository chain (failing first) The crate's migration pins expect 000024 sealed beside 000023 and admitting class large with stage math_rebuild; the integration chain is 000000-000024. The server's catalog pins expect the install row to read polis-queue/3, the stage and class CHECKs to admit math_rebuild and large (bound to each other), 29 pq_ functions and pq_class_depth granted to the executor. At this commit the migration does not exist, so these fail. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: the large worker class (migration 000024, polis-queue/3; nothing enqueues yet) On 000023's Delphi job table: worker_class admits large, stage admits math_rebuild, the two are bound to each other (pq_stage_large), delphi_jobs.kind admits math_rebuild so the one-active-job-per-scope guard covers a rebuild unchanged, runs admit contract polis-queue/3, and the install row reads polis-queue/3. pq_claim and pq_reap take the class as given (delphi or large) and see only jobs of that class; pd_enqueue admits math_rebuild (no report id; scope math:<label>:<zid>) under the guards it already applies; pd_queue_binding binds the rebuild row to its kind, class and contract; pq_result says polis-queue/3 for a rebuild. One new read, pq_class_depth(env, class): queued, leased, parked, dead and the oldest unresolved created_at, for the small poller's capacity line. No new job table; every job table stays empty; the /1 noop and /2 Delphi paths are unchanged. Sealed with its down script (down/000024-files.sha256); the down restores the /2 catalog from the recorded baseline and refuses with any /3 row present. Proven on a work box against the real chain by down/test_000024_down.sh (seal, a-i): forward, refused replays of 000019/000023/000024, the rebuild admitted, guarded, claimed by its class only, finalized and released, refused down with /3 data, byte-identical dump after the down, re-apply, clean failure where 000024 or 000023 is absent, and 000023's own down unwinding after it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * tests: polis-jobs as a worker of class large runs the math rebuild and nothing else (failing first) Against the repository's 000024 (polis-queue/3, template jobs_v3): a class-large daemon claims the math_rebuild job, runs the math poller's job entry with `--job` and the zid in the frame, finalizes a manifest with no DynamoDB outputs, and never claims a Delphi job; a class-delphi daemon on the same database runs the Delphi job and never the rebuild; both report the installed contract. On /2 a class-large daemon exits 3 (contract missing); a Delphi stage in a large worker's stage list exits 2. The fixture child gains the math poller's job entry. At this commit the daemon admits only class delphi, so these fail. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-jobs: the large worker class and the math rebuild stage (no deployment runs one yet) POLIS_JOBS_WORKER_CLASS admits large beside delphi, as a closed type with its own stage list and the contracts it may start on: delphi runs the two Delphi stages on polis-queue/2 or /3, large runs math_rebuild on /3 only. A worker claims only jobs of its class (the /3 claim takes the class as given) and admits only its class's stages. Stage math_rebuild maps to the child command scripts/math_poller.py --job, the math poller's job entry, with the zid in the frame; the frame and output-manifest schemas gain the stage; a rebuild's manifest lists no DynamoDB outputs. A class-large worker on a /2 database exits 3 naming the contract it needs; a stage of another class exits 2 before any connection. The readiness line reports the installed contract; the typed RPC table gains pq_class_depth. The math poller's job entry itself is a later change; the fixture child stands in for it here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * math poller: tests for oversized conversations as queue jobs (P-073 r2) The tests first, red on edge: the small poller inserts one math_rebuild job of worker class large through the queue's SQL contract when its estimator decides a rebuild would not fit; a second insert while that job is active finds it (the one-active-job-per-scope guard) and a finished job releases the scope; a queue child (`math_poller.py --job`) computes exactly the frame's conversation, stages it under the staged label and writes its output manifest; the capacity line carries large_demand and large_leased from pq_class_depth. Unit (tests/poller/test_capacity_queue.py): a fake of the closed queue contract. Postgres (tests/poller/test_capacity_queue_postgres.py): the real contract as an executor-member login, the guard, the depth read before and after a claim, the child as a subprocess, and the existing promotion of what it staged. The database the Postgres tests need does not exist on edge yet: the fixture applies the Delphi job table migration (000023, from the open pull request that adds it, copied verbatim under tests/poller/fixtures/queue/) and then polis_queue_3_standin.sql, a TEST STAND-IN for the polis-queue/3 migration that is its own pull request (class large, stage math_rebuild, pd_enqueue admitting it, pq_class_depth). The stand-in is marked as such in its header; when the real migration lands, the fixture points at it and the stand-in goes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * math poller: oversized conversations are jobs on the Postgres queue (P-073 r2) Observable changes: none with MATH_CAPACITY_ROUTING unset (default), except one new key on the capacity line and the readiness `capacity` object, large_leased (null without a queue). With routing on: oversized conversations become queue jobs instead of manifest entries; the S3 manifest and MATH_CAPACITY_MANIFEST_URI are gone. The small poller's side (polismath/poller/capacity_queue.py): when the estimator classifies a cold touch, rebuild or memory refusal `large`, one math_rebuild job of worker class large is inserted through a closed inventory of SQL functions (pd_enqueue, pq_class_depth, pq_job_status, pq_cancel; the executor's pattern: one short transaction per call, fixed casts, the executor-membership boundary re-checked on every connection) over a second DSN, MATH_CAPACITY_QUEUE_DSN, in the namespace MATH_CAPACITY_QUEUE_ENV. Idempotent under the queue's one-active-job-per- scope guard (scope math:<label>:<zid>); the promotion pass asks again whenever a large record has no staged bundle covering its input, never- asked records first. The record keeps the job id. A queue failure never changes the routing outcome. The large box's side (polismath/poller/rebuild_child.py, `math_poller.py --job`): the jobs daemon's child contract (the frame, the output manifest). Exactly the frame's conversation, a cold full- history rebuild under an exclusive reservation, published under the staged label with the ordinary writer; the manifest names the staged bundle's tick and newest vote. Refused before any connection (exit 2): version skew on the frame's source_commit, a served or target label, the small poller's settings in its environment, a declared large budget above its own. The label's single-writer lock is taken once; held by another writer, the attempt fails (exit 1). No readiness reporter, no heartbeat phrase. MATH_CAPACITY_CLASS=large refuses to start: the large class is a child, not a poller. The capacity line: large_demand (queued + retry_wait) and large_leased (running) from pq_class_depth(env, large) once per readiness tick when a queue DSN is set; without one, large_demand stays the records' count and large_leased is null. The TypeScript port of the validators (server/src/ops/readinessLine.ts) learns the key, since the ops tests read the shared fixture lines. Deleted: polismath/poller/capacity_manifest.py and its tests, polismath/poller/large_class.py and its tests, the manifest restore and flush in promotion.py, MATH_CAPACITY_MANIFEST_URI (code, compose, example.env, docs), the math-python-large compose service and its volume, the AWS client lines the manifest needed on math-python, and tests/poller/test_compose_large_parity.py (it ran the deleted service; its MinIO mode with it). The delphi-large deploy-hook branches start and stop nothing now (their worker is the daemon, whose service lands with it) and still write the readiness identity. Stacks on the Delphi job table migration (000023), the polis-queue/3 migration and the daemon's class-large support; until those land the Postgres tests use the fixture stand-in from the previous commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * server: polis-queue/3 gains the poison latch and the job view's scope key (000024 resealed) Two composed-contract defects from the operating-surface map, fixed where the contract lives: - pd_enqueue: when a scope's last three jobs all died under the code image being admitted now, the reply is outcome `poisoned` naming the latest dead job and no job is made; a new image, or a succeeded or cancelled job among the last three, admits again. Without it an automated producer re-admits a permanent failure forever, one attempt budget per pass. - pd_job_view gains `scope_key` (the guard the job's root holds, null once released), so the polis-jobs daemon can release the scope through pd_release_scope after a terminal attempt whose exit it proved. The safe release function is unchanged: terminal status alone never releases. test_000024_down.sh (h) witnesses both: the job view's scope before and after a release; three deaths, a poisoned fourth admission (no job, no guard), a new image admitting again. Both migration files resealed in down/000024-files.sha256. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-jobs: the typed math config crosses the child boundary; a terminal job frees its scope Two composed-contract defects from the operating-surface map (findings 2 and 3): - The frame of a math_rebuild job carried only the Delphi config keys, so the child could not check the source commit, the target label or the declared need. The admission's config is now a typed MathConfig (staged_label, target_label, need_bytes, input_through_ms, binding, source_commit), checked whole by the daemon before any child spawns (an admission without it is math_config_invalid, permanent, exit proof honest) and carried whole in the frame (schemas/job-frame-v1.json). The fixture child asserts the six typed keys arrived through the real frame. - Nobody called pd_release_scope, so the first terminal job occupied its conversation forever. The daemon now releases the scope after a terminal reply (succeeded, dead, cancelled) for an attempt whose exit it proved, on the task, journal-recovery and reaper paths, reading the scope from pd_job_view (scope_key, polis-queue/3). The SQL re-checks every condition and refuses otherwise; one transition says scope_released or scope_held with the reason; the readiness line counts released_total. Terminal status alone releases nothing. Integration (real daemons, real chain to 000024): the rebuild's typed config reaches the child and its scope is released by the daemon, a Delphi job on /3 too; an untyped rebuild is dead before any child and the scope is free; cancel -> re-enqueue succeeds; dead frees the scope; three deaths under one image make the fourth admission poisoned (no job, no guard) and a new image admits again. 39 unit, 23 integration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * math poller: the real queue contract end to end (depth decoder, scope release, typed child config, receipt-gated promotion) The four composed-contract defects from the operating-surface map, fixed against the real polis-queue/3 (000024) instead of the test stand-in, which is deleted along with the 000023 fixture copy: the Postgres tests now read the migration chain the service image bakes. 1. The depth decoder accepts exactly the reply pq_class_depth returns (queued, leased, parked, dead, oldest_unresolved_created_at; version polis-queue/3); the capacity line's large_demand and large_leased come from it. A failing-first unit test feeds the real reply literal. 2. Scope release and the poison latch: the daemon releases a scope after a terminal attempt it proved; when an admission hands the poller a terminal job still holding its guard, the poller calls the same guarded pd_release_scope and asks once more (a refusal leaves it to the daemon). A scope whose last three jobs died under this source commit is answered `poisoned` by the contract; the poller parks the record with the reason (large_poisoned on the capacity line and the readiness capacity object; the server's TypeScript port learns the key) until a new deploy or the restage nonce. Nothing is enqueued without a source commit. 3. The child validates the whole typed math config the daemon carries (staged_label = MATH_ENV = inputs.math_env, target_label, need_bytes, input_through_ms, binding, source_commit present and equal); the frame the daemon writes is pinned to one golden file from both sides (queue-rs child::frame and the child's tests). 4. Promotion is gated by the job's receipt: pq_job_status succeeded with an output_sha256, the attempt's manifest row hashing to it and naming the job and the staged bundle's label, tick and newest vote. A bundle committed before finalize, a manifest the database refused, or a failed job is not served; without a queue nothing is promoted. Tests: poller unit 154 passed; Postgres (real chain to 000024) 33 passed, including the interruption between the staged commit and finalize, cancel -> re-enqueue, a running job's scope kept until its exit is proven, dead x3 -> poisoned -> a new commit admits; queue-rs unit 40 with the golden pin; server opsEngine/opsCloudPages 58, tsc, contract:check, lint; vote-sign lint 0 new; test_gate 32. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * math poller: the composed chain proof script (real daemon, real child, one database) delphi/tests/poller/chain_proof.py, not collected by pytest: runs the real polis-jobs binary as a worker of class large, the real scripts/math_poller.py --job as its child and the small poller's router, queue client and promotion loop in-process against one Postgres holding the real chain to 000024, with one generated conversation sized to exceed a small budget. Asserts and prints every step: route, enqueue, claim, stage, finalize, release, receipt-gated promotion, served; then cancel of a queued job, the daemon stopped mid-job, the child killed, cancel of a running job, three deaths making the scope poisoned (parked, counted, not re-asked) and a new source commit admitting again. Removes what it wrote. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s * polis-jobs: module list in rustfmt order (cargo fmt --check) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
#2973 removed the math-large-state volume's name line but left its labels block, so math-backfill-state carried two labels keys and docker compose refuses the file ("mapping key \"labels\" already defined"). Every compose command that reads docker-compose.yml fails on edge, including the deploy's docker-compose up. Drop the stray two lines. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) server/postgres/bin/apply-migration.sh refused 000024 ("has no apply policy here"), so an existing database had no checked path from polis-queue/2 to /3. The wrapper now accepts 000024: - seal: down/000024-files.sha256 - rights: superuser or SET membership in polis_queue_owner (as 000023) - chain: 000023 installed and polis_queue_install reads polis-queue/2 (a second run is refused here, since the install then reads /3) - rows: every queue and job table is empty (000024 alters the queue tables' CHECK constraints, ACCESS EXCLUSIVE until COMMIT) - lock line: names the four queue tables; no lock on conversations - post-check: polis_queue_install reads polis-queue/3 The usage text now prints the whole header (it was cut at line 60). test_000024_down.sh gains check (w): refused without 000023 and with a queued job (catalog unchanged both times); --preflight-only passes and changes nothing; the apply exits 0 with post-check /3 and the same schema dump as a plain psql apply; a second run is refused at preflight. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…t) (#2977) The large class's scale-in alarm (cdk/workerClasses.ts, #2974) sums large_demand + large_leased + large_parked from the small poller's capacity line and fires only when every term is reported. The line had no large_parked, so the group could never scale in. - capacity.py: large_parked is a closed count, the queue's pq_class_depth `parked` for class large (the depth decoder already accepts it), null with no queue read this tick, like large_leased. - server/src/ops/readinessLine.ts (the TypeScript port the ops page parses with): the same key, nullable; prettier on the file. - The generated ops fixture lines are regenerated. - Tests: the count comes from the depth; an idle queue puts all three scale-in terms on the line as integer 0 and one parked job holds the sum at 1; null without a queue or after a failed read; a line without large_parked is refused (Python and TypeScript). Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…service (#2980) A queue worker box (service types delphi-large and delphi-worker from the worker launch templates) runs one thing, the polis-jobs daemon of its class as polis-jobs.service. after_install.sh had no delphi-worker branch (the unknown-type catch-all would start every service) and its delphi-large branch built no image for the unit to run. - after_install.sh: worker_daemon <class> checks /etc/app-info/worker_class.txt against the service type and that the unit is installed (else the deploy fails), builds the Delphi image (which carries the daemon) and queues `systemctl restart --no-block polis-jobs.service`; no compose service starts. delphi-large keeps writing the readiness identity. On those two types only, the cleanup spares the polis-jobs container so the unit's stop can drain its job. - application_stop.sh: both types stop nothing (the drain can outlast the hook; AfterInstall restarts the unit). - delphi/tests/test_after_install_hook.py runs the real hooks with every external command faked: server, delphi, math and an unknown type behave as before (the same cases pass against the previous hooks); the worker types build, restart only polis-jobs.service, and refuse a mismatched class or a missing unit. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…age witness (#2979) polis-jobs.service on a worker box (cdk/launchTemplates.ts) runs `docker run ... "$POLIS_JOBS_IMAGE" polis-jobs` against the Delphi image, which did not contain the binary. Now it does: - delphi/Dockerfile: a stage builds polis-jobs from the named build context `queue-rs` (rust 1.98.1, Debian 12 so the binary also runs on an older cached python:3.12-slim) and the final stage copies it to /usr/local/bin/polis-jobs. The image's CMD is unchanged; nothing but the unit runs the binary, and it exits 0 unless POLIS_JOBS_ENABLED=1. - docker-compose.yml, docker-compose.test.yml: additional_contexts queue-rs for every build of the Delphi image; ci/p027_rerecord_build.sh and delphi/Makefile pass --build-context. - polis-jobs --version prints "polis-jobs <crate version>" and exits 0. - queue-rs/tests/polis_jobs_cli.rs: --version, the disabled exit, and a missing password file refusing with exit 2. - queue-rs/image-witness.sh: runs the binary from a built image the way the unit does, against a throwaway postgres:17: --version; disabled 0; missing password file 2; a login that is not a plain executor member 2; class large on polis-queue/2 3; class delphi without the queue 3; an absent login role keeps retrying (no exit); class large on /3 starts and exits 0 on docker stop. Also carries the one-block docker-compose.yml fix from the compose labels PR so this branch builds. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Production waited the full 5 s translation timeout on every vote: Google Translate was refusing on quota and failures were not remembered, so each request (including English to English) paid the whole wait. - never call the provider when the statement is already in the requested language - remember a failed translation per (zid, tid, lang) for ten minutes - nextComment and participationInit stop waiting on translation after 300 ms; the translation finishes in the background and is stored for the next request Tests: a hanging-provider regression (13 fail before, pass after) plus the targeted server translation tests and the 251 vote-path recordings. Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tack flags off) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observable changes:
Proven on mm5 against a database at production's current schema (no queue tables): 304/304 targeted server tests incl. 251 vote-path checks, 96/96 report views, both pollers healthy, zero queue-table statements.
The branch is stable merged with edge; its tree equals edge. Merge with a merge commit.
🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s