server: the database declares its vote convention; a server refuses to start against a convention it was not built for

[colinmegill]

Observable changes:

Corrections from the connections map

The job-system connections map found that this PR's apply wrapper (000019, 000023, 000024, 000025) and #2978's (000019, 000023, 000024, 000026) each lacked the other's migration, and that 000026 had no ledger row, so this PR's 000025 down could not see it. Resolved without changing this branch:

• Where the combined wrapper lands: DRAFT — schema change — server: migration 000026, queue retention and the parked read (P-083, P-086) #2978. DRAFT — schema change — server: migration 000026, queue retention and the parked read (P-083, P-086) #2978 is now built on this branch (merged, not rebased) and merges after it. It carries this PR's 000025 entry unchanged and adds 000026's, so the two PRs do not conflict: this PR keeps the 000019/000023/000024/000025 wrapper, and DRAFT — schema change — server: migration 000026, queue retention and the parked read (P-083, P-086) #2978's version is a strict extension of it.

• Down order. 000026 now inserts its own schema_migrations row (self-checksum, the rule this PR introduced). This PR's down/000025_drop_vote_convention.sql already refuses while the ledger records a later migration, so over 000026 it refuses and changes nothing; 000026's down runs first and removes its row, then 000025's down runs. Proven in DRAFT — schema change — server: migration 000026, queue retention and the parked read (P-083, P-086) #2978's test_000026_down.sh (l), (i) and (p) on mm2.

• Census. This PR's census (000000..000025, 96 relations, 697 columns) stays as is; DRAFT — schema change — server: migration 000026, queue retention and the parked read (P-083, P-086) #2978 re-measures the combined chain.

• No file in this PR changed; this branch already contains edge 7987554ff.

• pol.is (our deployment): none at runtime once the row exists. The deploy itself writes nothing: production holds votes, so migration 000025 creates the table and leaves it empty, printing DECLARE_NEEDED; the owner then runs the declare command once (one row, one transaction, no vote touched). Both are the owner's explicit steps, before the images of this release are deployed: until the row exists, the new server, import worker, math poller, Delphi job poller, the Delphi job itself (before its reset) and the coordinator refuse to start (exit 1; the poller exit 2) with the messages below, while the running images never read the row and keep working. The checks are made at startup (and at the job boundary); a process already running when the row changes is not stopped by this release. After the declaration each component logs vote convention: version 0, agree stored as -1 at startup and nothing else changes: no vote value, no read or write path, no export byte, no API response. The vote-path recordings replay byte-identical (no expected-difference entry).

• Other deployments (and any dev or prodclone volume that already holds votes): on upgrade, applying the migration prints DECLARE_NEEDED, and the new containers refuse to start until the operator runs make vote-convention-declare AGREE=-1 once; old containers keep working meanwhile. A fresh install needs nothing (the migration seeds version 0). A fork that reversed its own signs can declare AGREE=+1, which records that fact; this release does not serve a +1 database (its components refuse it rather than read it inverted). Nothing else.

• New operator commands: make vote-convention-status, make vote-convention-declare AGREE=-1|+1 [REASON=...] (server/bin/vote-convention-declare.sh, server/postgres/operations/vote_convention_declare.sql); the apply of 000025 to an existing database goes through server/postgres/bin/apply-migration.sh (the wrapper that already applies 000019 and 000023). New guide docs/vote-convention-upgrade.md; CHANGELOG, README, docs/upgrading.md, docs/migrations.md, docs/vote-convention.md updated.

• A Delphi job run against an undeclared database now stops before its reset stage, so a conversation's previous results are kept; before this PR the reset ran first. The commentgraph CLI's test-postgres exits 1 (not 0) when the database refuses.

• The server waits for Postgres at startup. The startup convention check queries the database. If Postgres is not answering yet, the server retries every 2 s instead of exiting. A normal start costs one query and no delay; a cold start waits up to a minute. Without the wait, the server exited during a cold start, came back at a new address, and nginx answered 502 until it was restarted. A refusal of the convention itself (no table, no row, the other sign) still exits at once.

• The large-class rebuild job checks the convention too. The rebuild job (math_poller --job, from math poller: oversized conversations are jobs on the Postgres queue (P-073 r2) #2973) makes the same check before it reads a vote. A refusal exits 2.

• The characterization harness's seed no longer truncates vote_convention, vote_convention_history and schema_migrations (they refuse TRUNCATE and are not application data).

Renumbered to 000025

The draft was 000023_vote_convention.sql. Edge has since merged the Delphi job table as 000023_create_delphi_foundation.sql (#2968) and the large worker class as 000024_create_polis_queue_large_class.sql (#2971), so this migration, its reversal, its down test and its compose file are 000025. Every reference, pin and both ledger self-checksums were regenerated: migration ledger checksum 3c39aadd679d9c5aa4f4d5c1a42cf39cd0b15659cb1112ccdad4d8f438c10047, declare operation 60e17ec16e1e035c1645575978f1028c0f4f08c253880f6c51222f3377ece20c. The two files are sealed in server/postgres/migrations/down/000025-files.sha256, and the wrapper checks that seal before it sends anything.

• Chain. The chain before this file is 000000..000024 (there is no 000020). 000025 needs none of 000019, 000021, 000023 or 000024. It records each of them in the ledger like every other earlier file (below).
• Ledger rule. "Every migration file inserts its own row" starts at 000025. 000023 and 000024 predate the rule.
• Queue integration test. queue-rs's jobs_integration chain assertion requires 000000..000024 as a prefix and accepts the files after it.
• Schema census. The coordinator's census was measured again on PostgreSQL 17.11 with the whole chain applied to a fresh database. It now carries the Delphi job tables (000023) and the large-class install table and widened checks (000024), neither of which had a census. The census test enumerates the migrations directory, so it has been red on edge wherever it runs since server: the Delphi job table (foundation migration, flag off; DynamoDB keeps running every job) #2968. It also pins 000025. Totals: 96 relations, 697 columns.
• Apply wrapper. server/postgres/bin/apply-migration.sh is extended for 000025. Edge's apply wrapper: 000024 (polis-queue/3) gets its seal and preflight #2975 gave it 000024, and the two are merged.

Corrections from the operating-surface map

The second reviewer's map of this PR's operating surface (review round 2) ranked what had to change inside the PR before pol.is could take it. Each item, and what changed:

1. The ledger told an assumed story. The backfilled rows said pre-ledger, "assumed applied", and a database that had never run 000019 still got a 000019 row. Now 000025_vote_convention.sql probes the catalog for every earlier file (000000..000024) and records what it found: checksum verified (the file's signature object is present) or unverified (it is not), the probe named in the note, plus a NOTICE with the counts (server/postgres/migrations/000025_vote_convention.sql, the ledger section). The migration's own prerequisites are the vote tables from 000000 and 000006 only; without them it refuses P0780 and creates nothing. The down file refuses on ledger-era rows (64-hex checksums) rather than on the backfill (down/000025_drop_vote_convention.sql). Test: test_000025_down.py cases 16 (full chain: every earlier file verified), 17 (a copy shaped like pol.is, without the dormant 000019/000021/000023/000024: those four rows unverified with their probe, 000025 applies, declares and reverses without them) and 18 (no vote tables: P0780, nothing created).
2. The guide said the migration takes no lock on the vote tables. It takes AccessShareLock on votes and votes_latest_unique until COMMIT (its empty-table checks): compatible with ordinary traffic, and behind an ACCESS EXCLUSIVE holder it waits its 5 s per-lock budget and rolls back. The migration header, docs/vote-convention-upgrade.md#guard and docs/vote-convention.md say exactly that; the wrapper prints it; the down test proves it (case 19: an ACCESS EXCLUSIVE holder on votes makes the apply exit 5 after the budget with the catalog unchanged).
3. The apply goes through the wrapper. server/postgres/bin/apply-migration.sh (server: the Delphi job table (foundation migration, flag off; DynamoDB keeps running every job) #2968) covers 000025: preflight seal (down/000025-files.sha256), server (PostgreSQL 17), rights (the login owns the vote tables and can CREATE in public: vote_insert and the views run with the owner's rights), chain (the vote tables exist; nothing later required), rows (none of the thirteen objects exists yet: a first install, not a repair), xacts, disk; then the budgets (lock_timeout 5 s, statement_timeout 60 s, transaction_timeout 120 s, idle_in_transaction_session_timeout 30 s) ahead of the file; after it, the post-check (the ledger row with a 64-hex checksum) and the convention state (DECLARE_NEEDED with the next command, or GUARDED) with the verified/unverified counts. Exit codes 0/2/3/4/5/6 as for the queue files. The runbook, the guide and docs/migrations.md route the apply through it. Test: case 19 (--preflight-only applies nothing; an old transaction refuses at exit 4; the lock holder; the apply; a second run refused at exit 4; the seal verifies).
4. A Delphi job reset ran before the check. run_delphi.py ran reset_conversation.py (which removes a conversation's previous results) before its first guarded stage, so a job launched directly, or by an older poller that never checked at boot, could erase results and then print "Nothing has been changed". Now delphi/polismath/check_vote_convention.py is the job's first stage, before the reset (delphi/run_delphi.py); a refusal exits 1 with the operator message and Nothing has been reset or changed. Tests: delphi/tests/test_run_delphi_exit_codes.py::test_undeclared_convention_refuses_before_the_reset (the stage order: only convention ran), test_check_vote_convention.py (the three refusals and the pass, through a fake client), and the job-child protocol tests carry the new first stage.
5. The commentgraph CLI logged the refusal and exited 0. test-postgres now prints the operator message to stderr and exits 1 (delphi/umap_narrative/polismath_commentgraph/cli.py); only that command is claimed as gated at the CLI (lambda-local is gated where it reads votes, in the UMAP stage's fetch_conversation_data). Test: delphi/tests/test_commentgraph_cli_refusal.py.
6. Wording reconciled with the plan. The guide now says: the checks are made at startup (and at the job boundary), a running process is not stopped by a later change of the row; the server checks DATABASE_URL and a differing READ_ONLY_DATABASE_URL, the import worker and the Python processes check the one database they read (this body's earlier "both URLs" for the worker was wrong); a fork that declares +1 is recorded, not served, by this release. The plan carries the same table.

Not done here, on purpose (they belong to Colin's deploy step or to later PRs, as the map says): the deploy-day execution itself, +1 serving, tombstones and the write fence, the backup format, the flip rehearsal, the runner rewrite.

Rehearsal on composed images

This was run on a build box with images built from this PR's head (57bd51aa8): the server image and the Delphi image. The pre-release server image was built from edge 288163928.

The database was PostgreSQL 17.11 with the chain 000000..000024 applied and a generated fixture loaded: 5 participants, 5 statements and 25 votes stored in the original convention. Each component was run as its stock command would run it, against that database, and given 60 s:

• a the server
• b the import worker
• c the math poller (--once)
• d the Delphi job poller (the image's stock command)
• e a Delphi job (run_delphi.py --zid=501)
• f the commentgraph CLI (test-postgres)

The box has no AWS credentials or MinIO, so (b), (d) and (e) fail later on those once they are past the check. Before every step and after it, I took a pg_dump --schema-only (pg_dump's random \restrict lines set aside) and the sha256 of every vote row.

1. Images first: the release images against the database before 000025. Every component refused, and nothing changed.

a server              exit 1   Polis cannot start (server): this database has no vote_convention table, ...
b import worker       exit 1   Polis cannot start (import worker): this database has no vote_convention table, ...
c math poller         exit 2   Polis cannot start (math poller): this database has no vote_convention table, ...
d Delphi job poller   exit 1   Polis cannot start (Delphi job poller): this database has no vote_convention table, ...
e Delphi job          exit 1   Polis cannot start (Delphi job): this database has no vote_convention table, ...
                               Vote convention check failed with exit code 1. Nothing has been reset or changed. Aborting pipeline.
f commentgraph CLI    exit 1   Polis cannot start (commentgraph CLI): this database has no vote_convention table, ...
schema after: byte-identical to before; votes: identical (25 rows, sha256 b1a8a1bf…; votes_latest_unique 25 rows, sha256 eed32a2c…)

The full message is:

Polis cannot start (math poller): this database has no vote_convention table, so it does not record which stored vote value means "agree". Migration 000025 has not been applied. Nothing has been changed. Next: back up the database, apply server/postgres/migrations/000025_vote_convention.sql (see docs/migrations.md), then, if the database already holds votes, run "make vote-convention-declare AGREE=-1". Guide: docs/vote-convention-upgrade.md#guard

No "Resetting" line appears in (e): the reset never ran.

2. The migration through the wrapper, then the declaration, then the images.

$ server/postgres/bin/apply-migration.sh --free-bytes 129064505344 000025 -- docker exec -i <pg> psql -U postgres -d polis-dev
ok    seal: cfff57e4f416744d90d1beddc6daa48f730a6c0538cc9400a84a19bafc60c812
ok    server: PostgreSQL 17.11 (Debian 17.11-1.pgdg13+2)
ok    rights: superuser
ok    chain: the vote tables exist (000000, 000006); nothing later is required
ok    rows: no vote convention object exists yet (a first install)
ok    xacts: no other transaction older than 30s on polis-dev
ok    disk: 129064505344 free bytes reported, floor 5368709120 (database is 12 MB)
budgets: lock_timeout=5s statement_timeout=60s transaction_timeout=120s idle_in_transaction_session_timeout=30s
note: 000025 sets lock_timeout to 5s inside its transaction; that is its acquisition budget
lock: this file holds AccessShareLock on public.votes and public.votes_latest_unique until COMMIT (its empty-table checks); no vote table is rewritten
NOTICE:  vote convention: DECLARE_NEEDED. This database holds votes and records no sign. Polis components will not start until you declare it. Next: "make vote-convention-declare AGREE=-1" (the original convention) or AGREE=+1 only if your deployment reversed its vote signs itself. Guide: docs/vote-convention-upgrade.md#declare
NOTICE:  migration ledger: 24 earlier files verified in the catalog; unverified: none
applied: 000025_vote_convention.sql (post-check 1)
convention: DECLARE_NEEDED (earlier files in the ledger: 24 verified, 0 unverified)
next: the database holds votes; declare its sign once with server/bin/vote-convention-declare.sh (the command and the sign: docs/vote-convention-upgrade.md#declare)
wrapper exit: 0

Between the apply and the declaration, all six components refused again with the same exit codes. The message is:

Polis cannot start (server): this database does not record which stored vote value means "agree". Older Polis databases store agree as -1; this release no longer assumes it. Nothing has been changed. Next: back up the database, then declare its convention with "make vote-convention-declare AGREE=-1" (AGREE=+1 only if your deployment reversed its vote signs itself). Guide: docs/vote-convention-upgrade.md#declare

Then the declaration, a check, and the images:

$ server/bin/vote-convention-declare.sh -1 "rehearsal: the original convention"
GUARDED v0 agree -1 (contract 1; written by vote_convention_declare as postgres: rehearsal: the original convention)
declare exit: 0
vote_convention_current(): 0 -1 1
vote_insert dry run, rolled back (agree, semantic +1): stored -1, convention_version 0

a server              running   vote convention: version 0, agree stored as -1 / Server started on port 5000 / GET /api/v3/testConnection -> HTTP 200
b import worker       running   [Worker] vote convention: version 0, agree stored as -1   (then: SQS ... Could not load credentials: no AWS on the box)
c math poller         exit 0    math_poller: vote convention: version 0, agree stored as -1
d Delphi job poller   past the check: delphi_poller - INFO - vote convention: version 0, agree stored as -1   (then exits 1: no DynamoDB credentials on the box)
e Delphi job          past the check: vote convention: version 0, agree stored as -1 / Resetting all existing data for conversation 501 ...   (then exits 1: no DynamoDB credentials)
f commentgraph CLI    exit 0    Vote convention declared: version 0, agree stored as -1
votes after the apply, the declaration and every start: identical to before

3. The down path. The release containers were all gone first (0 left besides postgres).

$ psql -X -v ON_ERROR_STOP=1 -f server/postgres/migrations/down/000025_drop_vote_convention.sql      (declared (0, -1), one history row)
down exit: 0
schema after: byte-identical to the dump taken before the apply; votes identical
$ (the same file again)
NOTICE:  function public.vote_convention_record_history() does not exist, skipping
NOTICE:  table "schema_migrations" does not exist, skipping
down (second run) exit: 0
pre-release server image (edge 288163928): running; Server started on port 5000; GET /api/v3/testConnection -> HTTP 200
release server image: exit 1, Polis cannot start (server): this database has no vote_convention table, ...
cleanup: 0 containers, 0 networks left

The server's vote convention: and Server started lines are info level. The stock image logs at warn, so these runs set SERVER_LOG_LEVEL=info.

What this is

Migration 000025_vote_convention.sql: one table with at most one row naming the stored vote sign (agree_value; today -1, "version 0"), its append-only history, the migration ledger, vote_convention_current() ((version, agree_value, contract_version)), the pure conversions, vote_insert() and the two semantic views, as drafted, amended to plan P-078 for other deployments:

	draft	this PR
who writes the row	the migration, always	the migration only when both vote tables are empty; otherwise the operator, once, with the declare command
a database holding votes	silently seeded -1	left undeclared; NOTICE: vote convention: DECLARE_NEEDED ...; nothing reads a sign from the data
components without the row	assumed -1 (engine), constant (server)	refuse to start, naming the exact command
a row naming the other sign	followed by the engine; ignored by the server	refused by every component built for -1 ("would read and write every vote inverted")
contract	CHECK (contract_version = 1)	also carried by vote_convention_current(); an unknown contract is refused
first states	version 0 only	(0, -1) seeded or declared, or a declared (1, +1); then exactly version+1 with a sign change
history	state rows	plus operation and operation_checksum (which file wrote it)
ledger	assumed pre-ledger rows	every earlier file verified / unverified from the catalog
down file	refuses on no row	reverts an undeclared database (nothing is lost); still refuses once the sign moved
restore rule	no row = corrupt	no row = undeclared; a first-state +1 declaration = declared-agree-plus

Where the refusal lives: server/index.ts and src/workers/start-import-worker.ts (through server/src/votes/dbConvention.ts; the server on DATABASE_URL and a differing READ_ONLY_DATABASE_URL, the worker on DATABASE_URL); delphi/scripts/math_poller.py, delphi/polismath/run_math_pipeline.py, delphi/run_delphi.py (first stage, polismath/check_vote_convention.py), the UMAP stage (run_pipeline.fetch_conversation_data through storage.PostgresClient.require_declared_convention), the commentgraph CLI's test-postgres and delphi/scripts/job_poller.py (through delphi/polismath/utils/vote_convention_boot.py); coordinator-rs/src/vote_convention.rs at startup for run/once and inside every source snapshot transaction. math/ (Clojure) is retired and untouched. The sign each build is built for is its chokepoint's STORAGE_AGREE_VALUE; no literal sign is written in the checks.

Decisions taken as recommended

Each C-question the plan leaves to the owner, with the plan's recommendation taken; veto any line.

• C1 (R-A amended: the row, the empty-only seed, the explicit declaration, history operation columns, the contract in the read function, every component refusing, the verified ledger): yes, this PR, scoped: the handshake's structural layout check, the transition assertion (P0792), vote_import_batch(), the fence-marker SET clauses and the tombstone/fence functions are not in this PR (they belong with the flip tool and the server surface PRs).
• C2 (the flip's read pause under ACCESS EXCLUSIVE, once measured): yes for the plan; nothing in this PR flips anything.
• C3 (the write fence installed at the flip): yes for the plan; not in this PR.
• C4 (the weekly backup's format polis-data-backup/2): yes for the plan; nothing in AWS changes here.
• C5 (role separation behind the sanctioned functions): not now; documented as optional hardening only.
• C6 (the runner validates first, stops on the first error, skips applied files, refuses newer databases): yes for the plan; run-migrations.sh is unchanged in this PR; an existing database takes 000025 through the apply wrapper instead.
• C7 (where an undeclared import is recorded): the column on byod_import_jobs, in a later schema PR; nothing here.
• C8 (drop the mandatory re-flip of a restored version-0 copy): yes; the docs here say a restored v0 copy is served as is.
• C9 (wording of the operator messages): as written, adapted to what exists in this PR (no preflight yet; the declare command is named; exact texts below).

Taken under the delegation for D-level choices, beyond the plan's list: AGREE=+1 writes (1, +1) as a first state without the renames, tombstones and fence (those are the flip tool's), so this release's components refuse such a database rather than read it inverted; the engine's per-cycle "absent row means version 0" fallback in RowConventionSource is unchanged (plan PR 4), the refusal being at startup; the migration, its reversal, the declare operation and the down test are registered as the database's own chokepoint in the vote-sign lint (0 new literals, no allowlist entry); the number is 000025 (above).

The three upgrade paths, proven earlier on a build box with real databases

The built server (node dist/index.js) and the math poller (scripts/math_poller.py --once) against PostgreSQL 17.11 containers (recorded at the previous head; the messages are unchanged except the migration number).

(a) A fresh database. The migrations run at container init; the row is seeded; both start.

NOTICE:  vote convention: GUARDED v0 agree -1 (seeded: this database held no votes)
row: version 0, agree -1, operation seed-empty
server:  vote convention: version 0, agree stored as -1
server:  Server started on port 55480
poller:  vote convention: version 0, agree stored as -1      (exit 0)

(b) A database with existing votes and no row (another deployment upgrading): the chain applied, one vote inserted, then 000025.

NOTICE:  vote convention: DECLARE_NEEDED. This database holds votes and records no sign. Polis components will not start until you declare it. Next: "make vote-convention-declare AGREE=-1" (the original convention) or AGREE=+1 only if your deployment reversed its vote signs itself. Guide: docs/vote-convention-upgrade.md#declare
rows in vote_convention: 0

The server, exit code 1:

Polis cannot start (server): this database does not record which stored vote value means "agree". Older Polis databases store agree as -1; this release no longer assumes it. Nothing has been changed. Next: back up the database, then declare its convention with "make vote-convention-declare AGREE=-1" (AGREE=+1 only if your deployment reversed its vote signs itself). Guide: docs/vote-convention-upgrade.md#declare

The math poller, exit code 2, the same text with (math poller). Then server/bin/vote-convention-declare.sh --status:

DECLARE_NEEDED: this database records no vote convention. Polis components will not start until you declare it. Next: "make vote-convention-declare AGREE=-1" (the original convention) or AGREE=+1 only if your deployment reversed its vote signs itself. Guide: docs/vote-convention-upgrade.md#declare

The declaration (server/bin/vote-convention-declare.sh -1):

NOTICE:  vote convention: GUARDED v0 agree -1 (declared by postgres)
GUARDED v0 agree -1 (contract 1; written by vote_convention_declare as postgres: declared by the operator: the original Polis convention (agree stored as -1))

Then the server starts (vote convention: version 0, agree stored as -1 / Server started on port 55483), the poller's --once exits 0, and the stored vote is untouched (-1).

(c) A row naming version 1 (agree +1) with a server built for version 0 (declared with AGREE=+1). The server, exit code 1:

Polis cannot start (server): this database declares vote convention version 1 (agree = +1), but this release of the server is built for agree = -1. Running it would read and write every vote inverted. Nothing has been changed. Next: run a Polis release built for the declared convention, or restore the database this release was built for. Guide: docs/vote-convention-upgrade.md#mismatch

The math poller, exit code 2, the same text with (math poller). (The coordinator's decision is the same function, unit-tested; its live run needs a TLS database and was not exercised.)

The message when the migration is not applied at all: Polis cannot start (server): this database has no vote_convention table, so it does not record which stored vote value means "agree". Migration 000025 has not been applied. Nothing has been changed. Next: back up the database, apply server/postgres/migrations/000025_vote_convention.sql (see docs/migrations.md), then, if the database already holds votes, run "make vote-convention-declare AGREE=-1". Guide: docs/vote-convention-upgrade.md#guard

Test evidence

Since the run below:

• b38925a5f names the seven Rust tests in coordinator-rs/tests/vote_convention.rs in the coordinator campaign's inventory.

• 1a1404255 adds the startup wait above. I checked it on the build box with the e2e compose stack: postgres was held back 20 s; the server restarted 0 times; nginx answered 200 for / and for /api/v3/participationInit. One forced server restart on the previous head reproduces the 502.

• d47cbe2c3 merges edge (math poller: oversized conversations are jobs on the Postgres queue (P-073 r2) #2973, cdk: one worker group per queue class; demand and alarms read from the queue; manifest remnants removed #2974). It had one conflict in delphi/scripts/math_poller.py; the check stays in _build_service. Delphi pytest after the merge: 3851 passed, 0 failed.

• 5f99152c8 makes the campaign tests go through the convention row and adds the check to the rebuild job. On the build box at this head:

  • Delphi pytest: 3855 passed, 0 failed, including the rebuild job's new test.
  • Server jest, the CI way: vote-path recordings byte-identical. 4 tests failed in that run (conversation-stats "until", 3 in trashes); I re-ran those two suites alone and they passed 6/6.
  • Two-convention gate: v0 green, v1 red as pinned, ratchet 0 failures.
  • sign_lint: 0 new literals.
  • Coordinator campaign: cargo and clippy pass; pytest 290 of 291 (the edge failure below).

• 83ebe3967 merges edge again (apply wrapper: 000024 (polis-queue/3) gets its seal and preflight #2975 gave the apply wrapper a policy for 000024, plus math poller: the capacity line reports large_parked (scale-in reads it) #2977, delphi image: carries the polis-jobs daemon (queue-rs), --version, image witness #2979 and deploy hooks: delphi-large and delphi-worker restart only polis-jobs.service #2980). One conflict, in server/postgres/bin/apply-migration.sh: the wrapper now covers 000019, 000023, 000024 and 000025, each with its own policy. After the merge, on the build box:

  • test_000024_down.sh: all checks pass, including its wrapper case.
  • test_000025_down.sh: 24 passed, 0 failed, 1 skipped.
  • The chain 000019 → 000023 → 000024 → 000025 and back is byte-identical at every step.

Coordinator campaign: the sign changes through the row. The ruling is that the declared row is the only truth, and STORAGE_AGREE_VALUE never overrides it. Seven campaign tests used to reverse the votes with a raw UPDATE and launch the coordinator at +1 while the row still said (0, -1). They now go through the row:

• The polarity tests (three fixtures), the rebuild schedule and the tie key reverse the votes and move the row to the next version with +1. That is the transition the row's triggers admit, and it is recorded in the history. Then they run at +1.
• They also show that a -1 build refuses that database (#mismatch).
• Their "negative" control is now the reversed votes under the same declaration. Before, it was the same votes read under the wrong sign.
• The node reader's deliberately different third namespace is now the same conversation with every vote reversed, with the row unchanged.

No witness needed re-pinning. The reversed votes read under the right sign are byte for byte what the wrong-sign read produced, so every polarity witness on the build box (macOS arm64) equals its pin:

witness	a = b	negative
polarity-public-fixture	0b5f690d35ba…	4fa3b33d193d…
polarity-vw	523b48455a31…	23c7933c7b10…
polarity-biodiversity	50a50a0b1c11…	6fbd8cf04653…
semantic-tie-key	c40f36ef591f…	aec5181c022a…
polarity-rebuild-schedule	(per step)	01a9fdf049f7…, e2b674af0ba3…, 686486e4e1b1…

On the build box the campaign passed 290 of 291 cases. The one failure, test_step2_review_controls.py::test_actual_pca2_route_serves_generation_zero, also fails on edge itself: it expects ETag "0", and since 7d452d41c the route sends "rustproto-0". It is not fixed here, so "Coordinator S1/S2 required" stays red until edge fixes it.

All of these were run on the build box at head 57bd51aa8, with the same entry points the workflows use. Hosted CI is the second witness.

• Chain with the real files, forward and back. PostgreSQL 17.11, populated (25 votes), starting from 000000..000018, 000021 and 000022.
  • Forward: 000019 (wrapper), 000023 (wrapper), 000024 (psql -f; the wrapper has no policy for it), 000025 (wrapper: 24 verified, DECLARE_NEEDED), then the declaration.
  • Back: 000025's down file, 000024's, 000023's and 000019's. After each one, the schema dump is byte-identical to the dump taken before the matching up.
  • Forward again: 000025's schema is byte-identical to the first time.
  • The vote rows are unchanged throughout.
• Migration test server/postgres/migrations/down/test_000025_down.sh (PostgreSQL 17.11, the real chain 000000..000024 then 000025): 24 passed, 0 failed, 1 skipped (the held un-flip file is not in the tree). It covers:
  • the verified ledger on the full chain;
  • a pol.is-shaped copy without the dormant 000019/000021/000023/000024;
  • the prerequisite refusal;
  • the wrapper: an old transaction exits 4; an ACCESS EXCLUSIVE holder on votes rolls the apply back on its 5 s budget (exit 5, catalog unchanged); a second run exits 4; the seal verifies;
  • up/down/up and the down refusals.
• Server jest, the CI way (compose test stack, node 24, npm test -- --ci --coverage --maxWorkers=2): 1723 passed, 1 skipped, 0 failed.
  • An earlier run on the same head had one failure, conversation-stats "should accept until parameter". That case compares the host clock with database timestamps, and the same case failed once on the previous head too.
  • The vote-path recordings replay 249 cases byte-identical, with no expected-difference entry.
  • npm run contract:check and npm run lint are clean.
• Golden guards against origin/edge (vote path, collective statement, routing, job views): ok.
• Collective-statement replay: red by construction. I did not route around it.
  • main.cjs replay → postgres schema the recordings read changed: public.vote_convention_current(): absent at record time, now defined. The other 51 cases replayed, and its node tests pass.
  • The harness pins the one function the shared vote writer reads, and this migration defines that function.
  • expected-differences.json admits only per-case text entries, and the golden guard forbids re-recording together with server/postgres/ changes.
  • It needs a named schema-pin entry in the harness, or a re-record on edge once this merges. That is a finding for the harness, from edge.
• Delphi characterization: the codecs pass, generate.py --check passes, and main.cjs replay reproduces 272 cases, every recording byte for byte.
• Delphi, the CI way (the docker test stack, pytest over delphi/tests): 3902 passed, 250 skipped, 44 xfailed, 2 xpassed, 0 failed.
• Two-convention gate (ci/vote_convention/run.sh):
  • v0 green (336 pca2, 390 exports, 41 math, 65 folds).
  • v1 red as pinned (exit 3).
  • Ratchet: 0 failures.
• Vote-sign checks:
  • sign_lint.py: 0 new, no allowlist entry. Its 30 stale entries are pre-existing.
  • test_gate.py: 32 OK.
  • check_ledger_checksums.py: ok for the migration and the operation.
• Rust crates:
  • Coordinator: cargo clippy --all-targets -- -D warnings and cargo test --locked (every suite, including the schema census at 96/697) are clean.
  • Queue crate: clippy, unit tests and the 23 jobs_integration tests against PostgreSQL 17 pass.

Applying this migration to production is a separate, explicit step by the owner

Runbook in docs/vote-convention.md (production runbook): check the checksums and the seal, apply the file through the wrapper as the owner role (prints DECLARE_NEEDED), run the declare command once (prints GUARDED v0 agree -1), verify through the runtime connections, then deploy this release. Rollback: the down file, after stopping the components that read the row; the old images never needed it. The deploy-day order is migration → declare → images; the images refuse until the row exists, and nothing in the deploy hooks enforces the order, so the hold before any container stop is the operator's checklist.

Not in this PR

Any flip; the rehearsal of the flip (#2942); the read-only preflight and the author's-vote witness (plan PR 2); the runner hardening (PR 2); the server and engine surfaces over the views and vote_insert() (PRs 3–5); the handshake's structural check, the transition assertion, the write fence and the tombstones (PR 9); the backup format (PR 8); a continuous (per-cycle) check in any component.

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

[github-actions]

Delphi Coverage Report

File	Stmts	Miss	Cover
init.py	2	0	100%
benchmarks/bench_pca.py	128	107	16%
benchmarks/bench_repness.py	81	65	20%
benchmarks/bench_update_votes.py	38	28	26%
benchmarks/benchmark_utils.py	34	0	100%
check_vote_convention.py	33	8	76%
components/init.py	1	0	100%
components/config.py	165	70	58%
conversation/init.py	2	0	100%
conversation/conversation.py	1291	264	80%
conversation/group_vote_counts.py	38	2	95%
conversation/manager.py	131	42	68%
database/init.py	1	0	100%
database/dynamodb.py	395	178	55%
database/postgres.py	414	95	77%
delphi_storage/init.py	1	0	100%
delphi_storage/codec.py	295	23	92%
delphi_storage/golden_corpus.py	60	9	85%
empty_output.py	29	2	93%
engine_adapter.py	317	265	16%
job_child/init.py	325	49	85%
job_child/census.py	101	13	87%
pca_kmeans_rep/init.py	5	0	100%
pca_kmeans_rep/clusters.py	257	38	85%
pca_kmeans_rep/corr.py	98	17	83%
pca_kmeans_rep/group_k_smoother.py	28	0	100%
pca_kmeans_rep/legacy_kmeans.py	197	7	96%
pca_kmeans_rep/pca.py	162	19	88%
pca_kmeans_rep/repness.py	226	5	98%
poller/init.py	4	0	100%
poller/admission.py	318	15	95%
poller/backfill.py	936	32	97%
poller/capacity.py	485	27	94%
poller/capacity_queue.py	248	13	95%
poller/coordinator_bridge.py	317	317	0%
poller/math_writer.py	113	3	97%
poller/promotion.py	112	4	96%
poller/readiness.py	368	13	96%
poller/rebuild_child.py	133	6	95%
poller/service.py	665	52	92%
poller/worker_pool.py	140	0	100%
queue/executor.py	223	111	50%
regression/init.py	4	0	100%
regression/clojure_comparer.py	188	20	89%
regression/comparer.py	891	640	28%
regression/datasets.py	135	27	80%
regression/recorder.py	36	27	25%
regression/utils.py	130	85	35%
replay/init.py	2	0	100%
replay/attribution_capture.py	49	0	100%
replay/axis_continuity.py	347	24	93%
replay/certify.py	999	59	94%
replay/crosslang.py	128	2	98%
replay/decision_ties.py	55	8	85%
replay/diagnostics.py	50	7	86%
replay/driver.py	73	1	99%
replay/dsn_admission.py	36	1	97%
replay/equiv_query.py	10	0	100%
replay/event_ingress.py	95	1	99%
replay/fixture_bundle.py	815	62	92%
replay/fixture_config.py	266	11	96%
replay/fixture_extract.py	285	12	96%
replay/fixture_generate.py	103	1	99%
replay/fixture_samples.py	157	14	91%
replay/fixture_selection.py	119	0	100%
replay/fixture_survey.py	147	11	93%
replay/legacy_pca.py	29	17	41%
replay/polarity.py	319	25	92%
replay/poller_equiv.py	776	99	87%
replay/prodclone.py	193	2	99%
replay/real_data.py	146	10	93%
replay/schedule.py	205	10	95%
replay/served_math.py	159	4	97%
replay/shard_bench.py	189	65	66%
replay/stagecompare.py	890	161	82%
replay/stages.py	256	40	84%
replay/stepcompare.py	86	1	99%
replay/store.py	120	11	91%
replay/tie_capture.py	107	19	82%
replay/types.py	71	6	92%
run_math_pipeline.py	294	93	68%
tools/engine_rebuild.py	230	74	68%
types.py	22	0	100%
umap_narrative/501_calculate_comment_extremity.py	113	38	66%
umap_narrative/502_calculate_priorities.py	135	135	0%
umap_narrative/700_datamapplot_for_layer.py	502	502	0%
umap_narrative/701_static_datamapplot_for_layer.py	310	310	0%
umap_narrative/702_consensus_divisive_datamapplot.py	434	275	37%
umap_narrative/801_narrative_report_batch.py	837	620	26%
umap_narrative/803_check_batch_status.py	267	148	45%
umap_narrative/llm_factory_constructor/init.py	2	2	0%
umap_narrative/llm_factory_constructor/model_provider.py	234	234	0%
umap_narrative/polismath_commentgraph/init.py	1	0	100%
umap_narrative/polismath_commentgraph/cli.py	280	280	0%
umap_narrative/polismath_commentgraph/core/init.py	3	3	0%
umap_narrative/polismath_commentgraph/core/clustering.py	108	108	0%
umap_narrative/polismath_commentgraph/core/embedding.py	104	104	0%
umap_narrative/polismath_commentgraph/lambda_handler.py	219	219	0%
umap_narrative/polismath_commentgraph/schemas/init.py	2	0	100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py	160	9	94%
umap_narrative/polismath_commentgraph/tests/conftest.py	17	17	0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py	74	74	0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py	55	55	0%
umap_narrative/polismath_commentgraph/tests/test_storage.py	87	87	0%
umap_narrative/polismath_commentgraph/utils/init.py	3	0	100%
umap_narrative/polismath_commentgraph/utils/converter.py	283	237	16%
umap_narrative/polismath_commentgraph/utils/group_data.py	331	289	13%
umap_narrative/polismath_commentgraph/utils/storage.py	585	470	20%
umap_narrative/reset_conversation.py	159	50	69%
umap_narrative/run_pipeline.py	385	234	39%
umap_narrative/topic_naming.py	148	148	0%
utils/cli_flags.py	15	0	100%
utils/clj_hash.py	44	0	100%
utils/env_flags.py	13	0	100%
utils/general.py	63	39	38%
utils/output_profile.py	66	2	97%
utils/serialization.py	10	5	50%
utils/vote_convention.py	159	5	97%
utils/vote_convention_boot.py	72	4	94%
Total	24409	8252	66%