Repository navigation
worker boot: the large worker's unit carries its CA, staged label and commit check; rehearsed end to end - #2988
Open
colinmegill wants to merge 2 commits into
Open
colinmegill wants to merge 2 commits into
colinmegill wants to merge 2 commits into
Conversation
… check; rehearsed end to end The large worker's env document (/etc/app-info/polis-jobs.env, written by its launch template and passed to docker run after the shared app .env, so it wins) now carries: - POLIS_JOBS_CA_FILE and POLIS_JOBS_HOST_ALLOWLIST (the RDS instance endpoint, or context queueHostAllowlist): the start script fetches AWS's RDS CA bundle when the file is missing and mounts it read-only (#671); - the large child's MATH_ENV=python-large (STAGED_LABEL, pinned to the small poller's default staged label) with MATH_CAPACITY_ROUTING/PROMOTE off and RESTAGE empty, which the child otherwise refuses (#722); - POLIS_JOBS_REQUIRE_SOURCE_COMMIT=1: the start script refuses to start the daemon without MATH_POLLER_SOURCE_COMMIT in the app .env, so no job is claimed only for its child to refuse it. The env document, start script and unit are exported functions (workerEnvDocument, workerStartScript, workerUnit); cdk/scripts/ emit-worker-files.ts writes them for local rehearsal. The chain proof no longer sets the child's label: its base env carries the served label and --worker-env applies the generated env document. delphi/tests/poller/worker_boot_proof.py starts the final image through the real unit under systemd in a stand-in box, with the login read by name from a stand-in secret store, TLS against a stand-in CA, and checks the child's success and its refusals (no commit, another staged label, and with 000025 a vote convention it was not built for). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
postgres:17 declares a data volume; docker rm -f without -v left one behind per run on the build box. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Delphi Coverage Report
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observable changes: only with
-c enableLargeClass=true(off in production; the flag-off template is byte-identical). The worker boxes' user data changes: the env document/etc/app-info/polis-jobs.envgainsPOLIS_JOBS_CA_FILE=/etc/polis-jobs/rds-ca-bundle.pem,POLIS_JOBS_HOST_ALLOWLIST=<RDS instance endpoint>(or contextqueueHostAllowlist),POLIS_JOBS_REQUIRE_SOURCE_COMMIT, and for the math-large classMATH_ENV=python-large,MATH_CAPACITY_ROUTING=0,MATH_CAPACITY_PROMOTE=0,MATH_CAPACITY_RESTAGE=. The start script fetches AWS's RDS CA bundle (https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem) when the file is missing, mounts it read-only, and refuses to start the daemon (exit 1, systemd retries every 60 s) when the app.envhas noMATH_POLLER_SOURCE_COMMITon a class that needs one. No schema change. No Lambda. Not deployed by this PR.Why (connections map §5, findings #671 and #722)
.env, whoseMATH_ENVis the served labelpython; the child refuses that (it never writes a served label), and it also refuses any small-poller routing/promotion setting that is on. Nothing gave the worker its staged label: the composed chain proof setMATH_ENV=python-largeitself, which hid the gap..envlacks the source commit would claim jobs only for every child to refuse them (version-skew guard), burning attempts.What changes
cdk/workerClasses.ts:STAGED_LABEL = 'python-large'(a test pins it toDEFAULT_STAGED_LABELindelphi/polismath/poller/capacity.py); the math-large row'schildEnvandrequiresSourceCommit;queueHostAllowlistcontext.cdk/launchTemplates.ts: the env document, start script and unit are exported functions (workerEnvDocument,workerStartScript,workerUnit); the user data writes the env document from the first. The env document stays AFTER the app.envindocker run, so its keys win.cdk/lib/cdk-stack.ts: passes the RDS endpoint as the allowlist.cdk/scripts/emit-worker-files.ts: writes the three files exactly as the launch template installs them, for rehearsal.delphi/tests/poller/chain_proof.py: no longer sets the child's label. Its daemon env carries the served label (as the app.envdoes) and--worker-envapplies the generated env document.delphi/tests/poller/worker_boot_proof.py(+worker_boot/Dockerfile.box,worker_boot/aws-standin): the real boot path on a build box.Proof on the work box
Worker boot proof: the final Delphi image built from edge + this PR + #2944 (so the vote-convention guard is in), the unit/start script/env document from
emit-worker-files.ts, installed at their real paths in a stand-in box running systemd and started withsystemctl start polis-jobs. The secret store is a stand-in: a local file served by a stand-inawscommand, clearly marked, never AWS. The TLS CA is a stand-in local CA in place of the RDS bundle. 24 checks, all ok:TLSv1.3aspolis_jobs_proof, a plain login that is a member ofpolis_queue_executor. Inside the containerMATH_ENV=python-large(the app.envsayspython) andMATH_CAPACITY_ROUTING=0(the app.envsays 1). The cgroup limit is 53248 MiB.memory admission: limit_mb=53248 (cgroup),staged=python-largeand the source commit, and it rebuilt underpython-large. The bundle is complete underpython-large, nothing was written underpython, and the receipt binds the result.MATH_POLLER_SOURCE_COMMITin.env, the start script refuses and no daemon starts. The proof found a real bug here: underset -o pipefaila missing line exited before the refusal message was printed. Fixed (|| true).python-otheris refused by the child (stage_failed:2).agree = +1is refused by the child through the same unit ("this database declares vote convention version 1 (agree = +1) … built for agree = -1").Proof on edge + this PR alone: steps 1–4 hold (22 checks) and step 5 is skipped because the tree has no 000025. Composed chain proof (edge + this PR, real daemon + child + promotion, with the generated env document and no label override): 41 checks ok,
done: every step held.cdkjest 165 passed;cdk synth -c enableLargeClass=trueok; Delphi poller tests 865 passed. Every container, the box image and the VM paths were removed afterwards.Not covered: the CA fetch from AWS's URL was not exercised (the proof places the stand-in CA first). Drain and shutdown (#718), and the Delphi class's own child env, are out of scope. The unit still runs the hook-built mutable image tag.
🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s