Repository navigation
CI: Delphi tests run through one entry point, in parallel - #3036
Merged
Merged
Conversation
ci/local/delphi-python.sh now holds every step of the Delphi pytest job (stack, migrations, copy-in, projection-gate wiring, pytest, coverage), and the workflow calls it and nothing else, so the same check runs on any Docker host. It removes published host ports so several runs can share one host. pytest runs in two passes: a parallel pass (pytest-xdist, one worker per CPU, --dist loadgroup, each dataset's tests on one worker so its conversation is computed once), then the tests marked `serial` sequentially. The serial marker is on the tests that collide when run concurrently: fixed ids or cluster-wide roles in the shared test database, each with the observed failure in a comment. Hosted CI splits the suite into three shard jobs (--shard I/3, by expected seconds per file or dataset group) and the `test` job combines them: it checks that every collected test ran in exactly one shard, adds up the counts and combines coverage into the same PR comment. A plain `pytest` still runs sequentially (-n0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…indow The shard plan now weighs each unit by expected wall seconds: parallel-pass seconds shared by four workers, plus serial-pass seconds at full weight. The durations file holds [parallel, serial] seconds per unit, measured on a hosted run. The plan uses a fixed worker count, so every machine computes the same plan. test_engine_rebuild's generated conversations had vote timestamps at 2026-10-03, inside the 10-day start window of the stack's live math-python poller. The poller could compute them under its own label, and the assertion that only the rebuild label has rows then failed (assert 2 == 0). Whether it did depended on which tests ran first, so a shard hit it. The fixture now uses 2020 timestamps. The assertion is unchanged. Shard results no longer include the compose inputs (stack/). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Observable changes: none to product behaviour. The Delphi workflow's jobs become
test shard 1/3,2/3and3/3plustest.testkeeps the required-check name, posts the same coverage comment and fails if any shard fails.ci/local/delphi-python.shholds every step the Delphitestjob ran inline (stack, migrations, copy-in, projection-gate wiring, pytest with coverage, coverage comment). The workflow calls the script and nothing else; the same command runs on any Docker host. It removes published host ports, so several runs can share a host.--dist loadgroup; each dataset's tests stay on one worker so its conversation is computed once), then a sequential pass of the tests markedserial. A plainpyteststill runs sequentially (-n0).--shard 1/3,2/3,3/3, split by expected seconds per file or dataset group. Thetestjob (same check name as before) checks that every collected test ran in exactly one shard, adds up the counts, combines coverage and posts the same coverage comment.serial(marker only; no test thinned or skipped). Each collided when run concurrently:poller/test_capacity_queue_postgres.py: creates the cluster-wide rolemath_queue_test; parallel run: "role already exists".poller/test_backfill_postgres.py: same role, plus fixed zids 13500...; parallel run: duplicate keys, foreign-key failures.test_certify_bundle_integration.py: seeds fixed zid 1990000001...; parallel run: duplicate key onconversations_zid_key.test_engine_rebuild.py: seeds fixed zids 9101/9102; parallel run: duplicate keys onparticipants/comments.test_generator_vote_copy.py: seeds and deletes fixed zids per test; parallel run:assert 7 == 12(another worker's rows).test_prodclone_extract.py::test_survey_and_extract_round_trip: leaves zids 101001-101005 in the shared database; when it ran first, the certify-bundle survey picked zid 101005 instead of its planted conversation.Evidence
Counts are identical everywhere: 4116 passed, 228 skipped, 44 xfailed, 2 xpassed (4390 collected), the same as the latest edge run. Per-test outcomes were compared test by test with the serial run.
DELPHI_PYTEST_WORKERS=0), final codeCoverage: the combined shard coverage and the 12-worker run match the serial run line for line (25412 statements, 7950 missed). One 4-worker run differed by one line in
polismath/poller/service.py. That line depends on timing in the live poller.Hosted CI wall-clock (
Delphi Python Tests):testtestHosted counts: shards 1228+1474+1414 passed, 21+154+53 skipped, 22+0+22 xfailed, 2+0+0 xpassed = 4116 / 228 / 44 / 2, 4390 of 4390 collected tests, each in exactly one shard (checked by the
testjob). Coverage: 25412 statements, 7951 missed (edge: 7950). The one-line difference is the timing-dependent line inpolismath/poller/service.pydescribed above.Each shard still spends about 6 minutes building every image in
docker-compose.test.yml, plus disk cleanup and Buildx teardown. Most of the remaining time is that fixed cost. Layer caching and building only the services this job uses are separate changes.One test fixture changed (the test's assertions did not).
test_engine_rebuild.pygenerated its conversations with vote timestamps at 2026-10-03. That is inside the 10-day start window (POLL_FROM_DAYS_AGO) of the stack's livemath-pythonpoller, which runs during pytest. The poller could compute those conversations under its own label.test_dry_writes_nothing_and_digests_match_the_published_rowsthen saw rows outside the rebuild label (assert 2 == 0). Whether that happened depended on which tests ran first: it failed in shard 3 twice (once on a work box, once in hosted run 38220973217) and passed in every full run. The fixture now uses 2020 timestamps, which the poller never reads. Shard 3 then passed in every later run.Run it locally:
bash ci/local/delphi-python.sh(whole suite),--shard 2/3(one shard),combine DIR...(merge shard results).DELPHI_PYTEST_WORKERS=0gives the old single sequential pass.🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s