move PostgreSQL test lanes back to GitHub-hosted runners - #517
Merged
Conversation
roborev: Combined Review (
|
mjacobs
changed the title
Three consecutive main builds (aaba110, a8b4f4b, 930a891) failed in test-postgres and test-pgvector after #508 routed trusted Linux jobs to the managed public fleet. Both die on Go's default 10m per-package timeout in cmd/msgvault/cmd, and it is not a hung test: each failure has a different test in flight when the alarm fires, so the package as a whole is simply running too slowly to finish. The cause is the runner, not the tests. These two lanes are the only ones driving the full Go suite against a live PostgreSQL service container, and that container is badly I/O-bound on the managed fleet -- a routine checkpoint logged write=116.7s sync=55.7s over 11169 files, and individual CREATE TABLE statements crossed the store's slow-query threshold at 1.3-1.9s apiece. Same commits on GitHub-hosted runners: test-postgres 5m11s, test-pgvector 1m48s. Raising -timeout would have kept main green while leaving both lanes running 20+ minutes on every push and trusted PR, so pin them back instead and leave the rest of #508's routing untouched. setup-go regains Actions caching here because #513's cache: false exists only to reuse a self-hosted machine-local GOMODCACHE, which no longer applies once the job is hosted. This is a workaround, not a fix -- the fleet's service-container disk I/O still needs attention before these lanes can move back. Generated with Claude Code (claude-opus-5) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The explicit eight-way limits and twenty-minute timeout were added only to compensate for managed-runner slowness. Retaining them after moving these lanes back to GitHub-hosted workers would diverge from the last known-green configuration and make genuine stalls take longer to surface. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
wesm
force-pushed
the
fix/pg-lanes-hosted-runners
branch
from
July 27, 2026 11:13
a601d61 to
ca71bd1
Compare
roborev: Combined Review (
|
jesserobbins
pushed a commit
to jesserobbins/msgvault
that referenced
this pull request
Aug 6, 2026
`test-postgres` and `test-pgvector` have failed on every managed-runner main build since kenn-io#508. Host telemetry confirms the lanes are storage-bound: PostgreSQL checkpoint syncs take minutes while CPU remains mostly idle. Move both lanes back to GitHub-hosted runners, restore Actions caching, and restore the test commands from their last known-green hosted configuration. This keeps main reliable without hiding the managed fleet's service-container I/O problem behind longer timeouts. The workflow passes actionlint and all repository hooks. The fleet storage bottleneck remains a separate infrastructure follow-up. Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test-postgresandtest-pgvectorhave failed on every managed-runner main build since #508. Host telemetry confirms the lanes are storage-bound: PostgreSQL checkpoint syncs take minutes while CPU remains mostly idle.Move both lanes back to GitHub-hosted runners, restore Actions caching, and restore the test commands from their last known-green hosted configuration. This keeps main reliable without hiding the managed fleet's service-container I/O problem behind longer timeouts.
The workflow passes actionlint and all repository hooks. The fleet storage bottleneck remains a separate infrastructure follow-up.