fix(persistence): retry sidecar open on the rollback→WAL conversion BUSY race - #179
Merged
Conversation
…BUSY race Concurrent first opens of a stale (rollback-journal) sidecar raced and one failed with "sidecar schema: begin: database is locked (5) (SQLITE_BUSY)". The DSN sets journal_mode=WAL, so the first openers convert the file from rollback to WAL, which takes a brief EXCLUSIVE lock. SQLite does not consult the busy handler for a journal-mode change, so busy_timeout is bypassed and the loser gets an immediate SQLITE_BUSY (observed in ~0.02s, far under the 5s timeout). _txlock=immediate + busy_timeout only serialise the post-conversion BEGIN IMMEDIATE write lock, not the conversion itself. Add a bounded application-level BUSY/LOCKED retry (withSidecarBusyRetry, typed detection via the modernc *sqlite.Error code) and wrap runBaseSchema and runMigrations in OpenSidecar with it. Both are idempotent (CREATE ... IF NOT EXISTS / user_version-gated), so retry is safe. Verified load-bearing: TestOpenSidecarConcurrentProcesses fails reliably without the change and passes under -race (and a 180-open stress) with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Concurrent first opens of a stale (rollback-journal) sidecar raced and one process failed:
TestOpenSidecarConcurrentProcessesreproduces it (6 processes opening a pre-WAL DB at once).Root cause
The sidecar DSN sets
journal_mode=WAL, so the first openers convert the file rollback→WAL, which takes a brief EXCLUSIVE lock. SQLite does not consult the busy handler for a journal-mode change, sobusy_timeout(5000)is bypassed and the loser gets an immediateSQLITE_BUSY— observed failing in ~0.02s, far under the 5s timeout. The existing_txlock=immediate+busy_timeoutonly serialize the post-conversionBEGIN IMMEDIATEwrite lock, not the conversion itself.Fix
internal/persistence/sidecar_migrate.go+sidecar_sqlite.go:isSidecarBusyErr— typed detection via modernc's*sqlite.Error, maskingCode() & 0xffto match baseSQLITE_BUSY(5) /SQLITE_LOCKED(6) and their extended codes.withSidecarBusyRetry— bounded exponential backoff (40 attempts, 5ms→250ms).OpenSidecarwraps bothrunBaseSchemaandrunMigrationsin it. Both are idempotent (CREATE … IF NOT EXISTS/user_version-gated), so retry is safe.busy_timeoutcovers the conversion.Verification
-race(5× and 3×), a 30-iteration / 180-concurrent-open stress, and the fullinternal/persistencepackage under-race.go vetclean.