Skip to content

fix(storage): stop a cancelled begin() from poisoning the pool - #20

Merged
christianhuening merged 1 commit into
mainfrom
fix/idle-transaction-leak
Sep 6, 2026
Merged

fix(storage): stop a cancelled begin() from poisoning the pool#20
christianhuening merged 1 commit into
mainfrom
fix/idle-transaction-leak

Conversation

@christianhuening

Copy link
Copy Markdown
Contributor

Fixes the Postgres transaction leak flagged in #19. Independent of that PR —
different crates, no overlap — so the two can merge in either order.

Root cause

sqlx's pool.begin() is not cancellation-safe. If the future is dropped
after BEGIN has reached Postgres but before begin() returns, sqlx never
receives a Transaction to roll back and keeps no record that one is open.
The connection goes back to the pool looking clean, and every later query
handed that connection silently joins the orphaned transaction.

Nothing ever commits it. It accumulates locks across whatever tables the
reused connection happens to touch, pins a transaction id against vacuum, and
stalls any TRUNCATE or DDL on those tables until the process exits.

axum drops a handler future exactly this way when a client disconnects
mid-request — ordinary browser behaviour, and constant under Playwright.

The evidence

Turning on log_statement=all and reading one backend's whole history:

12 ×  statement: BEGIN
11 ×  statement: COMMIT
 0 ×  statement: ROLLBACK

The statements after the unmatched BEGIN were unrelated work for a dozen
different documents — session touches, workspace_members role checks,
document lookups, UpdatesStore::since for many doc_ids — all inside one
transaction that had been open for minutes. pg_locks for that backend:

table mode
doc_updates AccessShare
board_updates AccessShare
comments AccessShare
sessions RowExclusive

Four subsystems that no single operation touches.

The fix

An after_release hook that issues ROLLBACK on the way back into the pool.

This has to live at the pool rather than at a call site: the cancelled task
never receives a Transaction, so no caller can clean up after itself, and
any future begin() call site would reintroduce the bug. ROLLBACK is a
no-op when no transaction is open, which is the overwhelmingly common case.

Supporting changes

  • idle_in_transaction_session_timeout=30s and application_name. The
    timeout is a backstop and explicitly not the fix
    — the connection in this
    bug is in constant use and never idles, so it would never have fired. It is
    kept for anything that still escapes. application_name is what identified
    the connection as ours in the first place, and is why the next occurrence
    will be obvious.
  • e2e/support/reset.ts now retries for ~46s instead of ~25s. Its old window
    expired just before the 30s backstop would have released a lock, which
    made that backstop useless to this suite.

Tests

tests/cancel_safety.rs reproduces the leak against the real pool builder.
It fails in 0.31s without the after_release guard and survives ~3000
cancelled begin() calls with it — verified both ways by temporarily removing
the guard.

The other four tests in that file are hypotheses this investigation ruled out:
a cancelled fetch, a dropped Transaction, a transaction dropped by
cancellation, and a query cancelled mid-execution. sqlx handles all four
correctly. They are kept because they are cheap and because a future sqlx
upgrade that regresses any of them would reintroduce this class of leak with
nothing else in the suite noticing.

tests/pool_hygiene.rs pins the backstop: that the setting is applied, and
that Postgres really does reclaim a session left idle in a transaction.

Verification

cargo fmt, clippy -D warnings, 339 nextest tests, and the full
Playwright suite:

stranded txn after run failures duration
before (3 runs) 1 each 5, 5, 7 3.8–5.9m
after 0 2 3.0m

The two remaining failures are the known environmental pair: tree-reorder
"nest" (macOS-only dnd-kit, passes in CI) and ws-reconnect (toxiproxy's
admin API is unreachable on this machine).

The comment-anchors, two-users-converge and upload-image flakes that
recurred across the earlier runs are gone, and the suite is ~1-3 minutes
faster — both consistent with removing the lock contention. That is one run,
so: strongly suggestive rather than proven.

🤖 Generated with Claude Code

`sqlx`'s `pool.begin()` is not cancellation-safe. If the future is dropped
after BEGIN has reached Postgres but before `begin()` returns, sqlx never
receives a `Transaction` to roll back and keeps no record that one is open.
The connection goes back to the pool looking clean, and every later query
handed that connection silently joins the orphaned transaction.

Nothing ever commits it. It accumulates locks across whatever tables the
reused connection touches, pins a transaction id against vacuum, and stalls
any TRUNCATE or DDL on those tables until the process exits. axum drops a
handler future exactly this way when a client disconnects mid-request, which
is ordinary browser behaviour and constant under Playwright.

Caught by turning on `log_statement=all` and reading one backend's history:

    12 x  statement: BEGIN
    11 x  statement: COMMIT
     0 x  statement: ROLLBACK

The statements after the unmatched BEGIN were unrelated work for a dozen
different documents — session touches, workspace_members role checks,
document lookups, `UpdatesStore::since` for many doc_ids — all inside one
transaction that had been open for minutes. `pg_locks` for that backend held
AccessShare on doc_updates, board_updates and comments plus RowExclusive on
sessions: four subsystems no single operation touches.

The fix is an `after_release` hook that issues ROLLBACK on the way back into
the pool. It has to be here rather than at a call site: the cancelled task
never receives a `Transaction`, so no caller can clean up after itself.
ROLLBACK is a no-op when no transaction is open, which is the common case.

Two supporting changes:

  - `idle_in_transaction_session_timeout=30s` and `application_name`. The
    timeout is a backstop, explicitly NOT the fix — the connection in this bug
    is in constant use and never idles, so it would never have fired. The
    application_name is what identified the connection as ours in the first
    place.
  - `e2e/support/reset.ts` retries for ~46s instead of ~25s. Its old window
    expired just before the 30s backstop would have released a lock, which
    made the backstop useless to the suite.

`tests/cancel_safety.rs` reproduces the leak against the real pool builder —
it fails in under a second without the `after_release` guard and survives
~3000 cancelled `begin()` calls with it. The other four tests in that file
are hypotheses this investigation ruled out: a cancelled fetch, a dropped
`Transaction`, a transaction dropped by cancellation, and a query cancelled
mid-execution. sqlx handles all four correctly. They are kept because a
future sqlx upgrade that regresses any of them would reintroduce this class
of leak silently.

Verified: fmt, clippy, 339 nextest tests, and the full Playwright suite with
no stranded transaction left behind — where the three runs before this change
each left one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@christianhuening
christianhuening merged commit 43b6e36 into main Sep 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant