roachtest: psycopg failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.psycopg failed with artifacts on release-26.1.6-rc @ e9b188f70df1de360e9491bd6041adf95b149570:

Failed with:

(orm_helpers.go:225).summarizeFailed: 
Tests run on Cockroach v26.1.6-dev-e9b188f70df1de360e9491bd6041adf95b149570
Tests run against psycopg 3.2.8
3852 Total Tests Run
3851 tests passed
1 test failed
1895 tests skipped
25 tests ignored
0 tests passed unexpectedly
1 test failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: tests.pool.test_pool_common_async.test_check_init[asyncio-AsyncConnectionPool] - unknown (unexpected)
For a full summary look at the psycopg artifacts 
An updated blocklist (psycopgBlockList) is available in the artifacts' psycopg log
test artifacts and logs in: /artifacts/psycopg/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21391977-1781762853-77-n1cpu4-0001	34.24.159.46	10.142.0.253

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=expiration
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: psycopg failed #171528 roachtest: psycopg failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-26.1]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-64906

[spilchen]

Triager status

Field	Value
Stage	classify-issue
Run ID	2026-06-18-122
Updated	2026-06-18T12:12:42Z
Confidence (flake)	10/10
Confidence (fix)	—
Result	waiting

[spilchen]

/investigate

[github-actions]

Investigation: psycopg (test_check_init)

Investigated failure: #171770
Failure SHA: e9b188f70df1de360e9491bd6041adf95b149570
Confidence: high

What This Test Does

The psycopg roachtest runs the psycopg 3.2.8 Python driver test suite against a single-node CockroachDB cluster. Results are compared against a blocklist (empty) and an ignore list to identify unexpected failures. The specific failing test, test_check_init[asyncio-AsyncConnectionPool], creates an AsyncConnectionPool with a custom check callback and then immediately attempts to get a connection with a hardcoded 1-second timeout.

Where the Failure Occurs

The test fails at tests/pool/test_pool_common_async.py:631 with:

psycopg_pool.PoolTimeout: couldn't get a connection after 1.00 sec

The full traceback shows the pool's getconn method in psycopg_pool/pool_async.py:229 raising PoolTimeout because the async connection pool could not establish and deliver a connection within the 1-second deadline. The roachtest framework then reports this as an unexpected failure via orm_helpers.go:225.

Analysis

Hypothesis: CockroachDB's async connection establishment latency exceeds the test's 1-second timeout — missing ignore list entry (high confidence)

This is the same class of timing flake that was addressed in commit 96ecdd84f19, which added 7 psycopg pool/concurrency tests to the ignore list to resolve issues #168667, #168686, #168743, and #168747. That commit missed test_check_init[asyncio-AsyncConnectionPool].

Evidence supporting this hypothesis:

1. Identical error across all occurrences — every failure across issues roachtest: psycopg failed #171528 (release-26.1, 7+ failures), roachtest: psycopg failed #171650 (release-26.1.6-rc, closed), and roachtest: psycopg failed #171770 (release-26.1.6-rc, this issue) produces the exact same PoolTimeout: couldn't get a connection after 1.00 sec error.
2. 100% failure rate — the test fails on every nightly run, regardless of runtime assertion settings, metamorphic lease modes, or disk configuration.
3. Hardcoded 1-second timeout — the test uses p.connection(timeout=1.0), which is too tight for CockroachDB's TLS handshake + authentication + async pool initialization overhead.
4. Sync variant passes — test_check_init[ConnectionPool] and test_check_init[NullConnectionPool] both pass consistently (confirmed in the test.log from build 21391977), confirming this is specific to async pool initialization timing.
5. Same pattern as fixed tests — the 7 tests already in the ignore list all had the same root cause: psycopg pool tests with wall-clock timeouts too tight for CockroachDB in CI.

Recommended fix: Add test_check_init[asyncio-AsyncConnectionPool] to the psycopgIgnoreList in psycopg_blocklist.go:

`tests.pool.test_pool_common_async.test_check_init[asyncio-AsyncConnectionPool]`: "flaky timing; see #171770",

This needs to be applied to master, release-26.1, and release-26.1.6-rc branches (and potentially others where the same failure appears).

Related Issues and PRs

Issue/PR	Status	Relationship
#171528	Open	Same test_check_init failure on release-26.1, 7+ consecutive nightly failures
#171650	Closed	Same failure on release-26.1.6-rc, classified as duplicate of #171528
#168667	Closed	Previous batch of psycopg timing flakes on master (7 tests)
#168686	Closed	Same batch on release-26.1
#168743	Closed	Same batch on release-25.2
#168747	Closed	Same batch on release-25.4
96ecdd84f19	Merged	Fix that added 7 tests to ignore list but missed test_check_init

Timeline

• 2026-04-22: Commit 96ecdd84f19 adds 7 flaky psycopg pool timing tests to the ignore list, resolving issues roachtest: psycopg failed #168667, roachtest: psycopg failed #168686, roachtest: psycopg failed #168743, roachtest: psycopg failed #168747. test_check_init is not included.
• 2026-06-10: Issue roachtest: psycopg failed #171528 opened — test_check_init[asyncio-AsyncConnectionPool] starts failing on every release-26.1 nightly run.
• 2026-06-14: Issue roachtest: psycopg failed #171650 opened for release-26.1.6-rc (later closed as duplicate).
• 2026-06-15: Investigation on roachtest: psycopg failed #171528 confirms the timing flake hypothesis.
• 2026-06-18: Issue roachtest: psycopg failed #171770 opened (this issue) — failure continues on release-26.1.6-rc.

Detailed evidence and investigation notes

Log Excerpts

Full traceback from build 21391977 (this issue's failure):

_________________ test_check_init[asyncio-AsyncConnectionPool] _________________

pool_cls = <class 'psycopg_pool.pool_async.AsyncConnectionPool'>
dsn = 'host=localhost port=26257 user=roachprod password=cockroachdb dbname=defaultdb'

    async def test_check_init(pool_cls, dsn):
        checked = False

        async def check(conn):
            nonlocal checked
            checked = True

        async with pool_cls(dsn, check=check) as p:
>           async with p.connection(timeout=1.0) as conn:

tests/pool/test_pool_common_async.py:631:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.10/contextlib.py:199: in __aenter__
    return await anext(self.gen)
psycopg_pool/build/__editable__.psycopg_pool-3.3.0.dev1-py3-none-any/psycopg_pool/pool_async.py:194: in connection
    conn = await self.getconn(timeout=timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <psycopg_pool.pool_async.AsyncConnectionPool 'pool-77' at 0x73f6724fbf40>
timeout = 1.0

    async def getconn(self, timeout: float | None = None) -> ACT:
        ...
        except PoolTimeout:
>           raise PoolTimeout(
                f"couldn't get a connection after {timeout:.2f} sec"
            ) from None
E           psycopg_pool.PoolTimeout: couldn't get a connection after 1.00 sec

psycopg_pool/build/__editable__.psycopg_pool-3.3.0.dev1-py3-none-any/psycopg_pool/pool_async.py:229: PoolTimeout

Code References

• Roachtest definition: psycopg.go
• Ignore list: psycopg_blocklist.go — test_check_init is absent from both psycopgBlockList and psycopgIgnoreList
• ORM test results processing: orm_helpers.go:225

Things Ruled Out

• CockroachDB bug: Not a CockroachDB correctness issue. 3851 other tests pass. The sync pool variant of the same test passes. This is purely a timing sensitivity in psycopg's own async pool test.
• Runtime assertions overhead: Fails identically both with and without runtime assertions (confirmed across multiple builds in roachtest: psycopg failed #171528).
• Infrastructure/environment variance: Fails across different node configurations, disk setups, metamorphic lease modes — it is deterministic.
• Regression in CockroachDB: Fails across many different SHAs (cb4efe22, 2aef52a3, fd7b98ab, 63ef98af, 10b5d90a, 292f1f75, a91df537, e9b188f7). Not tied to any specific code change.

Other Failure Occurrences

All failures across the three related issues (#171528, #171650, #171770) show the identical failure mode:

• Same test: test_check_init[asyncio-AsyncConnectionPool]
• Same error: PoolTimeout: couldn't get a connection after 1.00 sec
• Same location: psycopg_pool/pool_async.py:229
• All other variants (ConnectionPool, NullConnectionPool, AsyncNullConnectionPool) pass consistently.

There is no evidence of different root causes across the failures.

Tooling Feedback

• Could not check out the failure SHA (e9b188f70df1de360e9491bd6041adf95b149570) because it is on the release-26.1.6-rc branch which is not reachable from this clone's remote. Analysis was performed using the default branch tip code (which has the same psycopg_blocklist.go content) and artifact logs from TeamCity build 21391977.

---

Investigation triggered by this workflow run.

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

Issue Classification

Field	Value
Classification	duplicate
Confidence	10/10
Scope	test-infrastructure
Duplicate of	#171528
Frequency	recurring

Reasoning: This is the exact same failure as #171528 — identical test (test_check_init[asyncio-AsyncConnectionPool]), identical error (PoolTimeout: couldn't get a connection after 1.00 sec), identical root cause (hardcoded 1-second timeout too tight for CockroachDB's async connection setup). Issue #171650 was already closed as a duplicate of #171528 for the same reason. The only difference is the branch (release-26.1.6-rc vs release-26.1), but the underlying issue and fix are the same: add this test to the psycopgIgnoreList.

[cockroach-teamcity]

roachtest.psycopg failed with artifacts on release-26.1.6-rc @ e9b188f70df1de360e9491bd6041adf95b149570:

Failed with:

(orm_helpers.go:225).summarizeFailed: 
Tests run on Cockroach v26.1.6-dev-e9b188f70df1de360e9491bd6041adf95b149570
Tests run against psycopg 3.2.8
3852 Total Tests Run
3851 tests passed
1 test failed
1895 tests skipped
25 tests ignored
0 tests passed unexpectedly
1 test failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: tests.pool.test_pool_common_async.test_check_init[asyncio-AsyncConnectionPool] - unknown (unexpected)
For a full summary look at the psycopg artifacts 
An updated blocklist (psycopgBlockList) is available in the artifacts' psycopg log
test artifacts and logs in: /artifacts/psycopg/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21393503-1781849441-112-n1cpu4-0001	104.196.116.118	10.142.1.65

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: psycopg failed #171528 roachtest: psycopg failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-26.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.psycopg failed with artifacts on release-26.1.6-rc @ e9b188f70df1de360e9491bd6041adf95b149570:

Failed with:

(orm_helpers.go:225).summarizeFailed: 
Tests run on Cockroach v26.1.6-dev-e9b188f70df1de360e9491bd6041adf95b149570
Tests run against psycopg 3.2.8
3852 Total Tests Run
3851 tests passed
1 test failed
1895 tests skipped
25 tests ignored
0 tests passed unexpectedly
1 test failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: tests.pool.test_pool_common_async.test_check_init[asyncio-AsyncConnectionPool] - unknown (unexpected)
For a full summary look at the psycopg artifacts 
An updated blocklist (psycopgBlockList) is available in the artifacts' psycopg log
test artifacts and logs in: /artifacts/psycopg/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21394358-1781935669-112-n1cpu4-0001	34.148.75.245	10.142.2.7

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: psycopg failed #171528 roachtest: psycopg failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-26.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.psycopg failed with artifacts on release-26.1.6-rc @ e9b188f70df1de360e9491bd6041adf95b149570:

Failed with:

(orm_helpers.go:225).summarizeFailed: 
Tests run on Cockroach v26.1.6-dev-e9b188f70df1de360e9491bd6041adf95b149570
Tests run against psycopg 3.2.8
3852 Total Tests Run
3851 tests passed
1 test failed
1895 tests skipped
25 tests ignored
0 tests passed unexpectedly
1 test failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: tests.pool.test_pool_common_async.test_check_init[asyncio-AsyncConnectionPool] - unknown (unexpected)
For a full summary look at the psycopg artifacts 
An updated blocklist (psycopgBlockList) is available in the artifacts' psycopg log
test artifacts and logs in: /artifacts/psycopg/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21394844-1782022044-88-n1cpu4-0001	104.196.206.72	10.142.0.138

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: psycopg failed #171528 roachtest: psycopg failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-26.1]

This test on roachdash | Improve this report!