release-26.3: pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed

[cockroach-teamcity]

pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestExecBuild_distsql_plan_locality_filter failed on release-26.3 @ 8a9a8e3305daa0890b7b083c13cb572f911bd6cc:

google.golang.org/grpc.(*Server).processStreamingRPC(0xc001f65a40, {0xc4c45a0, 0xc00f280820}, 0xc046992b40, 0xc0032b5290, 0x1286d100, 0x0)
	external/org_golang_google_grpc/server.go:1659 +0x20f5
google.golang.org/grpc.(*Server).handleStream(0xc001f65a40, {0xc4c45a0, 0xc00f280820}, 0xc046992b40, 0x0)
	external/org_golang_google_grpc/server.go:1739 +0xfa5
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:970 +0x173
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 19887
	external/org_golang_google_grpc/server.go:981 +0x253

goroutine 646601 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).runOnce(0xc4c2320, {0xc4586b8, 0xc050585d60})
	pkg/rpc/stream_pool.go:107 +0x18e
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).run(0xc4c2320, {0xc4586b8, 0xc050585d60})
	pkg/rpc/stream_pool.go:102 +0x105
github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream.func2()
	pkg/rpc/stream_pool.go:314 +0x13c
created by github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream in goroutine 9286
	pkg/rpc/stream_pool.go:312 +0x6bd

goroutine 654692 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	external/org_golang_google_grpc/stream.go:383 +0xe7
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 12154
	external/org_golang_google_grpc/stream.go:382 +0x1c9f

goroutine 655291 [runnable]:
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:967
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 58843
	external/org_golang_google_grpc/server.go:981 +0x253

goroutine 654202 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).runOnce(0xc4c2320, {0xc4586b8, 0xc05ebb3720})
	pkg/rpc/stream_pool.go:107 +0x18e
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).run(0xc4c2320, {0xc4586b8, 0xc05ebb3720})
	pkg/rpc/stream_pool.go:102 +0x105
github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream.func2()
	pkg/rpc/stream_pool.go:314 +0x13c
created by github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream in goroutine 654186
	pkg/rpc/stream_pool.go:312 +0x6bd

goroutine 654194 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).runOnce(0xc4c2320, {0xc4586b8, 0xc05ebb2cd0})
	pkg/rpc/stream_pool.go:107 +0x18e
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).run(0xc4c2320, {0xc4586b8, 0xc05ebb2cd0})
	pkg/rpc/stream_pool.go:102 +0x105
github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream.func2()
	pkg/rpc/stream_pool.go:314 +0x13c
created by github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream in goroutine 654183
	pkg/rpc/stream_pool.go:312 +0x6bd

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-64918

[michae2]

/investigate

[github-actions]

Investigation: TestExecBuild_distsql_plan_locality_filter

Investigated failure: issue body
Failure SHA: 8a9a8e3305daa0890b7b083c13cb572f911bd6cc
Confidence: high

Warning: Could not check out failure SHA 8a9a8e3305daa0890b7b083c13cb572f911bd6cc. Analysis is based on the default branch tip, which may differ from the code that produced the failure. However, the relevant test file and deflake fix are confirmed present on release-26.3.

What This Test Does

This test exercises the distsql_plan_locality_filter session variable in a 9-node, 3-region in-process cluster. It verifies that DistSQL flow placement respects locality filters by checking that EXPLAIN (VEC) plans only schedule flows on nodes matching the configured locality filter.

Where the Failure Occurs

The test times out after 1h16m40s during cluster setup in newCluster at logic.go:1945. The test goroutine is blocked waiting for a SQL response (SET CLUSTER SETTING sql.stats.automatic_collection.min_stale_rows), and the SQL connection is stuck in pq.(*conn).readBindResponse waiting for the server to respond. The server never responds because the 9-node cluster is severely overloaded under the race detector.

Analysis

Hypothesis 1: Race-detector overhead causes timeout on resource-constrained CI (likely cause, high confidence)

The test runs with race=true and spins up a 9-node in-process cluster. The race detector imposes extreme CPU and memory overhead. The cluster never stabilizes enough to complete even the setup phase. Evidence from the server logs (8,655 overload-related messages):

• 87 under-replicated ranges: ranges.underreplicated@s1=87
• 25+ second latch holds: has held latch for 25.2s
• 26+ second slow replica RPCs: slow replica RPC: have been waiting 26.79s
• Slow heartbeats: slow heartbeat took 1.297s
• "node might be overloaded" warnings from raft ready handling

The EngFlow invocation ran 3 attempts. Run 1 timed out after 1h16m40s with the goroutine dump shown in the issue body. Run 3 also exited with error code -1 (killed after ~30 minutes with no test output beyond initial setup).

Hypothesis 2: This is NOT the original test assertion flakiness (confirmed, high confidence)

The original assertion bug (exact plan matching that was sensitive to node selection) was fixed in commit 68071d846b0, backported to release-26.3 as 079d681ef22. That fix replaced the exact-plan assertion with a deterministic negative-match assertion. The current failure is a timeout, not an assertion mismatch.

Missing skip.UnderRace: The test file at generated_test.go:58 has skip.UnderDeadlock(t, "times out and/or hangs") but does not have skip.UnderRace. Adding skip.UnderRace would prevent this timeout from recurring. Running a 9-node multi-region cluster under the race detector provides little incremental value — this test validates query plan correctness, not race conditions.

Recommendation: Add skip.UnderRace(t, "9-node cluster times out under race detector") to the runExecBuildLogicTest function in generated_test.go (or equivalently to the test generation template). This would match the existing skip.UnderDeadlock pattern.

Related Issues and PRs

• #171633 (open, master) — Same test, same timeout-under-race failure mode. Has a prior investigation with identical findings and the same recommendation.
• #170347 (closed) — Original test assertion flakiness, fixed by 68071d846b0 / #170401. This fix is present on release-26.3.
• #170414 (open, P-3, master) — Initially filed for a DATA RACE in SetYield, but most recent failures are the same overloaded-cluster timeout pattern. The data race was separately fixed.
• #170847 (closed) — SetYield data race fix.

Timeline

Date	Event
2026-05-04	Test introduced on master
2026-05-15	Deflake fix merged (68071d846b0) — fixes assertion flakiness
2026-05-15	Deflake fix backported to release-26.3 (079d681ef22)
2026-06-04 → 2026-06-18	Repeated race-build timeout failures on master (#171633)
2026-06-18	This issue filed: same failure mode on release-26.3

Detailed evidence and investigation notes

Log Excerpts

Timeout panic (test.log run 1, line 82):

panic: test timed out after 1h16m40s
	running tests:
		TestExecBuild_distsql_plan_locality_filter (1h16m40s)

Blocked test goroutine (test.log run 1, lines 170-206):

github.com/lib/pq.(*conn).readBindResponse(0xc02a970008)
	external/com_github_lib_pq/conn.go:1886 +0x5e
...
github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).newCluster(...)
	pkg/sql/logictest/logic.go:1945 +0x2e15

Server overload (8,655 total occurrences across 6 log files):

W260618 14:45:55.580842 ... health alerts detected: ranges.underreplicated@s1=87
W260618 14:45:56.082394 ... has held latch for 25.2s ... node might be overloaded
W260618 14:45:47.596403 ... slow replica RPC: have been waiting 26.79s
W260618 14:45:52.294589 ... slow heartbeat took 1.297s; err=<nil>

Code References

• Test definition: generated_test.go:87-92
• Test data file: distsql_plan_locality_filter
• skip.UnderDeadlock (exists, line 58): no skip.UnderRace present
• Hang location: newCluster at logic.go:1945

Things Ruled Out

• Test assertion flakiness: The original assertion bug was fixed in 68071d846b0, confirmed present on release-26.3. This failure is a timeout, not an assertion mismatch.
• DATA RACE in SetYield: This was a separate issue (#170414) that has been fixed. No WARNING: DATA RACE message appears in the logs.
• Code logic changes: The test file and testdata have not changed since the deflake fix. No recent changes could have introduced a regression.

Other Failure Occurrences

This is the first failure of this test on release-26.3. The same failure mode (race-build timeout) has been occurring repeatedly on master since mid-June (tracked in #171633), with 6+ occurrences, all under race=true. All share the same overloaded-cluster timeout pattern.

Tooling Feedback

• Could not check out failure SHA 8a9a8e3305daa0890b7b083c13cb572f911bd6cc (private repo SHA not reachable from the public repo clone). The test files were unchanged between the failure SHA's branch and the default branch tip, so this did not affect the investigation.

---

Workflow run

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[cockroach-teamcity]

pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestExecBuild_distsql_plan_locality_filter failed on release-26.3 @ 769f38f9d3492a70416a81f5ed38c5f09f4b4b83:

=== RUN   TestExecBuild_distsql_plan_locality_filter
    test_log_scope.go:172: test logs captured to: outputs.zip/logTestExecBuild_distsql_plan_locality_filter1717864030
    test_log_scope.go:82: use -show-logs to present logs inline

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestExecBuild_distsql_plan_locality_filter failed on release-26.3 @ 9caafebcafcd887c18107a3a718e5a07f38938d0:

github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00391d408, {0xc45dc18, 0xc0578e30e8}, {0x6?}, 0xc0364cb4a0)
	pkg/server/node.go:1734 +0xbc5
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00391d408, {0xc45bc70, 0xc0398b1500}, 0xc0364cb4a0)
	pkg/server/node.go:1927 +0x3f6
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0})
	bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12302 +0xbf
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00, 0xc052be2e70)
	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60 +0x74c
google.golang.org/grpc.getChainUnaryHandler.func1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0})
	external/org_golang_google_grpc/server.go:1177 +0x23d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00, 0xc04c2cc780)
	pkg/rpc/context.go:182 +0xf1
google.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0})
	external/org_golang_google_grpc/server.go:1177 +0x11f
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc002d3a000, {{0x1}, {0xc4c5a20, 0xc00289fda0}}, 0x0}, {0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00, ...)
	pkg/rpc/auth.go:96 +0x325
google.golang.org/grpc.getChainUnaryHandler.func1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0})
	external/org_golang_google_grpc/server.go:1177 +0x23d
github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00?, 0xc04c2cc700)
	pkg/rpc/metrics.go:803 +0x183
google.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0})
	external/org_golang_google_grpc/server.go:1177 +0x11f
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00, 0xc04c2cc6c0)
	pkg/rpc/metrics.go:589 +0x3a2
google.golang.org/grpc.getChainUnaryHandler.func1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0})
	external/org_golang_google_grpc/server.go:1177 +0x23d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xc45bc70, 0xc0398b1500})
	pkg/rpc/context.go:139 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001e1a280, {0xc45bc70, 0xc0398b1500}, {0xc0004c2, 0x21}, 0xc05a1e36c8)
	pkg/util/stop/stopper.go:358 +0x237
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00, 0xc04c2cc680)
	pkg/rpc/context.go:137 +0x14c
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0xc45bc70, 0xc0398b1500}, {0xbe7c4a0, 0xc0364cb4a0}, 0xc03b0abe00, 0xc052be2e70)
	external/org_golang_google_grpc/server.go:1168 +0xe8
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0xbe6f300, 0xc00391d408}, {0xc45bc70, 0xc0398b1500}, 0xc0381e0380, 0xc002f21cc0)
	bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12304 +0x1e7
google.golang.org/grpc.(*Server).processUnaryRPC(0xc002dedc20, {0xc4c7b80, 0xc00729c340}, 0xc035aaad80, 0xc0021418f0, 0x12877a40, 0x0)
	external/org_golang_google_grpc/server.go:1358 +0x1871
google.golang.org/grpc.(*Server).handleStream(0xc002dedc20, {0xc4c7b80, 0xc00729c340}, 0xc035aaad80, 0x0)
	external/org_golang_google_grpc/server.go:1735 +0xff3
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:970 +0x173
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 11201
	external/org_golang_google_grpc/server.go:981 +0x253

goroutine 511237 [runnable]:
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:967
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 9810
	external/org_golang_google_grpc/server.go:981 +0x253

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestExecBuild_distsql_plan_locality_filter failed on release-26.3 @ 8449e457346fdc20966c6f9acde86056a50dafd4:

github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0xbe6fb40, 0xc0040a2008}, {0xc49e4e0, 0xc02bb832c0}, 0xc0203414a0, 0xc060a05540)
	pkg/rpc/context.go:147 +0x15a
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0xbe6fb40, 0xc0040a2008}, {0xc49e4e0, 0xc02bb832c0}, 0xc0203414a0, 0xc3acdd0)
	external/org_golang_google_grpc/server.go:1495 +0xdd
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00217a5a0, {0xc4c83e0, 0xc009ae4d00}, 0xc0312acc60, 0xc0040921b0, 0x12871120, 0x0)
	external/org_golang_google_grpc/server.go:1659 +0x20f5
google.golang.org/grpc.(*Server).handleStream(0xc00217a5a0, {0xc4c83e0, 0xc009ae4d00}, 0xc0312acc60, 0x0)
	external/org_golang_google_grpc/server.go:1739 +0xfa5
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:970 +0x173
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 8564
	external/org_golang_google_grpc/server.go:981 +0x253

goroutine 557743 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc03395db30, {0xc026904850, 0x5, 0x5})
	external/org_golang_google_grpc/internal/transport/transport.go:195 +0x130
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc03395db30, {0xc026904850, 0x5, 0x5})
	external/org_golang_google_grpc/internal/transport/transport.go:175 +0x35b
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0269047e0, {0xc026904850, 0x5, 0x5})
	external/org_golang_google_grpc/internal/transport/transport.go:514 +0x62
io.ReadAtLeast({0xc405cc0, 0xc0269047e0}, {0xc026904850, 0x5, 0x5}, 0x5)
	GOROOT/src/io/io.go:335 +0xd1
io.ReadFull(...)
	GOROOT/src/io/io.go:354
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc035f3bb00, {0xc026904850, 0x5, 0x5})
	external/org_golang_google_grpc/internal/transport/transport.go:498 +0x11f
google.golang.org/grpc.(*parser).recvMsg(0xc026904840, 0x7fffffff)
	external/org_golang_google_grpc/rpc_util.go:600 +0x6a
google.golang.org/grpc.recvAndDecompress(0xc026904840, 0xc035f3bb00, {0x0, 0x0}, 0x7fffffff, 0xc027d4f9b0, {0xc43e610, 0x13e995c0})
	external/org_golang_google_grpc/rpc_util.go:730 +0x7c
google.golang.org/grpc.recv(0xc026904840, {0x7f9fa66c6e20, 0xc00149a250}, 0xc035f3bb00, {0x0, 0x0}, {0xbcc7e00, 0xc03eca6180}, 0x7fffffff, 0xc027d4f9b0, ...)
	external/org_golang_google_grpc/rpc_util.go:796 +0xc8
google.golang.org/grpc.(*csAttempt).recvMsg(0xc022bff520, {0xbcc7e00, 0xc03eca6180}, 0x0)
	external/org_golang_google_grpc/stream.go:1080 +0x52e
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc022bff520)
	external/org_golang_google_grpc/stream.go:917 +0x51
google.golang.org/grpc.(*clientStream).withRetry(0xc035f3b0e0, 0xc027d4fcd0, 0xc027d4fcc0)
	external/org_golang_google_grpc/stream.go:752 +0x439
google.golang.org/grpc.(*clientStream).RecvMsg(0xc035f3b0e0, {0xbcc7e00, 0xc03eca6180})
	external/org_golang_google_grpc/stream.go:916 +0x1df
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*internalBatchStreamClient).Recv(0xc01c93ee00)
	bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:11995 +0x66
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).runOnce(0xc4c6160, {0xc45c498, 0xc03395d9f0})
	pkg/rpc/stream_pool.go:116 +0x21d
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).run(0xc4c6160, {0xc45c498, 0xc03395d9f0})
	pkg/rpc/stream_pool.go:102 +0x105
github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream.func2()
	pkg/rpc/stream_pool.go:314 +0x13c
created by github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream in goroutine 104895
	pkg/rpc/stream_pool.go:312 +0x6bd

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestExecBuild_distsql_plan_locality_filter failed on release-26.3 @ 8449e457346fdc20966c6f9acde86056a50dafd4:

=== RUN   TestExecBuild_distsql_plan_locality_filter
    test_log_scope.go:172: test logs captured to: outputs.zip/logTestExecBuild_distsql_plan_locality_filter1475430641
    test_log_scope.go:82: use -show-logs to present logs inline

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[michae2]

Dupe of #171633, closing