release-26.3: roachtest: ldr/unique_constraint/workload=source/update_benchmark failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ldr/unique_constraint/workload=source/update_benchmark failed with artifacts on release-26.3 @ e83d5ed5cc10a4601448eec2a6810ed98de6cab5:

Failed with:

(cluster.go:2816).Run: full command output in run_213054.802092276_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/ldr/unique_constraint/workload=source/update_benchmark/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21398766-1782281435-275-n7cpu8-0001	35.185.32.163	10.142.0.124
teamcity-21398766-1782281435-275-n7cpu8-0002	34.75.29.235	10.142.0.63
teamcity-21398766-1782281435-275-n7cpu8-0003	34.26.51.110	10.142.0.38
teamcity-21398766-1782281435-275-n7cpu8-0004	104.196.131.84	10.142.0.122
teamcity-21398766-1782281435-275-n7cpu8-0005	34.24.24.8	10.142.0.113
teamcity-21398766-1782281435-275-n7cpu8-0006	34.26.78.193	10.142.0.40
teamcity-21398766-1782281435-275-n7cpu8-0007	35.196.243.50	10.142.0.127

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=8
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• metamorphicLeases=leader
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachlabs/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-65113

[blathers-crl]

cc @cockroachdb/cdc

[celiala]

/investigate

[github-actions]

Investigation: ldr/unique_constraint/workload=source/update_benchmark

Warning: Could not check out failure SHA e83d5ed5cc10a4601448eec2a6810ed98de6cab5. Analysis is based on the default branch tip, which may differ from the code that produced the failure.

Investigated failure: #171978
Failure SHA: e83d5ed5cc10a4601448eec2a6810ed98de6cab5
Confidence: high

What This Test Does

The ldr/unique_constraint/workload=source/update_benchmark roachtest sets up unidirectional logical data replication (LDR) between two 3-node clusters and runs the uniqueindex workload to benchmark UPDATE performance with unique constraints. The test metamorphically picks a predecessor version (v26.2.x) for one of the clusters when mixedVersionMinimum allows it.

Where the Failure Occurs

The test fails during workload initialization at logical_data_replication.go:975, when running:

./cockroach workload init uniqueindex {pgurl:1-3:system} --data-size=1024 --rows=600 --scatter --split-every=1000

The workload binary (current version, on the workload node) sends an IMPORT INTO ... CSV DATA ('workload://...') statement to the left cluster (running v26.2.1). The v26.2.1 server cannot resolve the uniqueindex workload generator because it was only added in v26.3:

Error: importing fixture: importing table uniqueindex: pq: unknown generator: uniqueindex

Analysis

Confirmed cause (high confidence): Missing backport of mixedVersionMinimum fix to release-26.3.

This is the same root cause as #171732 (master) and #171838 (release-26.3).

The test spec originally set mixedVersionMinimum: clusterversion.V26_2, which allows the test to run one of the clusters on v26.2.x. However, the uniqueindex workload was introduced in v26.3. When workload init runs against a v26.2.1 server, the IMPORT uses the workload:// scheme which requires server-side resolution of the generator — and v26.2.1 doesn't have uniqueindex registered.

The fix (commit 31e68b0a495 — "roachtest: bump mixed version minimum for unique index update benchmark", 2026-06-18 by Keith Chow) changed mixedVersionMinimum from clusterversion.V26_2 to clusterversion.V26_3. This fix is present on master but has not been backported to release-26.3.

The prior issue #171838 (also on release-26.3) was closed as "Fixed by cockroachdb/cockroach-private#926", but that private PR apparently either didn't include the mixedVersionMinimum fix or didn't land on the release-26.3 branch.

Fix: Backport commit 31e68b0a495 to release-26.3. This is a test-only change (no product code impact).

Related Issues and PRs

Issue/PR	Description	Same failure mode?
#171732	Same test failure on master	Yes, identical (unknown generator: uniqueindex)
#171838	Same test failure on release-26.3	Yes, identical. Closed via private PR #926 but fix didn't fully land.
31e68b0a495	Fix: bump mixedVersionMinimum to V26_3	Fix on master, not backported to release-26.3

Timeline

• 2026-04-28: uniqueindex workload and this test added (both given mixedVersionMinimum: V26_2 incorrectly).
• 2026-06-17: First failure reported as #171732 (master) when random seed chose v26.2.1 predecessor.
• 2026-06-18: Fix commit 31e68b0a495 merged to master.
• 2026-06-20: #171838 filed for same failure on release-26.3, closed via private PR.
• 2026-06-24: This issue #171978 — same failure recurs on release-26.3 (fix still not present).

Detailed evidence and investigation notes

Log Excerpts

From test.log (predecessor version selection):

2026/06/24 21:30:10 logical_data_replication.go:307: random seed is -200254166155954788
2026/06/24 21:30:10 logical_data_replication.go:1227: using predecessor version v26.2.1 for left cluster

From failure_1.log (actual error):

Node 7. Command with error:
./cockroach workload init uniqueindex {pgurl:1-3:system} --data-size=1024 --rows=600 --scatter --split-every=1000

stderr:
I260624 21:30:55.595060 1 workload/cli/run.go:702  [-] 1  random seed: 4692980293295468314
I260624 21:30:55.618414 1 workload/fixture/fixture.go:320  [-] 2  starting import of 1 tables
I260624 21:30:55.645933 1 workload/fixture/fixture.go:406  [-] 3  created 1/1 tables
I260624 21:30:55.645986 1 workload/fixture/fixture.go:422  [-] 4  importing with concurrency limit 9223372036854775807
Error: importing fixture: importing table uniqueindex: pq: unknown generator: uniqueindex

Code References

The test spec on master (already fixed):

// logical_data_replication.go:280-290
{
    name:        "ldr/unique_constraint/workload=source/update_benchmark",
    clusterSpec: defaultLdrCluster(),
    ldrConfig: ldrConfig{
        createTables: true,
    },
    // The uniqueindex workload was added in v26.3 PR: #167174.
    mixedVersionMinimum: clusterversion.V26_3,  // fixed on master
    run:                 TestLDRUniqueConstraintUpdate,
    monitor:             true,
},

The version selection logic at logical_data_replication.go:1224:

• case 0: uses predecessor for left cluster (this run)
• case 1: uses predecessor for right cluster
• case 2, 3: both clusters run current version

Things Ruled Out

• New bug or regression: This is the exact same known issue from roachtest: ldr/unique_constraint/workload=source/update_benchmark failed #171732/release-26.3: roachtest: ldr/unique_constraint/workload=source/update_benchmark failed #171838 — incorrect mixedVersionMinimum allowing v26.2.x predecessor that doesn't have the uniqueindex workload.
• LDR replication issue: The test never reaches the replication phase; fails during workload init.
• Infrastructure issue: All nodes started successfully; the error is a clean SQL error from IMPORT.
• Runtime assertions: The runtimeAssertionsBuild=true flag is unrelated; the failure is a workload compatibility issue.
• Private PR fix: The private PR MultiRaft needs to return a retriable error to clients waiting for a command while the group is deleted #926 that closed release-26.3: roachtest: ldr/unique_constraint/workload=source/update_benchmark failed #171838 either didn't include this fix or hasn't merged to release-26.3 yet.

Other Failure Occurrences

All three occurrences (#171732, #171838, #171978) have the identical failure mode: unknown generator: uniqueindex when the random seed selects a v26.2.x predecessor version. The failure is deterministic whenever v26.2.x is chosen (2/4 probability per the random switch at line 1224).

Tooling Feedback

• Could not check out the failure SHA e83d5ed5cc10a4601448eec2a6810ed98de6cab5 from this blobless clone (the SHA is from the release-26.3 branch; transparent blob fetches failed for objects exclusive to that branch). The relevant test code on master is essentially identical except that the fix is already applied, which was helpful for confirming the root cause.

---

Investigation triggered by this workflow run.

Was this investigation helpful? Leave a 👍 or 👎 on this comment.