roachtest: backup-restore/mixed-version failed

[cockroach-teamcity]

roachtest.backup-restore/mixed-version failed with artifacts on release-24.3 @ 714fac9c5c714e1d628334960aced7f07a6efc06:

(mixedversion.go:791).Run: mixed-version test failure while running step 8 (run "configure gc policies"): pq: unimplemented: operation is unsupported within a virtual cluster
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=true
• fs=ext4
• localSSD=true
• mvtDeploymentMode=shared-process
• mvtVersions=v23.1.30 → v23.2.30 → v24.1.28 → release-24.3
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: backup-restore/mixed-version failed #168594 roachtest: backup-restore/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.4]
• roachtest: backup-restore/mixed-version failed #167512 roachtest: backup-restore/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.2]
• roachtest: backup-restore/mixed-version failed #157896 roachtest: backup-restore/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.4.0-rc release-blocker]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-63376

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.backup-restore/mixed-version failed with artifacts on release-24.3 @ 714fac9c5c714e1d628334960aced7f07a6efc06:

(mixedversion.go:791).Run: mixed-version test failure while running step 10 (run "configure gc policies"): pq: unimplemented: operation is unsupported within a virtual cluster
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=true
• fs=ext4
• localSSD=true
• mvtDeploymentMode=shared-process
• mvtVersions=v23.1.30 → v23.2.30 → v24.1.28 → release-24.3
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: backup-restore/mixed-version failed #168594 roachtest: backup-restore/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.4]
• roachtest: backup-restore/mixed-version failed #167512 roachtest: backup-restore/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.2]
• roachtest: backup-restore/mixed-version failed #157896 roachtest: backup-restore/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.4.0-rc release-blocker]

This test on roachdash | Improve this report!

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.

[dt]

/investigate

[github-actions]

Investigation: backup-restore/mixed-version

Investigated failure: Initial report and second occurrence
Failure SHA: 714fac9c5c714e1d628334960aced7f07a6efc06
Confidence: high

What This Test Does

The backup-restore/mixed-version roachtest exercises backup and restore operations during a multi-version upgrade (v23.1.30 → v23.2.30 → v24.1.28 → release-24.3). It runs workloads (bank and tpcc), takes backups at various points during the upgrade, and verifies them. The test is parameterized to run in both system-only and shared-process (virtual cluster / multi-tenant) deployment modes.

Where the Failure Occurs

The test fails at step 10 ("configure gc policies") with:

pq: unimplemented: operation is unsupported within a virtual cluster

The failing SQL statement is ALTER DATABASE bank CONFIGURE ZONE USING gc.ttlseconds = $1 in configureGCPolicies.

The statement is executed via u.Exec() (backup_restore_roundtrip.go:336-342), which picks a random cached connection. These connections are created via h.Connect(node) (mixed_version_backup.go:2899), which delegates to h.DefaultService().Connect(node) (helper.go:187-188).

In shared-process deployment mode, DefaultService() returns h.Tenant (the virtual cluster), not h.System. CONFIGURE ZONE is not supported on virtual clusters, causing the error.

Analysis

Likely cause (high confidence): The configureGCPolicies function was introduced in commit bb1e96dc286 ("roachtest: add tighter GC for bank database in mixed version backup/restore"), backported to release-24.3 as 2b119a41fe5. This commit was intended to fix timeout issues (#168139) by reducing GC TTL on the bank database.

The CONFIGURE ZONE statement requires a connection to the system tenant, but the code uses CommonTestUtils.Exec() which connects to the default service (the virtual cluster in shared-process mode). The fix should route this specific operation through a system-tenant connection (e.g., h.System.Exec()), or skip the zone configuration when running in shared-process deployment mode.

Both failures in this issue thread have identical error signatures and the same SHA — they are the same bug.

Potential fix approaches:

1. Use h.System.Exec() for zone configuration instead of the tenant connection. This requires the configureGCPolicies function to access the helper's system service directly rather than going through CommonTestUtils.
2. Skip GC TTL configuration when h.IsMultitenant() returns true, since zone configs are a system-level concern.
3. Use ALTER VIRTUAL CLUSTER ... CONFIGURE ZONE ... syntax if available in the target version.

Related Issues and PRs

• release-26.2.0-rc: roachtest: backup-restore/mixed-version failed #168139 — The timeout issue that configureGCPolicies was added to fix.
• roachtest: backup-restore/mixed-version failed #168594 — backup-restore/mixed-version failure on release-25.4, but with a different failure mode (timeout, not virtual cluster error).
• roachtest: backup-restore/mixed-version failed #167512 — Failure on release-25.2 with a different failure mode (backup timeout/ambiguous result).
• roachtest: backup-restore/mixed-version failed #157896 — Failure on release-25.4.0-rc with a timeout (separate-process mode, different issue).
• backup: mixed version roachtest never creates a backup schedule #169334 — Tracking issue noting the mixed-version backup test never creates backup schedules.
• roachtest: backup-restore/mixed-version failed #168234 — Closed failure on release-24.1 with a different failure mode (issue backupccl: offline table data in revision history backups can leak into restored cluster #88042 IMPORT backup corruption).

None of the related issues share this same "unsupported within a virtual cluster" failure mode. This is a new failure introduced by the GC policy commit.

Timeline

• 2026-04-17: Commit bb1e96dc286 lands on master, adding configureGCPolicies to fix release-26.2.0-rc: roachtest: backup-restore/mixed-version failed #168139.
• 2026-04-27: Backported to release-24.3 as 2b119a41fe5.
• 2026-04-30: First failure in this issue (build 21331012), non-assertion build, shared-process deployment mode.
• 2026-04-30: Second failure (build 21332757), runtime-assertions build, same shared-process deployment mode, same error.

Detailed evidence and investigation notes

Log Excerpts

From test.log (build 21332757):

[mixed-version-test] Deployment mode:    shared-process

[mixed-version-test/10_run-configure-gc-policies] 2026/04/30 05:30:01 runner.go:427: ---------- STARTING (10): run "configure gc policies" ----------
[mixed-version-test/10_run-configure-gc-policies] 2026/04/30 05:34:11 runner.go:381: mixed-version test failure while running step 10 (run "configure gc policies"): pq: unimplemented: operation is unsupported within a virtual cluster

Full stack trace from failure_1.log:

mixed-version test failure while running step 10 (run "configure gc policies"): pq: unimplemented: operation is unsupported within a virtual cluster
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*Test).Run
  |     pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go:791
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackupMixedVersion.func1
  |     pkg/cmd/roachtest/tests/mixed_version_backup.go:2749
Wraps: (3) mixed-version test failure while running step 10 (run "configure gc policies")
Wraps: (4) pq: unimplemented: operation is unsupported within a virtual cluster
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.withPrefix (4) *pq.Error

Code References

The connection path in shared-process mode:

1. configureGCPolicies calls mvb.CommonTestUtils(ctx, h) which creates connections via h.Connect(node) — mixed_version_backup.go:2899
2. h.Connect(node) returns h.DefaultService().Connect(node) — helper.go:187-188
3. DefaultService() returns h.Tenant when h.Tenant != nil (shared-process mode) — helper.go:171-177
4. u.Exec() picks a random connection and runs ALTER DATABASE bank CONFIGURE ZONE... — backup_restore_roundtrip.go:336-342

The failing SQL in configureGCPolicies:

return u.Exec(
    ctx, rng, `ALTER DATABASE bank CONFIGURE ZONE USING gc.ttlseconds = $1`, bankTTLSeconds,
)

Things Ruled Out

• Infrastructure/flake issue: Not a flake — this is a deterministic failure when the test runs in shared-process deployment mode. Both occurrences had identical error messages.
• Related to other failure modes in sibling issues: The other open issues (roachtest: backup-restore/mixed-version failed #168594, roachtest: backup-restore/mixed-version failed #167512, roachtest: backup-restore/mixed-version failed #157896) have different failure modes (timeouts, backup errors). This is a distinct bug.
• Regression in CockroachDB server code: The error is expected behavior — CONFIGURE ZONE is by design unsupported in virtual clusters. The bug is in the test code.

Other Failure Occurrences

Both failures in this issue thread (builds 21331012 and 21332757) have the same:

• SHA: 714fac9c5c714e1d628334960aced7f07a6efc06
• Deployment mode: shared-process
• Error: pq: unimplemented: operation is unsupported within a virtual cluster
• Failing step: step 10 (first failure) / step 8 (second failure) — both "configure gc policies"
• Versions: v23.1.30 → v23.2.30 → v24.1.28 → release-24.3

The step number differs because the startup hooks run concurrently with randomized delays; the ordering is non-deterministic but the same function fails.

Tooling Feedback

No issues with available tooling. The investigation was straightforward with the available tools.

---

Workflow run

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.