Merge #155804 #155958 #156130 #156265

155804: workload: enable rangefeed with --changefeed r=wenyihu6 a=wenyihu6

This commit enables rangefeed when the --changefeed flag is specified with the
workload command.

Epic: https://cockroachlabs.atlassian.net/browse/CRDB-55203
Release note: none 

155958: roachtestutil: use DETACHED option for INSPECT jobs r=spilchen a=spilchen

Previously, CheckInspectDatabase used a statement timeout hack to background INSPECT jobs - it set a 5-second timeout and relied on the timeout error to leave jobs running.

Now that INSPECT supports the DETACHED option, we use `INSPECT DATABASE <name> WITH OPTIONS DETACHED` to properly run jobs in the background. This provides a cleaner way to background the job.

Informs #155676

Epic: CRDB-55075
Release note: none

156130: clusterversion: sync documentation from release-25.4 r=rail a=celiala

The release-25.4 branch has enhanced documentation for release preparation processes. This PR syncs those improvements to master.

## Changes

- **New file**: `pkg/clusterversion/CLAUDE.md` - Detailed runbook with step-by-step instructions for R.1 (beta prep) and R.2 (minting releases)
- **Updated**: `pkg/clusterversion/README.md` 
  - Added Claude prompt examples for R.1 and R.2 sections
  - Updated R.1 example PR reference from #113912 to #153906

## Context

These documentation improvements make the release preparation process clearer and more accessible, particularly for AI-assisted workflows.

Release note: None

Epic: None

156265: kvnemesis: remove TestKVNemesisMultiNode_BufferedWritesNoPipelining r=miraradeva a=stevendanna

This configuration is no longer relevant since the TxnCoordSender disables pipelining when buffered writes is in use.

Epic: none
Release note: None

Co-authored-by: wenyihu6 <[email protected]>
Co-authored-by: Matt Spilchen <[email protected]>
Co-authored-by: Celia La <[email protected]>
Co-authored-by: Steven Danna <[email protected]>

Author: 5 people

pkg/clusterversion/CLAUDE.md

@@ -0,0 +1,217 @@
+# CockroachDB Release Preparation Guide
+
+This document provides step-by-step instructions for preparing the CockroachDB source tree for beta releases, specifically following the R.1: Prepare for beta checklist from the clusterversion README.md.
+
+## Beta Release Preparation (R.1 Checklist)
+
+When preparing for a beta release (e.g., 25.4 beta1), the following files must be updated:
+
+### 1. Core Version Changes
+
+**Set Development Branch Flag:**
+- File: `pkg/clusterversion/cockroach_versions.go`
+- Change: Set `const DevelopmentBranch = false` (line ~334)
+
+**Update Version String:**
+- File: `pkg/build/version.txt`
+- Change: Update from alpha to beta version (e.g., `v25.4.0-alpha.2` → `v25.4.0-beta.1`)
+
+### 2. Regenerate Documentation
+
+Run the following command to update generated documentation:
+```bash
+./dev gen docs
+```
+
+This updates:
+- `docs/generated/settings/settings-for-tenants.txt`
+- `docs/generated/settings/settings.html`
+
+### 3. Update Test Data
+
+**System Schema Tests:**
+Run this command to regenerate bootstrap test data:
+```bash
+./dev test pkg/sql/catalog/systemschema_test --rewrite
+```
+
+This updates:
+- `pkg/sql/catalog/systemschema_test/testdata/bootstrap_system`
+- `pkg/sql/catalog/systemschema_test/testdata/bootstrap_tenant`
+
+**Bootstrap Hash Test:**
+Run this command and manually update hash values if the test fails:
+```bash
+./dev test pkg/sql/catalog/bootstrap --rewrite -f TestInitialValuesToString
+```
+
+If the rewrite fails, manually update the hash values in:
+- `pkg/sql/catalog/bootstrap/testdata/testdata`
+  - Update `system hash=` value
+  - Update `tenant hash=` value
+  - Update binary data values as shown in test output
+
+### 4. CLI Test Data Updates
+
+**Declarative Rules Tests:**
+- File: `pkg/cli/testdata/declarative-rules/deprules`
+  - Change: Update version reference to current release branch (e.g., `debug declarative-print-rules 25.2 dep` → `debug declarative-print-rules 25.3 dep`)
+
+- File: `pkg/cli/testdata/declarative-rules/invalid_version`
+  - Change: Update supported versions list to reflect current releases (use short format like `25.2`, `25.3` not `1000025.x`)
+
+### 5. Logic Test Updates
+
+**Internal Catalog Test:**
+- File: `pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog`
+- Change: Update `majorVal` from `1000025` to `25` in systemDatabaseSchemaVersion entries
+
+## Example PR Reference
+
+For reference, see PR #148382 which demonstrates the complete set of changes needed for beta release preparation.
+
+## Expected File Count
+
+A typical beta release preparation should modify approximately 10-12 files:
+1. `pkg/clusterversion/cockroach_versions.go`
+2. `pkg/build/version.txt`
+3. `docs/generated/settings/settings-for-tenants.txt`
+4. `docs/generated/settings/settings.html`
+5. `pkg/sql/catalog/systemschema_test/testdata/bootstrap_system`
+6. `pkg/sql/catalog/systemschema_test/testdata/bootstrap_tenant`
+7. `pkg/sql/catalog/bootstrap/testdata/testdata`
+8. `pkg/cli/testdata/declarative-rules/deprules`
+9. `pkg/cli/testdata/declarative-rules/invalid_version`
+10. `pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog`
+
+## Verification
+
+After making all changes:
+1. Run the bootstrap test to ensure it passes: `./dev test pkg/sql/catalog/bootstrap -f TestInitialValuesToString`
+2. Check git status to verify expected number of modified files
+3. All tests should pass before proceeding with the release
+
+## Notes
+
+- Some test data files contain binary encoded data that may change when version numbers are updated
+- Hash values in bootstrap test data are expected to change when the development branch flag is modified
+- The logic test updates are necessary to reflect the new major version in system database schema version metadata
+- CLI declarative rules tests use short version format (e.g., `25.3`) not the long format (e.g., `1000025.3`)
+- Always run the CLI tests after updating declarative rules files to ensure correct version format
+
+---
+
+## R.2: Mint Release (Creating Final Version)
+
+This change finalizes the cluster version for the release. It should be done when you are absolutely sure that no additional version gates are needed - right before cutting the first RC (typically rc.1).
+
+**Important timing note:** The minting happens before the final v25.X.0 release. It's typically done when preparing rc.1, which is shipped before the final release.
+
+### Critical Step: Update SystemDatabaseSchemaBootstrapVersion
+
+**IMPORTANT:** This is the most commonly missed step and will cause test failures if forgotten.
+
+**File:** `pkg/sql/catalog/systemschema/system.go` (line ~1445)
+
+**Change:** Update from the last internal version to the final minted version:
+
+```go
+// Before (last internal version - e.g., V25_4_AddSystemStatementHintsTable)
+var SystemDatabaseSchemaBootstrapVersion = clusterversion.V25_4_AddSystemStatementHintsTable.Version()
+
+// After (final minted version - e.g., V25_4)
+var SystemDatabaseSchemaBootstrapVersion = clusterversion.V25_4.Version()
+```
+
+### Full Checklist
+
+1. **Update cockroach_versions.go:**
+   - Add the final version key (e.g., `V25_4`) with `Internal: 0`
+   - Set `finalVersion` constant to this key (e.g., `const finalVersion Key = V25_4`)
+
+2. **Update SystemDatabaseSchemaBootstrapVersion** (see above - critical!)
+
+3. **Update version.txt:**
+   ```bash
+   # Update pkg/build/version.txt to RC version (e.g., v25.4.0-rc.1)
+   # Note: Minting happens before the final release, typically when cutting rc.1
+   ```
+
+4. **Regenerate documentation:**
+   ```bash
+   ./dev gen docs
+   ```
+
+5. **Regenerate bootstrap test data:**
+   ```bash
+   ./dev test pkg/sql/catalog/systemschema_test --rewrite
+   ```
+   This updates:
+   - `pkg/sql/catalog/systemschema_test/testdata/bootstrap_system`
+   - `pkg/sql/catalog/systemschema_test/testdata/bootstrap_tenant`
+
+6. **Update bootstrap hash test data:**
+
+   Run the test to see what values need updating:
+   ```bash
+   ./dev test pkg/sql/catalog/bootstrap -f TestInitialValuesToString
+   ```
+
+   The test will fail showing the expected hash values. Update in `pkg/sql/catalog/bootstrap/testdata/testdata`:
+   - Line 1: `system hash=<new_hash_from_test_output>`
+   - Line ~228: `tenant hash=<new_hash_from_test_output>`
+   - Update the binary data on the line following each hash (the `{"key":"8b89898a89","value":"..."}` entry)
+
+7. **Update logic test data:**
+
+   After updating `SystemDatabaseSchemaBootstrapVersion`, you need to update the expected test output:
+
+   - File: `pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog`
+   - Change: Update the `systemDatabaseSchemaVersion` in the test data to match the new minted version
+
+   **Before (with internal version):**
+   ```
+   1           {"database": {"id": 1, "name": "system", ... "systemDatabaseSchemaVersion": {"internal": 14, "majorVal": 25, "minorVal": 3}, ...}}
+   ```
+
+   **After (final minted version):**
+   ```
+   1           {"database": {"id": 1, "name": "system", ... "systemDatabaseSchemaVersion": {"majorVal": 25, "minorVal": 4}, ...}}
+   ```
+
+   Note: The `internal` field should be removed when minting the final version.
+
+8. **Verify all tests pass:**
+   ```bash
+   ./dev test pkg/sql/catalog/bootstrap -f TestInitialValuesToString
+   ./dev test pkg/sql/catalog/systemschema_test
+   ./dev test pkg/sql/logictest -f crdb_internal_catalog
+   ```
+
+### Common Errors and Solutions
+
+**Error: "Unexpected hash value for system"**
+- **Cause:** Forgot to update `SystemDatabaseSchemaBootstrapVersion` in step 2
+- **Fix:** Complete step 2, then re-run steps 5-7
+
+**Error: "output didn't match expected" with binary data diff**
+- **Cause:** The binary-encoded system database descriptor needs updating
+- **Fix:** Copy the exact binary value from the test output diff (shown after the `+` sign) and paste it into the testdata file
+
+### Expected Files Modified
+
+A typical R.2 mint should modify:
+1. `pkg/clusterversion/cockroach_versions.go` (add final version, set finalVersion)
+2. `pkg/sql/catalog/systemschema/system.go` (update SystemDatabaseSchemaBootstrapVersion)
+3. `pkg/build/version.txt` (update to final version)
+4. `docs/generated/settings/settings-for-tenants.txt`
+5. `docs/generated/settings/settings.html`
+6. `pkg/sql/catalog/systemschema_test/testdata/bootstrap_system`
+7. `pkg/sql/catalog/systemschema_test/testdata/bootstrap_tenant`
+8. `pkg/sql/catalog/bootstrap/testdata/testdata` (hashes and binary data)
+9. `pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog` (update systemDatabaseSchemaVersion)
+
+### Example PRs
+
+- For reference: [#112347](https://github.com/cockroachdb/cockroach/pull/112347)
+- 25.3 mint: [#150211](https://github.com/cockroachdb/cockroach/pull/150211)

pkg/clusterversion/README.md

@@ -97,13 +97,15 @@ code for more details.
 
 **When**: When we are ready to select the first beta candidate.
 
+**Claude Prompt**: "Please create a PR to prepare the release-25.4 branch for beta.1 following the R.1 checklist in pkg/clusterversion/README.md and the detailed runbook in pkg/clusterversion/CLAUDE.md"
+
 **Checklist**:
  - [ ] Set `developmentBranch` constant to `false`
  - [ ] Update `version.txt` to the beta version, e.g. `24.1.0-beta.1`
  - [ ] Regenerate docs (`./dev gen docs`)
  - [ ] Regenerate expected test data results as needed
 
-**Example PR:** [#113912](https://github.com/cockroachdb/cockroach/pull/113912)
+**Example PR:** [#153906](https://github.com/cockroachdb/cockroach/pull/153906)
 
 ### R.2: Mint release
 
@@ -112,6 +114,8 @@ code for more details.
 **When**: When we are absolutely sure that we no longer need additional version
 gates - right before the final RC at the latest.
 
+**Claude Prompt**: "Please create a PR to mint the 25.4 release following the R.2 checklist in pkg/clusterversion/README.md and the detailed runbook in pkg/clusterversion/CLAUDE.md. We're preparing rc.1."
+
 **Checklist**:
 - [ ] Replace temporary constant for current release (e.g. `V24_1`) with a
   cluster version key, associated with a "final" (`Internal=0`) version (e.g.

pkg/cmd/roachtest/roachtestutil/validation_check.go

@@ -206,17 +206,6 @@ func discoverUserDatabases(ctx context.Context, db *gosql.DB) ([]string, error)
 	return databases, rows.Err()
 }
 
-// isStatementTimeoutError returns true if the error is a statement timeout error.
-// Statement timeout errors are expected when launching INSPECT jobs since we only
-// want to start the job, not wait for it to complete.
-func isStatementTimeoutError(err error) bool {
-	var pqErr *pq.Error
-	if errors.As(err, &pqErr) {
-		return pgcode.MakeCode(string(pqErr.Code)) == pgcode.QueryCanceled
-	}
-	return false
-}
-
 // isFeatureNotSupportedError returns true if the error is a feature not supported error.
 // This can occur when the cluster version is not yet upgraded to support INSPECT.
 func isFeatureNotSupportedError(err error) bool {
@@ -229,8 +218,7 @@ func isFeatureNotSupportedError(err error) bool {
 
 // launchInspectJobs launches INSPECT DATABASE commands in parallel for all
 // provided databases using task manager for concurrency control. Each INSPECT
-// command enables the inspect command for that connection. Statement timeout
-// errors are ignored as they indicate the job was successfully started.
+// command is launched with the DETACHED option to run in the background.
 // Feature not supported errors are returned to the caller, indicating the
 // cluster version does not support INSPECT.
 func launchInspectJobs(
@@ -242,7 +230,6 @@ func launchInspectJobs(
 		return errors.Wrap(err, "failed to disable INSPECT admission control")
 	}
 
-	statementTimeout := 5 * time.Second
 	tm := task.NewManager(ctx, l)
 	g := tm.NewErrorGroup()
 
@@ -253,30 +240,13 @@ func launchInspectJobs(
 
 			statements := []string{
 				"SET enable_inspect_command = true",
-				fmt.Sprintf("SET statement_timeout = '%s'", statementTimeout.String()),
-				fmt.Sprintf("INSPECT DATABASE %s", lexbase.EscapeSQLIdent(dbName)),
+				fmt.Sprintf("INSPECT DATABASE %s WITH OPTIONS DETACHED", lexbase.EscapeSQLIdent(dbName)),
 			}
 
-			var stmtErr error
 			for _, stmt := range statements {
 				if _, err := db.ExecContext(ctx, stmt); err != nil {
-					stmtErr = err
-					break
-				}
-			}
-
-			// Always reset statement timeout back to default.
-			if _, err := db.ExecContext(ctx, "RESET statement_timeout"); err != nil {
-				l.Printf("Warning: failed to reset statement timeout: %v", err)
-			}
-
-			// Check for errors from the statements loop.
-			if stmtErr != nil {
-				// Statement timeout is expected - it means the job started but didn't complete
-				// within the timeout. The job is still running in the background.
-				if !isStatementTimeoutError(stmtErr) {
-					l.Printf("INSPECT DATABASE %s failed to start: %v", dbName, stmtErr)
-					return errors.Wrapf(stmtErr, "failed to start INSPECT DATABASE %s", dbName)
+					l.Printf("INSPECT DATABASE %s failed to start: %v", dbName, err)
+					return errors.Wrapf(err, "failed to start INSPECT DATABASE %s", dbName)
 				}
 			}
 

pkg/kv/kvnemesis/kvnemesis_test.go

@@ -430,7 +430,7 @@ func TestKVNemesisSingleNode_ReproposalChaos(t *testing.T) {
 }
 
 // TestKVNemesisMultiNode_BufferedWritesNoLockDurabilityUpgrades runs KVNemesis
-// with write buffering enabled and no lock durability ugprades. We leave splits
+// with write buffering enabled and no lock durability upgrades. We leave splits
 // to be metamorphic since those are all handled in-memory.
 func TestKVNemesisMultiNode_BufferedWritesNoLockDurabilityUpgrades(t *testing.T) {
 	defer leaktest.AfterTest(t)()
@@ -465,25 +465,6 @@ func TestKVNemesisMultiNode_BufferedWritesLockDurabilityUpgrades(t *testing.T) {
 	testKVNemesisImpl(t, cfg)
 }
 
-// TestKVNemesisMultiNode_BufferedWritesNoPipelining turns on buffered
-// writes and turns off write pipelining.
-func TestKVNemesisMultiNode_BufferedWritesNoPipelining(t *testing.T) {
-	defer leaktest.AfterTest(t)()
-	defer log.Scope(t).Close(t)
-
-	cfg := defaultTestConfiguration(3)
-	cfg.seedOverride = 0
-	cfg.bufferedWriteProb = 0.7
-	cfg.testSettings = func(ctx context.Context, st *cluster.Settings) {
-		kvcoord.BufferedWritesEnabled.Override(ctx, &st.SV, true)
-		kvcoord.PipelinedWritesEnabled.Override(ctx, &st.SV, false)
-		concurrency.UnreplicatedLockReliabilityLeaseTransfer.Override(ctx, &st.SV, true)
-		concurrency.UnreplicatedLockReliabilityMerge.Override(ctx, &st.SV, true)
-		concurrency.UnreplicatedLockReliabilitySplit.Override(ctx, &st.SV, true)
-	}
-	testKVNemesisImpl(t, cfg)
-}
-
 func TestKVNemesisMultiNode_Partition_Safety(t *testing.T) {
 	defer leaktest.AfterTest(t)()
 	defer log.Scope(t).Close(t)
@@ -829,25 +810,25 @@ func setAndVerifyZoneConfigs(
 ) {
 	// Set constraints on the system database; GeneratorDataTableID inherits from it.
 	sqlRunner.Exec(
-		t, `ALTER DATABASE system CONFIGURE ZONE USING 
-			num_replicas = 3, 
+		t, `ALTER DATABASE system CONFIGURE ZONE USING
+			num_replicas = 3,
 			num_voters = 3,
 			constraints = '{"+node=n1": 1, "+node=n2": 1}',
 			voter_constraints = '{"+node=n1": 1, "+node=n2": 1}'`,
 	)
 
 	// Ensure the liveness and meta ranges are also constrained appropriately.
 	sqlRunner.Exec(
-		t, `ALTER RANGE meta CONFIGURE ZONE USING 
-			num_replicas = 3, 
+		t, `ALTER RANGE meta CONFIGURE ZONE USING
+			num_replicas = 3,
 			num_voters = 3,
 			constraints = '{"+node=n1": 1, "+node=n2": 1}',
 			voter_constraints = '{"+node=n1": 1, "+node=n2": 1}'`,
 	)
 
 	sqlRunner.Exec(
-		t, `ALTER RANGE liveness CONFIGURE ZONE USING 
-			num_replicas = 3, 
+		t, `ALTER RANGE liveness CONFIGURE ZONE USING
+			num_replicas = 3,
 			num_voters = 3,
 			constraints = '{"+node=n1": 1, "+node=n2": 1}',
 			voter_constraints = '{"+node=n1": 1, "+node=n2": 1}'`,