release-26.3: roachtest: pg_dump failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.pg_dump failed with artifacts on release-26.3 @ 8a9a8e3305daa0890b7b083c13cb572f911bd6cc:

Failed with:

(pg_dump.go:1069).func1: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/pg_dump/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21392022-1781763104-17-n1cpu4-0001	34.74.252.170	10.142.0.143

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=leader
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-64913

[spilchen]

Triager status

Field	Value
Stage	classify-issue
Run ID	2026-06-19-126
Updated	2026-06-19T11:59:43Z
Confidence (flake)	9/10
Confidence (fix)	—
Result	waiting

[spilchen]

/investigate

[github-actions]

Investigation: roachtest.pg_dump

Investigated failure: #171777 (issue body)
Failure SHA: 8a9a8e3305daa0890b7b083c13cb572f911bd6cc
Confidence: high

What This Test Does

The pg_dump roachtest verifies round-trip compatibility between CockroachDB and PostgreSQL's pg_dump tool. It creates test databases with representative schemas, dumps them using PostgreSQL 18's pg_dump with pg_dump_compatibility=postgres, restores the dumps into target databases, re-dumps, and compares output. The test also includes a data export (COPY) round-trip that creates tables with data and verifies the COPY output survives a dump-restore-dump cycle.

Where the Failure Occurs

The failure occurs at pg_dump.go:1069 in the runPGDumpDataRoundTrip function. The pg_dump binary itself crashes with exit status 1 when attempting to dump the pgdump_data_target database:

pg_dump: error: unexpected column data for table "pg_partitioned_table"

This is a different failure mode from the data_parent_pkey duplication seen in the related master branch issue #171695.

Analysis

Likely cause (high confidence): OID collision between user table data_parent and pg_catalog table pg_partitioned_table.

The test creates 3100 filler tables in a separate filler_db database to push the global descriptor ID counter into the range of PostgreSQL catalog OIDs (1247–6104). This tests that the OID remapping in pg_dump_compatibility mode works correctly. However, it introduces a collision:

1. The filler tables consume descriptor IDs from ~100 up to ~6300.
2. The pgdump_data_target database is created later (after schema restore into pgdump_target). When tables are restored into pgdump_data_target via psql, they get the next available descriptor IDs.
3. From the schema change log: data_parent in pgdump_data_target gets DescriptorID 3350.
4. When pg_dump_compatibility=postgres is enabled, the virtual table pg_partitioned_table in pg_catalog is remapped to OID 3350 (its standard PostgreSQL RelationId, defined in pg_catalog_oids.go:54).
5. The tableOid() function (pg_catalog.go:6712-6713) simply casts the raw descriptor ID to an OID — it does not apply any collision avoidance for user tables.
6. Result: both data_parent (user table, OID 3350) and pg_partitioned_table (pg_catalog, remapped OID 3350) appear in pg_class with OID 3350.
7. When pg_dump processes the database, it encounters this OID collision. It attempts to use the data from data_parent (columns: id, label) as if it were pg_partitioned_table (columns: partrelid, partstrat, partnatts, etc.), producing the "unexpected column data" error.

Why the source dump succeeded: data_parent in pgdump_data_src was created earlier and got DescriptorID 3279, which doesn't collide with any remapped pg_catalog OID. The collision only occurs in pgdump_data_target because more descriptors were allocated between source and target database creation (for pgdump_target schema objects).

Why this is deterministic, not flaky: Given the same cluster state and the same number of filler tables (3100), the descriptor ID allocation is deterministic. The collision will happen every time on release-26.3 with this exact test setup. The specific colliding OID (3350) depends on how many descriptors are allocated before pgdump_data_target tables are created.

Root Cause

The OID remapping in RemapPgCatalogOid (pg_catalog_oids.go:87-94) correctly remaps pg_catalog table descriptor IDs to their PostgreSQL OIDs. However, user tables whose raw descriptor IDs happen to equal one of these remapped OIDs will collide in pg_class, causing pg_dump to malfunction.

The filler table mechanism in the test was specifically designed to exercise this OID range, but it inadvertently exposed this collision bug. The test was ironically designed to verify that remapping works, but it revealed that the remapping is incomplete — it doesn't prevent user table OIDs from colliding with the remapped pg_catalog OIDs.

Proposed Fixes

Option A (product fix — proper): When pg_dump_compatibility=postgres is enabled, add a secondary remapping for user tables whose descriptor IDs collide with any remapped pg_catalog OID. This could offset the user table OID to an unused range (e.g., add a large constant). This fix belongs in the tableOid() function or wherever user table OIDs are computed for pg_catalog virtual tables.

Option B (product fix — simpler): In pg_class population, when compat mode is on, skip emitting pg_catalog stub tables that are empty (like pg_partitioned_table). Since CockroachDB doesn't support partitioning in the PostgreSQL sense, pg_partitioned_table will always be empty, and omitting it would prevent the collision.

Option C (test fix — workaround): Reduce the filler table count or add padding tables between the schema-only and data round-trip phases to ensure pgdump_data_target tables don't land on colliding descriptor IDs. This only papers over the issue.

Related Issues and PRs

Issue	Title	Relation
#171695	roachtest: pg_dump failed (master)	Same test, different failure mode — data_parent_pkey duplication due to descriptor lease race. Occurs on master with much higher frequency.
#171586	roachtest: pg_dump failed (master)	Prior data_parent_pkey duplication issue; partially fixed by waitForSchemaChangeJobs (commit 3327337a3c0).
#171551	roachtest: pg_dump failed	Different failure: "table is being added" — fixed by splitting DDL/DML (commit 06009b76c54).
47c62762b9c	sql: centralize and extend pg_catalog OID remapping	Introduced the OID remapping mechanism and filler table test. The remapping doesn't handle user-table-vs-catalog OID collisions.

Timeline

• 2026-04-05: Commit 47c62762b9c centralized OID remapping and added filler tables to the pg_dump test.
• 2026-06-09: Commit 3829d6cc862 added the data export round-trip test (pgDumpDataSchemaDDL/pgDumpDataSchemaDML), which creates additional tables that push descriptor IDs further into the collision range.
• 2026-06-18: This failure on release-26.3 at SHA 8a9a8e3305d — data_parent in pgdump_data_target gets descriptor ID 3350, colliding with pg_partitioned_table's remapped OID.

Detailed evidence and investigation notes

Log Excerpts

From test.log:

2026/06/18 12:44:08 pg_dump.go:1068: pg_dump (pgdump_data_target) stderr: pg_dump: error: unexpected column data for table "pg_partitioned_table"
2026/06/18 12:44:09 test_impl.go:486: test failure #1: full stack retained in failure_1.log: (pg_dump.go:1069).func1: COMMAND_PROBLEM: exit status 1

From failure_1.log:

stderr:pg_dump: error: unexpected column data for table "pg_partitioned_table"

From cockroach-sql-schema.log — the critical descriptor ID allocations:

# data_parent in pgdump_data_src gets DescriptorID 3279 (no collision)
DescriptorID:3279,"TableName":"pgdump_data_src.public.data_parent"

# data_parent in pgdump_data_target gets DescriptorID 3350 (COLLISION!)
DescriptorID:3350,"TableName":"pgdump_data_target.public.data_parent"

The pg_catalog OID remapping maps pg_partitioned_table to OID 3350:

// pkg/sql/sem/catconstants/pg_catalog_oids.go:54
PgCatalogPartitionedTableTableID: 3350,

The tableOid() function that exposes user table OIDs without collision protection:

// pkg/sql/pg_catalog.go:6712-6713
func tableOid(id descpb.ID) *tree.DOid {
    return tree.NewDOid(oid.Oid(id))
}

Code References

File	Lines	Description
pg_catalog_oids.go	54	PgCatalogPartitionedTableTableID: 3350 — the colliding OID
pg_catalog_oids.go	87-94	RemapPgCatalogOid — only remaps pg_catalog, not user tables
pg_catalog.go	6712-6713	tableOid() — raw descriptor ID cast, no collision avoidance
pg_catalog.go	1005-1007	relOid in pg_class — uses RemapPgCatalogOid for pg_catalog tables but raw ID for user tables
pg_dump.go	544-575	Filler table creation that pushes IDs into PG catalog OID range
pg_dump.go	1056-1069	runDump function where pg_dump fails

Things Ruled Out

1. Descriptor lease race / data_parent_pkey duplication (roachtest: pg_dump failed #171695): That failure mode produces a diff with duplicate PK constraints. This failure is pg_dump crashing entirely with "unexpected column data for table pg_partitioned_table" — a fundamentally different error.
2. "table is being added" race (roachtest: pg_dump failed #171551): Already fixed by commit 06009b76c54 splitting DDL/DML. The DDL/DML are separate in the code at this SHA.
3. Runtime assertions: The error is a pg_dump crash, not a CockroachDB assertion failure.
4. pg_partitioned_table schema mismatch: CockroachDB's stub pg_partitioned_table has 8 columns matching the PostgreSQL definition. The issue is not a column count mismatch in the virtual table itself, but pg_dump confusing data_parent's data with pg_partitioned_table.

Other Failure Occurrences

This is the only occurrence of this specific failure mode ("unexpected column data for table pg_partitioned_table"). All other recent pg_dump failures (#171695, #171586) have the data_parent_pkey duplication error, which is a separate issue.

This failure occurs on release-26.3 while the data_parent_pkey duplication failures occur on master, suggesting the specific descriptor ID allocation and thus the collision target differs between branches.

Tooling Feedback

• PostgreSQL source code (pg_dump.c) could not be fetched via WebFetch (file too large, returns 404). Being able to read the pg_dump source would have allowed confirming exactly how pg_dump handles the OID collision internally.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

Issue Classification

Field	Value
Classification	production-bug
Confidence	9/10
Scope	production-behavioral

Reasoning: The OID remapping in pg_dump_compatibility=postgres mode has a real collision bug: user tables whose raw descriptor IDs happen to equal a remapped pg_catalog OID appear twice in pg_class with the same OID, causing pg_dump to malfunction. The fix requires changing production code — either adding collision avoidance in tableOid() / pg_class population, or offsetting user table OIDs when they overlap with remapped catalog OIDs. This is not a test flake (the failure is deterministic given the descriptor ID state) and is distinct from all related issues which have different root causes.

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.pg_dump failed with artifacts on release-26.3 @ 769f38f9d3492a70416a81f5ed38c5f09f4b4b83:

Failed with:

(pg_dump.go:1069).func1: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/pg_dump/cpu_arch=fips/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21393548-1781849444-18-n1cpu4-0001	34.73.131.68	10.142.0.24

Parameters:

• arch=fips
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.pg_dump failed with artifacts on release-26.3 @ 9caafebcafcd887c18107a3a718e5a07f38938d0:

Failed with:

(pg_dump.go:1069).func1: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/pg_dump/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21394403-1781935825-72-n1cpu4-0001	34.73.219.216	10.142.1.102

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.pg_dump failed with artifacts on release-26.3 @ 8449e457346fdc20966c6f9acde86056a50dafd4:

Failed with:

(pg_dump.go:1069).func1: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/pg_dump/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21394889-1782022160-17-n1cpu4-0001	34.74.186.29	10.142.1.170

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.pg_dump failed with artifacts on release-26.3 @ 8449e457346fdc20966c6f9acde86056a50dafd4:

Failed with:

(pg_dump.go:1069).func1: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/pg_dump/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21395224-1782104923-137-n1cpu4-0001	34.24.169.58	10.142.1.113

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=default
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!