Proposal: REPLACE TABLE for the Hive Metastore catalog

[ErikBPF]

Problem

The Hive Metastore catalog returns NotSupported for CREATE OR REPLACE / REPLACE TABLE.

Proposal

alter_table (single Thrift call) under an exclusive table lock — no drop+create data-loss window. Partition keys are carried forward (a requested change errors); a metadata_location precondition is re-checked under the lock for lakehouse tables.

Open questions for maintainers

1. Is alter_table-under-lock acceptable given HMS has no CAS primitive? The residual race for plain (non-lakehouse) tables is documented — tolerable?
2. Erroring on a requested partition change (vs silently keeping the old spec) — correct behavior?

[linhr]

Using alter_table for REPLACE TABLE doesn't seem right to me if we cannot replace partitions. I feel we should allow the new table to be completely unrelated to the original table.

How does Spark handle this if HMS doesn't have CAS primitives? Could you do some research?

[ErikBPF]

Follow-up with the research you asked for. Spark's CREATE OR REPLACE TABLE is atomic-or-nothing only when the catalog implements StagingTableCatalog; otherwise Spark itself does a non-atomic dropTable+createTable (its own javadoc notes the "cannot roll back the drop" risk). Hive Metastore 4.0.0 offers no atomic table-swap primitive — rename-over-existing throws, alter_table can't change partition keys, and metastore DDL isn't transactional — so drop+create is the only mechanism, matching Spark's non-staging path.

I've hardened it to atomic-or-nothing as far as HMS allows: under the table lock, the replacement is pre-validated before the drop (client-detectable errors like duplicate columns fail with the original intact), and if the create still fails the original is restored from a snapshot captured under the lock (only a restore-failure surfaces a "recover manually" error). Partition/schema redefinition works (fresh table, per your suggestion); owner is carried forward. Verified live on apache/hive:4.0.0 (10 tests, incl. a create-failure regression asserting the original survives).