Setup Utils for CCv2 tables #5477

TimothyW553 · 2025-11-10T23:46:28Z

🥞 Stacked PR

Use this link to review incremental changes.

catalogtableutils-ccv2 [Files changed]
- stack/ccv2-uc-utils [Files changed]
  - stack/snapshotmanager-wireframe [Files changed]
    - stack/snapshotmanager-implementation [Files changed]
      - stack/snapshotmanager-factory-wireup [Files changed]

Which Delta project/connector is this regarding?

Description

This PR adds utils for CatalogTable, Scala, and for catalogtable testing. In particular, CatalogTableUtils is used for determining if a table is managed/owned by UC -- which will determine the source of truth for operations.

How was this patch tested?

tested locally via build/sbt -DsparkVersion=master "++ 2.13.16" clean sparkV2/test
passing CI tests

Does this PR introduce any user-facing changes?

No.

huan233usc · 2025-11-12T18:41:20Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+
+  private CatalogTableUtils() {}
+
+  public static boolean isCCv2Table(CatalogTable table) {


Let's call it catalogOwned/Managed

Sounds good, this will also help align with the existing terminology!

huan233usc · 2025-11-12T18:44:28Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+
+  private static boolean isCatalogOwnedFeatureSupported(
+      Map<String, String> tableProperties, String featureKey) {
+    if (tableProperties == null) {


Can we just check tableProperties non null?

huan233usc · 2025-11-12T18:46:47Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+ * properties.
+ */
+public final class CatalogTableUtils {
+  static final String UNITY_CATALOG_PROPERTY_PREFIX = "delta.unityCatalog.";


Do we really put this properties?

This specific property I thought existed but actually does not, changed it now to UC_TABLE_ID_KEY which is ucTableId right now - I believe it is changing to catalogManaged.unityCatalog.tableId

huan233usc · 2025-11-14T00:59:27Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+  // if the catalogManaged/catalogOwned-preview flags are 'supported'
+  public static boolean isCatalogManaged(CatalogTable table) {
+    requireNonNull(table, "table is null");
+    Map<String, String> tableProperties = toJavaMap(table.properties());


We should look at table.storage.properties() storage's class name is CatalogStorageFormat, I think delta(which is a stoarg)'s related properties should like here if upstream implements the semantic correctly, like https://github.com/unitycatalog/unitycatalog/blob/main/connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala#L279

I currently merge both outer properties() and storage().properties() in case. I looked into AbstractDeltaCatalog, it seems we will have to merge them to get all desired properties.

huan233usc · 2025-11-14T01:00:44Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+    return featureValue.equalsIgnoreCase(SUPPORTED);
+  }
+
+  private static Map<String, String> toJavaMap(


Put it in the ScalaUtils.java

Gotcha, moved to ScalaUtils

huan233usc · 2025-11-17T21:00:17Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+
+  private CatalogTableUtils() {}
+
+  // Checks whether *any* catalog manages this table via CCv2 semantics by checking


nit: please use
/**
*

@param xxxx
*/
for method documentation

Sounds good, updated!

huan233usc · 2025-11-17T21:01:42Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+    requireNonNull(table, "table is null");
+    Map<String, String> merged = new HashMap<>();
+    merged.putAll(ScalaUtils.toJavaMap(table.storage().properties()));
+    merged.putAll(ScalaUtils.toJavaMap(table.properties()));


can we just add table.storage().properties()?

huan233usc · 2025-11-17T21:05:11Z

kernel-spark/src/test/java/io/delta/kernel/spark/utils/CatalogTableUtilsTest.java

+  /**
+   * Creates a CatalogTable with the given properties. This is a helper method to create a
+   * CatalogTable for testing purposes - see interface {@link CatalogTable} for more details.
+   *
+   * @param properties the properties to set on the CatalogTable
+   * @return a CatalogTable with the given properties
+   */
+  private static CatalogTable catalogTableWithProperties(Map<String, String> properties) {
+    return catalogTableWithProperties(properties, Collections.emptyMap());
+  }
+
+  private static CatalogTable catalogTableWithProperties(
+      Map<String, String> properties, Map<String, String> storageProperties) {
+    return CatalogTableTestUtils$.MODULE$.catalogTableWithProperties(properties, storageProperties);
+  }


maybe not needed? I feel like these two utils methods are a bit verbose

Yeah the first one with just properties() is not needed and verbose I agree. I'll keep the one with two parameters as it makes the testing code cleaner.

huan233usc · 2025-11-17T21:07:13Z

kernel-spark/src/test/java/io/delta/kernel/spark/utils/CatalogTableUtilsTest.java

+class CatalogTableUtilsTest {
+
+  @Test
+  void catalogManagedFlagEnablesDetection() {


can we name the test case like
testIsCatalogManaged_CatalogManagedEnable_returnsTrue

testIntent_condition_expectation.

Also split

assertTrue(
CatalogTableUtils.isCatalogManaged(table), "Should detect catalog management with flag");
assertFalse(
CatalogTableUtils.isUnityCatalogManagedTable(table), "Should not detect Unity without ID");
to keep unit test focused.

Ah, thanks for the test case naming review. I'll keep that in mind

huan233usc · 2025-11-17T23:29:29Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+   * @param table Spark {@link CatalogTable} descriptor
+   * @return Java map view of the storage properties
+   */
+  public static Map<String, String> getStorageProperties(CatalogTable table) {


can be private method?

huan233usc · 2025-11-17T23:29:35Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+    return ScalaUtils.toJavaMap(table.storage().properties());
+  }
+
+  public static boolean isCatalogManagedFeatureEnabled(


Updated to private.

huan233usc · 2025-11-17T23:30:27Z

kernel-spark/src/test/scala/io/delta/kernel/spark/utils/CatalogTableTestUtils.scala

+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.types.StructType
+
+/** Helpers for constructing [[CatalogTable]] instances inside Java tests. */


Document why scala is needed. CatalogTable may accept different params on different spark version?

Yeah added docs on why scala is needed. To summarize, we need it because if we were to construct it in Java, then we need to fill even the optional parameters in. By constructing in scala we dont need to worry about new optional parameters.

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

huan233usc

LGTM

gengliangwang · 2025-11-18T00:50:58Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+   */
+  public static boolean isCatalogManaged(CatalogTable table) {
+    requireNonNull(table, "table is null");
+    Map<String, String> storageProperties = getStorageProperties(table);


QQ: is storageProperties always not null? Given the method returns a boolean, shall we simply return false if storageProperties is null?

Yep! that's a good idea. I've updated it so that if the table.storage().properties() is empty we return an emptyMap (less code, instead of checking for null each time) the empty map will result in a false anyway.

gengliangwang · 2025-11-18T00:53:50Z

kernel-spark/src/test/scala/io/delta/kernel/spark/utils/CatalogTableTestUtils.scala

why adding a scala file?

The reason I added scala file was for the CatalogTable construction. Construction the CatalogTable in java required me to explicitly pass in all parameters including the optional ones. If we add more optional parameters in CatalogTable constructor then it will also require changing the test here.

Signed-off-by: TimothyW553 <[email protected]>

Use this [link](https://github.com/delta-io/delta/pull/5477/files) to review incremental changes. - [**catalogtableutils-ccv2**](delta-io#5477) [[Files changed](https://github.com/delta-io/delta/pull/5477/files)] - [stack/ccv2-catalog-config](delta-io#5520) [[Files changed](https://github.com/delta-io/delta/pull/5520/files/6359268dbee8d1a114e3f66620c6585bc0bdb6eb..4a1d8fa93e56d68b5971fb32970bdeaa5799abdc)] ---------   - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) This PR adds utils for CatalogTable, Scala, and for catalogtable testing. In particular, CatalogTableUtils is used for determining if a table is managed/owned by UC -- which will determine the source of truth for operations.  - tested locally via `build/sbt -DsparkVersion=master "++ 2.13.16" clean sparkV2/test` - passing CI tests  No.  --------- Signed-off-by: TimothyW553 <[email protected]> Signed-off-by: Timothy Wang <[email protected]>

scottsand-db · 2025-12-10T21:51:21Z

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java

+
+/**
+ * Utility helpers for inspecting Delta-related metadata persisted on Spark {@link CatalogTable}
+ * instances by Unity Catalog.


I don't think we should assume or default to Unity - eh?

huan233usc reviewed Nov 12, 2025

View reviewed changes

TimothyW553 marked this pull request as ready for review November 14, 2025 00:44

huan233usc reviewed Nov 14, 2025

View reviewed changes

TimothyW553 requested a review from huan233usc November 17, 2025 04:43

TimothyW553 mentioned this pull request Nov 17, 2025

[Kernel] Add Unity Catalog client factory for CCv2 #5503

Closed

5 tasks

huan233usc reviewed Nov 17, 2025

View reviewed changes

TimothyW553 force-pushed the catalogtableutils-ccv2 branch from 9e3af8f to 4da2a45 Compare November 17, 2025 21:33

This was referenced Nov 17, 2025

Setup UC Client Factory #5504

Closed

CCv2 CatalogManagedSnapshotManager #5506

Closed

Create Snapshot Manager Factory #5507

Closed

huan233usc reviewed Nov 17, 2025

View reviewed changes

kernel-spark/src/main/java/io/delta/kernel/spark/utils/CatalogTableUtils.java Show resolved Hide resolved

huan233usc approved these changes Nov 17, 2025

View reviewed changes

huan233usc requested a review from raveeram-db November 17, 2025 23:31

TimothyW553 force-pushed the catalogtableutils-ccv2 branch from a523d7d to 57d813a Compare November 18, 2025 00:18

huan233usc requested a review from gengliangwang November 18, 2025 00:18

huan233usc mentioned this pull request Nov 18, 2025

SC-211135: Implement analyzer rule for V2 streaming reads #5475

Draft

5 tasks

TimothyW553 force-pushed the catalogtableutils-ccv2 branch from eefc8dc to 57d813a Compare November 18, 2025 00:23

gengliangwang reviewed Nov 18, 2025

View reviewed changes

TimothyW553 requested a review from gengliangwang November 18, 2025 03:42

TimothyW553 force-pushed the catalogtableutils-ccv2 branch from 4335df1 to ae14d13 Compare November 18, 2025 03:54

TimothyW553 added 15 commits November 21, 2025 06:26

storage props

b2b4428

Signed-off-by: TimothyW553 <[email protected]>

Add CatalogTable test helper

0832138

merge props

5830196

Signed-off-by: TimothyW553 <[email protected]>

address comments

8b5c962

fmt

4463c3e

address second round comments

fed1e0a

add reason for scala usage

f88f46a

remove private method test

aab04bb

null check for storage properties

ec1a35b

fmt

1b1fe59

address comments

6686a97

address comments

26ad1cd

constants

3f3cd9c

space

f4d4f13

fmt

6359268

TimothyW553 force-pushed the catalogtableutils-ccv2 branch 2 times, most recently from 6605534 to 6359268 Compare November 24, 2025 23:25

zachschuermann merged commit dc92aa2 into delta-io:master Nov 25, 2025
40 checks passed

TimothyW553 mentioned this pull request Nov 25, 2025

Wire SparkTable to snapshot manager factory #5580

Draft

5 tasks

This was referenced Dec 9, 2025

[Kernel-Spark][DSv2] Wireframe for UCManagedSnapshotManager #5655

Open

[Kernel-Spark][DSv2] Implement UCManagedSnapshotManager operations using UCCatalogManagedClient #5656

Draft

[Kernel-Spark][DSv2] Add CCv2 routing to SparkTable #5660

Draft

scottsand-db reviewed Dec 10, 2025

View reviewed changes

This was referenced Dec 11, 2025

[Kernel-Spark][DSv2] Implement UCManagedSnapshotManager operations using UCCatalogManagedClient #5677

Draft

[Kernel-Spark][DSv2] Add CCv2 routing to SparkTable and SnapshotManagerFactory #5678

Draft


		private CatalogTableUtils() {}

		public static boolean isCCv2Table(CatalogTable table) {


		private CatalogTableUtils() {}

		// Checks whether any catalog manages this table via CCv2 semantics by checking

Setup Utils for CCv2 tables #5477

Setup Utils for CCv2 tables #5477

Conversation

TimothyW553 commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

huan233usc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

TimothyW553 commented Nov 10, 2025 •

edited

Loading

TimothyW553 Nov 17, 2025 •

edited

Loading

TimothyW553 Nov 18, 2025 •

edited

Loading