[Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 #5616

allisonport-db · 2025-12-02T21:53:34Z

Which Delta project/connector is this regarding?

Description

PART OF #5326

Contains the following changes:

Removes Spark 3.5 support
Adds explicit Spark 4.0 support
Removes a "master" build for now
Merges shims from the 3.5 vs 4.0 breaking changes into the src code

In a future PR

we will add Spark 4.1.0-SNAPSHOT support (in preparation for the Spark 4.1 release)
we will add back a "master" build tracking Spark master
(these will require adding new shims, but in different areas)

How was this patch tested?

Unit tests + ran integration tests locally (python, scala + pip)

Tracking open TODOs at #5326

This reverts commit 8fd880f.

allisonport-db · 2025-12-05T23:08:41Z

In theory I could do the src code shims + test code shims separately if that would help. Let me know if that makes reviews easier (not sure if anyone wants to review the shim code changes closely, or if tests pass & code compiles that's enough).

kernel-spark/src/main/java/io/delta/kernel/spark/catalog/SparkTable.java

.github/workflows/kernel_test.yaml

scottsand-db · 2025-12-06T00:45:47Z

.github/workflows/spark_master_test.yaml

@@ -1,59 +0,0 @@
-name: "Delta Spark Master"


I suggest we update the PR title to say drop support for spark 3.5 and spark master compilation ?

I mean we haven't actually been compiling with spark master in a while... (as we're using a very stale snapshot). But I can make the title more clear

Hm. Sorry, I'm still confused. Here we are deleting our job to compile against spark "master" right? (perhaps it was a stale master ..)

But does Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 reflect that?

That seems like an important highlight, sorry, and I want to make sure my understanding is correct

I think calling it spark master before was misleading, in fact, in the previous PR we renamed the spark version spec to spark40Snapshot instead of master. I think saying we are removing spark master is misleading considering we never were compiling with Spark master. We will be fixing that in future PRs.

It would be more correct to say spark_master_test.yaml was incorrectly named this whole time.

should we remove this action completely? eventually we will be addng back spark master support. then what should we be doing .. finding it from history and adding it back?
alternative is to just make this a no-op in some way

yeah for iceberg action I just made it a no-op.

I'm fine with just removing this though since this work i'm tracking directly and will add it back (need to update the CI in a few ways for multi-versions anyways)

setup.py

allisonport-db · 2025-12-08T23:17:42Z

project/SparkMimaExcludes.scala

+
+      // Changes in 4.1.0
+      // TODO: change in type hierarchy due to removal of DeltaThrowableConditionShim
+      ProblemFilters.exclude[MissingTypesProblem]("io.delta.exceptions.*")


@reviewers this seems safe to me, considering no one should be catching DeltaThrowableConditionShim... but would like additional opinions

allisonport-db · 2025-12-08T23:21:03Z

build.sbt

  ).configureUnidoc()

+/*
+TODO: readd delta-iceberg on Spark 4.0+


@lzlfred Hey Fred, we will be releasing in on both Spark 4.0 and Spark 4.1 next release, we will need to update this build to work for that

also tracking the todo at #5326

allisonport-db · 2025-12-09T00:48:00Z

build.sbt

  ).configureUnidoc()

+/*
+TODO: compilation broken for Spark 4.0


tracking at #5326

@linzhou-db @littlegrasscao FYI can you please look into fixing this once I merge this PR

allisonport-db · 2025-12-09T00:48:47Z

examples/scala/build.sbt

 val lookupSparkVersion: PartialFunction[(Int, Int), String] = {
-  // version 4.0.0-preview1
-  case (major, minor) if major >= 4 => "4.0.0-preview1"
+  // TODO: how to run integration tests for multiple Spark versions


tracking at #5326

allisonport-db · 2025-12-09T00:49:03Z

setup.py

 with open("python/README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

+# TODO: once we support multiple Spark versions update this to be compatible with both


tracking at #5326

raveeram-db

build changes look good to me overall

tdas · 2025-12-09T19:48:45Z

.github/workflows/kernel_unitycatalog_test.yaml

        uses: actions/setup-java@v3
        with:
          distribution: "zulu"
-          java-version: "11"


general question @scottsand-db why is the kernel unitycatalog is a separate github action from kernel?

tdas · 2025-12-09T19:50:26Z

...nel-defaults/src/test/scala/io/delta/kernel/defaults/engine/DefaultParquetHandlerSuite.scala


      // Try to write as same file and expect an error
-      intercept[FileAlreadyExistsException] {
+      val e = intercept[IOException] {


why is remove spark 3.5 causing all these kernel code changes?

I upgraded our hadoop version to match Spark 4.0. More details on #5616 (comment)

Kernel changes are only related to this.

tdas · 2025-12-09T20:00:40Z

lets merge this quick. the java version is adding all sort of complications in a different place.

…d Spark 4.0.1 (delta-io#5616)  #### Which Delta project/connector is this regarding?  - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description PART OF delta-io#5326 Contains the following changes: - Removes Spark 3.5 support - Adds explicit Spark 4.0 support - Removes a "master" build for now - Merges shims from the 3.5 vs 4.0 breaking changes into the src code In a future PR - we will add Spark 4.1.0-SNAPSHOT support (in preparation for the Spark 4.1 release) - we will add back a "master" build tracking Spark master (these will require adding new shims, but in different areas) ## How was this patch tested? Unit tests + ran integration tests locally (python, scala + pip) Tracking open TODOs at delta-io#5326

allisonport-db added 16 commits December 2, 2025 13:48

Initial build changes

3501613

remove 3.5 shims

77d305b

Remove LoggingShims and DeltaSqlParserShims

83ebe6b

Variant shim

90e1e1f

DecimalPrecisionTypeCoercionShims

2a0f940

LogKeyShims

86fea31

Some more shims

20f72f3

TypeWideningShim

77a9a42

MergeIntoMaterializeSourceShims

13ac618

Throwable, TimeTravelSpec, LogicalRelation

5abacfa

DeltaInvariantCheckerExecShims

a8c2804

Misc

e3dd2b9

Misc shims

ec34fd8

Remove connectors

8fd880f

Update test_cross_spark_publish.py

70effca

Revert "Remove connectors"

11cec06

This reverts commit 8fd880f.

allisonport-db changed the title ~~[Spark][Infra][WIP] Drop support for Spark 3.5 in master~~ [Spark][Infra] Drop support for Spark 3.5 in master Dec 3, 2025

allisonport-db added 4 commits December 2, 2025 18:03

Remove shim dir from build

73deabf

add default

c9dc7fa

drop iceberg for now

36b8472

Misc job fixes

9e41fec

allisonport-db requested review from raveeram-db, scottsand-db and tdas December 3, 2025 02:40

allisonport-db added 6 commits December 2, 2025 19:33

Fix kernel tests

ee907b7

Fix mima for now

73bea33

Type widening shims

6aa8f58

SnapshotManagementSuiteShims

1c58630

DeltaGenerateSymlinkManifestSuiteShims

6bb2c54

DeltaHistoryManagerSuiteShims

d1f8616

raveeram-db requested a review from TimothyW553 December 5, 2025 23:12

Merge remote-tracking branch 'delta-io/master' into update-build-1

4bc7d73

scottsand-db reviewed Dec 6, 2025

View reviewed changes

kernel-spark/src/main/java/io/delta/kernel/spark/catalog/SparkTable.java Outdated Show resolved Hide resolved

scottsand-db reviewed Dec 6, 2025

View reviewed changes

.github/workflows/kernel_test.yaml Outdated Show resolved Hide resolved

scottsand-db reviewed Dec 6, 2025

View reviewed changes

Fix scalastyle

a6fd612

allisonport-db changed the title ~~[Spark][Infra] Drop support for Spark 3.5 in master~~ [Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 Dec 6, 2025

scottsand-db reviewed Dec 6, 2025

View reviewed changes

setup.py Outdated Show resolved Hide resolved

allisonport-db added 2 commits December 8, 2025 13:16

Fix sharing test compile

0de65e8

Merge remote-tracking branch 'delta-io/master' into update-build-1

8d6d117

allisonport-db commented Dec 8, 2025

View reviewed changes

allisonport-db added 2 commits December 8, 2025 15:27

Update setup.py

4dbad62

Skip iceberg test for now

f8c5c3f

allisonport-db mentioned this pull request Dec 9, 2025

[Infra][Feature Request] Upgrade master branch to support Spark 4.0+ and remove remaining standalone-based connectors #5326

Open

5 tasks

Fixes

264e629

allisonport-db requested a review from scottsand-db December 9, 2025 00:46

allisonport-db commented Dec 9, 2025

View reviewed changes

raveeram-db approved these changes Dec 9, 2025

View reviewed changes

Update test_cross_spark_publish.py

ae7363c

tdas reviewed Dec 9, 2025

View reviewed changes

tdas approved these changes Dec 9, 2025

View reviewed changes

Move comments in kernel_test

49ff48c

allisonport-db merged commit db826cf into delta-io:master Dec 9, 2025
14 checks passed

[Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 #5616

[Spark][Infra] Drop support for Spark 3.5 and formally pin to released Spark 4.0.1 #5616

Uh oh!

Conversation

allisonport-db commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Uh oh!

allisonport-db commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

allisonport-db Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raveeram-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

allisonport-db Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas commented Dec 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

allisonport-db commented Dec 2, 2025 •

edited

Loading

allisonport-db Dec 9, 2025 •

edited

Loading

allisonport-db Dec 9, 2025 •

edited

Loading