Pom refactor - Add Spark 3.5 connector by stevelordbq · Pull Request #152 · GoogleCloudDataproc/spark-spanner-connector

stevelordbq · 2026-01-20T07:24:31Z

Creates a separate build for each Spark version.
The approach was not chosen, however Jiang asked that we preserve this branch in case it is adopted later.

…vider acceptance error

…endencies between Spark versions.

…efactoring.

gemini-code-assist · 2026-01-20T07:26:15Z

Summary of Changes

Hello @stevelordbq, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the project's build system to improve modularity and facilitate multi-version Spark support. By introducing dedicated core and common Spark modules, it streamlines the process of adding new Spark connector versions. The primary outcome is the successful integration and support for Spark 3.5, complete with updated documentation and comprehensive test coverage.

Highlights

Spark 3.5 Connector Addition: Introduced a new connector specifically for Spark 3.5, expanding compatibility for the Spark Spanner connector.
Maven POM Refactoring: The project's Maven Project Object Model (POM) structure has been significantly refactored to support separate builds for each Spark version. This includes the introduction of a new root reactor POM.
Module Restructuring: Core, Spark-independent code has been moved into a new spanner-core module, and Spark-dependent but version-agnostic code has been moved into a spanner-spark-common module. This modularization simplifies maintenance and future Spark version upgrades.
Documentation Updates: The README.md file has been updated to reflect the new Spark 3.5 compatibility and include updated Dataproc image compatibility matrices.
Acceptance Test Enhancements: Acceptance tests have been updated and new test classes added across various Spark versions (3.1, 3.2, 3.3, 3.5) to align with the new module structure and ensure proper functionality for the newly added Spark 3.5 connector.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request is a significant refactoring of the project's build structure to support multiple Spark versions from a single codebase, including the addition of support for Spark 3.5. The changes introduce a modular approach with spanner-core, spanner-spark-common, and version-specific modules, which is a solid strategy. My review focuses on improving maintainability by reducing code duplication in both the build configuration and the source code, fixing some minor issues like wildcard imports and invalid SQL in test files, and suggesting best practices for logging and code formatting enforcement.

# Conflicts: # spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/Spark31SpannerTableProvider.java

jiangzzhu · 2026-01-20T22:14:46Z

/gcbrun

jiangzzhu · 2026-01-20T22:37:43Z

/gcbrun

jiangzzhu

Could you also update https://github.com/GoogleCloudDataproc/spark-spanner-connector/blob/main/.github/workflows/dev-ci.yml to build 3.5 as well? thanks!

jiangzzhu · 2026-01-21T21:55:36Z

Could you also update https://github.com/GoogleCloudDataproc/spark-spanner-connector/blob/main/.github/workflows/dev-ci.yml to build 3.5 as well? thanks!

And also

spark-spanner-connector/cloudbuild/presubmit.sh

Line 26 in 68c66ee

$MVN install -DskipTests -P3.1,3.2,3.3

jiangzzhu · 2026-01-22T02:52:03Z

-| spark-3.2-spanner          |         |         |         | &check; | &check; | &check; | &check;                 |                         | &check;                 |                         |
-| spark-3.3-spanner          |         |         |         | &check; | &check; | &check; | &check;                 |                         | &check;                 |                         |
-| spark-3.5-spanner          |         |         |         |         |         | &check; |                         | &check;                 |                         | &check;                 |
+| Connector \ Dataproc Image | 1.3     | 1.4     | 1.5     | 2.0     | 2.1     | 2.2     | Serverless<br>Image 1.1 | Serverless<br>Image 1.2 | Serverless<br>Image 2.0 | Serverless<br>Image 2.1 | Serverless<br>Image 2.2 |


Why does spark 3.1,3.2,3.3 not work with serverless 1.2 and 2.2?

We haven't tested that those combinations work. So it may work.
https://docs.cloud.google.com/dataproc-serverless/docs/concepts/versions/spark-runtime-1.2

Did you want us to increase the cross version compatibility testing?

If adding the test is simple (by acceptance test), maybe we could add it, so we could check on those entries.
Or we could say it is supported but not tested, to avoid confusion that it is not supported. What do you think?

jiangzzhu · 2026-01-22T04:13:29Z

A few questions on the current PR.

Before the PR, we can execute mvn clean install -P3.1,3.2,3.3 -DskipTests successfully. After the PR, we will encounter the following errors when executing the same cmd:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:testCompile (default-testCompile) on project spark-3.3-spanner-lib: Compilation failure
[ERROR] /usr/local/google/home/jiangzzhu/Projects/spark-spanner-connector/spark-3.3-spanner-lib/src/test/java/com/google/cloud/spark/spanner/SpannerDataWriter33IntegrationTest.java:[26,22] cannot find symbol
[ERROR] symbol: method apply(org.apache.spark.sql.types.StructType)
[ERROR] location: class org.apache.spark.sql.catalyst.encoders.RowEncoder

Could you fix this? Otherwise, the release process will fail. Thanks!

When executing mvn install -P3.1,3.2,3.3,3.5,integration failsafe:integration-test, encounter the following errors:

[ERROR] WriteIntegrationTest.testEmptyDataFrameWrite:140 » SparkClassNotFound [DATA_SOURCE_NOT_FOUND] Failed to find the data source: cloud-spanner. Please find packages at [https://spark.apache.org/third-party-projects.html](https://spark.apache.org/third-party-projects.html%60).

Please also take a look at this. It also breaks the release process. Thanks!

Please also verify if the following acceptance tests work or not. Thanks!
mvn install -P3.1,3.2,3.3,3.5,acceptance failsafe:integration-test

stevelordbq · 2026-01-22T18:06:36Z

A few questions on the current PR.

Before the PR, we can execute mvn clean install -P3.1,3.2,3.3 -DskipTests successfully. After the PR, we will encounter the following errors when executing the same cmd:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:testCompile (default-testCompile) on project spark-3.3-spanner-lib: Compilation failure [ERROR] /usr/local/google/home/jiangzzhu/Projects/spark-spanner-connector/spark-3.3-spanner-lib/src/test/java/com/google/cloud/spark/spanner/SpannerDataWriter33IntegrationTest.java:[26,22] cannot find symbol [ERROR] symbol: method apply(org.apache.spark.sql.types.StructType) [ERROR] location: class org.apache.spark.sql.catalyst.encoders.RowEncoder

Could you fix this? Otherwise, the release process will fail. Thanks!

When executing mvn install -P3.1,3.2,3.3,3.5,integration failsafe:integration-test, encounter the following errors:

[ERROR] WriteIntegrationTest.testEmptyDataFrameWrite:140 » SparkClassNotFound [DATA_SOURCE_NOT_FOUND] Failed to find the data source: cloud-spanner. Please find packages at [https://spark.apache.org/third-party-projects.html](https://spark.apache.org/third-party-projects.html%60).

Please also take a look at this. It also breaks the release process. Thanks!

Please also verify if the following acceptance tests work or not. Thanks!
mvn install -P3.1,3.2,3.3,3.5,acceptance failsafe:integration-test

Please note @jiangzzhu , now we have separate builds for each Spark version, it doesn't make sense to aggregate the profiles together, since that would use the Spark version set by the last profile, which would be Spark 3.3 or 3.5 in your examples above.

However, I will investigate this failure for a single Spark version.

…ovider context

stevelordbq added 19 commits January 13, 2026 14:48

Add libraries for Spark 3.5

4d6c8cb

Remove hard coding of jar names/paths

aeb248a

Remove Spark 3.5 Spanner Table Provider as causing multiple Table Pro…

1b73333

…vider acceptance error

Move Spark dependencies to Spark version specific modules. Remove dep…

cde85f2

…endencies between Spark versions.

Make Spanner Table options case insensitive.

ce92171

Refactored Maven project to change parent pom ready for Spark build r…

ada76f5

…efactoring.

Add spanner-spark-common to dependencies of Spark 3.2, 3.3 and 3.5

edc8bf5

Rationalize Spark 3.1

99b14b7

Rationalize Spark 3.1 - unit/integration tests pass

9f03990

Spark 3.2 rationalize code duplication

135a83a

Spark 3.3 rationalize code duplication

a2fc9e4

Spark 3.5 add Spark dependency

f61bd66

Spark 3.5 rationalization of code duplication

5c0ec34

Add Spark 3.2, 3.3 specific tests

8b3bf44

Update maven-shade-plugin for 3.5

780d367

Update maven-shade-plugin for 3.1 & 3.5

0af1917

Update maven-shade-plugin for 3.2 & 3.3

ca7830b

Update README for Spark 3.5

6be6bea

Add Spanner Table properties case insensitivity to each Spark version.

2ea346e

stevelordbq closed this Jan 20, 2026

stevelordbq reopened this Jan 20, 2026

gemini-code-assist Bot reviewed Jan 20, 2026

View reviewed changes

stevelordbq added 3 commits January 20, 2026 13:40

PR review updates

f0f84e4

Merge branch 'main' into pom-refactor

88e20c4

# Conflicts: # spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/Spark31SpannerTableProvider.java

Merge from main to resolve merge conflicts

2026367

Move revision property back to top level pom.xml

5bc1d84

jiangzzhu self-assigned this Jan 21, 2026

jiangzzhu requested review from davidrabinowitz and jiangzzhu January 21, 2026 17:28

jiangzzhu removed their assignment Jan 21, 2026

jiangzzhu reviewed Jan 21, 2026

View reviewed changes

Comment thread README.md

Comment thread spanner-spark-common/src/test/resources/acceptance/read_test_table.py Outdated

PR updates

f5b2e51

jiangzzhu reviewed Jan 21, 2026

View reviewed changes

Comment thread README-template.md

stevelordbq added 2 commits January 21, 2026 17:02

Clean up unused resources

19fd72c

PR update to README-template.md

36a0c40

jiangzzhu reviewed Jan 22, 2026

View reviewed changes

Comment thread README.md

Rename tests to end with IntegrationTest if they are integration tests

7e5f033

Subclass integration tests so they run in the correct *SpannerTablePr…

28c371c

…ovider context

Uh oh!

Conversation

stevelordbq commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Jan 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiangzzhu commented Jan 20, 2026

Uh oh!

jiangzzhu commented Jan 20, 2026

Uh oh!

jiangzzhu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jiangzzhu commented Jan 21, 2026

Uh oh!

Uh oh!

jiangzzhu Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

stevelordbq Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

jiangzzhu Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jiangzzhu commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevelordbq commented Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stevelordbq commented Jan 20, 2026 •

edited

Loading

jiangzzhu commented Jan 22, 2026 •

edited

Loading