[WIP] [BUILD] Add Gradle build support and replace Maven in CI by liuneng1994 · Pull Request #11576 · apache/gluten

liuneng1994 · 2026-02-06T06:55:12Z

What changes are proposed in this pull request?

Add a complete Gradle build system that coexists with the existing Maven build. CI workflows for Spark UT (velox_backend_x86.yml, velox_backend_enhanced.yml) remain on Maven.

Gradle Build System:

Root Gradle configuration with version catalog (libs.versions.toml)
Convention plugins for common patterns (Scala, shading, native, spotless)
All core modules with proper dependencies
Multi-version support (Spark 3.3-4.1, Scala 2.12/2.13, Java 8-21)
Backend selection (Velox, ClickHouse) via -Pbackend property
Native C++ build integration via Gradle tasks (full Velox build pipeline)
Dependency shading with Shadow plugin
ScalaTest integration for running non-ut test suites via Gradle
Common Spark and test dependencies centralized in convention plugins

Default configuration: Spark 4.1, Scala 2.13, Java 17, Velox backend

Spark UT: Gradle Limitations (why Maven is kept for gluten-ut)

Running gluten-ut tests via Gradle was attempted with a per-suite JVM forking approach (ScalaTestPerSuiteAction) but failed due to the following issues:

LOCATION_ALREADY_EXISTS errors from shared spark-warehouse directories: Parallel suite JVMs all resolve getClass.getResource("/") to the same test-classes dir, causing Spark's managed table creation to fail with warehouse directory collisions. Even with per-suite working directories (gluten.test.workDir), the classpath-based resource resolution in GlutenTestsBaseTrait.basePath makes true isolation difficult.
Excessive memory consumption: Per-suite JVM forking with the required 4g heap per JVM scales linearly (e.g., 6 forks × 4g = 24GB+), exceeding CI runner memory limits. Reducing parallelism defeats the purpose of per-suite forking.
Spark's JVM-global state leaking: Spark uses JVM-global singletons (SparkContext, daemon threads, native library loading) that leak between suites in a shared JVM. The gradle-scalatest plugin's default Runner -R <dir> mode runs all suites in one JVM, causing failures when one suite's SparkContext is not properly stopped before the next suite starts.
Tag filtering complexity: ScalaTest tag annotations are class-level but per-test Tag objects (e.g., test("name", MyTag)) only produce runtime-visible metadata, requiring constant pool scanning heuristics for class-level pre-filtering that is fragile and incomplete.

Maven's scalatest-maven-plugin handles these issues with battle-tested process isolation, making it the pragmatic choice for gluten-ut. Non-ut modules (backends-velox, gluten-ras, shims, etc.) continue to run tests via Gradle successfully.

Usage examples:

./gradlew build                              # Default build
./gradlew build -PsparkVersion=3.5           # Spark 3.5
./gradlew build -Pbackend=clickhouse         # ClickHouse backend
./gradlew build -Pdelta=true -Piceberg=true  # With optional modules

# Run specific test suites (non-ut modules)
./gradlew :backends-velox:test --tests "org.apache.spark.sql.execution.VeloxParquetReadSuite"

# CI helper script (translates Maven args to Gradle)
bash dev/ci-gradle-build.sh -Pspark-3.5 -Pjava-8 -Pbackends-velox -DskipTests

# Native C++ build (replaces dev/builddeps-veloxbe.sh)
./gradlew :backends-velox:buildNativeAll     # Full pipeline: fetch + build Velox + build Gluten C++
./gradlew :backends-velox:buildNative        # Just rebuild Gluten C++ (if Velox already built)

# Native build with options
./gradlew :backends-velox:buildNativeAll -PnativeBuildType=Debug -PenableS3=ON -PnativeBuildTests=ON

Native build Gradle tasks

Task	Description
`getVelox`	Fetch Velox source code (skips if already present)
`buildVelox`	Build the Velox C++ library (skips if already built)
`configureNative`	Configure CMake for Gluten C++ (skips if already configured)
`buildNative`	Build libgluten.so + libvelox.so
`buildNativeAll`	Full pipeline: all of the above

Configurable properties: -PveloxHome=, -PnativeBuildType=Debug|Release, -PnativeThreads=N, -PenableHdfs=ON, -PenableS3=ON, -PenableGcs=ON, -PenableAbfs=ON, -PenableQat=ON, -PenableGpu=ON, -PnativeBuildTests=ON, -PnativeBuildBenchmarks=ON

Key Gradle build fixes (Maven compatibility)

Several fixes were needed to match Maven's behavior in Gradle:

Issue	Root Cause	Fix
`No Spark Shim Provider found` when switching Spark versions	`generateBuildInfo` task had no declared inputs; Gradle cached stale `gluten-build-info.properties`	Added `inputs.property()` declarations for sparkVersion, scalaVersion, etc.
`VeloxParquetWriteSuite` failures in Spark 3.3	Shim's `FileFormatWriter` not loaded before Spark's own version	Reorder test classpath: project outputs before external JARs
`VeloxExpandSuite` NPE in CI	Leaked SparkContext from prior suite in shared JVM	`forkEvery = 1` — fork new JVM per test class (matches Maven)
Symbol literal deprecation errors in Scala 2.13	Scala 2.13 `-Wconf` rules not matching Maven config	Added `-Wconf:cat=deprecation&msg=symbol literal is deprecated:s`
Assertion failures in Spark's `UnsafeRow`	Gradle enables assertions by default; Maven does not	`enableAssertions = false` in test configuration
Test resource path mismatches	Gradle separates classes/resources; Maven merges them	Redirect test resources to Scala classes directory
ARM native lib `FileNotFoundException`	Gradle mapped `aarch64` → `arm64`; JNI loader uses raw `aarch64`	Changed Gradle mapping to `aarch64` to match Maven and JNI
Checkstyle `${config_loc}` not set in Maven	Maven's checkstyle plugin doesn't auto-set `${config_loc}` like Gradle does	Added `<propertyExpansion>` to pom.xml; Gradle uses `configDirectory`

How was this patch tested?

Compilation verified across Spark 3.3, 3.4, 3.5, 4.0, 4.1 with corresponding Scala/Java versions
Full CI run on all three Velox workflows (x86, enhanced, arm) — all jobs passing
Non-ut test suite isolation verified with gradle-scalatest plugin
Native build tasks verified: task chain executes correctly, onlyIf guards skip already-built artifacts
CI helper script tested across all Spark/Java/Scala matrix combinations
gluten-ut tests verified disabled in Gradle (enabled = false), run via Maven in CI

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4 (Anthropic)

fix ci

Add a complete Gradle build system that coexists with the existing Maven build. This enables developers to choose their preferred build tool while maintaining full feature parity. Features: - Root Gradle configuration with version catalog (libs.versions.toml) - Convention plugins for common patterns (Scala, shading, native, spotless) - All core modules with proper dependencies - Multi-version support (Spark 3.3-4.1, Scala 2.12/2.13, Java 8-21) - Backend selection (Velox, ClickHouse) via -Pbackend property - Native C++ build integration via CMake - Dependency shading with Shadow plugin Default configuration: Spark 4.1, Scala 2.13, Java 17, Velox backend Usage examples: - ./gradlew build # Default build - ./gradlew build -PsparkVersion=3.5 # Spark 3.5 - ./gradlew build -Pbackend=clickhouse # ClickHouse backend - ./gradlew build -Pdelta=true -Piceberg=true # With optional modules Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

github-actions · 2026-02-06T06:55:41Z