spark-3.5-scala-2.12/3.5.4 package update

[octo-sts]

[octo-sts]

Gen AI suggestions to solve the build error:

Let me help analyze and fix the patch failure.

• Detected Error: "Hunk #1 FAILED at 56. 1 out of 2 hunks FAILED -- saving rejects to file build/mvn.rej"

• Error Category: Build Configuration/Patch

• Failure Point: Patch application for build/mvn script

• Root Cause Analysis: The patch for build/mvn is failing because the context lines in the patch file don't match the target file, likely due to version differences between when the patch was created and the current source code version (3.5.4).

• Suggested Fix:

1. Update the mvn.patch file to match the current source version:

--- a/build/mvn
+++ b/build/mvn
@@ -53,7 +53,7 @@
 MVN_ARGS=("$@")
 
 # First, install any dependencies needed by this module.
-$MVN clean install -DskipTests "${MVN_ARGS[@]}"
+$MVN clean install -DskipTests -Dmaven.repo.local=/tmp/maven_repo "${MVN_ARGS[@]}"
 
 # Now, continue with the regular build.
 exec $MVN "$@"

• Explanation:

• The patch is failing because line numbers/context don't match the current version
• The fix adds the maven local repository path specification while preserving the original functionality
• Using /tmp/maven_repo ensures clean builds in ephemeral environments
• The modified patch maintains the original intent while working with the current source

• Additional Notes:

• Other patches applied successfully with some fuzz
• Consider reviewing and updating all patches for 3.5.4 compatibility
• The maven repository path modification is important for reproducible builds

• References:

• Apache Spark build documentation: https://spark.apache.org/docs/3.5.4/building-spark.html
• Maven build isolation best practices: https://maven.apache.org/guides/mini/guide-multiple-repositories.html

[OddBloke]

The two issues found in ci-cve-scan are covered by advisories already: dnsjava, logback-core

[octo-sts]

⚠️ EXPERIMENTAL

Please use 👍 or 👎 on this comment to indicate if you agree or disagree with the recommendation.

To provide more detailed feedback please comment on the recommendation prefixed with /ai-verify:

e.g. /ai-verify partially helpful but I also added bash to the build environment

Gen AI suggestions to solve the build error:

• Detected Error: "[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.5.0:shade (default) on project spark-kvstore_2.12: Error creating shaded jar: null: NullPointerException"

• Error Category: Build Configuration

• Failure Point: Maven shade plugin execution during build of spark-kvstore_2.12 module

• Root Cause Analysis:
The Maven shade plugin is failing with a NullPointerException while trying to create the shaded jar for the spark-kvstore module. This typically occurs when there are issues with dependency management or conflicting resources during the shading process.

• Suggested Fix:
Update the Maven shade plugin configuration in the build section:

pipeline:
  - uses: maven/pombump
    with:
      settings: |
        <settings>
          <profiles>
            <profile>
              <id>shade-config</id>
              <build>
                <plugins>
                  <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-shade-plugin</artifactId>
                    <version>3.5.0</version>
                    <configuration>
                      <createDependencyReducedPom>true</createDependencyReducedPom>
                      <filters>
                        <filter>
                          <artifact>*:*</artifact>
                          <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                          </excludes>
                        </filter>
                      </filters>
                    </configuration>
                  </plugin>
                </plugins>
              </build>
            </profile>
          </profiles>
          <activeProfiles>
            <activeProfile>shade-config</activeProfile>
          </activeProfiles>
        </settings>

• Explanation:
This configuration:

1. Adds explicit filters for META-INF resources to prevent conflicts
2. Creates a dependency-reduced POM to handle transitive dependencies properly
3. Configures the shade plugin with more robust error handling
4. Activates the configuration through a dedicated Maven profile

• Additional Notes:

• The error occurs in the reactor build during the spark-kvstore module
• Other modules built successfully before the failure
• The shade plugin version 3.5.0 is being used
• Maven is skipping tests during the build

• References:

• Maven Shade Plugin documentation: https://maven.apache.org/plugins/maven-shade-plugin/
• Apache Spark build documentation: https://spark.apache.org/docs/latest/building-spark.html
• Similar issue in Maven Shade Plugin: https://issues.apache.org/jira/browse/MSHADE-424

[OddBloke]

There's an ongoing internal discussion on reworking how these packages get built, so I'll hold off on any further work here.

[powersj]

@OddBloke is there a link to the discussion or any resolution yet?

[OddBloke]

@powersj #39643 is the outcome of that conversation, it replaces the separate files for different Scala versions with a single spark-3.5.yaml.