Remove old Spark 3.3.1 through 3.4 shim sources

[gerashegalov]

Related to #14834.

Description

This PR is one reviewable layer in the unshim stack introduced by #15025. It removes old Spark 3.3.1 through Spark 3.4 shim sources that are now provided by shared helpers or the new helper modules.

Stack context

• Stack introduction: Add default common unshim packaging flow #15025
• Previous layer: Remove old Spark 3.3 DBR shim sources #15053
• Next layer: Remove old Spark 3.5 shim sources #15037

Testing and validation notes

• No standalone behavior change is intended in this layer. It is covered by the full-stack packaging/build validation described in Add default common unshim packaging flow #15025 and the existing tests for the affected subsystem.
• The full split stack was verified to be tree-equivalent to the pre-split stack top.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Covered by the validation notes in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR is one layer in a broader "unshim stack" that consolidates version-specific Spark shim sources into shared helpers. It removes duplicate shim sources for Spark 3.3.1–3.4.x that are now provided by new helper modules introduced in the preceding stack layer.

• 11 SparkShimServiceProvider boilerplate files deleted (spark331–spark344, including Databricks variants); their version registration is now handled by shared infrastructure from the previous stack layer.
• Several classes migrated off Spark's internal Logging trait (RapidsShuffleIterator, RapidsCachingReader, HiveFileUtil, GpuWindowGroupLimitingIterator) in favour of direct SLF4J, which is required when source files are compiled into multiple shim JARs without access to Spark internals.
• OrcProtoWriterShim refactored to reflection to support both org.apache.orc.protobuf and com.google.protobuf variants across ORC library generations; GpuGroupedPythonRunnerFactory converted from case class to class extends Serializable with explicit argNames parameter; blank lines added to Python runner files for binary-dedupe alignment.

Confidence Score: 4/5

The PR is a structural cleanup with no intended behavioral change; the full-stack build was validated tree-equivalent to the pre-split stack, so merging carries low risk.

The deletions are mechanical and the refactorings (Logging → SLF4J, WriteFilesExecRule delegation, shim provider removal) are straightforward. The OrcProtoWriterShim reflection rewrite is the most complex change: it is logically correct but surfaces an opaque error message when neither protobuf library is resolvable at runtime. The GpuGroupedPythonRunnerFactory API break (removed argNames default) is an internal shim concern that should be covered by the full-stack build.

sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shims/OrcProtoWriterShim.scala deserves a second look for the reflection-based proto dispatch and its error-path diagnostics.

Important Files Changed

Filename	Overview
sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shims/OrcProtoWriterShim.scala	Refactored to use reflection to support both org.apache.orc.protobuf and com.google.protobuf variants; introduces mutable per-instance state and a potentially misleading error message when no proto library is found
sql-plugin/src/main/spark332db/scala/com/nvidia/spark/rapids/shims/WriteFilesExecRule.scala	Renamed/replaced from SparkUpgradeExceptionShims.scala; provides WriteFilesExec GPU rule shared across many shim versions; copyright year truncated to single year 2026
sql-plugin/src/main/spark341db/scala/org/apache/spark/sql/rapids/execution/python/shims/GpuGroupedPythonRunnerFactory.scala	Changed from case class to class with explicit Serializable; removed default argNames=None, which is a breaking API change for callers not explicitly passing argNames
sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleIterator.scala	Migrated from Spark internal Logging trait to direct SLF4J; only single-argument log helpers are added (no throwable overloads)
sql-plugin/src/main/spark340/scala/org/apache/spark/sql/rapids/RapidsCachingReader.scala	Migrated from Spark internal Logging trait to direct SLF4J; only logInfo and logDebug helpers are added
sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/GpuWindowGroupLimitExec.scala	Removed unused Logging trait import and mixin from GpuWindowGroupLimitingIterator; no log calls present in the class so removal is clean
sql-plugin/src/main/spark341db/scala/org/apache/spark/sql/rapids/execution/python/shims/GpuArrowPythonRunner.scala	Added blank lines to align executable line numbers with pre-Spark-4 shims for binary deduplication; intentional and documented in comment
sql-plugin/src/main/spark332db/scala/org/apache/spark/sql/hive/rapids/shims/HiveFileUtil.scala	Removed Spark internal Logging mixin and replaced with direct SLF4J; only logWarning is used in the file and is correctly replicated
sql-plugin/src/main/spark331/scala/com/nvidia/spark/rapids/shims/Spark331PlusNonDBShims.scala	CheckOverflowInTableInsert expression registration delegated to shared CheckOverflowInTableInsertShims.exprs helper
sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shims/Spark340PlusNonDBShims.scala	WriteFilesExec rule registration delegated to WriteFilesExecRule.execs; CreateDataSourceTableAsSelectCommand uses new runnableCmdFromShim helper

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["SparkShimServiceProvider (deleted per version)\nspark331 / 332 / 332db / 333 / 334 / 340 / 341 / 341db / 342 / 343 / 344"]
    B["Shared helper module\n(previous stack layer #15053)"]
    A -->|"replaced by"| B

    C["Spark.internal.Logging (removed)"]
    D["Direct SLF4J logger"]
    C -->|"replaced by"| D
    C --> RapidsShuffleIterator
    C --> RapidsCachingReader
    C --> HiveFileUtil
    C --> GpuWindowGroupLimitingIterator
    RapidsShuffleIterator --> D
    RapidsCachingReader --> D
    HiveFileUtil --> D
    GpuWindowGroupLimitingIterator --> D

    E["OrcProtoWriterShim (static CodedOutputStream)"]
    F["OrcProtoWriterShim (reflection over protoApis)\nSupports org.apache.orc.protobuf + com.google.protobuf"]
    E -->|"refactored to"| F

    G["WriteFilesExecRule (new shared object)\nWriteFilesExecShims.exec → GpuWriteFilesMeta"]
    H["Spark332PlusDBShims.getExecs"]
    I["Spark340PlusNonDBShims.getExecs"]
    G --> H
    G --> I

Loading

Comments Outside Diff (1)

1. sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shims/OrcProtoWriterShim.scala, line 557-565 (link)

   Misleading error when no protobuf library is loadable

   If both org.apache.orc.protobuf and com.google.protobuf are unavailable at runtime (e.g., due to a misconfigured classpath), protoApis will be empty and every call to writeAndFlush will throw IllegalArgumentException("Unexpected protobuf message type: …"). That message implies the object is the wrong type, when the real problem is that neither codec library could be loaded. It would improve diagnostics to assert protoApis.nonEmpty at initialization (or inside apply) and surface a clear "no protobuf implementation found on classpath" error early.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "Remove old Spark 3.3.1 through 3.4 shim ..." | Re-trigger Greptile}