Adapt SQL RAPIDS callers to columnar helpers

[gerashegalov]

Related to #14834.

Description

This PR is one reviewable layer in the unshim stack introduced by #15025. It updates SQL RAPIDS callers to use the moved columnar helpers. This follows the lower-level execution caller changes and keeps SQL-facing call-site changes grouped together.

Stack context

• Stack introduction: Add default common unshim packaging flow #15025
• Previous layer: Adapt core execution callers to columnar helpers #15051
• Next layer: Adapt remaining SQL plugin callers to Java helpers #15034

Testing and validation notes

• No standalone behavior change is intended in this layer. It is covered by the full-stack packaging/build validation described in Add default common unshim packaging flow #15025 and the existing tests for the affected subsystem.
• The full split stack was verified to be tree-equivalent to the pre-split stack top.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Covered by the validation notes in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR is the SQL-layer step in the "unshim stack" (#15025): it decouples SQL-facing callers from versioned shim imports by replacing direct class/object references with reflective lazy-val dispatch, converting case class to plain class throughout, and migrating org.apache.spark.internal.Logging mixins to RapidsLocalLog or local SLF4J loggers. No behavioral change is intended.

• Reflective dispatch introduced for TrampolineConnectShims, DFUDFShims, and PerfIO; RapidsAnalysisException is converted from a subclass of AnalysisException to a factory object that probes the constructor shape at call time.
• case class → class for DFUDF*, JDFUDF*, PartialFile, CompressedRecord, BatchState, and several converter types in InternalColumnarRddConverter, removing auto-generated equals/hashCode/copy to satisfy cross-module visibility rules.
• Three new ShimPredicate shim files added for spark330–358, spark400–402, and spark411 to cover contextIndependentFoldable across all supported Spark versions.

Confidence Score: 4/5

The change is a pure mechanical refactor with no intended behavior change; the vast majority of diffs are import/mixin swaps, case-class-to-class conversions, and local logger migrations.

Two call sites in RapidsAnalysisException re-run reflective constructor scans and re-allocate the emptyQueryContextArray on every error throw rather than caching them at initialization. The joinStrategy/buildSideSelection accessors rely on unchecked asInstanceOf casts on Enumeration.Value fields whose types are defined in a lower stack layer not visible in this diff. Both issues are on non-hot paths and consistent with the layered unshim design, but they warrant a second look.

TrampolineUtil.scala (RapidsAnalysisException factory — constructor caching), GpuHashJoin.scala (asInstanceOf casts on JoinOptions enum fields)

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/TrampolineUtil.scala	Replaces direct import of TrampolineConnectShims and JsonAST with reflective dispatch; converts class RapidsAnalysisException into a factory object with multi-version constructor probing — the emptyQueryContextArray helper is an uncached def that repeats class-lookup and array-allocation on every error throw.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala	Removes case class JoinOptions and JoinCardinalityStats (now in a lower layer); adds joinStrategy / buildSideSelection accessor properties with asInstanceOf casts on Enumeration.Value; replaces joinOptions.strategy / buildSideSelection at all 14 call sites.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuTaskMetrics.scala	Removes NanoTime/SizeInBytes/PerfIO direct imports; accesses PerfIO through reflective lazy-val dispatch; replaces Logging mixin with a static SLF4J logger. Looks correct.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/RapidsShuffleInternalManagerBase.scala	Switches Logging mixin to RapidsLocalLog; converts three internal case classes (PartialFile, CompressedRecord, BatchState) to plain class with explicit val fields. Purely mechanical change; behavior unchanged.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/BridgeGenerateUnsafeProjection.scala	Decouples BridgeUnsafeProjection from CodeGeneratorWithInterpretedFallback; introduces a private BridgeUnsafeProjectionCodegen object that wraps the NonFatal-based fallback pattern and delegates logging via SLF4J. Clean refactor.
sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/ShimPredicate.scala	New shim file covering spark330–358; defines contextIndependentFoldable as a plain def (no override) delegating to .foldable, which is correct for Spark versions that lack the method in Predicate.
sql-plugin/src/main/spark411/scala/com/nvidia/spark/rapids/shims/ShimPredicate.scala	New shim file for spark411; overrides contextIndependentFoldable and delegates to children.forall(_.contextIndependentFoldable), matching Spark 4.11's base Predicate API. Correct override keyword.
sql-plugin/src/main/scala/org/apache/spark/sql/nvidia/LogicalPlanRules.scala	Replaces direct DFUDFShims and RapidsConf calls with reflective lazy dispatch; reads DFUDF enabled flag directly from SQLConf rather than through RapidsConf. Correct mechanical refactor.
sql-plugin/src/main/scala/org/apache/spark/sql/nvidia/dataframe_udfs.scala	Converts all DFUDF0–DFUDF10 and JDFUDF0–JDFUDF10 from case class to plain class with explicit val f. No behavioral change; necessary for cross-module compatibility in the unshim stack.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/InternalColumnarRddConverter.scala	Converts internal case classes (MapConverter, ArrayConverter, StructConverter, etc.) to plain classes; switches from Spark Logging to a local SLF4J logger with guard. Mechanical refactor.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[SQL RAPIDS Callers\nthis PR] -->|reflective lazy dispatch| B[TrampolineConnectShims]
    A -->|reflective lazy dispatch| C[DFUDFShims]
    A -->|reflective lazy dispatch| D[PerfIO]
    A -->|RapidsLocalLog mixin| E[SLF4J Logger]
    A -->|factory apply| F[RapidsAnalysisException\nobject - probes AnalysisException\nconstructor shape]
    A -->|case class to class| G[DFUDF*/JDFUDF*\nPartialFile/CompressedRecord\nBatchState/Converters]
    A -->|new shim files| H[ShimPredicate\nspark330-spark411]
    B --> I[Lower Execution Layer\nprev PR 15051]
    C --> I
    D --> I

Loading

_{Reviews (1): Last reviewed commit: "Keep ShimPredicate version-specific" | Re-trigger Greptile}

[greptile-apps]

emptyQueryContextArray re-creates on every call

emptyQueryContextArray is a plain def that calls Class.forName("org.apache.spark.QueryContext") and java.lang.reflect.Array.newInstance on each invocation. Because rawMessageCtor7, rawMessageCtor8, and rawMessageCtorWithStringParameters each call it afresh (and they themselves rescan classOf[AnalysisException].getConstructors every time), every single RapidsAnalysisException(msg) site performs multiple full reflective constructor scans plus class-lookup + array-allocation per call. The AnalysisException constructor shape is fixed at JVM load time, so the lookup and the resolved constructor should be cached in lazy vals at object initialization. The empty context-array can similarly be a private lazy val.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

asInstanceOf on Enumeration.Value fields is fragile

joinOptions.strategy.asInstanceOf[JoinStrategy.JoinStrategy] (and the buildSideSelection equivalent) casts an AnyRef-typed field to a Value type alias from a specific Enumeration. If the actual stored value was ever populated from the wrong Enumeration singleton (e.g., due to a cross-layer type mismatch introduced by moving JoinOptions to a lower module), this silently succeeds at the cast but fails later at call sites that distinguish values. Could you clarify why the cast is necessary here rather than simply typing the accessors as joinOptions.strategy directly? If the new JoinOptions class exposes strategy as AnyRef intentionally for cross-module compatibility, a comment explaining this design choice would help future maintainers. Why does joinOptions.strategy need an asInstanceOf[JoinStrategy.JoinStrategy] cast rather than being directly typed as JoinStrategy.JoinStrategy in the new JoinOptions definition?