Add Spark 3.3.0 SQL shim module sources#15043
Conversation
b311578 to
de49e32
Compare
3b14485 to
b5c391f
Compare
Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
de49e32 to
e6f5a41
Compare
Greptile SummaryThis PR populates the
Confidence Score: 3/5The bulk of the shim wiring is safe; two files have defects that could surface silently at runtime. SparkDateTimeExceptionShims on Databricks 3.3 silently discards messageParameters, producing exceptions with empty substitution values. SparkSessionUtils.invokeNoArg leaves NoSuchMethodException unhandled so callers targeting Spark 3.3 see a raw propagated exception with no fallback opportunity. SparkDateTimeExceptionShims.scala (spark330db) and SparkSessionUtils.scala (shared) Important Files Changed
|
|
|
||
| import org.apache.spark.{QueryContext, SparkDateTimeException} | ||
|
|
||
| object SparkDateTimeExceptionShims { | ||
|
|
||
| def newSparkDateTimeException( | ||
| errorClass: String, | ||
| messageParameters: Map[String, String], | ||
| context: Array[QueryContext], | ||
| summary: String): SparkDateTimeException = { | ||
| new SparkDateTimeException( | ||
| errorClass, | ||
| None, |
There was a problem hiding this comment.
messageParameters silently dropped
The messageParameters: Map[String, String] argument is accepted by the method but never forwarded to the SparkDateTimeException constructor — Array.empty is passed in its place. Any caller that supplies message parameters expecting them to appear in the exception message will silently get an exception with empty/blank substitution values, producing a misleading or incomplete error string at runtime on Databricks 3.3.0.
| } catch { | ||
| case _: ClassNotFoundException | _: NoSuchFieldException => | ||
| throw new UnsupportedOperationException() | ||
| case e: InvocationTargetException => | ||
| throw e.getCause | ||
| } |
There was a problem hiding this comment.
Hardcoded
null passed to unknown parameter type
The code finds any overflowInConvError overload with exactly one parameter and invokes it with null. If that parameter is a primitive-boxed type, JVM unboxing will throw a NullPointerException inside invoke, masking the intended error. Either matching the exact parameter type in the find predicate or adding a comment documenting the nullable assumption would reduce the ambiguity.
Related to #14834.
Description
This PR is one reviewable layer in the unshim stack introduced by #15025. It adds the Spark 3.3.0 SQL shim module sources after the shared helper module wiring is in place. Keeping this as its own layer makes the Spark-family source population easy to inspect.
Stack context
Testing and validation notes
Checklists
Documentation
Testing
(Covered by the validation notes in the PR description.)
Performance