Invalid CAST does not surface Spark's CAST_INVALID_INPUT condition (Arrow type names in message)

[Robin-Everaars]

Sail is a drop-in Spark replacement, and invalid string-to-numeric casts correctly raise under ANSI mode (matching Spark 4). The raised error itself, however, doesn't match Spark's, which breaks code and users that rely on Spark's error contract.

Repro

Server:

sail spark server --ip 127.0.0.1 --port 50052

Client (pyspark 4.1.2):

from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:50052").getOrCreate()
try:
    spark.sql("SELECT CAST('not_a_long' AS BIGINT)").collect()
except Exception as e:
    print(type(e).__name__, "|", e.getCondition(), "|", str(e).splitlines()[0])

Actual (Sail 0.6.6)

NumberFormatException | None | Cannot cast string 'not_a_long' to value of Int64 type

• getCondition(), getErrorClass(), getSqlState() and getMessageParameters() all return None.
• The message uses Arrow type names (Int64 here; AS INT gives Int32) rather than Spark SQL type names.

Expected (Apache Spark 4.1.2, same query)

SparkNumberFormatException | CAST_INVALID_INPUT | [CAST_INVALID_INPUT] The value 'not_a_long' of the type "STRING" cannot be cast to "BIGINT" ...

• getCondition() returns CAST_INVALID_INPUT; the exception also carries the SQLSTATE and message parameters.
• Type names are Spark SQL types (STRING, BIGINT).

The exception class already matches (NumberFormatException), so this is only about the missing error condition and the message format.

Why it matters

Error handling on the client keys on the condition, not the message text:

except NumberFormatException as e:
    if e.getCondition() == "CAST_INVALID_INPUT":
        ...  # tolerate / route bad rows

Against Sail that branch never matches (getCondition() is None), so Spark code that distinguishes an invalid-cast from other failures can't tell them apart. The Arrow type names in the message also leak the underlying engine through the Spark-compatible surface.

Environment

• Sail 0.6.6 (pysail), Spark Connect
• pyspark client 4.1.2, Python 3, Linux

[linhr]

Thanks for reporting this!

Yeah I feel that better error message and full support for error methods are indeed super helpful for diagnostic purposes. This is an area where we haven't figured out a good plan. Right now we mostly receive errors from DataFusion and Arrow and return them directly to the client (with some heuristics to map error types). I feel to handle errors rigorously and robustly, we need some structured error representation internally so that we obtain or parse structured information from libraries we depend on, and convert them to a representation that better matches Spark semantics before returning the error to the client.

[Robin-Everaars]

@linhr you're very welcome! If I can help in any way, let me know!