[VL] Fall back to vanilla Delta write for VariantType columns#12444
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prevents Delta Lake writes from failing when the table schema contains Spark VARIANT columns by bypassing the Velox native columnar write path (which cannot represent VariantType in Arrow) and delegating those writes to the vanilla Delta writer instead.
Changes:
- Add a recursive schema check (
schemaContainsVariant) and guardGlutenOptimisticTransaction.writeFilesto fall back tosuper.writeFileswhen anyVARIANTis present (including nested). - Add a Delta 4.0 Velox-backend test suite validating top-level, struct-nested, and
UPDATEwrites involvingVARIANT.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/GlutenOptimisticTransaction.scala | Adds a schema-based guard to route VARIANT-containing writes through vanilla Delta instead of the native Velox writer. |
| backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaVariantWriteSuite.scala | Adds coverage ensuring VARIANT writes and updates succeed and round-trip correctly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
b7c684f to
d97a33e
Compare
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
d97a33e to
bff7ffa
Compare
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
bff7ffa to
c40c3c7
Compare
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
c40c3c7 to
84e9591
Compare
felipepessoto
added a commit
to felipepessoto/gluten
that referenced
this pull request
Jul 4, 2026
…thout the fix) Adds only GlutenDeltaVariantWriteSuite, WITHOUT the GlutenOptimisticTransaction guard from apache#12444, to demonstrate that the suite fails on an unpatched main: with native Delta write enabled, the Velox write path throws `UnsupportedOperationException: Unsupported data type: variant` at runtime. Do not merge -- this PR exists only to validate that the test is an effective regression test for apache#12444. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Validation that it fails without the fix is here: #12445 (comment) |
Contributor
Author
|
@baibaichen could you please review? |
zml1206
reviewed
Jul 9, 2026
| // RowToVeloxColumnarExec transition whose SparkArrowUtil.toArrowSchema call throws | ||
| // `Unsupported data type: variant` at runtime. Delegate writes whose schema contains a | ||
| // variant column to the vanilla Delta write path instead of offloading them. | ||
| if (inputData.schema.existsRecursively(_.isInstanceOf[VariantType])) { |
Contributor
There was a problem hiding this comment.
Thanks, could we reuse the existing VeloxValidatorApi.validateSchema / doSchemaValidate for Delta/native write schema validation instead of adding another one-off VariantType guard? That would make unsupported Spark types fail consistently before RowToVeloxColumnarExec / Arrow schema conversion.
Contributor
Author
There was a problem hiding this comment.
Done. Thanks for the suggestion
Contributor
Author
There was a problem hiding this comment.
@zml1206 I also addressed some Copilot comments. Could you review again please?
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
84e9591 to
f113e26
Compare
felipepessoto
added a commit
to felipepessoto/gluten
that referenced
this pull request
Jul 9, 2026
…thout the fix) Adds only GlutenDeltaVariantWriteSuite, WITHOUT the GlutenOptimisticTransaction guard from apache#12444, to demonstrate that the suite fails on an unpatched main: with native Delta write enabled, the Velox write path throws `UnsupportedOperationException: Unsupported data type: variant` at runtime. Do not merge -- this PR exists only to validate that the test is an effective regression test for apache#12444. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
5db07eb to
c143d1d
Compare
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
c143d1d to
95d8dcf
Compare
zml1206
approved these changes
Jul 10, 2026
Contributor
There was a problem hiding this comment.
Nit: Could we rename this suite to DeltaVariantWriteSuite? Other Delta suites in this source set do not use the Gluten prefix, and the file already lives under Gluten's Velox Delta tests, so DeltaVariantWriteSuite would be more consistent. cc @zhztheplayer
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
95d8dcf to
e7b4d4c
Compare
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
e7b4d4c to
6d77625
Compare
Velox cannot write every Spark type. When native Delta write is enabled, Gluten offloads certain Delta write commands to the native writer via OffloadDeltaCommand -- DataFrameWriter.save (append/overwrite), CREATE/REPLACE TABLE AS SELECT, UPDATE, DELETE and OPTIMIZE. The native write path inserts a RowToVeloxColumnarExec transition whose SparkArrowUtil.toArrowSchema call throws `UnsupportedOperationException: Unsupported data type: <type>` at runtime for any type it has no Arrow mapping for (VariantType is the motivating example). (Plain SQL INSERT INTO and MERGE are not offloaded and were unaffected.) Guard GlutenOptimisticTransaction.writeFiles with the backend's existing schema validator: if VeloxValidatorApi.validateSchema reports the input schema is not supported, delegate to super.writeFiles (the vanilla Delta write path) instead of offloading. This reuses the same check Velox already applies to scan schemas, so unsupported types fall back consistently before Arrow schema conversion rather than adding a one-off VariantType guard. Supported writes are unaffected, and the validator's reason is logged when it falls back. Adds DeltaVariantWriteSuite, which exercises offloaded writes: top-level and struct-nested variant columns via DataFrameWriter.save, plus an UPDATE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
felipepessoto
force-pushed
the
vl-variant-write-fallback
branch
from
6d77625 to
decb51c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Velox cannot write every Spark type. When native Delta write is enabled, Gluten offloads certain
Delta write commands to the native columnar writer (
OffloadDeltaCommand) --DataFrameWriter.save(append/overwrite),CREATE/REPLACE TABLE AS SELECT,UPDATE,DELETEand
OPTIMIZE. The native write path inserts aRowToVeloxColumnarExectransition that convertsthe schema via
SparkArrowUtil.toArrowSchema, which throwsUnsupportedOperationException: Unsupported data type: <type>at runtime for any type it has noArrow mapping for (
VariantTypeis the motivating example). (Plain SQLINSERT INTOandMERGEare not offloaded and were unaffected.)
This PR guards
GlutenOptimisticTransaction.writeFileswith the backend's existing schemavalidator: if
VeloxValidatorApi.validateSchema(inputData.schema)reports the schema is notsupported, it delegates to
super.writeFiles(the vanilla Delta write path) instead of offloadingto Velox, and logs an info message when it falls back. This reuses the same validator Velox already
applies to scan schemas (
VeloxBackend.validateDataSchema), so unsupported types fall backconsistently before the Arrow schema conversion -- rather than adding a one-off
VariantTypeguard(per review feedback). Supported writes are unaffected.
How was this patch tested?
Adds
DeltaVariantWriteSuite(backends-velox, delta-4.0). It exercises offloaded writes --top-level and struct-nested variant columns via
DataFrameWriter.save, plus anUPDATE-- andasserts the data is written and read back correctly. The suite runs with the Gluten plugin and
spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true, so these writes go throughthe native path that previously failed.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 4.8)