Repository navigation
Fix DBR 18.3 GPU MERGE type mismatch with schema evolution [databricks] - #16642
Merged
Merged
Conversation
Signed-off-by: Rahul Prabhu <raprabhu@nvidia.com>
Contributor
[Medium impact] No blocking code issue was found in the changes since the previous review. SummaryTypes schema-evolved nulls before nondeterministic update values are evaluated, so nested fields keep their target types.
Reviews (4) · Last reviewed commit: "Move struct evaluation" · Reviewed by Greptile |
nartal1
requested changes
Oct 28, 2026
Collaborator
Author
|
build |
nartal1
approved these changes
Oct 29, 2026
sdrp713
enabled auto-merge (squash)
October 29, 2026 18:37
nartal1
added a commit
to nartal1/spark-rapids
that referenced
this pull request
Oct 29, 2026
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #16314.
Description
With
MERGE WITH SCHEMA EVOLUTIONon DBR 18.3, an UPDATE clause that does not set the newly added column gets an untypedNULLliteral for it. The GPU merge processor materializes that literal as INT8, thenGpuRapidsProcessDeltaMergeJoinExecfails while concatenating the batches from the different clauses, where the column is STRING:CudfException: Type mismatch in columns to concatenateThis PR types those
NULLliterals with the target column's type inGpuMergeIntoCommand.updateOutput. The file is shared by the 400db173 and 410db183 shims. The fix mirrors the existing typed null handling for the source row id column in the same file.Tests in
delta_lake_merge_test.py:test_delta_merge_not_matched_by_source_schema_evolution_db173is now parametrized overuse_cdf, so the CDC pre/post-image path is covered.test_delta_merge_partial_update_schema_evolution_db173: a MATCHEDUPDATE SET dest.b = ...plusINSERT *that evolves the schema, with and without CDF.On DBR 18.3 the 5 selected tests pass with the fix. With the fix reverted, 4 failed and 1 passed, with the error above in the logs.
Performance tests are not required because the change only sets the type of a null literal in a query that previously failed. It adds no work on the data path.
Checklists
Documentation
Testing
(Please provide the names of the existing tests in the PR description.)
Performance