Skip to content

Fix DBR 18.3 GPU MERGE type mismatch with schema evolution [databricks] - #16642

Merged
sdrp713 merged 4 commits into
NVIDIA:mainfrom
sdrp713:delta-schema-evolution-arrays
Oct 29, 2026
Merged

sdrp713 merged 4 commits into
NVIDIA:mainfrom
sdrp713:delta-schema-evolution-arrays

Conversation

@sdrp713

@sdrp713 sdrp713 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Fixes #16314.

Description

With MERGE WITH SCHEMA EVOLUTION on DBR 18.3, an UPDATE clause that does not set the newly added column gets an untyped NULL literal for it. The GPU merge processor materializes that literal as INT8, then GpuRapidsProcessDeltaMergeJoinExec fails while concatenating the batches from the different clauses, where the column is STRING: CudfException: Type mismatch in columns to concatenate

This PR types those NULL literals with the target column's type in GpuMergeIntoCommand.updateOutput. The file is shared by the 400db173 and 410db183 shims. The fix mirrors the existing typed null handling for the source row id column in the same file.

Tests in delta_lake_merge_test.py:

  • test_delta_merge_not_matched_by_source_schema_evolution_db173 is now parametrized over use_cdf, so the CDC pre/post-image path is covered.
  • New test_delta_merge_partial_update_schema_evolution_db173: a MATCHED UPDATE SET dest.b = ... plus INSERT * that evolves the schema, with and without CDF.

On DBR 18.3 the 5 selected tests pass with the fix. With the fix reverted, 4 failed and 1 passed, with the error above in the logs.

Performance tests are not required because the change only sets the type of a null literal in a query that previously failed. It adds no work on the data path.

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Signed-off-by: Rahul Prabhu <raprabhu@nvidia.com>
@sdrp713
sdrp713 requested review from a team and nartal1 October 27, 2026 22:34
@sdrp713 sdrp713 self-assigned this Oct 27, 2026
@sdrp713 sdrp713 added the bug Something isn't working label Oct 27, 2026
@greptile-apps

greptile-apps Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] No blocking code issue was found in the changes since the previous review.

Summary

Types schema-evolved nulls before nondeterministic update values are evaluated, so nested fields keep their target types.

  • GPU MERGE types unset evolved fields before it combines clause outputs.

Reviews (4) · Last reviewed commit: "Move struct evaluation" · Reviewed by Greptile

@sdrp713

sdrp713 commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author

build

@sdrp713
sdrp713 requested a review from nartal1 October 29, 2026 17:42
@sdrp713
sdrp713 enabled auto-merge (squash) October 29, 2026 18:37
@sdrp713
sdrp713 merged commit 1d450a1 into NVIDIA:main Oct 29, 2026
62 of 63 checks passed
nartal1 added a commit to nartal1/spark-rapids that referenced this pull request Oct 29, 2026
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG][DBR 18.3] Delta MERGE schema evolution fails with a cuDF concatenate type mismatch

2 participants