Skip to content

Test Delta 4.3 dynamic partition overwrite partition value normalization on GPU [databricks] - #16611

Open
sdrp713 wants to merge 3 commits into
NVIDIA:mainfrom
sdrp713:delta-dpo-normalization
Open

sdrp713 wants to merge 3 commits into
NVIDIA:mainfrom
sdrp713:delta-dpo-normalization

Conversation

@sdrp713

@sdrp713 sdrp713 commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Fixes #16610.

Description

Delta Lake 4.3 normalizes partition values when dynamic partition overwrite decides which files to remove, so equivalent but differently formatted values (timestamps, dates, decimals) match. The GPU Delta 4.3 write path already passes the new AddFiles to Delta's txn.filterFiles(Seq[AddFile]), which does the normalization, so no plugin code changes are needed. This PR adds integration tests showing the GPU matches the CPU.

New tests in delta_lake_write_test.py rewrite the stored partition strings in the table logs, then run a DPO write on CPU and GPU:

  • timestamp, date and decimal partitions with null partitions, under UTC and America/Los_Angeles, with mixed timestamp spellings in one table
  • a renamed physical partition column (column mapping)
  • parsePartitionValues=false, where raw string comparison must match between CPU and GPU
  • an unparseable partition value, where both fall back to raw string comparison and succeed
  • SQL INSERT OVERWRITE on a table with two partition columns, including null bearing partitions

Performance tests are not required because this change only adds regression tests and does not modify runtime code or the existing GPU write path.

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Signed-off-by: Rahul Prabhu <raprabhu@nvidia.com>
@sdrp713
sdrp713 requested a review from a team October 26, 2026 17:39
@sdrp713 sdrp713 self-assigned this Oct 26, 2026
@sdrp713 sdrp713 added the test Only impacts tests label Oct 26, 2026
@greptile-apps

greptile-apps Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; the Python 3.8 collection issue is fixed.

Summary

Adds Delta Lake 4.3 tests that compare CPU and GPU dynamic partition overwrite behavior.

  • Delta overwrite tests check matches across alternate partition spellings.
  • Delta overwrite tests cover raw and unparseable partition strings.
  • SQL overwrite tests cover matching across two partition columns.

Reviews (3) · Last reviewed commit: "Fix test import" · Reviewed by Greptile

Comment thread integration_tests/src/main/python/delta_lake_write_test.py Outdated
Comment thread integration_tests/src/main/python/delta_lake_write_test.py Outdated
Comment thread integration_tests/src/main/python/delta_lake_write_test.py Outdated
@sdrp713 sdrp713 changed the title Test Delta 4.3 dynamic partition overwrite partition value normalization on GPU Test Delta 4.3 dynamic partition overwrite partition value normalization on GPU [databricks] Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Only impacts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Verify GPU dynamic partition overwrite normalizes partition values for Delta 4.3

2 participants