[res-life]

Contributes to #15437.

Description

This PR supports Iceberg v3 initial-default semantics in GPU reads, including fields required implicitly by equality-delete files.

It materializes supported primitive and nested defaults while preserving parent/container nullability, and falls back during planning for unsupported default types or nested equality-delete keys.

Earlier planning-only measurement, before the reader fixes: on an AMD EPYC 7R13 host with Spark local[16], the opt-in traversal added a median paired 0.694 ms (0.71 us/task) across 978 scan tasks, with a paired p95 of 4.687 ms. This used 30 measured pairs after five warm-up pairs. Both modes asserted CPU BatchScan fallback, so this measures planning rather than GPU reader performance.

Reader-only runtime measurements at 259d6b98c, before the shim/visitor refactor and the latest base integration, show the cost of retaining helper leaves and bypassing unsafe pruning. The control substitutes reader.scala and GpuParquetReaderPostProcessor.scala from be4e548e4 while holding the other PR dependencies fixed. Spark 3.5.6 / Scala 2.12 / Iceberg 1.10.1, one NVIDIA L4, local[4], PERFILE, file cache disabled, two million rows in 16 files; A/B/B/A ordering, five warm-ups per JVM and 20 measured samples per artifact/shape.

Warm-cache workload	Control median	Candidate median	Spark input bytes: control → candidate
Normal projection, narrow helper	541.8 ms	542.6 ms	79.85 MB → 79.85 MB
Normal projection, wide helper	541.2 ms	544.1 ms	79.85 MB → 79.85 MB
Missing-default predicate with no matches, narrow helper	115.4 ms	286.4 ms	0.10 MB → 2.43 MB
Missing-default predicate with no matches, wide helper	114.0 ms	358.5 ms	0.10 MB → 39.37 MB

The missing-default predicate returns the same correct empty result in both versions; the candidate reads old row groups conservatively before evaluating the residual. Input metrics report requested Spark bytes, not physical disk I/O. These measurements do not establish the performance of the final head, which has not been rebenchmarked after the shim/visitor refactor and VARIANT integration.

• Version-dispatched Parquet projection preserves physical container paths and validity leaves when only newly added fields are requested. Row-group filtering retains row groups when a predicate references an absent non-null initial default, so Spark can evaluate residuals after materialization. Iceberg 1.6–1.11 retain the compatibility implementation; future shims can override these boundaries with validated native behavior.
• A physical-schema visitor decodes unprojected helpers by physical type, including opaque VARIANT groups, while retaining projected logical types and LIST/MAP wrapper structure. Helpers receive IDs outside the requested schema to prevent accidental binding, and legacy two-level LISTs retain their element interpretation.
• Planning recursively checks structs, lists and maps and includes implicit equality-delete fields. Unsupported initial defaults and nested equality-delete keys trigger CPU fallback. GPU write-default support is tracked separately in Support Iceberg v3 write defaults in GPU writes res-life/spark-rapids#16.
• Read regressions live in iceberg/iceberg_read_test.py, since they cover both format v2 and v3. Delete regressions are consolidated in iceberg/iceberg_delete_test.py. The move preserves all 47 function/class definitions, and all 83 moved parametrized case IDs match the prior runtime report.

Latest base integration: merged 98d729483 (#16443) without rewriting the reviewed commits. The merge retains both shim APIs and adapts two new unit-test calls to the version-dispatched ParquetUnshader.

Combined-head validation at 1513612ac against base 98d729483:

• JDK 17 builds passed for Spark 3.5.6 / Scala 2.12 and Spark 4.0.4 + 4.1.1 / Scala 2.13; Spark 3.5.0 / Iceberg 1.6.1 compilation/package compatibility also passed. The converter modules were cleaned before rebuilding their moved generated sources.
• GpuPostProcessorSuite: 46/46; GpuTypeToSparkTypeSuite: 9/9; Spark 4.1 GpuIcebergVariantSchemaSuite: 10/10. No unit-test failures, errors or skips.
• Integration runs: 287 passed, 6 expected VARIANT skips on Spark 3.5.6, zero failures/errors. Normal and system-classpath runs cover Spark 3.5.6 / Iceberg 1.10.1 and Spark 4.0.4 / Iceberg 1.11.0; Spark 3.5.6 / 1.9.2 passed the selected v2 helper cases. Spark 4.1.1 / 1.11.0 passed its default-only struct, primitive-default and VARIANT-helper cases in both deployment modes.
• All 13 affected shim/reader classes remain byte-identical across the overlapping Iceberg modules and match the packaged aggregator/distribution copies.
• Source-selection preflight passed all 32 profiles on the current-base combination. The bounded cross-PR advisory checked five combinations: three had no new modelled static interactions; Pin shared Spark installation dependency generator [databricks] #16549 and Use shared shim source inheritance in public builds [databricks] #16595 could not be combined with the base because of merge conflicts. Remaining candidates were not checked within the tool limit. This advisory models sql-plugin source selection rather than API or full-runtime compatibility.

The test reorganization additionally collected 153 cases across 37 test functions, preserving all 83 moved case IDs and all 47 function/class definitions; 17 targeted post-reorganization cases passed before the base merge. Earlier America/New_York runs at 259d6b98c passed four cases on each of Spark 3.5.6 and 4.0.4; they were not repeated for the merged head.

Baseline checks reproduced Unknown logical type annotation: VARIANT(1), unmapped helpers supplying 99/100 instead of default 11, and Not a struct type: long on a legacy LIST. The corresponding regressions now pass. Legacy LIST tests use Spark's raw Parquet CPU reader as their oracle because Iceberg's CPU row reader separately fails on that layout.

The title retains [databricks] for filesystem-path and execution-plan coverage. AI assistance: Codex assisted with implementation, tests and local validation under Chong Gao's direction.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The reviewed code appears safe to merge; no blocking defect was established.

Summary

Adds GPU reads for supported Iceberg v3 initial defaults.

• GPU reads fill missing Iceberg fields with supported initial defaults.
• Scan planning checks defaults and equality-delete keys before using the GPU path.
• Parquet reads keep container validity while using physical types for helper fields.

No actionable new issue was established. No previous review threads were supplied.

Shim compatibility evidence is incomplete: the referenced skill file is missing, and GitHub access requires authentication. Exact-head core and related-PR reports could not be checked. No builds or runtime tests were run during this review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Iceberg scan] --> B[Check defaults and equality-delete keys]
  B -->|Unsupported| C[CPU fallback]
  B -->|Supported| D[Project physical fields and retain helpers]
  D --> E[Decode Parquet]
  E --> F[Fill missing defaults and preserve parent nulls]
  F --> G[Apply deletes and remaining predicates]

Loading

_{Reviews (19) · Last reviewed commit: "Merge origin/main for VARIANT schema con..." · Reviewed by Greptile}

[liurenjie1024]

One blocking correctness issue remains in the planning-time default validation; see the inline comment. The updated head now correctly rejects UUID/FIXED defaults and adds direct coverage for supported primitive types.

Non-blocking test note: the end-to-end write-default section only runs outside Spark 3.5, while the reported successful integration validation is Spark 3.5.x, so that branch has not yet been exercised in the listed validation.

[nvauto]

NOTE: release/26.10 has been created from main. Please retarget your PR to release/26.10 if it should be included in the release.

[liurenjie1024]

One blocking correctness issue remains for projections that select only a newly defaulted child of an existing struct; see the inline comment.

AI-assisted review. The schema-projection case was reproduced with Iceberg 1.10.1. Full GPU validation was unavailable locally because of an NVIDIA driver/library version mismatch.