Skip to content

Conversation

@SergeiPatiakin
Copy link
Contributor

This PR improves append/overwrite compatibility by relaxing the compatibility check between the inserted Arrow batches and the existing Iceberg table:

  • Ignore field IDs
  • Ignore field names
  • Only compare field types and nullability. Match by field position/index.

@SergeiPatiakin SergeiPatiakin requested review from a team and gruuya December 12, 2024 13:38
@SergeiPatiakin SergeiPatiakin changed the title Allow immaterial schema differences Allow immaterial schema differences for Iceberg append Dec 12, 2024
@SergeiPatiakin SergeiPatiakin force-pushed the lakehouse-loader/schema-alignment branch from d90700c to ecc682a Compare December 16, 2024 08:27
@SergeiPatiakin SergeiPatiakin merged commit 0b390f3 into main Dec 16, 2024
1 check passed
@SergeiPatiakin SergeiPatiakin deleted the lakehouse-loader/schema-alignment branch December 16, 2024 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants