Equality delete column constraints are not enforced

[weijiii]

Apache Iceberg version

1.8.1

Query engine

None

Please describe the bug 🐞

From the equality-delete-files doc, only primitive types are allowed to be used as equality delete columns, excluding FLOAT and DOUBLE types.

We bumped into a use case where downstream is generating delete files directly with DeleteWriteBuilder and the delete column is not supported. This issue was only discovered when reading the table.

It seems that DeleteWriteBuilder::buildEqualityWriter for all the formats should check on the constraints based on the schema and equalityFieldIds

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[JeonDaehong]

Hello, just checking—are you planning to work on a PR for this yourself?

[RussellSpitzer]

Do we say in the spec you can’t use those types? I was checking and didn’t see anything specific

…

On Thu, May 8, 2025 at 8:24 AM JeonDaehong ***@***.***> wrote: *JeonDaehong* left a comment (apache/iceberg#12971) <#12971 (comment)> Hello, just checking—are you planning to work on a PR for this yourself? — Reply to this email directly, view it on GitHub <#12971 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADE2YPCPCEGBSUXQQLPLHT25NLIFAVCNFSM6AAAAAB4NQUM5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNRTGA2TMMBZGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[weijiii]

Do we say in the spec you can’t use those types? I was checking and didn’t
see anything specific

@RussellSpitzer I was referencing the equality-delete doc which says it has the same constraints as the identifier-fields with the exception that optional columns and fields nested under optional structs are allowed. The identifier-fields doc says that only primitive fields are allowed, and I assume this means they should of primitive type (struct is a nested type)? Do I understand the doc correctly?

[weijiii]

Hello, just checking—are you planning to work on a PR for this yourself?

@JeonDaehong Yes I would like to give it a try. Thanks for checking!

[stevenzwu]

The problem is that the identifier fields rule (e.g. excluding FLOAT and DOUBLE types) is not enforced in your case? Yes, I would agree that it should be enforced, as floating numbers can exhibit differences across platforms and hence are not suitable for.

Schema class already validate the field types of identifier fields.
https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/Schema.java#L157

It seems that DeleteWriteBuilder::buildEqualityWriter for all the formats should check on the constraints based on the schema and equalityFieldIds

I am not sure we should add such enforcement to Parquet/Orc writer.

It seems that In your case someone is producing the equality delete file with custom code using Iceberg Java SDK. Is the identifier fields set for the table schema? Validation above should fail such schema change. If a writer is not conforming to the spec, it is an implementation bug of the writer.

[JeonDaehong]

Hello, just checking—are you planning to work on a PR for this yourself?

@JeonDaehong Yes I would like to give it a try. Thanks for checking!

I was cheering for the PR back then ! :D

[weijiii]

The problem is that the identifier fields rule (e.g. excluding FLOAT and DOUBLE types) is not enforced in your case? Yes, I would agree that it should be enforced, as floating numbers can exhibit differences across platforms and hence are not suitable for.

Schema class already validate the field types of identifier fields. https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/Schema.java#L157

It seems that DeleteWriteBuilder::buildEqualityWriter for all the formats should check on the constraints based on the schema and equalityFieldIds

I am not sure we should add such enforcement to Parquet/Orc writer.

It seems that In your case someone is producing the equality delete file with custom code using Iceberg Java SDK. Is the identifier fields set for the table schema? Validation above should fail such schema change. If a writer is not conforming to the spec, it is an implementation bug of the writer.

@stevenzwu If we directly use the file format writers to write the delete files, then yes I think FLOAT / DOUBLE would be allowed right now. The case I am talking about is a struct type though. Could you help clarify if I am understanding the spec correctly, that nested types are also not allowed for both the identifier fields and equality delete columns? From identifier-fields doc

A schema can optionally track the set of primitive fields that identify rows in a table, 

Or is it allowed when that struct type is optional?

Is it allowed to have an equality delete, e.g.

row  = (1, 2, 'string')

equality_ids=[1]

 1: row            | 
-------------------|
 (1, ,2, 'string') |

if column row is optional?

[weijiii]

It seems that In your case someone is producing the equality delete file with custom code using Iceberg Java SDK. Is the identifier fields set for the table schema? Validation above should fail such schema change. If a writer is not conforming to the spec, it is an implementation bug of the writer.

Ah interesting I take it you mean that when we use RowDelta::addDeletes (assuming this is the correct API) to add a DeleteFile, the identifier-field-ids would also be configured if not already? And this would constitute a schema change? I am fairly new to the code base so I will have to read more to get familiar here 😄 Is it required to set the identifier fields for any equality deletes to work?

Schema class already validate the field types of identifier fields.
https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/Schema.java#L157

But I am curious isn't this method validating the constraints for the identifier fields? From the equality-delete spec it says there are some exceptions right?

The column restrictions for columns used in equality delete files are the same as those for [identifier fields](https://iceberg.apache.org/spec/?h=equality#identifier-field-ids) with the exception that optional columns and columns nested under optional structs are allowed (if a parent struct column is null it implies the leaf column is null).

FWIW the case I am seeing is directly using the delete writer, e.g. you can see this usage for Trino testing; I was working on this Trino PR to allow equality deletes on columns of struct type (row type in Trino).