Skip to content

row_tuples_to_arrow switches UUID columns from string to extension<arrow.uuid> on pyarrow 24+ #3893

Description

@burnash

On pyarrow 24.0.0, UUID columns with no explicit dlt data_type are inferred as the native extension<arrow.uuid> type instead of the string type dlt has produced in prior version.

CI: https://github.com/dlt-hub/dlt/actions/runs/24839586689/job/73159319854?pr=3889#step:18:439

Repro:

from uuid import uuid4
from dlt.common.libs.pyarrow import row_tuples_to_arrow
from dlt.common.destination import DestinationCapabilitiesContext

rows = [(uuid4(),), (uuid4(),)]
result = row_tuples_to_arrow(
    rows,
    DestinationCapabilitiesContext().generic_capabilities(),
    columns={"uuid_col": {"name": "uuid_col"}},
    tz="UTC",
)
print(result[0].type)  # pyarrow 23: string,  pyarrow 24: extension<arrow.uuid>

Likely introduced by apache/arrow#48727

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Fields

No fields configured for Bug.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions