On pyarrow 24.0.0, UUID columns with no explicit dlt data_type are inferred as the native extension<arrow.uuid> type instead of the string type dlt has produced in prior version.
CI: https://github.com/dlt-hub/dlt/actions/runs/24839586689/job/73159319854?pr=3889#step:18:439
Repro:
from uuid import uuid4
from dlt.common.libs.pyarrow import row_tuples_to_arrow
from dlt.common.destination import DestinationCapabilitiesContext
rows = [(uuid4(),), (uuid4(),)]
result = row_tuples_to_arrow(
rows,
DestinationCapabilitiesContext().generic_capabilities(),
columns={"uuid_col": {"name": "uuid_col"}},
tz="UTC",
)
print(result[0].type) # pyarrow 23: string, pyarrow 24: extension<arrow.uuid>
Likely introduced by apache/arrow#48727
On pyarrow 24.0.0, UUID columns with no explicit dlt
data_typeare inferred as the nativeextension<arrow.uuid>type instead of thestringtype dlt has produced in prior version.CI: https://github.com/dlt-hub/dlt/actions/runs/24839586689/job/73159319854?pr=3889#step:18:439
Repro:
Likely introduced by apache/arrow#48727