From 48be3fd7833d3c9c40a3da687f1ae296e6ddb811 Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Tue, 27 May 2025 14:21:28 -0500 Subject: [PATCH 1/6] update doc --- docs/source/cpp/parquet.rst | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/source/cpp/parquet.rst b/docs/source/cpp/parquet.rst index 421b051b1c05..380c877868af 100644 --- a/docs/source/cpp/parquet.rst +++ b/docs/source/cpp/parquet.rst @@ -482,6 +482,14 @@ physical type. +-------------------+-----------------------------+----------------------------+---------+ | FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | | +-------------------+-----------------------------+----------------------------+---------+ +| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (arrow.uuid) | | ++-------------------+-----------------------------+----------------------------+---------+ +| JSON | BYTE_ARRAY | Extension (arrow.json) | | ++-------------------+-----------------------------+----------------------------+---------+ +| GEOMETRY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) | ++-------------------+-----------------------------+----------------------------+---------+ +| GEOGRAPHY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) | ++-------------------+-----------------------------+----------------------------+---------+ * \(1) On the write side, the Parquet physical type INT32 is generated. @@ -496,9 +504,11 @@ physical type. in contradiction with the `Parquet specification `__. -*Unsupported logical types:* JSON, BSON, UUID. If such a type is encountered +* \(6) Requires that the geoarrow.wkb extension type is registered. + +*Unsupported logical types:* BSON. If such a type is encountered when reading a Parquet file, the default physical type mapping is used (for -example, a Parquet JSON column may be read as Arrow Binary or FixedSizeBinary). +example, a Parquet BSON column may be read as Arrow Binary or FixedSizeBinary). Converted types ~~~~~~~~~~~~~~~ From 61fe0fc5c45c3400ee43f68b2c5df050fa975ecd Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Fri, 30 May 2025 09:58:18 -0500 Subject: [PATCH 2/6] more accurate footnotes --- docs/source/cpp/parquet.rst | 85 +++++++++++++++++++------------------ 1 file changed, 44 insertions(+), 41 deletions(-) diff --git a/docs/source/cpp/parquet.rst b/docs/source/cpp/parquet.rst index 380c877868af..64ea839892d0 100644 --- a/docs/source/cpp/parquet.rst +++ b/docs/source/cpp/parquet.rst @@ -450,46 +450,46 @@ Logical types Specific logical types can override the default Arrow type mapping for a given physical type. -+-------------------+-----------------------------+----------------------------+---------+ -| Logical type | Physical type | Mapped Arrow type | Notes | -+===================+=============================+============================+=========+ -| NULL | Any | Null | \(1) | -+-------------------+-----------------------------+----------------------------+---------+ -| INT | INT32 | Int8 / UInt8 / Int16 / | | -| | | UInt16 / Int32 / UInt32 | | -+-------------------+-----------------------------+----------------------------+---------+ -| INT | INT64 | Int64 / UInt64 | | -+-------------------+-----------------------------+----------------------------+---------+ -| DECIMAL | INT32 / INT64 / BYTE_ARRAY | Decimal128 / Decimal256 | \(2) | -| | / FIXED_LENGTH_BYTE_ARRAY | | | -+-------------------+-----------------------------+----------------------------+---------+ -| DATE | INT32 | Date32 | \(3) | -+-------------------+-----------------------------+----------------------------+---------+ -| TIME | INT32 | Time32 (milliseconds) | | -+-------------------+-----------------------------+----------------------------+---------+ -| TIME | INT64 | Time64 (micro- or | | -| | | nanoseconds) | | -+-------------------+-----------------------------+----------------------------+---------+ -| TIMESTAMP | INT64 | Timestamp (milli-, micro- | | -| | | or nanoseconds) | | -+-------------------+-----------------------------+----------------------------+---------+ -| STRING | BYTE_ARRAY | String / LargeString / | | -| | | StringView | | -+-------------------+-----------------------------+----------------------------+---------+ -| LIST | Any | List | \(4) | -+-------------------+-----------------------------+----------------------------+---------+ -| MAP | Any | Map | \(5) | -+-------------------+-----------------------------+----------------------------+---------+ -| FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | | -+-------------------+-----------------------------+----------------------------+---------+ -| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (arrow.uuid) | | -+-------------------+-----------------------------+----------------------------+---------+ -| JSON | BYTE_ARRAY | Extension (arrow.json) | | -+-------------------+-----------------------------+----------------------------+---------+ -| GEOMETRY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) | -+-------------------+-----------------------------+----------------------------+---------+ -| GEOGRAPHY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) | -+-------------------+-----------------------------+----------------------------+---------+ ++-------------------+-----------------------------+----------------------------+----------+ +| Logical type | Physical type | Mapped Arrow type | Notes | ++===================+=============================+============================+==========+ +| NULL | Any | Null | \(1) | ++-------------------+-----------------------------+----------------------------+----------+ +| INT | INT32 | Int8 / UInt8 / Int16 / | | +| | | UInt16 / Int32 / UInt32 | | ++-------------------+-----------------------------+----------------------------+----------+ +| INT | INT64 | Int64 / UInt64 | | ++-------------------+-----------------------------+----------------------------+----------+ +| DECIMAL | INT32 / INT64 / BYTE_ARRAY | Decimal128 / Decimal256 | \(2) | +| | / FIXED_LENGTH_BYTE_ARRAY | | | ++-------------------+-----------------------------+----------------------------+----------+ +| DATE | INT32 | Date32 | \(3) | ++-------------------+-----------------------------+----------------------------+----------+ +| TIME | INT32 | Time32 (milliseconds) | | ++-------------------+-----------------------------+----------------------------+----------+ +| TIME | INT64 | Time64 (micro- or | | +| | | nanoseconds) | | ++-------------------+-----------------------------+----------------------------+----------+ +| TIMESTAMP | INT64 | Timestamp (milli-, micro- | | +| | | or nanoseconds) | | ++-------------------+-----------------------------+----------------------------+----------+ +| STRING | BYTE_ARRAY | String / LargeString / | | +| | | StringView | | ++-------------------+-----------------------------+----------------------------+----------+ +| LIST | Any | List | \(4) | ++-------------------+-----------------------------+----------------------------+----------+ +| MAP | Any | Map | \(5) | ++-------------------+-----------------------------+----------------------------+----------+ +| FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | | ++-------------------+-----------------------------+----------------------------+----------+ +| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (arrow.uuid) | \(6) | ++-------------------+-----------------------------+----------------------------+----------+ +| JSON | BYTE_ARRAY | Extension (arrow.json) | \(6) | ++-------------------+-----------------------------+----------------------------+----------+ +| GEOMETRY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) \(7)| ++-------------------+-----------------------------+----------------------------+----------+ +| GEOGRAPHY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) \(7)| ++-------------------+-----------------------------+----------------------------+----------+ * \(1) On the write side, the Parquet physical type INT32 is generated. @@ -504,7 +504,10 @@ physical type. in contradiction with the `Parquet specification `__. -* \(6) Requires that the geoarrow.wkb extension type is registered. +* \(6) Requires that ``arrow_extensions_enabled`` in ``ArrowReaderProperties`` is ``true``. + When ``false``, the underlying storage type is read. + +* \(7) Requires that the geoarrow.wkb extension type is registered. *Unsupported logical types:* BSON. If such a type is encountered when reading a Parquet file, the default physical type mapping is used (for From 80e1659147bd5f65ab35a1d586291d8276d29157 Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Fri, 30 May 2025 10:04:38 -0500 Subject: [PATCH 3/6] one more --- docs/source/cpp/parquet.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/source/cpp/parquet.rst b/docs/source/cpp/parquet.rst index 64ea839892d0..a69f763b94a3 100644 --- a/docs/source/cpp/parquet.rst +++ b/docs/source/cpp/parquet.rst @@ -526,7 +526,10 @@ Special cases An Arrow Extension type is written out as its storage type. It can still be recreated at read time using Parquet metadata (see "Roundtripping Arrow -types" below). +types" below). Some extension types have Parquet LogicalType equivalents +(e.g., UUID, JSON, GEOMETRY, GEOGRAPHY). These are created automatically +if the appropriate option is set in the ``ArrowReaderProperties`` even if +there was no Arrow schema stored in the Parquet metadata. An Arrow Dictionary type is written out as its value type. It can still be recreated at read time using Parquet metadata (see "Roundtripping Arrow From 4defb642341e4d123d797e88edd65e5d22d64d50 Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Mon, 2 Jun 2025 09:40:16 -0500 Subject: [PATCH 4/6] Update docs/source/cpp/parquet.rst Co-authored-by: Antoine Pitrou --- docs/source/cpp/parquet.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/cpp/parquet.rst b/docs/source/cpp/parquet.rst index a69f763b94a3..305c953f42e7 100644 --- a/docs/source/cpp/parquet.rst +++ b/docs/source/cpp/parquet.rst @@ -507,7 +507,7 @@ physical type. * \(6) Requires that ``arrow_extensions_enabled`` in ``ArrowReaderProperties`` is ``true``. When ``false``, the underlying storage type is read. -* \(7) Requires that the geoarrow.wkb extension type is registered. +* \(7) Requires that the ``geoarrow.wkb`` extension type is registered. *Unsupported logical types:* BSON. If such a type is encountered when reading a Parquet file, the default physical type mapping is used (for From c9458a4ff63470a57a1f5d5c95ab2c195e067d0c Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Mon, 2 Jun 2025 09:40:22 -0500 Subject: [PATCH 5/6] Update docs/source/cpp/parquet.rst Co-authored-by: Antoine Pitrou --- docs/source/cpp/parquet.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/cpp/parquet.rst b/docs/source/cpp/parquet.rst index 305c953f42e7..8e1eec145208 100644 --- a/docs/source/cpp/parquet.rst +++ b/docs/source/cpp/parquet.rst @@ -482,13 +482,13 @@ physical type. +-------------------+-----------------------------+----------------------------+----------+ | FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | | +-------------------+-----------------------------+----------------------------+----------+ -| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (arrow.uuid) | \(6) | +| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (``arrow.uuid``) | \(6) | +-------------------+-----------------------------+----------------------------+----------+ -| JSON | BYTE_ARRAY | Extension (arrow.json) | \(6) | +| JSON | BYTE_ARRAY | Extension (``arrow.json``) | \(6) | +-------------------+-----------------------------+----------------------------+----------+ -| GEOMETRY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) \(7)| +| GEOMETRY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7)| +-------------------+-----------------------------+----------------------------+----------+ -| GEOGRAPHY | BYTE_ARRAY | Extension (geoarrow.wkb) | \(6) \(7)| +| GEOGRAPHY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7)| +-------------------+-----------------------------+----------------------------+----------+ * \(1) On the write side, the Parquet physical type INT32 is generated. From 6efc1eaf42a68c5e949032651c8445f17f90df3a Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Mon, 2 Jun 2025 09:43:57 -0500 Subject: [PATCH 6/6] reformat table --- docs/source/cpp/parquet.rst | 80 ++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/docs/source/cpp/parquet.rst b/docs/source/cpp/parquet.rst index 8e1eec145208..162e8db9da62 100644 --- a/docs/source/cpp/parquet.rst +++ b/docs/source/cpp/parquet.rst @@ -450,46 +450,46 @@ Logical types Specific logical types can override the default Arrow type mapping for a given physical type. -+-------------------+-----------------------------+----------------------------+----------+ -| Logical type | Physical type | Mapped Arrow type | Notes | -+===================+=============================+============================+==========+ -| NULL | Any | Null | \(1) | -+-------------------+-----------------------------+----------------------------+----------+ -| INT | INT32 | Int8 / UInt8 / Int16 / | | -| | | UInt16 / Int32 / UInt32 | | -+-------------------+-----------------------------+----------------------------+----------+ -| INT | INT64 | Int64 / UInt64 | | -+-------------------+-----------------------------+----------------------------+----------+ -| DECIMAL | INT32 / INT64 / BYTE_ARRAY | Decimal128 / Decimal256 | \(2) | -| | / FIXED_LENGTH_BYTE_ARRAY | | | -+-------------------+-----------------------------+----------------------------+----------+ -| DATE | INT32 | Date32 | \(3) | -+-------------------+-----------------------------+----------------------------+----------+ -| TIME | INT32 | Time32 (milliseconds) | | -+-------------------+-----------------------------+----------------------------+----------+ -| TIME | INT64 | Time64 (micro- or | | -| | | nanoseconds) | | -+-------------------+-----------------------------+----------------------------+----------+ -| TIMESTAMP | INT64 | Timestamp (milli-, micro- | | -| | | or nanoseconds) | | -+-------------------+-----------------------------+----------------------------+----------+ -| STRING | BYTE_ARRAY | String / LargeString / | | -| | | StringView | | -+-------------------+-----------------------------+----------------------------+----------+ -| LIST | Any | List | \(4) | -+-------------------+-----------------------------+----------------------------+----------+ -| MAP | Any | Map | \(5) | -+-------------------+-----------------------------+----------------------------+----------+ -| FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | | -+-------------------+-----------------------------+----------------------------+----------+ -| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (``arrow.uuid``) | \(6) | -+-------------------+-----------------------------+----------------------------+----------+ -| JSON | BYTE_ARRAY | Extension (``arrow.json``) | \(6) | -+-------------------+-----------------------------+----------------------------+----------+ -| GEOMETRY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7)| -+-------------------+-----------------------------+----------------------------+----------+ -| GEOGRAPHY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7)| -+-------------------+-----------------------------+----------------------------+----------+ ++-------------------+-----------------------------+------------------------------+-----------+ +| Logical type | Physical type | Mapped Arrow type | Notes | ++===================+=============================+==============================+===========+ +| NULL | Any | Null | \(1) | ++-------------------+-----------------------------+------------------------------+-----------+ +| INT | INT32 | Int8 / UInt8 / Int16 / | | +| | | UInt16 / Int32 / UInt32 | | ++-------------------+-----------------------------+------------------------------+-----------+ +| INT | INT64 | Int64 / UInt64 | | ++-------------------+-----------------------------+------------------------------+-----------+ +| DECIMAL | INT32 / INT64 / BYTE_ARRAY | Decimal128 / Decimal256 | \(2) | +| | / FIXED_LENGTH_BYTE_ARRAY | | | ++-------------------+-----------------------------+------------------------------+-----------+ +| DATE | INT32 | Date32 | \(3) | ++-------------------+-----------------------------+------------------------------+-----------+ +| TIME | INT32 | Time32 (milliseconds) | | ++-------------------+-----------------------------+------------------------------+-----------+ +| TIME | INT64 | Time64 (micro- or | | +| | | nanoseconds) | | ++-------------------+-----------------------------+------------------------------+-----------+ +| TIMESTAMP | INT64 | Timestamp (milli-, micro- | | +| | | or nanoseconds) | | ++-------------------+-----------------------------+------------------------------+-----------+ +| STRING | BYTE_ARRAY | String / LargeString / | | +| | | StringView | | ++-------------------+-----------------------------+------------------------------+-----------+ +| LIST | Any | List | \(4) | ++-------------------+-----------------------------+------------------------------+-----------+ +| MAP | Any | Map | \(5) | ++-------------------+-----------------------------+------------------------------+-----------+ +| FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | | ++-------------------+-----------------------------+------------------------------+-----------+ +| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (``arrow.uuid``) | \(6) | ++-------------------+-----------------------------+------------------------------+-----------+ +| JSON | BYTE_ARRAY | Extension (``arrow.json``) | \(6) | ++-------------------+-----------------------------+------------------------------+-----------+ +| GEOMETRY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7) | ++-------------------+-----------------------------+------------------------------+-----------+ +| GEOGRAPHY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7) | ++-------------------+-----------------------------+------------------------------+-----------+ * \(1) On the write side, the Parquet physical type INT32 is generated.