Skip to content

GH-46599: [C++][Doc][Parquet] Update supported types documentation #46620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
paleolimbot merged 6 commits into from
Jun 2, 2025
Merged

GH-46599: [C++][Doc][Parquet] Update supported types documentation #46620

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
48be3fd
update doc
paleolimbot May 27, 2025
61fe0fc
more accurate footnotes
paleolimbot May 30, 2025
80e1659
one more
paleolimbot May 30, 2025
4defb64
Update docs/source/cpp/parquet.rst
paleolimbot Jun 2, 2025
c9458a4
Update docs/source/cpp/parquet.rst
paleolimbot Jun 2, 2025
6efc1ea
reformat table
paleolimbot Jun 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 51 additions & 35 deletions docs/source/cpp/parquet.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,38 +450,46 @@ Logical types
Specific logical types can override the default Arrow type mapping for a given
physical type.

+-------------------+-----------------------------+----------------------------+---------+
| Logical type | Physical type | Mapped Arrow type | Notes |
+===================+=============================+============================+=========+
| NULL | Any | Null | \(1) |
+-------------------+-----------------------------+----------------------------+---------+
| INT | INT32 | Int8 / UInt8 / Int16 / | |
| | | UInt16 / Int32 / UInt32 | |
+-------------------+-----------------------------+----------------------------+---------+
| INT | INT64 | Int64 / UInt64 | |
+-------------------+-----------------------------+----------------------------+---------+
| DECIMAL | INT32 / INT64 / BYTE_ARRAY | Decimal128 / Decimal256 | \(2) |
| | / FIXED_LENGTH_BYTE_ARRAY | | |
+-------------------+-----------------------------+----------------------------+---------+
| DATE | INT32 | Date32 | \(3) |
+-------------------+-----------------------------+----------------------------+---------+
| TIME | INT32 | Time32 (milliseconds) | |
+-------------------+-----------------------------+----------------------------+---------+
| TIME | INT64 | Time64 (micro- or | |
| | | nanoseconds) | |
+-------------------+-----------------------------+----------------------------+---------+
| TIMESTAMP | INT64 | Timestamp (milli-, micro- | |
| | | or nanoseconds) | |
+-------------------+-----------------------------+----------------------------+---------+
| STRING | BYTE_ARRAY | String / LargeString / | |
| | | StringView | |
+-------------------+-----------------------------+----------------------------+---------+
| LIST | Any | List | \(4) |
+-------------------+-----------------------------+----------------------------+---------+
| MAP | Any | Map | \(5) |
+-------------------+-----------------------------+----------------------------+---------+
| FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | |
+-------------------+-----------------------------+----------------------------+---------+
+-------------------+-----------------------------+------------------------------+-----------+
| Logical type | Physical type | Mapped Arrow type | Notes |
+===================+=============================+==============================+===========+
| NULL | Any | Null | \(1) |
+-------------------+-----------------------------+------------------------------+-----------+
| INT | INT32 | Int8 / UInt8 / Int16 / | |
| | | UInt16 / Int32 / UInt32 | |
+-------------------+-----------------------------+------------------------------+-----------+
| INT | INT64 | Int64 / UInt64 | |
+-------------------+-----------------------------+------------------------------+-----------+
| DECIMAL | INT32 / INT64 / BYTE_ARRAY | Decimal128 / Decimal256 | \(2) |
| | / FIXED_LENGTH_BYTE_ARRAY | | |
+-------------------+-----------------------------+------------------------------+-----------+
| DATE | INT32 | Date32 | \(3) |
+-------------------+-----------------------------+------------------------------+-----------+
| TIME | INT32 | Time32 (milliseconds) | |
+-------------------+-----------------------------+------------------------------+-----------+
| TIME | INT64 | Time64 (micro- or | |
| | | nanoseconds) | |
+-------------------+-----------------------------+------------------------------+-----------+
| TIMESTAMP | INT64 | Timestamp (milli-, micro- | |
| | | or nanoseconds) | |
+-------------------+-----------------------------+------------------------------+-----------+
| STRING | BYTE_ARRAY | String / LargeString / | |
| | | StringView | |
+-------------------+-----------------------------+------------------------------+-----------+
| LIST | Any | List | \(4) |
+-------------------+-----------------------------+------------------------------+-----------+
| MAP | Any | Map | \(5) |
+-------------------+-----------------------------+------------------------------+-----------+
| FLOAT16 | FIXED_LENGTH_BYTE_ARRAY | HalfFloat | |
+-------------------+-----------------------------+------------------------------+-----------+
| UUID | FIXED_LENGTH_BYTE_ARRAY | Extension (``arrow.uuid``) | \(6) |
+-------------------+-----------------------------+------------------------------+-----------+
| JSON | BYTE_ARRAY | Extension (``arrow.json``) | \(6) |
+-------------------+-----------------------------+------------------------------+-----------+
| GEOMETRY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7) |
+-------------------+-----------------------------+------------------------------+-----------+
| GEOGRAPHY | BYTE_ARRAY | Extension (``geoarrow.wkb``) | \(6) \(7) |
+-------------------+-----------------------------+------------------------------+-----------+

* \(1) On the write side, the Parquet physical type INT32 is generated.

Expand All @@ -496,9 +504,14 @@ physical type.
in contradiction with the
`Parquet specification <https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps>`__.

*Unsupported logical types:* JSON, BSON, UUID. If such a type is encountered
* \(6) Requires that ``arrow_extensions_enabled`` in ``ArrowReaderProperties`` is ``true``.
When ``false``, the underlying storage type is read.

* \(7) Requires that the ``geoarrow.wkb`` extension type is registered.

*Unsupported logical types:* BSON. If such a type is encountered
when reading a Parquet file, the default physical type mapping is used (for
example, a Parquet JSON column may be read as Arrow Binary or FixedSizeBinary).
example, a Parquet BSON column may be read as Arrow Binary or FixedSizeBinary).

Converted types
~~~~~~~~~~~~~~~
Expand All @@ -513,7 +526,10 @@ Special cases

An Arrow Extension type is written out as its storage type. It can still
be recreated at read time using Parquet metadata (see "Roundtripping Arrow
types" below).
types" below). Some extension types have Parquet LogicalType equivalents
(e.g., UUID, JSON, GEOMETRY, GEOGRAPHY). These are created automatically
if the appropriate option is set in the ``ArrowReaderProperties`` even if
there was no Arrow schema stored in the Parquet metadata.

An Arrow Dictionary type is written out as its value type. It can still
be recreated at read time using Parquet metadata (see "Roundtripping Arrow
Expand Down
Loading