GH-49946: [Format] Better document equivalence between IPC file and streams

Author: pitrou

docs/source/format/Columnar.rst

@@ -1209,8 +1209,8 @@ of these types:
 * DictionaryBatch
 
 We specify a so-called *encapsulated IPC message* format which
-includes a serialized Flatbuffer type along with an optional message
-body. We define this message format before describing how to serialize
+includes a serialized Flatbuffers metadata message along with an optional
+message body. We define this message format before describing how to serialize
 each constituent IPC message type.
 
 .. _ipc-message-format:
@@ -1238,7 +1238,7 @@ The encapsulated binary message format is as follows:
 Schematically, we have: ::
 
     <continuation: 0xFFFFFFFF>
-    <metadata_size: int32>
+    <metadata_size: little-endian int32>
     <metadata_flatbuffer: bytes>
     <padding>
     <message body>
@@ -1248,8 +1248,8 @@ can be relocated between streams. Otherwise the amount of padding between the
 metadata and the message body could be non-deterministic.
 
 The ``metadata_size`` includes the size of the ``Message`` plus
-padding. The ``metadata_flatbuffer`` contains a serialized ``Message``
-Flatbuffer value, which internally includes:
+padding. The ``metadata_flatbuffer`` contains a Flatbuffers-serialized
+``Message`` value, which internally includes:
 
 * A version number
 * A particular message value (one of ``Schema``, ``RecordBatch``, or
@@ -1264,11 +1264,11 @@ can be read.
 Schema message
 --------------
 
-The Flatbuffers files `Schema.fbs`_ contains the definitions for all
+The Flatbuffers definition file `Schema.fbs`_ contains the definitions for all
 built-in data types and the ``Schema`` metadata type which represents
 the schema of a given record batch. A schema consists of an ordered
 sequence of fields, each having a name and type. A serialized ``Schema``
-does not contain any data buffers, only type metadata.
+does not contain any body, only metadata.
 
 The ``Field`` Flatbuffers type contains the metadata for a single
 array. This includes:
@@ -1287,7 +1287,7 @@ array. This includes:
 
 We additionally provide both schema-level and field-level
 ``custom_metadata`` attributes allowing for systems to insert their
-own application defined metadata to customize behavior.
+own application-defined metadata to customize behavior.
 
 .. _ipc-recordbatch-message:
 
@@ -1302,17 +1302,18 @@ thus no memory copying.
 
 The serialized form of the record batch is the following:
 
-* The ``data header``, defined as the ``RecordBatch`` type in
-  `Message.fbs`_.
-* The ``body``, a flat sequence of memory buffers written end-to-end
-  with appropriate padding to ensure a minimum of 8-byte alignment
+* The metadata describing the record batch layout, defined as the ``RecordBatch``
+  type in `Message.fbs`_.
+* The record batch body, a flat sequence of binary buffers written end-to-end,
+  with appropriate padding between each of them to ensure a minimum of 8-byte
+  alignment.
 
-The data header contains the following:
+The record batch metadata contains the following:
 
 * The length and null count for each flattened field in the record
-  batch
+  batch.
 * The memory offset and length of each constituent ``Buffer`` in the
-  record batch's body
+  record batch's body.
 
 Fields and buffers are flattened by a pre-order depth-first traversal
 of the fields in the record batch. For example, let's consider the
@@ -1333,22 +1334,21 @@ The flattened version of this is: ::
 For the buffers produced, we would have the following (refer to the
 table above): ::
 
-    buffer 0: field 0 validity
-    buffer 1: field 1 validity
-    buffer 2: field 1 values
-    buffer 3: field 2 validity
-    buffer 4: field 2 offsets
-    buffer 5: field 3 validity
-    buffer 6: field 3 values
-    buffer 7: field 4 validity
-    buffer 8: field 4 values
-    buffer 9: field 5 validity
-    buffer 10: field 5 offsets
-    buffer 11: field 5 data
-
-The ``Buffer`` Flatbuffers value describes the location and size of a
-piece of memory. Generally these are interpreted relative to the
-**encapsulated message format** defined below.
+    buffer 0: field 0 ('col1') validity
+    buffer 1: field 1 ('col1.a') validity
+    buffer 2: field 1 ('col1.a') values
+    buffer 3: field 2 ('col1.b') validity
+    buffer 4: field 2 ('col1.b') offsets
+    buffer 5: field 3 ('col1.b.item') validity
+    buffer 6: field 3 ('col1.b.item') values
+    buffer 7: field 4 ('col1.c') validity
+    buffer 8: field 4 ('col1.c') values
+    buffer 9: field 5 ('col2') validity
+    buffer 10: field 5 ('col2') offsets
+    buffer 11: field 5 ('col2') data
+
+The ``Buffer`` Flatbuffers value describes the location and size of a buffer's
+data, relatively to the start of the RecordBatch message's body.
 
 The ``size`` field of ``Buffer`` is not required to account for padding
 bytes. Since this metadata can be used to communicate in-memory pointer
@@ -1391,7 +1391,6 @@ have two entries in each RecordBatch. For a RecordBatch of this schema with
     buffer 12: col2    data
     buffer 13: col2    data
 
-
 Compression
 -----------
 
@@ -1452,18 +1451,19 @@ serialized form is as follows:
    ratios.
 
 Byte Order (`Endianness`_)
----------------------------
+--------------------------
 
-The Arrow format is little endian by default.
+The Arrow IPC format is little-endian by default.
 
-Serialized Schema metadata has an endianness field indicating
-endianness of RecordBatches. Typically this is the endianness of the
-system where the RecordBatch was generated. The main use case is
-exchanging RecordBatches between systems with the same Endianness.  At
-first we will return an error when trying to read a Schema with an
-endianness that does not match the underlying system. The reference
-implementation is focused on Little Endian and provides tests for
-it. Eventually we may provide automatic conversion via byte swapping.
+Serialized Schema metadata has an endianness field indicating the endianness of
+Arrow data in all the RecordBatch and DictionaryBatch message bodies.
+Typically this is the endianness of the system where the Arrow data was generated.
+While some IPC reader implementations may allow for byte-swapping when reading
+an IPC Stream or File with non-native endianness, other implementations may simply
+refuse reading such data.
+
+Note that the endianness field only applies to RecordBatch and DictionaryBatch
+message bodies, not to message metadata or any other signaling in the IPC formats.
 
 IPC Streaming Format
 --------------------
@@ -1483,7 +1483,7 @@ a ``RecordBatch`` it should be defined in a ``DictionaryBatch``. ::
     <DICTIONARY k - 1>
     <RECORD BATCH 0>
     ...
-    <DICTIONARY x DELTA>
+    <DICTIONARY x REPLACEMENT>
     ...
     <DICTIONARY y DELTA>
     ...
@@ -1502,9 +1502,14 @@ message flatbuffer is read, you can then read the message body.
 
 The stream writer can signal end-of-stream (EOS) either by writing 8 bytes
 containing the 4-byte continuation indicator (``0xFFFFFFFF``) followed by 0
-metadata length (``0x00000000``) or closing the stream interface. We
-recommend the ".arrows" file extension for the streaming format although
-in many cases these streams will not ever be stored as files.
+metadata length (``0x00000000``) or closing the stream interface.
+
+File extension and MIME type
+''''''''''''''''''''''''''''
+
+IPC Streams are not always stored as files, but when they are, we recommend
+the ".arrows" file extension. The registered MIME type for IPC Streams is
+`vnd.apache.arrow.stream`_.
 
 IPC File Format
 ---------------
@@ -1524,22 +1529,55 @@ Schematically we have: ::
     <empty padding bytes [to 8 byte boundary]>
     <STREAMING FORMAT with EOS>
     <FOOTER>
-    <FOOTER SIZE: int32>
+    <FOOTER SIZE: little-endian int32>
     <magic number "ARROW1">
 
+Equivalence with the IPC Streaming Format
+'''''''''''''''''''''''''''''''''''''''''
+
+While it is theoretically possible for the IPC File footer to list RecordBatch
+messages in a differing order from the embedded IPC Stream's sequential order
+(or even to repeat or omit some of the IPC Stream's RecordBatch messages),
+compliant writers SHOULD arrange the IPC File footer so that an IPC File can be
+read using an IPC Stream reader with equivalent results.
+
+Deviations from the IPC Streaming Format
+''''''''''''''''''''''''''''''''''''''''
+
+The random accessible nature of IPC Files leads to semantic differences when
+decoding dictionary-encoded data:
+
+1. While the IPC Streaming format requires that all initial dictionary batches
+   are emitted before any record batch, there is no such requirement in the IPC
+   File format.
+
+2. The IPC File format does not support dictionary replacement, i.e. only one
+   non-delta dictionary batch can be emitted for a given dictionary ID.
+
+3. Delta dictionary batches in an IPC File are applied in the order they appear
+   in the file footer.
+
 In the file format, there is no requirement that dictionary keys
 should be defined in a ``DictionaryBatch`` before they are used in a
 ``RecordBatch``, as long as the keys are defined somewhere in the
 file. Further more, it is invalid to have more than one **non-delta**
 dictionary batch per dictionary ID (i.e. dictionary replacement is not
 supported). Delta dictionaries are applied in the order they appear in
-the file footer. We recommend the ".arrow" extension for files created with
+the file footer.
+
+We recommend the ".arrow" extension for files created with
 this format. Note that files created with this format are sometimes called
 "Feather V2" or with the ".feather" extension, the name and the extension
 derived from "Feather (V1)", which was a proof of concept early in
 the Arrow project for language-agnostic fast data frame storage for
 Python (pandas) and R.
 
+File extension and MIME type
+''''''''''''''''''''''''''''
+
+We recommend the ".arrow" extension for IPC Files. The registered MIME type for
+IPC Files is `vnd.apache.arrow.file`_.
+
 Dictionary Messages
 -------------------
 
@@ -1728,3 +1766,5 @@ the Arrow spec.
 .. _SIMD: https://www.intel.com/content/www/us/en/docs/cpp-compiler/developer-guide-reference/2021-8/simd-data-layout-templates.html
 .. _Parquet: https://parquet.apache.org/docs/
 .. _UmbraDB: https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf
+.. _vnd.apache.arrow.stream: https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+.. _vnd.apache.arrow.file: https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file