diff --git a/docs/source/cpp/security.rst b/docs/source/cpp/security.rst index ee35f7b296ff..b622607b98cf 100644 --- a/docs/source/cpp/security.rst +++ b/docs/source/cpp/security.rst @@ -120,6 +120,22 @@ variants, but these typically fall into two categories: arguments (this is typical of the :ref:`array builders ` and :ref:`buffer builders `). +Controlling and restricting memory allocation +--------------------------------------------- + +By construction, many Arrow C++ APIs can allocate large amounts of memory, depending +on their input parameters. Arrow C++ allows customizing the memory allocator for +such large data requests through the :ref:`MemoryPool ` interface. + +You can therefore implement a MemoryPool class enforcing the restrictions +of your choice (for example to limit the total number of allocated bytes), and pass +it to any Arrow C++ APIs you use. + +.. note:: + Unlike memory used for Arrow data, smaller metadata structures (such as field + names, etc.) instead rely on the C++ standard library allocators for convenience. + They will therefore be invisible to the MemoryPool memory accounting. + Ingesting untrusted data ======================== @@ -144,6 +160,13 @@ from an untrusted source), you **must** follow these steps: 2. If the API returned successfully, validate the returned Arrow data in full (see "Full validity" above) +Furthermore, both the IPC and the Parquet format allow for powerful forms of +compression, and can therefore exhibit large expansion factors when reading. +If you need to guard against potential denial-of-service attacks that would +exhaust available memory, we recommend you enforce memory allocation limits +using a dedicated MemoryPool implementation (see "Controlling and restricting +memory allocation" above). + CSV reader ---------- diff --git a/docs/source/format/Columnar.rst b/docs/source/format/Columnar.rst index 2e81bd0f9424..adf197887eb7 100644 --- a/docs/source/format/Columnar.rst +++ b/docs/source/format/Columnar.rst @@ -1391,6 +1391,8 @@ have two entries in each RecordBatch. For a RecordBatch of this schema with buffer 12: col2 data buffer 13: col2 data +.. _buffer-compression: + Compression ----------- diff --git a/docs/source/format/Security.rst b/docs/source/format/Security.rst index 8e630ea9a55f..d1fa38de723e 100644 --- a/docs/source/format/Security.rst +++ b/docs/source/format/Security.rst @@ -180,6 +180,11 @@ their own risks. For example, buffer offsets and sizes encoded in IPC messages may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads may carry incorrect offsets pointing outside of the designated metadata area. +In addition, the IPC format provides optional :ref:`buffer compression ` +using general-purpose compression algorithms. It is therefore possible to craft an IPC +stream or file that acts as a decompression bomb by consuming all available memory, +opening a potential channel for denial-of-service attacks. + Advice for users ---------------- @@ -194,6 +199,9 @@ It is **extremely recommended** to run dedicated validation checks when decoding the IPC format, to make sure that the decoding can not induce unwanted behavior. Failing those checks should return a well-known error to the caller, not crash. +It is **recommended** to provide facilities for users to control the memory +allocation behavior when reading an IPC file or stream (for example by making +the allocator customizable). Extension Types ===============