Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/source/cpp/security.rst
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,22 @@ variants, but these typically fall into two categories:
arguments (this is typical of the :ref:`array builders <cpp-api-array-builders>`
and :ref:`buffer builders <cpp-api-buffer-builders>`).

Controlling and restricting memory allocation
---------------------------------------------

By construction, many Arrow C++ APIs can allocate large amounts of memory, depending
on their input parameters. Arrow C++ allows customizing the memory allocator for
such large data requests through the :ref:`MemoryPool <cpp_memory_pool>` interface.

You can therefore implement a MemoryPool class enforcing the restrictions
of your choise (for example to limit the total number of allocated bytes), and pass
Comment thread
pitrou marked this conversation as resolved.
Outdated
it to any Arrow C++ APIs you use.

.. note::
Unlike memory used for Arrow data, smaller metadata structures (such as field
names, etc.) instead rely on the C++ standard library allocators for convenience.
They will therefore be invisible to the MemoryPool memory accounting.

Ingesting untrusted data
========================

Expand All @@ -144,6 +160,13 @@ from an untrusted source), you **must** follow these steps:
2. If the API returned successfully, validate the returned Arrow data in full
(see "Full validity" above)

Furthermore, both the IPC and the Parquet format allow for powerful forms of
compression, and can therefore exhibit large expansion factors when reading.
If you need to guard against potential denial-of-service attacks that would
exhaust available memory, we recommend you enforce memory allocation limits
using a dedicated MemoryPool implementation (see "Controlling and restricting
memory allocation" above).

CSV reader
----------

Expand Down
2 changes: 2 additions & 0 deletions docs/source/format/Columnar.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1391,6 +1391,8 @@ have two entries in each RecordBatch. For a RecordBatch of this schema with
buffer 12: col2 data
buffer 13: col2 data

.. _buffer-compression:

Compression
-----------

Expand Down
8 changes: 8 additions & 0 deletions docs/source/format/Security.rst
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,11 @@ their own risks. For example, buffer offsets and sizes encoded in IPC messages
may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
may carry incorrect offsets pointing outside of the designated metadata area.

In addition, the IPC format provides optional :ref:`buffer compression <buffer-compression>`
using general-purpose compression algorithms. It is therefore possible to craft an IPC
stream or file that acts as a decompression bomb by consuming all available memory,
opening a potential channel for denial-of-service attacks.

Advice for users
----------------

Expand All @@ -194,6 +199,9 @@ It is **extremely recommended** to run dedicated validation checks when decoding
the IPC format, to make sure that the decoding can not induce unwanted behavior.
Failing those checks should return a well-known error to the caller, not crash.

It is **recommended** to provide facilities for users to control the memory
allocation behavior when reading an IPC file or stream (for example by making
the allocator customizable).

Extension Types
===============
Expand Down
Loading