Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,223 changes: 798 additions & 425 deletions cpp/apidoc/Doxyfile

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion cpp/src/arrow/result.h
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ class [[nodiscard]] Result : public util::EqualityComparable<Result<T>> {
/// contents of a `Result<U>`. `T` must be implicitly constructible from `U
/// &&`.
///
/// Sets `other` to contain a non-OK status with a`StatusError::Invalid`
/// Sets `other` to contain a non-OK status with a `StatusError::Invalid`
/// error code.
///
/// \param other The Result object to move from and set to a non-OK status.
Expand Down
6 changes: 0 additions & 6 deletions cpp/src/arrow/table.h
Original file line number Diff line number Diff line change
Expand Up @@ -269,11 +269,6 @@ class ARROW_EXPORT TableBatchReader : public RecordBatchReader {
int64_t max_chunksize_;
};

/// \defgroup concat-tables ConcatenateTables function.
///
/// ConcatenateTables function.
/// @{

/// \brief Controls the behavior of ConcatenateTables().
struct ARROW_EXPORT ConcatenateTablesOptions {
/// If true, the schemas of the tables will be first unified with fields of
Expand Down Expand Up @@ -308,7 +303,6 @@ struct ARROW_EXPORT ConcatenateTablesOptions {
/// \param[in] memory_pool MemoryPool to be used if null-filled arrays need to
/// be created or if existing column chunks need to endure type conversion
/// \return new Table

ARROW_EXPORT
Result<std::shared_ptr<Table>> ConcatenateTables(
const std::vector<std::shared_ptr<Table>>& tables,
Expand Down
5 changes: 4 additions & 1 deletion docs/source/cpp/api/table.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,5 +44,8 @@ Tables
.. doxygenfunction:: arrow::ConcatenateTables
:project: arrow_cpp

.. doxygenfunction:: arrow::PromoteTableToSchema
.. doxygenfunction:: arrow::PromoteTableToSchema(const std::shared_ptr<Table> &table, const std::shared_ptr<Schema> &schema, MemoryPool *pool)
:project: arrow_cpp

.. doxygenfunction:: arrow::PromoteTableToSchema(const std::shared_ptr<Table> &table, const std::shared_ptr<Schema> &schema, const compute::CastOptions &options, MemoryPool *pool)
:project: arrow_cpp
4 changes: 3 additions & 1 deletion docs/source/cpp/arrays.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ example a ``std::vector``. Instead, several strategies can be used:

.. note:: For cases where performance isn't important such as examples or tests,
you may prefer to use the ``*FromJSONString`` helpers which can create
Arrays using a JSON text shorthand. See :ref:`fromjson_helpers`.
Arrays using a JSON text shorthand. See :ref:`fromjsonstring-helpers`.

Using ArrayBuilder and its subclasses
-------------------------------------
Expand Down Expand Up @@ -227,6 +227,8 @@ to some logical subsequence of the data. This is done by calling the
:func:`arrow::Array::Slice` and :func:`arrow::ChunkedArray::Slice` methods,
respectively.

.. _fromjsonstring-helpers:

FromJSONString Helpers
======================

Expand Down
6 changes: 3 additions & 3 deletions docs/source/cpp/compute.rst
Original file line number Diff line number Diff line change
Expand Up @@ -214,15 +214,15 @@ the input to a single output value.
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| count_distinct | Unary | Non-nested types | Scalar Int64 | :struct:`CountOptions` | \(2) |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| first | Unary | Numeric, Binary | Scalar Input type | :struct:`ScalarAggregateOptions` | \(3) |
| first | Unary | Numeric, Binary | Scalar Input type | :struct:`ScalarAggregateOptions` | \(3) |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| first_last | Unary | Numeric, Binary | Scalar Struct | :struct:`ScalarAggregateOptions` | \(3) |
| first_last | Unary | Numeric, Binary | Scalar Struct | :struct:`ScalarAggregateOptions` | \(3) |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| index | Unary | Any | Scalar Int64 | :struct:`IndexOptions` | \(4) |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| kurtosis | Unary | Numeric | Scalar Float64 | :struct:`SkewOptions` | \(11) |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| last | Unary | Numeric, Binary | Scalar Input type | :struct:`ScalarAggregateOptions` | \(3) |
| last | Unary | Numeric, Binary | Scalar Input type | :struct:`ScalarAggregateOptions` | \(3) |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
| max | Unary | Non-nested types | Scalar Input type | :struct:`ScalarAggregateOptions` | |
+--------------------+---------+-----------------------------------------------+------------------------+----------------------------------+-------+
Expand Down
4 changes: 2 additions & 2 deletions docs/source/cpp/tutorials/datasets_tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ One way we can do something with :class:`Datasets <dataset::Dataset>` is getting
them into a :class:`Table`, where we can do anything we’ve learned we can do to
:class:`Tables <Table>` to that :class:`Table`.

.. seealso:: :doc:`/cpp/streaming_execution` for execution that avoids manifesting the entire dataset in memory.
.. seealso:: :doc:`/cpp/acero` for execution that avoids manifesting the entire dataset in memory.

In order to move a :class:`Dataset’s <dataset::Dataset>` contents into a :class:`Table`,
we need a :class:`dataset::Scanner`, which scans the data and outputs it to the :class:`Table`.
Expand Down Expand Up @@ -441,7 +441,7 @@ Now you can get larger-than-memory data mapped for use!
Which means that now we have to be able to process this data without
pulling it all into memory at once. For this, try Acero.

.. seealso:: :doc:`/cpp/streaming_execution` for more information on Acero.
.. seealso:: :doc:`/cpp/acero` for more information on Acero.

Refer to the below for a copy of the complete code:

Expand Down
1 change: 1 addition & 0 deletions docs/source/developers/cpp/emscripten.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
.. highlight:: console

.. _developers-cpp-emscripten:

===============================================
Cross compiling for WebAssembly with Emscripten
===============================================
Expand Down
72 changes: 36 additions & 36 deletions docs/source/developers/java/building.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ To build the default modules, go to the project root and execute:
Maven
~~~~~

.. code-block::
.. code-block:: text

$ cd arrow/java
$ export JAVA_HOME=<absolute path to your java home>
Expand All @@ -74,7 +74,7 @@ Maven
Docker compose
~~~~~~~~~~~~~~

.. code-block::
.. code-block:: text

$ cd arrow/java
$ export JAVA_HOME=<absolute path to your java home>
Expand All @@ -84,7 +84,7 @@ Docker compose
Archery
~~~~~~~

.. code-block::
.. code-block:: text

$ cd arrow/java
$ export JAVA_HOME=<absolute path to your java home>
Expand Down Expand Up @@ -120,7 +120,7 @@ Maven

- To build only the JNI C Data Interface library (Windows):

.. code-block::
.. code-block:: text

$ cd arrow/java
$ mvn generate-resources -Pgenerate-libs-cdata-all-os -N
Expand All @@ -142,7 +142,7 @@ Maven

- To build all JNI libraries (Windows) except the JNI C Data Interface library:

.. code-block::
.. code-block:: text

$ cd arrow/java
$ mvn generate-resources -Pgenerate-libs-jni-windows -N
Expand Down Expand Up @@ -172,7 +172,7 @@ CMake

- To build only the JNI C Data Interface library (Windows):

.. code-block::
.. code-block:: text

$ cd arrow
$ mkdir java-dist, java-cdata
Expand All @@ -190,7 +190,7 @@ CMake

- To build all JNI libraries (macOS / Linux) except the JNI C Data Interface library:

.. code-block::
.. code-block:: text

$ cd arrow
$ brew bundle --file=cpp/Brewfile
Expand Down Expand Up @@ -240,7 +240,7 @@ CMake

- To build all JNI libraries (Windows) except the JNI C Data Interface library:

.. code-block::
.. code-block:: text

$ cd arrow
$ mkdir java-dist, cpp-jni
Expand Down Expand Up @@ -306,14 +306,14 @@ Building Java JNI Modules

- To compile the JNI bindings, use the ``arrow-c-data`` Maven profile:

.. code-block::
.. code-block:: text

$ cd arrow/java
$ mvn -Darrow.c.jni.dist.dir=<absolute path to your arrow folder>/java-dist/lib -Parrow-c-data clean install

- To compile the JNI bindings for ORC / Gandiva / Dataset, use the ``arrow-jni`` Maven profile:

.. code-block::
.. code-block:: text

$ cd arrow/java
$ mvn \
Expand All @@ -335,7 +335,7 @@ Configuring Maven toolchains
To be able to use a JDK version for testing, it needs to be registered first in Maven ``toolchains.xml``
configuration file usually located under ``${HOME}/.m2`` with the following snippet added to it:

.. code-block::
.. code-block:: xml

<?xml version="1.0" encoding="UTF8"?>
<toolchains>
Expand Down Expand Up @@ -364,7 +364,7 @@ To run Arrow tests with a specific JDK version, use the ``arrow.test.jdk-version

For example, to run Arrow tests with JDK 17, use the following snippet:

.. code-block::
.. code-block:: text

$ cd arrow/java
$ mvn -Darrow.test.jdk-version=17 clean verify
Expand Down Expand Up @@ -408,15 +408,15 @@ Common Errors

* When working with the JNI code: if the C++ build cannot find dependencies, with errors like these:

.. code-block::
.. code-block:: text

Could NOT find Boost (missing: Boost_INCLUDE_DIR system filesystem)
Could NOT find Lz4 (missing: LZ4_LIB)
Could NOT find zstd (missing: ZSTD_LIB)

Specify that the dependencies should be downloaded at build time (more details at `Dependency Resolution`_):

.. code-block::
.. code-block:: text

-Dre2_SOURCE=BUNDLED \
-DBoost_SOURCE=BUNDLED \
Expand Down Expand Up @@ -551,35 +551,35 @@ Installing Manually
$ tree ~/.m2/repository/org/apache/arrow
.
├── arrow-flight
   ├── 9.0.0.dev501
   │   └── arrow-flight-9.0.0.dev501.pom
├── 9.0.0.dev501
└── arrow-flight-9.0.0.dev501.pom
├── arrow-format
   ├── 9.0.0.dev501
   │   ├── arrow-format-9.0.0.dev501.jar
   │   └── arrow-format-9.0.0.dev501.pom
├── 9.0.0.dev501
├── arrow-format-9.0.0.dev501.jar
└── arrow-format-9.0.0.dev501.pom
├── arrow-java-root
   ├── 9.0.0.dev501
   │   └── arrow-java-root-9.0.0.dev501.pom
├── 9.0.0.dev501
└── arrow-java-root-9.0.0.dev501.pom
├── arrow-memory
   ├── 9.0.0.dev501
   │   └── arrow-memory-9.0.0.dev501.pom
├── 9.0.0.dev501
└── arrow-memory-9.0.0.dev501.pom
├── arrow-memory-core
   ├── 9.0.0.dev501
   │   ├── arrow-memory-core-9.0.0.dev501.jar
   │   └── arrow-memory-core-9.0.0.dev501.pom
├── 9.0.0.dev501
├── arrow-memory-core-9.0.0.dev501.jar
└── arrow-memory-core-9.0.0.dev501.pom
├── arrow-memory-netty
   ├── 9.0.0.dev501
   │   ├── arrow-memory-netty-9.0.0.dev501.jar
   │   └── arrow-memory-netty-9.0.0.dev501.pom
├── 9.0.0.dev501
├── arrow-memory-netty-9.0.0.dev501.jar
└── arrow-memory-netty-9.0.0.dev501.pom
├── arrow-vector
   ├── 9.0.0.dev501
   │   ├── _remote.repositories
   │   ├── arrow-vector-9.0.0.dev501.jar
   │   └── arrow-vector-9.0.0.dev501.pom
├── 9.0.0.dev501
├── _remote.repositories
├── arrow-vector-9.0.0.dev501.jar
└── arrow-vector-9.0.0.dev501.pom
└── flight-core
├── 9.0.0.dev501
│   ├── flight-core-9.0.0.dev501.jar
│   └── flight-core-9.0.0.dev501.pom
├── 9.0.0.dev501
├── flight-core-9.0.0.dev501.jar
└── flight-core-9.0.0.dev501.pom

6. Compile your project like usual with ``mvn clean install``.

Expand Down
1 change: 1 addition & 0 deletions docs/source/developers/python.rst
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,7 @@ The presets are provided as a convenience, but you may instead opt to
specify the individual components:

.. code-block::

$ cmake -S arrow/cpp -B arrow/cpp/build \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DCMAKE_BUILD_TYPE=Debug \
Expand Down
1 change: 1 addition & 0 deletions docs/source/developers/release_verification.rst
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ along with the vote the command that was executed and the local versions used.
As an example:

.. code-block::

+1

I've verified successfully the sources and binaries with:
Expand Down
54 changes: 28 additions & 26 deletions docs/source/format/DissociatedIPC.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,32 +83,34 @@ Goals
Definitions
-----------

IPC Metadata
The Flatbuffers message bytes that encompass the header of an Arrow IPC message

Tag
A little-endian ``uint64`` value used for flow control and used in determining
how to interpret the body of a message. Specific bits can be masked to allow
identifying messages by only a portion of the tag, leaving the rest of the bits
to be used for control flow or other message metadata. Some transports, such as
UCX, have built-in support for such tag values and will provide them in CPU
memory regardless of whether or not the body of the message may reside on a
non-CPU device.

Sequence Number
A little-endian, 4-byte unsigned integer starting at 0 for a stream, indicating
the sequence order of messages. It is also used to identify specific messages to
tie the IPC metadata header to its corresponding body since the metadata and body
can be sent across separate pipes/streams/transports.

If a sequence number reaches ``UINT32_MAX``, it should be allowed to roll over as
it is unlikely there would be enough unprocessed messages waiting to be processed
that would cause an overlap of sequence numbers.

The sequence number serves two purposes: To identify corresponding metadata and
tagged body data messages and to ensure we do not rely on messages having to arrive
in order. A client should use the sequence number to correctly order messages as
they arrive for processing.
.. glossary::

IPC Metadata
The Flatbuffers message bytes that encompass the header of an Arrow IPC message

Tag
A little-endian ``uint64`` value used for flow control and used in determining
how to interpret the body of a message. Specific bits can be masked to allow
identifying messages by only a portion of the tag, leaving the rest of the bits
to be used for control flow or other message metadata. Some transports, such as
UCX, have built-in support for such tag values and will provide them in CPU
memory regardless of whether or not the body of the message may reside on a
non-CPU device.

Sequence Number
A little-endian, 4-byte unsigned integer starting at 0 for a stream, indicating
the sequence order of messages. It is also used to identify specific messages to
tie the IPC metadata header to its corresponding body since the metadata and body
can be sent across separate pipes/streams/transports.

If a sequence number reaches ``UINT32_MAX``, it should be allowed to roll over as
it is unlikely there would be enough unprocessed messages waiting to be processed
that would cause an overlap of sequence numbers.

The sequence number serves two purposes: To identify corresponding metadata and
tagged body data messages and to ensure we do not rely on messages having to arrive
in order. A client should use the sequence number to correctly order messages as
they arrive for processing.

The Protocol
============
Expand Down
2 changes: 1 addition & 1 deletion docs/source/format/Intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ offered by the language and common in-memory data processing algorithms
provided by the Apache Arrow implementation in their programming
language of choice. Some implementations are further ahead and feature a
vast set of algorithms for in-memory analytical data processing. More detail
about how implementations differ can be found on the :ref:`status` page.
about how implementations differ can be found on the :doc:`../status` page.

Apart from this initial vision, Arrow has grown to also develop a
multi-language collection of libraries for solving problems related to
Expand Down
3 changes: 2 additions & 1 deletion docs/source/python/getstarted.rst
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,11 @@ Loading back the partitioned dataset will detect the chunks
and will lazily load chunks of data only when iterating over them

.. ipython:: python
:okexcept:

import datetime

current_year = datetime.datetime.utcnow().year
current_year = datetime.datetime.now(datetime.UTC).year
for table_chunk in birthdays_dataset.to_batches():
print("AGES", pc.subtract(current_year, table_chunk["years"]))

Expand Down
2 changes: 1 addition & 1 deletion docs/source/python/integration/substrait.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The ``arrow-substrait`` module implements support for the Substrait_ format,
enabling conversion to and from Arrow objects.

The ``arrow-dataset`` module can execute Substrait_ plans via the
:doc:`Acero <../cpp/acero>` query engine.
:doc:`Acero <../../cpp/acero>` query engine.

.. contents::

Expand Down
Loading