Skip to content

Commit 43fde45

Browse files
committed
doc: add pyarrow.parquet.filters_to_expression example
Signed-off-by: ChiLin Chiu <chilin.chiou@gmail.com>
1 parent b7ed8df commit 43fde45

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

docs/source/python/dataset.rst

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -569,6 +569,28 @@ calculate the average of a column without loading the entire column into memory:
569569
... count += batch.num_rows
570570
>>> mean_a = col2_sum/count
571571
572+
The ``filter`` argument of :meth:`Dataset.to_batches` (and :func:`~Dataset.to_table`)
573+
expects a boolean :class:`~pyarrow.dataset.Expression`, which can be constructed using
574+
:func:`pyarrow.dataset.field` and its operator overloads. However, if you already have
575+
filters in the DNF (Disjunctive Normal Form) list-of-tuples format accepted by
576+
:class:`pyarrow.parquet.ParquetDataset`, you can convert them to an ``Expression``
577+
using :func:`pyarrow.parquet.filters_to_expression`:
578+
579+
.. code-block:: python
580+
581+
>>> import pyarrow.parquet as pq
582+
>>> import pyarrow.compute as pc
583+
>>> filters = [("a", ">=", 5), ("c", "==", 2)]
584+
>>> filter_expr = pq.filters_to_expression(filters)
585+
>>> filter_expr
586+
<pyarrow.compute.Expression ((a >= 5) and (c == 2))>
587+
>>> a_sum = 0
588+
>>> for batch in dataset.to_batches(columns=["a"], filter=filter_expr):
589+
... if batch.num_rows:
590+
... a_sum += pc.sum(batch.column("a")).as_py()
591+
>>> a_sum
592+
21
593+
572594
Customizing the batch size
573595
~~~~~~~~~~~~~~~~~~~~~~~~~~
574596

0 commit comments

Comments
 (0)