Skip to content

GH-47686: [Docs][Python] Split the Python Parquet docs into separate items#50419

Open
AlenkaF wants to merge 9 commits into
apache:mainfrom
AlenkaF:gh-47686-split-parquet-docs
Open

GH-47686: [Docs][Python] Split the Python Parquet docs into separate items#50419
AlenkaF wants to merge 9 commits into
apache:mainfrom
AlenkaF:gh-47686-split-parquet-docs

Conversation

@AlenkaF

@AlenkaF AlenkaF commented Jul 8, 2026

Copy link
Copy Markdown
Member

Rationale for this change

The current parquet section is one-page and covering lots of topics. As per #46601 the parquet section is most visited and would therefore need a bit of maintenance/content update.

What changes are included in this PR?

  • Parquet documentation is split in 4 different subpages (index.rst, parquet.rst, parquet_datasets.rst and parquet_encryption.rst).
  • Some additional content is added

Are these changes tested?

Via the docs preview

Are there any user-facing changes?

No

@AlenkaF

AlenkaF commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit preview-docs

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Revision: 156850c

Submitted crossbow builds: ursacomputing/crossbow @ actions-937dad9db2

Task Status
preview-docs GitHub Actions

@AlenkaF AlenkaF marked this pull request as ready for review July 9, 2026 11:25
Copilot AI review requested due to automatic review settings July 9, 2026 11:25
@AlenkaF

AlenkaF commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

I have used Claude for suggesting the split of pyarrow parquet page and to look through the code and the git log history for any features/changes that might be missing from the docs. From what was suggested I continued with the work on my own.

I have split the changes in different commits so the change will be easier to review (due to the docs split, the change is bigger than it really is).

commit 1: restructure of the content, no content changes @thisisnic, do you mind having a look?
commit 2: add content about filtering and ability to write bloom filters @mapleFU, do you mind having a look?
commit 3: add section on iter_batches @raulcd, do you mind having a look?
commit 4: add section on binary/list_type option on read, reading extension types @pitrou @paleolimbot, do you mind having a look?
commit 5: datasets note on single file partitions not being included
commit 6: update parquet writer's parameter list @raulcd, do you mind having a look?
commit 7: update encryption section adding info on key rotation and direct encryption @adamreeve, do you mind having a look?

Rendered docs are available here: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Splits the PyArrow Parquet user-guide documentation into multiple focused pages to improve navigability and enable incremental maintenance, aligning with the broader Python docs revision effort.

Changes:

  • Replaces the single large Parquet guide page with a section index plus dedicated subpages (single-file IO, datasets, type handling, encryption).
  • Updates the Python docs toctree to point at the new Parquet section entry point.
  • Adds/relocates some additional Parquet-related content during the split (e.g., content-defined chunking).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
docs/source/python/parquet/parquet.rst New “single files” Parquet IO page (read/write, metadata, filtering, etc.).
docs/source/python/parquet/parquet_type_handling.rst New page for Parquet read/write type-handling options.
docs/source/python/parquet/parquet_datasets.rst New page for partitioned datasets / multi-file Parquet usage.
docs/source/python/parquet/parquet_encryption.rst New page for Parquet modular encryption docs.
docs/source/python/parquet/index.rst New Parquet section landing page with toctrees to subpages.
docs/source/python/parquet.rst Removes the previous monolithic Parquet documentation page.
docs/source/python/index.rst Updates the Python guide toctree to reference the new Parquet section entry.

Comment on lines +25 to +26
Multiple Parquet files constitute a Parquet *dataset*. These may present in a
number of ways:
(when writing version 1.0 Parquet files), the nanoseconds will be cast to
microseconds ('us').

In addition, We provide the ``coerce_timestamps`` option to allow you to select
Key Management System (KMS), deployed in the user's organization. Using Parquet
encryption requires implementation of a client class for the KMS server.
Any KmsClient implementation should implement the informal interface
defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:

>>> crypto_factory = pe.CryptoFactory(kms_client_factory)

An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`
Comment on lines 55 to +57
timestamps
orc
parquet/index
dataset
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants