GH-50421: [C++][Parquet] Add LevelDecoder Skip and Count by AntoinePrv · Pull Request #50422 · apache/arrow

AntoinePrv · 2026-07-08T14:51:41Z

Rationale for this change

Adding new methods to the LevelDecoder to start reducing the complexity in ColumnReaderImplBase, TypedColumnReaderImpl, and TypedRecordReader.

What changes are included in this PR?

Simplify LevelDecoder members based on a single variant
Add LevelDecoder::Skip
Add LevelDecoder::CountUpTo
Simplify Skip implementation in TypedColumnReaderImpl
Add SkippableTypedDecoder to wrap TypedDecoder with the remaining usage of the scratch_for_skip_ buffer

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [C++][Parquet] Improve LevelDecoder #50421

github-actions · 2026-07-08T14:53:25Z

⚠️ GitHub issue #50421 has been automatically assigned in GitHub to PR creator.

Copilot

Pull request overview

This PR extends Parquet C++ decoding by moving more “skip/count” functionality into LevelDecoder and underlying RLE/bit-packed utilities, with the goal of simplifying skip logic in column/record readers.

Changes:

Refactors LevelDecoder to a single std::variant-backed implementation and adds Skip / CountUpTo.
Reworks TypedColumnReaderImpl::Skip to skip within a page by advancing rep/def levels and skipping only the required physical values.
Adds/updates Arrow RLE/bit-packed decoder internals and tests for Advance / CountUpTo.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

File	Description
cpp/src/parquet/column_reader.h	Updates `LevelDecoder` public API/docs to add `Skip` and `CountUpTo` and hide implementation details behind a pimpl.
cpp/src/parquet/column_reader.cc	Implements new `LevelDecoder` internals, introduces `SkippableTypedDecoder`, and simplifies `TypedColumnReaderImpl::Skip`.
cpp/src/arrow/util/rle_encoding_internal.h	Adds `Advance` / `CountUpTo` plumbing, introduces `BitPackedDecoder`, and refactors run processing into `ProcessValues`.
cpp/src/arrow/util/rle_encoding_test.cc	Adds targeted tests for `CountUpTo` and `Advance` behavior.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

pitrou · 2026-07-09T14:13:58Z

@ursabot please benchmark lang=C++

rok · 2026-07-09T14:14:05Z

Benchmark runs are scheduled for commit b7bd198. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

AntoinePrv · 2026-07-09T15:37:43Z

@pitrou this is ready again.

pitrou

A couple minor comments, otherwise LGTM

pitrou · 2026-07-09T16:55:18Z

Finished benchmark runs don't show any worrying regression:

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

conbench-apache-arrow · 2026-07-09T23:38:17Z

Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit b7bd198.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

AntoinePrv · 2026-07-10T08:04:31Z

@pitrou this is looking good

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

+      // Advance the definition levels, counting how many correspond to present
+      // (non-null) values that must be skipped in the data decoder.
+      int64_t non_null_values_to_skip = batch_size;
+      if (this->max_def_level() > 0) {
+        const auto count =
+            this->definition_level_decoder_.CountUpTo(this->max_def_level(), batch_size);
+        non_null_values_to_skip = count.matching_count;
+        ARROW_DCHECK_EQ(count.processed_count, batch_size);
+      }
+      // Advance the repetition levels; their values are not needed.
+      if (this->max_rep_level() > 0) {
+        this->repetition_level_decoder_.Skip(batch_size);
+      }
+      // Skip the corresponding data values.
+      this->current_decoder_.Skip(non_null_values_to_skip);
+


+  /// Advance and count the number of occurrence of a values.
+  ///
+  /// The count is limited to at most the next `batch_size` items.
+  /// @return The matching value count and number of of element that were processed.


pitrou · 2026-07-10T13:49:13Z

@github-actions crossbow submit -g cpp

pitrou

+1, this is ready to merge now :)

github-actions · 2026-07-10T13:52:08Z

Revision: 6e5c7a2

Submitted crossbow builds: ursacomputing/crossbow @ actions-2073a4b27b

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

AntoinePrv added 12 commits July 8, 2026 16:07

Simplify LevelReader

c38f9c6

Proper TypedColumn def/rep level accessors

a5104ac

in object level storage

9902d19

Add SkippableTypedDecoder

749d345

Add RleBitPackedDecoder::Advance

cfd7997

Add RleBitPackedToBitmapDecoder::Advance

2f060ad

Add legacy BitPackedDecoder

e31ca0d

Simplify LevelDecoder::Impl

d6f43d4

Add LevelDecoder::Skip

06464db

Add RleBitPackedDecoder::CountUpTo

481cf9a

Add LevelDecoder::CountUpTo

bd244ad

Remove scratch_for_skip_

cbbe6d6

github-actions Bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Jul 8, 2026

Remove RleBiPackedToBitmap changes

34c1b15

AntoinePrv marked this pull request as ready for review July 8, 2026 15:12

Copilot AI review requested due to automatic review settings July 8, 2026 15:12

AntoinePrv requested review from pitrou and wgtmac as code owners July 8, 2026 15:12

Copilot started reviewing on behalf of AntoinePrv July 8, 2026 15:12 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 9, 2026

Comments typos

b8abe56

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 9, 2026 08:46

Copilot started reviewing on behalf of AntoinePrv July 9, 2026 08:46 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_encoding_internal.h Outdated

Comment thread cpp/src/parquet/column_reader.cc

Comment thread cpp/src/parquet/column_reader.cc Outdated

Comment thread cpp/src/parquet/column_reader.cc

Copilot AI review requested due to automatic review settings July 9, 2026 08:52

AntoinePrv added 2 commits July 9, 2026 17:03

Review improvements

9dd5b89

Add test for RleBitPackedDecoder::CountUpTo

5fc5bf0

Copilot AI review requested due to automatic review settings July 9, 2026 15:19

Copilot started reviewing on behalf of AntoinePrv July 9, 2026 15:19 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread cpp/src/parquet/column_reader.cc

Comment thread cpp/src/parquet/column_reader.cc

Comment thread cpp/src/parquet/column_reader.cc

pitrou reviewed Jul 9, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_encoding_internal.h

pitrou reviewed Jul 9, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_encoding_test.cc Outdated

Comment thread cpp/src/arrow/util/rle_encoding_test.cc Outdated

Comment thread cpp/src/parquet/column_reader.cc

Address comments

83ef69e

pitrou reviewed Jul 9, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_encoding_internal.h Outdated

Copilot AI review requested due to automatic review settings July 9, 2026 16:47

Make max_read_byte requireed in BitPackedRun

7dc5354

Copilot started reviewing on behalf of AntoinePrv July 9, 2026 16:57 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread cpp/src/parquet/column_reader.cc

Comment thread cpp/src/parquet/column_reader.cc

Comment thread cpp/src/parquet/column_reader.cc

Mandatory BitPackedDecoder num_values

6e5c7a2

Copilot AI review requested due to automatic review settings July 10, 2026 12:37

Copilot started reviewing on behalf of AntoinePrv July 10, 2026 12:37 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

pitrou approved these changes Jul 10, 2026

View reviewed changes

pitrou merged commit 5815dc0 into apache:main Jul 10, 2026
60 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Jul 10, 2026

pitrou mentioned this pull request Jul 10, 2026

[C++][Parquet] Improve LevelDecoder #50421

Closed

AntoinePrv deleted the rle-bitmap-skip branch July 10, 2026 14:51

Uh oh!

Conversation

AntoinePrv commented Jul 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jul 9, 2026

Uh oh!

rok commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AntoinePrv commented Jul 9, 2026

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Jul 9, 2026

Uh oh!

AntoinePrv commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

pitrou commented Jul 10, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

AntoinePrv commented Jul 8, 2026 •

edited by github-actions Bot

Loading