[branch-54] backport #22857 (Skip loading Parquet page index when row-group statistics already prove it cannot prune)#23088
Merged
Conversation
…ve it cannot prune (apache#22857) - Closes apache#22795 The Parquet opener was loading the page index (ColumnIndex + OffsetIndex) before row-group statistics pruning. When all surviving row groups are fully matched by row-group statistics (for example, `IS NOT NULL` on a non-null column), page index I/O cannot prune further and is wasted. - Reorder the opener state machine: `PrepareFilters → PruneWithStatistics → LoadPageIndex? → LoadBloomFilters` - Skip `load_page_index` when there is no page-pruning predicate, no surviving row groups, or every surviving row group is fully matched - Add unit and integration tests for the gate and the fully-matched `IS NOT NULL` case - `cargo test -p datafusion-datasource-parquet should_load` - `cargo test -p datafusion-datasource-parquet page_index_skip` - `cargo test -p datafusion-datasource-parquet opener::test::test_page_pruning` - `cargo test -p datafusion --test parquet_integration` - `cargo clippy -p datafusion-datasource-parquet --all-targets -- -D warnings` No user-facing API changes. This reduces unnecessary Parquet page index I/O during scan planning when row-group statistics already prove no further pruning is possible. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 1fd29c9)
14 tasks
github-actions
Bot
added
sqllogictest
Contributor
|
The backport fails on compilation issues |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backport #22857 to
branch-54in anticipation of 54.1.0 release. DataFusion Comet encountered this issue while trying to upgrade to DF 54.0.0, so it would ease that upgrade for Comet if this were backported.This one had conflicts on cherry-pick, so I could use a sanity check on the review by comparing to the original PR.