feat(core): PBWAG — parallel BWAG wrapper - #2253
Merged
Merged
Conversation
Wraps DataFusion's `BoundedWindowAggExec` and overrides its `SinglePartition` distribution requirement to `Unspecified`. Hides BWAG from tree walkers by returning only the wrapper's input from `children()`, so `EnforceDistribution` can't reinsert an SPM(K→1) beneath it. Safe iff the input is already range-repartitioned + halo covers frame boundaries (see module doc). This is a Ballista-side placeholder for the upstream draft at apache/datafusion#23026 ("Parallel bounded RANGE-frame window functions without PARTITION BY"). Once that lands and Ballista bumps its DF pin past it, this wrapper collapses and callers target DF's BWAG directly. - `ballista_core::execution_plans::partitioned_bounded_window_agg`: the new operator. `InputOrderMode` and `can_repartition` are hardcoded (`Sorted` / `false`) — the only planned caller is a no-PARTITION-BY + single-Column-ORDER-BY range-window rule. - `BallistaPhysicalPlanNode::PartitionedBoundedWindowAgg`: proto message carrying only `window_expr` — the rest is implicit from the caller's shape gates. Round-trip goes through DF's `serialize_physical_window_expr` / `parse_physical_window_expr`. - Unit test `per_partition_execute_running_sum_no_cross_partition_leak` proves BWAG actually aggregates (partition 0's running sums are [1, 3, 6], not [1, 2, 3]) and that the K→K partitioning doesn't leak across partition boundaries (partition 1's first sum is 100, not 106). No in-tree caller yet — this is prep for the parallel-window rule extracted from apache#2223. Landing it separately keeps that PR's diff focused on the scheduler rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@phillipleblanc FYI |
milenkovicm
approved these changes
Aug 8, 2026
milenkovicm
left a comment
Contributor
There was a problem hiding this comment.
LGTM thanks @avantgardnerio, very cool initiative
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
purely additive
PartitionedBoundedWindowAggExecwraps DataFusion'sBoundedWindowAggExecand overrides itsrequired_input_distributionfromSinglePartitiontoUnspecifiedDistribution. It hides BWAG from tree walkers by returning only the wrapper's input fromchildren(), soEnforceDistributioncan't reinsert anSPM(K→1)beneath it.This is extracted from #2223 — the full parallel-BWAG stack for h2o's range-window Q8 shape. #2223 has already demonstrated the shape works end-to-end and delivers a 4.7× speedup at 100M rows on a 2-pod EKS cluster. Landing the pieces incrementally keeps each PR's diff focused and reviewable.
What this operator is for
DataFusion's
BoundedWindowAggExecdeclaresDistribution::SinglePartitionwhen no PARTITION BY is present — a correctness guard because a window frame's semantics can span rows across the whole input. If the input is already range-repartitioned (each partition is a globally-disjoint slice of the ORDER BY key + halo covering frame boundaries), BWAG can safely run per-partition and produce K correct outputs.EnforceDistributiondoesn't know that, so it plants anSPM(K→1)above BWAG and collapses everything to one vcore. This wrapper is the mechanism that tellsEnforceDistributionto stand down.The wrapper doesn't (and can't) verify that its input actually is range-repartitioned. Optimizer rules which plant this operator are responsible for the invariant. Today that is zero. When we merge #2223 there will be one.
Status: temporary
This is a Ballista-side placeholder for apache/datafusion#23026 or similar. Once upstream has this functionality, this wrapper should collapse and callers target DF's BWAG directly.
What's in this PR
ballista_core::execution_plans::partitioned_bounded_window_agg: the new operator.InputOrderModeandcan_repartitionare hardcoded (Sorted/false) — the only planned caller is a no-PARTITION-BY + single-Column-ORDER-BY range-window rule, so hardcoding keeps the wire format and the type small.BallistaPhysicalPlanNode::PartitionedBoundedWindowAgg: proto message carrying onlywindow_expr— the rest is implicit from the caller's shape gates. Round-trip goes through DF'sserialize_physical_window_expr/parse_physical_window_expr.per_partition_execute_running_sum_no_cross_partition_leakbuilds a two-partition memory source (rows 1–3 in partition 0, rows 100–102 in partition 1) with aRANGE 2.0 PRECEDINGframe that spans within a partition but not across, and asserts:[1, 3, 6], not[1, 2, 3]).100, not106).required_input_distributionisUnspecifiedDistribution.children()returns 1 element (BWAG hidden from tree walkers).What's NOT in this PR
No in-tree caller. This is prep for the
ParallelWindowRulefollow-up (also extracted from #2223), which is what actually plants this operator in the plan. Landing them separately keeps each diff focused.Test plan
cargo test -p ballista-core --lib per_partition_executepasses locally.cargo fmt --all -- --checkclean.cargo clippy --all-targets --workspaceclean.cargo check --workspace --all-targetsclean.