Skip to content

feat(core): PBWAG — parallel BWAG wrapper - #2253

Merged
avantgardnerio merged 1 commit into
apache:mainfrom
avantgardnerio:brent/partitioned-bwag
Aug 8, 2026
Merged

feat(core): PBWAG — parallel BWAG wrapper#2253
avantgardnerio merged 1 commit into
apache:mainfrom
avantgardnerio:brent/partitioned-bwag

Conversation

@avantgardnerio

@avantgardnerio avantgardnerio commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

purely additive

PartitionedBoundedWindowAggExec wraps DataFusion's BoundedWindowAggExec and overrides its required_input_distribution from SinglePartition to UnspecifiedDistribution. It hides BWAG from tree walkers by returning only the wrapper's input from children(), so EnforceDistribution can't reinsert an SPM(K→1) beneath it.

This is extracted from #2223 — the full parallel-BWAG stack for h2o's range-window Q8 shape. #2223 has already demonstrated the shape works end-to-end and delivers a 4.7× speedup at 100M rows on a 2-pod EKS cluster. Landing the pieces incrementally keeps each PR's diff focused and reviewable.

What this operator is for

DataFusion's BoundedWindowAggExec declares Distribution::SinglePartition when no PARTITION BY is present — a correctness guard because a window frame's semantics can span rows across the whole input. If the input is already range-repartitioned (each partition is a globally-disjoint slice of the ORDER BY key + halo covering frame boundaries), BWAG can safely run per-partition and produce K correct outputs. EnforceDistribution doesn't know that, so it plants an SPM(K→1) above BWAG and collapses everything to one vcore. This wrapper is the mechanism that tells EnforceDistribution to stand down.

The wrapper doesn't (and can't) verify that its input actually is range-repartitioned. Optimizer rules which plant this operator are responsible for the invariant. Today that is zero. When we merge #2223 there will be one.

Status: temporary

This is a Ballista-side placeholder for apache/datafusion#23026 or similar. Once upstream has this functionality, this wrapper should collapse and callers target DF's BWAG directly.

What's in this PR

  • ballista_core::execution_plans::partitioned_bounded_window_agg: the new operator. InputOrderMode and can_repartition are hardcoded (Sorted / false) — the only planned caller is a no-PARTITION-BY + single-Column-ORDER-BY range-window rule, so hardcoding keeps the wire format and the type small.
  • BallistaPhysicalPlanNode::PartitionedBoundedWindowAgg: proto message carrying only window_expr — the rest is implicit from the caller's shape gates. Round-trip goes through DF's serialize_physical_window_expr / parse_physical_window_expr.
  • Unit test per_partition_execute_running_sum_no_cross_partition_leak builds a two-partition memory source (rows 1–3 in partition 0, rows 100–102 in partition 1) with a RANGE 2.0 PRECEDING frame that spans within a partition but not across, and asserts:
    • BWAG actually aggregates (partition 0's running sums are [1, 3, 6], not [1, 2, 3]).
    • K→K partitioning doesn't leak (partition 1's first sum is 100, not 106).
    • required_input_distribution is UnspecifiedDistribution.
    • children() returns 1 element (BWAG hidden from tree walkers).

What's NOT in this PR

No in-tree caller. This is prep for the ParallelWindowRule follow-up (also extracted from #2223), which is what actually plants this operator in the plan. Landing them separately keeps each diff focused.

Test plan

  • cargo test -p ballista-core --lib per_partition_execute passes locally.
  • cargo fmt --all -- --check clean.
  • cargo clippy --all-targets --workspace clean.
  • cargo check --workspace --all-targets clean.

Wraps DataFusion's `BoundedWindowAggExec` and overrides its
`SinglePartition` distribution requirement to `Unspecified`. Hides BWAG
from tree walkers by returning only the wrapper's input from
`children()`, so `EnforceDistribution` can't reinsert an SPM(K→1)
beneath it. Safe iff the input is already range-repartitioned + halo
covers frame boundaries (see module doc).

This is a Ballista-side placeholder for the upstream draft at
apache/datafusion#23026 ("Parallel bounded RANGE-frame window functions
without PARTITION BY"). Once that lands and Ballista bumps its DF pin
past it, this wrapper collapses and callers target DF's BWAG directly.

- `ballista_core::execution_plans::partitioned_bounded_window_agg`: the
  new operator. `InputOrderMode` and `can_repartition` are hardcoded
  (`Sorted` / `false`) — the only planned caller is a
  no-PARTITION-BY + single-Column-ORDER-BY range-window rule.
- `BallistaPhysicalPlanNode::PartitionedBoundedWindowAgg`: proto
  message carrying only `window_expr` — the rest is implicit from the
  caller's shape gates. Round-trip goes through DF's
  `serialize_physical_window_expr` / `parse_physical_window_expr`.
- Unit test `per_partition_execute_running_sum_no_cross_partition_leak`
  proves BWAG actually aggregates (partition 0's running sums are
  [1, 3, 6], not [1, 2, 3]) and that the K→K partitioning doesn't leak
  across partition boundaries (partition 1's first sum is 100, not 106).

No in-tree caller yet — this is prep for the parallel-window rule
extracted from apache#2223. Landing it separately keeps that PR's diff
focused on the scheduler rule.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@avantgardnerio

Copy link
Copy Markdown
Contributor Author

@phillipleblanc FYI

@avantgardnerio avantgardnerio changed the title feat(core): PartitionedBoundedWindowAggExec — parallel BWAG wrapper feat(core): PBWAG — parallel BWAG wrapper Aug 7, 2026

@milenkovicm milenkovicm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @avantgardnerio, very cool initiative

@avantgardnerio
avantgardnerio merged commit 5333cd6 into apache:main Aug 8, 2026
29 checks passed
@avantgardnerio
avantgardnerio deleted the brent/partitioned-bwag branch August 8, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants