refactor: make join projection pushdown schema-aware via ColumnIndex/… by Phoenix500526 · Pull Request #23185 · apache/datafusion

Phoenix500526 · 2026-06-25T12:17:54Z

Which issue does this PR close?

Closes Refactor: Make join projection pushdown schema-aware via ColumnIndex / JoinSide #23010

Rationale for this change

try_pushdown_through_join (datafusion/physical-plan/src/projection.rs)
decided whether each projected join-output column belonged to the left or right
child by comparing its output index against the left child's field count
(join_table_borders) — i.e. it assumed the join output is a plain
left ++ right concatenation.

That assumption does not hold for all join types. A LeftMark / RightMark
join appends a synthetic mark boolean column that has JoinSide::None and
originates from neither child, so reasoning from output position can route the
mark column to the wrong child. This is the panic that #22902 worked around by
disabling projection pushdown for LeftMark / RightMark in HashJoinExec
and NestedLoopJoinExec — correct, but it left the helper reasoning from output
position rather than from the join's actual output-origin contract.

What changes are included in this PR?

Thread the join's output-origin metadata (&[ColumnIndex], already computed by
build_join_schema) into try_pushdown_through_join.
Replace the output-position split (join_table_borders +
index < left_field_count) with grouping by ColumnIndex.side:
- Left / Right columns are collected per child using the child-relative
  ColumnIndex.index;
- a JoinSide::None (synthetic, e.g. mark) column makes the pushdown decline
  (Ok(None)), so the projection is embedded into the join instead — no panic,
  no mis-routing.
Remove the LeftMark | RightMark bypass from
HashJoinExec::try_swapping_with_projection and
NestedLoopJoinExec::try_swapping_with_projection; they now call
try_pushdown_through_join uniformly.
The shared helpers used by cross / symmetric-hash / sort-merge join pushdown
(new_join_children, join_table_borders, join_allows_pushdown,
update_join_on, update_join_filter) keep their signatures. The side-grouped
path uses a new private new_join_children_from_groups and reuses
update_join_on / update_join_filter with a zero column-index offset (the
groups already carry child-relative indices).

Are these changes tested?

Yes.

New subquery.slt cases place a subset/reordering projection over a mark join
— hash LeftMark (IN under OR), negated mark (NOT EXISTS under OR),
and nested-loop mark (non-equi correlated) — asserting both the query result
and the EXPLAIN plan shape.
Existing coverage is preserved: cargo test -p datafusion-physical-plan projection (incl. the filter-pushdown and join_table_borders unit tests),
the join *.slt suite, and subquery.slt all pass.
cargo clippy -p datafusion-physical-plan -- -D warnings and cargo fmt --all
are clean.

Are there any user-facing changes?

No behavior change: the produced physical plans are unchanged for the supported
cases (mark joins keep falling back to the same embedded-projection plan as
before, regular joins push down as before).

One signature change to a pub helper: try_pushdown_through_join gains a
column_indices: &[ColumnIndex] parameter (all in-tree callers are updated). If
the project treats this helper as part of the public API surface, please add the
api change label.

…JoinSide try_pushdown_through_join decided whether each projected join-output column belonged to the left or right child by comparing its output index against the left child's field count, assuming a `left ++ right` output schema. That is wrong for joins whose output is not a plain concatenation -- mark joins append a synthetic `mark` column (JoinSide::None) belonging to neither child -- which is why apache#22902 had to bypass projection pushdown for LeftMark / RightMark. Group the projected columns by the join's output-origin metadata (ColumnIndex.side) instead of output position: Left/Right columns push to the matching child via the child-relative ColumnIndex.index, and a synthetic (JoinSide::None) column makes the pushdown decline so the projection is embedded into the join instead -- no panic, no mis-routing. This lets the LeftMark / RightMark bypass be removed from HashJoinExec and NestedLoopJoinExec. The shared helpers used by cross / symmetric / sort-merge joins (new_join_children, join_table_borders, update_join_on/filter) keep their signatures; the side-grouped path uses a new new_join_children_from_groups and reuses update_join_on/filter with a zero column-index offset. Mark joins still fall back to embedding (pushing their child columns down is a follow-up). Closes apache#23010 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

It is a crate-internal helper, called only by HashJoinExec and NestedLoopJoinExec within datafusion-physical-plan, is not re-exported, and has no external consumers. Reducing its visibility keeps it off the public API surface so future signature changes no longer trip cargo-semver-checks. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

kosiew

@Phoenix500526
Thanks for working on this!
I think the projection pushdown approach makes sense overall, but I have a couple of API surface concerns that I'd like to address before merging.

Remove public column_indices accessors from hash and nested-loop join execution nodes. Projection pushdown now passes the internal column_indices slice from within each impl, avoiding new public API surface and keeping the Vec representation private. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

kosiew

@Phoenix500526
Thanks for the follow-up. The accessor comments look addressed now. I think there is still one blocking semver issue to fix.

Restore the public try_pushdown_through_join signature for semver compatibility. Move the ColumnIndex-aware implementation behind a crate-private helper for join execution internals. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

kosiew

@Phoenix500526
This iteration looks good to me.

alamb

Thanks @kosiew and @Phoenix500526

alamb · 2026-07-11T00:19:30Z

-                projected_right_child,
-                join_filter,
-                join_on,
-            }) = try_pushdown_through_join(


Do we still need try_pushdown_through_join after this PR? Perhaps we should remove it (or migrate other sites over that still use it)?

alamb · 2026-07-11T00:20:18Z

    }))
 }

+pub(crate) fn try_pushdown_through_join_with_column_indices(


this looks pretty similar to try_pushdown_through_join above it -- maybe it could be consolidated so we dn't have two copies 🤔

try_pushdown_through_join no longer has any in-tree callers. I kept it only because it is public API and removing it was flagged by cargo-semver-checks.
Would you prefer to remove it and treat this as an API change, or keep it as a thin compatibility wrapper that delegates to the ColumnIndex-aware implementation? The wrapper would eliminate the duplicated logic noted below while preserving compatibility.
I can also migrate SymmetricHashJoinExec, since it already stores the required column_indices.

Would you prefer to remove it and treat this as an API change, or keep it as a thin compatibility wrapper that delegates to the ColumnIndex-aware implementation? The wrapper would eliminate the duplicated logic noted below while preserving compatibility.

I suggest keep as a compatibily wrapper and delegate and then mark it as "deprecate" per https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines

The ColumnIndex-aware path was added beside the public positional helper to preserve API compatibility, leaving duplicate implementations that could diverge as join output schemas evolve. Keep the public API as a compatibility wrapper over the schema-aware path, and use existing output-origin metadata for symmetric hash joins. This preserves semver compatibility while keeping ownership decisions in one implementation. Refs apache#23010 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

The public helper infers column ownership from output position, so it cannot represent synthetic join columns. It also encourages callers to rely on the left-then-right schema assumption. Keep it as a compatibility wrapper and expose the ColumnIndex-aware helper as its replacement. Document and validate the public mapping so malformed metadata returns planning errors instead of panicking. Refs apache#23010 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

github-actions · 2026-07-12T14:47:10Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  38.270s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.136s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  37.514s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.138s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.664s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure function_marked_deprecated: function #[deprecated] added ---

Description:
A function is now #[deprecated]. Downstream crates will get a compiler warning when using this function.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/function_marked_deprecated.ron

Failed in:
  function datafusion_physical_plan::projection::try_pushdown_through_join in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/projection.rs:649

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  78.437s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 183.938s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 183.801s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.089s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 372.373s] datafusion-sqllogictest

github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jun 25, 2026

Phoenix500526 force-pushed the refactor/23010 branch from 09a1d17 to dca264a Compare June 25, 2026 12:21

github-actions Bot added the auto detected api change Auto detected API change label Jun 25, 2026

kosiew requested changes Jun 29, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/joins/hash_join/exec.rs Outdated

Comment thread datafusion/physical-plan/src/joins/nested_loop_join.rs Outdated

Phoenix500526 requested a review from kosiew July 1, 2026 07:53

kosiew requested changes Jul 3, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/projection.rs Outdated

refactor(physical-plan): preserve join pushdown API

a62b5c6

Restore the public try_pushdown_through_join signature for semver compatibility. Move the ColumnIndex-aware implementation behind a crate-private helper for join execution internals. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

github-actions Bot removed the auto detected api change Auto detected API change label Jul 3, 2026

kosiew approved these changes Jul 7, 2026

View reviewed changes

alamb reviewed Jul 11, 2026

View reviewed changes

Phoenix500526 added 2 commits July 11, 2026 11:16

github-actions Bot added the auto detected api change Auto detected API change label Jul 12, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor: make join projection pushdown schema-aware via ColumnIndex/…#23185

refactor: make join projection pushdown schema-aware via ColumnIndex/…#23185
Phoenix500526 wants to merge 6 commits into
apache:mainfrom
Phoenix500526:refactor/23010

Phoenix500526 commented Jun 25, 2026

Uh oh!

kosiew left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

kosiew left a comment •

edited

Loading

Uh oh!

Uh oh!

kosiew left a comment

Uh oh!

alamb left a comment

Uh oh!

alamb Jul 11, 2026

Uh oh!

alamb Jul 11, 2026

Uh oh!

Phoenix500526 Jul 11, 2026

Uh oh!

alamb Jul 12, 2026

Uh oh!

Phoenix500526 Jul 12, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Phoenix500526 commented Jun 25, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

Phoenix500526 Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 12, 2026

Choose a reason for hiding this comment

Uh oh!

Phoenix500526 Jul 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kosiew left a comment •

edited

Loading

kosiew left a comment •

edited

Loading