chore: upgrade DataFusion to 54.1.0 - #2126
Merged
Merged
Conversation
Move the lockfile to the DataFusion 54.1.0 patch release and re-sync the vendored DataFusion proto files, which picks up the new `null_aware` field on `JoinNode`. The workspace requirements stay at `54` rather than pinning `54.1.0`. The python workspace has its own lockfile but depends on the ballista crates by path, so an exact requirement here forces it onto 54.1.0 as well, and datafusion-python 54.0.0 does not compile against 54.1.0 -- that release added a `schema` field to the public `RecursiveQuery` struct. Leaving the requirement at `54` lets the main workspace build against 54.1.0 while python/Cargo.lock stays on 54.0.0 until datafusion-python catches up.
andygrove
force-pushed
the
bump-datafusion-54.1.0
branch
from
July 21, 2026 18:17
e773886 to
3c76e80
Compare
andygrove
changed the title
andygrove
marked this pull request as ready for review
July 21, 2026 20:26
phillipleblanc
approved these changes
Jul 22, 2026
andygrove
added a commit
to andygrove/datafusion-ballista
that referenced
this pull request
Jul 22, 2026
Resolves three conflicts: * `Cargo.toml` / `Cargo.lock` — drop the `[patch.crates-io]` DataFusion git pin. It existed only to pick up apache/datafusion#22785 (logical `JoinNode` proto round-trip preserving `null_equality`), which was on `branch-54` but not in the released 54.0.0 tag. main is now on released DataFusion 54.1.0 (apache#2126), which contains the fix, so the pin retires as its own comment instructed. Verified: `logical_join_roundtrip_preserves_null_equality` passes against datafusion-proto 54.1.0 from crates.io with no patch block. * `ballista/core/src/serde/mod.rs` — both sides appended tests at the same point in the `test` module. Kept both: this branch's `logical_join_roundtrip_preserves_null_equality` plus main's `RuntimeStatsExec` / `BufferExec` codec round-trip tests. * `benchmarks/src/bin/tpcds.rs` — both sides removed different entries from the same `SKIP` group. This branch un-skipped q38/q87 (fixed by the `null_equality` change); main un-skipped q4/q78 (fixed by apache#2072, which removed the incorrect SortMergeJoinExec broadcast conversion). Took the union, so all four are now in the correctness gate and the `apache#2046` group and its comment are gone.
andygrove
added a commit
to andygrove/datafusion-ballista
that referenced
this pull request
Jul 22, 2026
Resolves three conflicts: * `Cargo.toml` / `Cargo.lock` — drop the `[patch.crates-io]` DataFusion git pin. It existed only to pick up apache/datafusion#22785 (logical `JoinNode` proto round-trip preserving `null_equality`), which was on `branch-54` but not in the released 54.0.0 tag. main is now on released DataFusion 54.1.0 (apache#2126), which contains the fix, so the pin retires as its own comment instructed. Verified: `logical_join_roundtrip_preserves_null_equality` passes against datafusion-proto 54.1.0 from crates.io with no patch block. * `ballista/core/src/serde/mod.rs` — both sides appended tests at the same point in the `test` module. Kept both: this branch's `logical_join_roundtrip_preserves_null_equality` plus main's `RuntimeStatsExec` / `BufferExec` codec round-trip tests. * `benchmarks/src/bin/tpcds.rs` — both sides removed different entries from the same `SKIP` group. This branch un-skipped q38/q87 (fixed by the `null_equality` change); main un-skipped q4/q78 (fixed by apache#2072, which removed the incorrect SortMergeJoinExec broadcast conversion). Took the union, so all four are now in the correctness gate and the `apache#2046` group and its comment are gone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
DataFusion 54.1.0 was released, so Ballista should pick it up and stay current with the upstream patch release.
What changes are included in this PR?
Cargo.lockto the DataFusion 54.1.0 patch release.dev/update_datafusion_proto.py, which adds the newnull_awarefield toJoinNodeinballista/core/proto/datafusion.proto.Cargo.tomlexplaining why thedatafusion*requirements stay at54.The workspace requirements deliberately stay at
54rather than pinning54.1.0. Thepythonworkspace has its own lockfile but depends on the ballista crates by path, so an exact requirement here drags it onto 54.1.0 too, anddatafusion-python54.0.0 does not compile against 54.1.0:DataFusion 54.1.0 added a
schemafield to the publicRecursiveQuerystruct, and there is no 54.1.0 release ofdatafusion-pythonyet. Keeping the requirement at54lets the main workspace build against 54.1.0 whilepython/Cargo.lockstays on 54.0.0 untildatafusion-pythoncatches up.python/is unchanged by this PR.Are there any user-facing changes?
No API changes in Ballista itself. The scheduler, executor, CLI and benchmarks now build against DataFusion 54.1.0. The Python wheel continues to build against 54.0.0.