chore: update datafusion to 51.0 - #1345
Conversation
| edition = "2021" | ||
| # we should try to follow datafusion version | ||
| rust-version = "1.86.0" | ||
| rust-version = "1.88.0" |
There was a problem hiding this comment.
Try to align with DataFusion rust version, feel free to let me know if we dont want to bump it right now
https://github.com/apache/datafusion/blob/main/Cargo.toml#L80
There was a problem hiding this comment.
I think it makes sense
|
Hey @milenkovicm — I continued your work by adding the new metrics type support, adjusting DF API usages, and updating the related tests. CI looks fine, appreciate your feedback whenever you get a chance. Thanks! |
|
Thanks @danielhumanmod will have a look later today |
milenkovicm
left a comment
milenkovicm
left a comment
There was a problem hiding this comment.
I think pr makes sense, thanks a lot! just few minor comments. Also, could you please update datafusion and datafusion-common.proto files to reflect df 51 versions (we usually do it with every df update)
| .build(); | ||
| let ctx = SessionContext::new_with_state(session_state); | ||
| // | ||
| let ctx = SessionContext::new_with_state(session_state).task_ctx(); |
There was a problem hiding this comment.
can we create TaskContext directly? we should have all required info for TaskContext
| // | ||
| // | ||
| // | ||
| let ctx = SessionContext::new_with_state(state).task_ctx(); |
There was a problem hiding this comment.
same comment, can we create task context directly ?
| edition = "2021" | ||
| # we should try to follow datafusion version | ||
| rust-version = "1.86.0" | ||
| rust-version = "1.88.0" |
There was a problem hiding this comment.
I think it makes sense
Thanks for the feedback! Updated all the comments in latest commit, feel free to merge it or let me know any further improvement |
|
thanks @danielhumanmod it looks good! |
2 participants
Which issue does this PR close?
Closes #1343 .
Rationale for this change
DataFusion 51 introduces new changes in its physical plan serde APIs, metrics system, and tonic/prost/Arrow dependencies
Reference: https://datafusion.apache.org/library-user-guide/upgrading.html#datafusion-51-0-0
What changes are included in this PR?
(no semantic changes)
Are there any user-facing changes?
No user-facing behavior changes.
This PR only updates internal dependencies and adapts Ballista to the breaking changes introduced by DataFusion 51.
All external APIs, CLI behavior, and user-visible features remain unchanged.