-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
March 4, 2025: This week(s) in DataFusion #15005
Comments
This was referenced Mar 4, 2025
@XiangpengHao made a new post on more optimal predicate pushdown (we'll get it on in DataFusion finally!) |
Next week: #15269 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge?
Introduction
This ticket is my weekly-ish summary of interesting things happening in DataFusion. Note this is not a complete list (it is what I remember / can find). Please leave comments on this ticket about things that I may have missed or you think should get wider attention by the community.
Community Highlights
Releases!
Performance
DataFusion's core value proposition is great performance without having to re-implement it yourself
to_hex
2x faster: Speedupto_hex
(~2x faster) #14686to_hex
4x faster: Speed upuuid
UDF (40x faster) #14675 (no string copies for the win!)date_trunc
to be 2x faster: Speedupdate_trunc
(~20% time reduction) #14593substr
faster: Always useStringViewArray
as output ofsubstr
#14498Quality
Testing
Bug Fixes
DataFusion is in the "we are finding all the corner case bugs now" phase of its life and people are now bashing them down
LogicalPlanBuilder
or when building logical plans from Substrait #14860return_type_from_args
instead ofreturn_type
#14852 @rluvaton 🙏Docs
Build time
Cleanups 🧹
invoke_args
etcFeatures
Features under way
Better Out of Core Support
In general, DataFusion is getting better at handling datasets that are larger than can fit in memory.
StringView
due to shared buffers #14823We can have nice things! (Explain plans)
tree
/ pretty explain mode #14677. I'll give you a teaser below. Come help with the follow on work on [EPIC] CompleteSQL EXPLAIN
Tree Rendering #14914Better Error Messages
@eliaperantoni is working with various contributors to make the error messages better. This work is tracked in
Diagnostic
to more errors #14429DataFusionError::Collection
to return multipleDataFusionError
s #14439Misc
range
table function #14830UNION ALL BY NAME
feat: Implement UNION ALL BY NAME #14538Looking to get more involved? Please help review code! 🎣
DataFusion has a long history of community members contributing in all aspects of the project. Reviewing PRs is an especially great way to get introduced to the project, help the community and grow your own knowledge -- researching and understanding the code enough to review PRs also often inspires additional ideas for improvements.
We have docs about reviews. TLDR is: look for test coverage, if the change is understandable and well documented, and if the code can be improved. When you think the PR looks good to merge, try
@
mentioning one of the committers.Help wanted
Please feel leave your own comments on this ticket if you are looking for help
Community
Upcoming meetups:
The text was updated successfully, but these errors were encountered: