Blog for DataFusion 46.0.0

[alamb]

Is your feature request related to a problem or challenge?

• We are in the final phases of releasing 46.0.0: Release DataFusion 46.0.0 #14123
  It would be great to write a blog post about the release

Describe the solution you'd like

I would like a blog about DataFusion 46, similarly to how @Omega359 wrote

• DataFusion 45 blog post datafusion-site#57

Except I think it should be easier to write such a post as 46 is only 1 month of work, rather than 8months

Describe alternatives you've considered

In terms of content, I think we can draw from

Changelog: https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md

• March 4, 2025: This week(s) in DataFusion #15005
• Feb 4, 2025: This week(s) in DataFusion #14491

Some suggested things to point out:
Upgrade Guide for help updating to DataFusion 46.0.0

Performance related:

feat: Add ScalarUDF support in FFI crate #14579 (timsaucer)

Better Error Messages

Quality improvements

Community Highlights

Additional context

FYI @oznur-synnada

[alamb]

FYI @oznur-synnada perhaps you are willing to try writing this post?

[berkaysynnada]

FYI @oznur-synnada perhaps you are willing to try writing this post?

Yes, she is. I'm reaching out her now

[oznur-synnada]

FYI @oznur-synnada perhaps you are willing to try writing this post?

Yes, @alamb, noted. I'm going to take this one

[alamb]

Thank you! I will love to help write it too! Just let me know!

[berkaysynnada]

Hi @alamb. We are considering highlighting the following items:

Major Items:

• Add related source code locations to errors by eliaperantoni

• Introduce unified DataSourceExec for provided datasources, remove ParquetExec, CsvExec, etc by mertak-synnada

• feat: Add ScalarUDF support in FFI crate by timsaucer

• StatisticsV2: initial statistics framework redesign by Fly-Style

• Feature: AggregateMonotonicity by mertak-synnada and Window Functions Order Conservation -- Follow-up On Set Monotonicity by berkaysynnada

• GSoC Acceptance of Datafusion

Minor Mentions:

• feat: Implement UNION ALL BY NAME by rkrishn7

• perf: Improve median with no grouping by 2X by 2010YOUY01

• Improve performance 10%-100% in FIRST_VALUE / LAST_VALUE by not sort rows in FirstValueAccumulator by blaginin

• Add range table function by simonvandel

• optimize performance of the repeat function (up to 50% faster) by zjregee

Would you like to make any update to this list?

[alamb]

Thanks @berkaysynnada

In general I suggest emphasizing things that many users of the crate will see / appreciate and mentioning, but not too deeply, things that developers of the crate care more about

I also suggest that until we enable a feature by default we should not make a big deal about it. The rationale is that most users won't use it until it is on by default, and we can make a big deal about it when it is available

I suggest emphasizing:

• feat: Add ScalarUDF support in FFI crate #14579
• GSOC for sure

I would also mention that we are working on statistics / monotonicity, but until there is something major we can point at that is user facing (e.g. an important SQL query that now goes faster with the monotonicity changes) I probably wouldn't emphasize them

I think these ones are worth mentioning but don't make a huge deal:

• Introduce unified DataSourceExec for provided datasources, remove ParquetExec, CsvExec, etc #14224

These ones are in progress so I would suggest we wait t:

• Add related source code locations to errors #13664
• StatisticsV2: initial statistics framework redesign #14699

I would suggesting highlighting the performance improvements

• perf: Improve median with no grouping by 2X perf: Improve median with no grouping by 2X  #14399 (2010YOUY01)
• Improve performance 10%-100% in FIRST_VALUE / LAST_VALUE by not sort rows in FirstValueAccumulator Improve performance 10%-100% in FIRST_VALUE / LAST_VALUE by not sort rows in FirstValueAccumulator #14402 (blaginin)
• Speed up uuid UDF (40x faster) Speed up uuid UDF (40x faster) #14675 (simonvandel)
• perf: Drop RowConverter from GroupOrderingPartial perf: Drop RowConverter from GroupOrderingPartial #14566 (ctsk)
  Speedup to_hex (~2x faster) Speedup to_hex (~2x faster) #14686 (simonvandel)
• Implement predicate pruning for not like expressions Implement predicate pruning for not like expressions #14567 (UBarney)
• Speed up chr UDF (~4x faster) Speed up chr UDF (~4x faster) #14700 (simonvandel)

Also I suggest pointing people at the new upgrade guide: https://github.com/apache/datafusion/blob/main/docs/source/library-user-guide/upgrading.md#datafusion-4600

And maybe we can refer people to https://github.com/apache/datafusion/blob/main/dev/changelog/46.0.0.md for more details

[alamb]

Completed in

• Blog post for DataFusion 46.0.0 datafusion-site#64

Published: https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0/