Release DataFusion 47.0.0 (April 2025)

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be https://crates.io/crates/datafusion/45.0.0 (likely Feb 1, 2025) December 31, 2024 so next major release would be around March 1, 2025

Steps:

• Update version and changelog
• Test with DataFusion Python:
• Test with DataFusion Comet
• Test with delta.rs
• Test with SailHQ:
• Test with parquet viewer
• Voting Thread:
• Create ticket for next release:

Prior release tickets:

• 45.0.0: Release DataFusion 45.0.0 #14008
• 46.0.0: Release DataFusion 46.0.0 #14123

Changes to add to upgrade guide

These PRs made changes that deserve a mention in the upgrade guide

• chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) #15123
• chore: remove ScalarUDFImpl::return_type_from_exprs #15130
• Fix type coercion for unsigned and signed integers (Int64 vs UInt64, etc) #15341
• Refactor: add FileGroup structure for Vec<PartitionedFile> #15379

Features to mention in the blog (if they make it)

• [EPIC] Complete SQL EXPLAIN Tree Rendering #14914
• [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
• Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096
• feat: introduce JoinSetTracer trait for tracing context propagation in spawned tasks #14547

Bugs that would be good to fix

• Regression: eager evaluation of expressions inside CASE conditional expression #15384

Community Wishlist

• Comparison Operators for Decimals of Different Precisions and Scales #15174

@alamb 's wishlist

• Avoid casting columns when comparing ints and strings #15035
• [EPIC] Complete SQL EXPLAIN Tree Rendering #14914
• Incorrect cast of integer columns to utf8 when comparing with utf8 constant #15161
• [DISCUSS] Switch to tree explain by default #15343

[xudong963]

@alamb, I'll also be in charge of this release.

[alamb]

@XiangpengHao also offered to test with the parquet viewer prior to 47: #15102 (comment)

[xudong963]

@XiangpengHao also offered to test with the parquet viewer prior to 47: #15102 (comment)

That's great, added it to release steps

[shehabgamin]

I feel like this may be important enough to try to get into the release. Does anyone else have thoughts?

• Comparison Operators for Decimals of Different Precisions and Scales #15174

[alamb]

I feel like this may be important enough to try to get into the release. Does anyone else have thoughts?

• Comparison Operators for Decimals of Different Precisions and Scales #15174

Seems reasonable to me -- I have added it to the "good to get in " list

[alamb]

I believe the voting deadline has passed and I will now will prmote / publish it.

Update: wrong ticket (I was looking for 46.0.0