[DISCUSS] Lower Friction / Lower Ceremony (faster?) releases

[alamb]

Is your feature request related to a problem or challenge?

Releasing DataFusion versions is described here:

• https://github.com/apache/datafusion/blob/main/dev/release/README.md

We have been releasing about once a month for the last three years. You can see the history here:

• https://crates.io/crates/datafusion/versions

However, there is some non trivial overhead to doing releases as @findepi has observed, such as the tarball creation and voting process.

Interestingly I believe that one of @jorgecarleitao 's concerns about arrow-rs when he created arrow2 was the (slow) pace of the Apache release model

Describe the solution you'd like

I would like to discuss what we can do to make releasing easier (and therefore potentially more frequent)

Describe alternatives you've considered

No response

Additional context

• [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) #13648

[alamb]

I have spent quite a bit of effort working on the 54.0.0

• [DISCUSS] Lower Friction / Lower Ceremony (faster?) releases #14428

However in my mind most of that additional effort comes from the pre-release testing we are doing with various downstream projects rather than the specifics of releases. See this for more detail

• [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) #13648

[alamb]

Another thing @findepi has mentioned is potentially publishing "non official" / "nightly" releases to crates.io to make it easier for downstream projects to test pre-releases.

[xudong963]

Another thing @findepi has mentioned is potentially publishing "non official" / "nightly" releases to crates.io to make it easier for downstream projects to test pre-releases.

The suggestion makes sense to me.

[westonpace]

Another thing @findepi has mentioned is potentially publishing "non official" / "nightly" releases to crates.io to make it easier for downstream projects to test pre-releases.

I don't think crates.io (or cargo) have support for a nightly release concept? What we've done in lance, for Rust, is to create pre-release tags. We still trigger them manually (e.g. it's not a nightly thing) but it's just a button press and very low friction.

Downstream users have to then specify the dependency with the tag:

lance = { "version" = "=0.23.0", git = "https://github.com/lancedb/lance.git", tag = "v0.23.0-beta.4" }

Overall, I'd say it works well. It also avoids any implication of an official release which is nice.

[westonpace]

(ah, I see you can do -beta, -alpha tags, so definitely supported, just not in the same way as python)

[findepi]

Another thing @findepi has mentioned is potentially publishing "non official" / "nightly" releases to crates.io to make it easier for downstream projects to test pre-releases.

Not sure I deserve credit for this idea. It's tracked by #5023.

Regardless, I think we can easily agree that having lower-overhead releases is a good thing.
Also, per se, It is better not to have two different types of releases (official and unofficial).

1. What things from release script can be further automated?
2. What human processes around releases (such as voting or deciding whether to release) can be considered subject to change?

[alamb]

What things from release script can be further automated?

I think we could automate creating the tarball / tags perhaps.

For what it is worth, this past release by far most of my time was spent chasing down bugs / tickets /testing (aka the "deciding whether to release" in your description).

[findepi]

can "deciding whether to release" be "automated" by marking mandatory issues and PRs as release-blockers?
The more external testing we want to do, the harder is to release. Yes, this gives more confidence in particular release, but it costs precious time resources.
(downstream projects should be well motivated to contribute good tests for Datafusion, so that eventually the net result is more confidence without having to chase)

[alamb]

The more external testing we want to do, the harder is to release. Yes, this gives more confidence in particular release, but it costs precious time resources.

I agree this is the fundamental tradeoff