Release cycle, long term support

[jykae]

Would like to open discussion about release cycle.

Users of Airflow would probably like to know from which release to jump to next release without breaking things too much.
Imho Django has nice release cycle, similar to Ubuntu, long term support releases, that are quite safe points to jump from LTS to next LTS

See Supported Versions section https://www.djangoproject.com/download/

Could Airflow have release process/cycle like that in the future?

[potiuk]

Airflow already has release process. See below.

But also just FYI - this is not the place to discuss it - if you want to bring it, it should go to devlist discussion if you have some proposals. This is where all things impacting Airflow as a product should be discussed see the information about comunity here
https://airflow.apache.org/community/

The current release process is based on semver and described here https://airflow.apache.org/docs/apache-airflow/stable/release-process.html - which means that until Airlfow 3 (which is not even planned). Airflow is backwards compatible and you should be able to migrate without hassle to latest version.

Also in Airflow 2.6.0 (which will be released in a fe days) our pages will have this clarification on what is the public interface that you can rely as a user on (and what is just an implemetnation detail) https://github.com/apache/airflow/blob/main/docs/apache-airflow/public-airflow-interface.rst

Some more "developer" related details about it can also be found here https://github.com/apache/airflow#semantic-versioning

We have discussed various approaches in the past (you can search our devlist archives for that), but generally speaking the approach is that you are supposed (and it should be easy) to always migrate to latest version in MAJOR branch (currently 2.*) without worrying about Airflow core "product" compatibilty.

Note that Providers have separate life-cycle and each of them can independently introduce their own breaking changes but you should be generally upgrade and downgrade them indepently from Airlow following the limitations described in https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers.

[jykae]

Thanks for your answer @potiuk !

I will check out material you pointed out. As amount of maintained DAGs rises, we need to plan how often we should upgrade Airflow to keep up with Airflow but optimize also the maintenance work.

[potiuk]

The more often the better. That's the general rule. If you do your upgrades in smaller increments, your mIgration work will be easier and the maintenance effort overall is smaller because it will be easier to rollback if you find an issue and ieasier to diagnose the root cause. Also by doing it more often you learn n how your deployment works and you automate more of the tasks that can be automated because you have bigger incentive. This is general rules for applocation like airflow.

You are like mixing OS upgrade strategy with application upgrade strategy. For applocations generally the .ore.often you upgrade the better (release early/release often) because there are no other apps that depend on the app as a base and you can test everything in staging. For OS-es it's a different story because usualy you have many apps using basic, foundational features that many apps expect from the OS.

When you look at OS migration and mobile app or browser migration that happens almost behind the scenes). Airflow is a bit between - you still want to test it before migration and have a staging environmen. And migrate it almost immediately after you can test that the new version works.

Also we are only releasing security fixes in the latest 2.x line (for example as of Sunday we will only release security bugfixes in 2.6.* and you are anyway expected to migrate to 2.6 if you want to apply jt- so if there will be an important security fix you will have to migrate anyway - the longer you delay and the more version you are behind, the more difficult it will be for you).

I think comparing it to OS migration is wrong parallel as there are completely different basic assumptions for OS-es.

We are using strict semver. Which means that we only release bugfixes in patchlevel versions. My personal recommendation if you care a lot about stability - when .0 version is released wait for those who are eager users and early reports and bugfixes that lead to .1 patchlevel usually in weeks. Install .0 in your staging, test it and report any issues you found, this will also contribute back to the community (Airflow is mainrai Ed by almost 2500 people - most of them work for free and you pay exactly 0 USD to use it so contributing back this way is the least you can do). Then once .1 is out test it with the goal of migration and migrate right after - do the same with any other patchlevel version.

This is the pattern i recommend to follow.