How to deploy to docker swarm with new compose file in the docs?

[trydalch]

I've been trying to deploy airflow onto a docker swarm cluster, but have been unsuccessful thus far. It's a brand new docker swarm cluster using Docker version 24.0.5, build ced0996.

I think I've gathered that the compose file in the docs relies on a newer version of docker-compose than the version that docker swarm uses. Specifically, it seems to use the depends_on.condition field instead of just a list of service names. I tried reverting to the old format, but while it tried to spin things up, redis and postgres weren't ready by the time docker tried to start up the other services.

But perhaps I'm doing something wrong. Is there anyone that can point me in the right direction.

[Victor384]

I have same issues, plz some one can give some hint!

[potiuk]

You can use any of the older docker-composes - just look at older versions of Airflow - we've been publishing "quick start" for years.

Just one note here for anyone looking for similar issues.

It's absolutely up to you what you do with that docker-compose. You can twist and change it and even manually develop parts of it as you see fit. Do not treat the docker-compose of Airflow as an "official" compse. This is merely an example and quick start that you have to adapt and update to your needs.

Peoople often mistake docker-compose wiith "a turn-key" solution that you can configure and use as black-box - nothing like that. When you decide to use docker-compose, you sign up to learn how it works and to modify it to adapt to whatever usage scenario you have. There are no promises nor expectations that it will work with wide range of options or versions.

In the upcoming version of Airflow, the docker-compose of ours will have even higher min-docker and min-docker-compose version:

MIN_DOCKER_VERSION = "23.0.0"
MIN_DOCKER_COMPOSE_VERSION = "2.14.0"

Those are 3-6 months old. And we stop supporting docker-compose v1 (it stopped receiving updates last month). And this is by design.

The docker-compose of ours is merely an example, not an "official" release of any sort. So while you can get help from other users, maintainers focus is not on that, you are mostly on your own and other user's help (but I recommend just study and learn the version of docker-compose and swarm you are using, release notes docker-compose and docker-swarm releases and figuring out the needed changes based on those studies). To be perfectly honest - you will definitely need to understand a lot there if you sign up to use both swarm and compose.

And you are perfectly free to modify and adapt the docker-compose to older versions - remove the depends_on etc. - look at old versions that have been released years ago in previous versions with even lower docker min-compatibility

On the other hand - the maintainers and community work on and release Helm Chart that uses K8S and this has much bigger potential for being a configurable turn-key solution if you need one. You still need to understand K8S etc. but many more things are configurable and we even run tests in CI that cover vast number of configuration parameters we have there.