Kubernetes - Use dynamic images for worker pods

[ralphotowo]

Hi There,

We have an implementation of Airflow using the Kubernetes Executor. We have a pipeline that builds custom docker images with our custom tools and dependencies using the airflow image as a base, the image is pushed to our internal registry and used for all pods that are created by Airflow. This is done using the following configuration:

[kubernetes_executor]
   multi_namespace_mode = False
   namespace = airflow
   worker_container_repository = us-east4-docker.pkg.dev/project/airflow/airflow
   worker_container_tag = qa

Currently, our CI step continuously publishes images to the same qa tag however we'd like to stop doing that and use unique tags for every CI execution. Our CD tool, ArgoCD Image Updater is able to automatically update the triggerer, scheduler and webserver deployments but not the worker pods since they do not exist until they are required.

We tried removing the worker_container_repository configuration in hopes that Airflow will default to the active images being used for the triggerer, scheduler or webserver but this doesn't work. Does anyone have any ideas on how to dynamically set the image tag of worker pods?

[potiuk]

You can set it as usual for all configuration variables when you restart scheduler

AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG - that could be dynamically set for your scheduler Pod when ArgoCD restart it.