Container registry as version control and dynamic load sharing at runtime.

[philipx]

Dear All,
How do airflow batch controlling version? by git hub?
I dunno if it is possible
If a container registry host all contain which enclose the task
And each DAG just use container image tag to specify the task.

On runtime airflow pull the container from register to the cluster and run the task.
Then the airflow batch is independent on worker hardware nor setup.
and updating the batch flow is just pushing new container to registry and update the config.
I think kubenretes executor almost done this job but I think just select feature of kubenretes and helm is needed in this case

[potiuk]

Regarding versioning there is a proposal (not fully hashed out yet) about it. https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning . The proposal has not yet been accepted, it will need to be picked again and updated and eventually voted on during the devlist discussion.

Versioning using Docker images/registry was one of the options considered and proposed in the past, however since KubernetesExecutor is not the only one used by Airflow (see https://airflow.apache.org/blog/airflow-survey-2020/ - CeleryExecutor is ~50% of use) and we need to support different scenarios (Including the new combined CeleryKubernetesExecutor), it is highly unlikely container registry will be used for that (it's not really compatible with Celery way of running Airflow).

Some of that will be discussed at the upcoming Airflow Summit's talk Airflow Loves Kubernetes in two weeks.

[PhilipMok-cc]

Thank for your reply
I think I did not think it clearly before, container registry is only apply for kubenretes.
However helm itself is a package manager could we use something like helm chart to support different type of executer?
that is universal package manager config by YAML and different implementation of package pulling (git/folder/container register) and we have different type of asset pool?

[potiuk]

Airflow has the Official Airlfow Community Helm Chart released recently - with support for all the different kinds of executor, git-sync and others: https://airflow.apache.org/docs/helm-chart/stable/index.html

[philipx]

I think I was not having the big picture.
There is a task finger print concept already introduced but mind you. I would like to explain my concept again with a long story

Support I have production and UAT version which have 1 DAG
Prd:
DAG-Apple:1.0
Task1:1.0 -> Task2:1.0 -> Task3:1:0

However in UAT I have a updated version of task2 which mean my DAG has to update too?
UAT:
DAG-Apple:1.1
Task1:1.0 -> Task2:1.1 -> Task3:1:0

My requested feature is, for smooth deployment of the DAG-Apple:1.1 to prd env.
We should store all the package ( all version of Tasks and DAG) in somewhere (repository?)
Therefore when everything was checked in:

a) If we would like to promote the version 1.1 to prd
We should update a config file at prd
with sth like
Run DAG-Apple:1.1
and version control itself could pull the DAG and implementation detail from repository to the worker machine?

When rollback
We rollback the config file and the airflow could load the old program from repository base on the tag.

I do think this kind of version control is very useful as sometime we would conduct a before and after comparison to check the different of output of old and new version.

Is it already out of scope of airflow which I need to think of an integration of another product?

[potiuk]

That's what exactly what is part of discussion in AiP-36 and related discussions (it will be implemented soon) . You can package dag and related files in .zip file if you want, but that is not solving all issues. So if you need to do it now then you have to roll out your own solution.