AbstractOperator should have broader support when resolving Jinja templates

[shane-lab]

Description

At the moment the Jinja templates are resolved at the AbstractOperator class during render_template. It is best practice to use the provided Kubernetes client package for defining custom resources when using the derived KubernetesPodOperator. This gives issues when passing through class instances/objects because the assertions in the method that interpolates the templates currently only supports native types and instances of DagParam or XComArg.

Use case/motivation

To give a better example: say I would like to invoke an init container with templated values for my KubernetesPodOperator task using the corresponding classes from the provided k8s client package. The currently present assertions will ignore the class instances and require me to convert these types of objects over to native dictionaries.

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import models as k8s_models

....

init_container = k8s_models.V1Container(
    name="init",
    image="alpine:latest",
    args=["echo", "{{ ti.xcom_pull(key=\"my_xcom_value\") }}"]
)

task = KubernetesPodOperator(
    init_containers=[init_container],
    ...
)

task.template_fields = task.template_fields + ("init_containers",)

Fortunately, the KubernetesPodOperator is able to map from dict to the corresponding classes of the kubernetes client package already, which solves my "issue", but maybe a two-way conversion could be a solution to the mentioned inconvenience above.

Related issues

It would also be great if it was possible to overload/extend the template_fields through the constructor. At the moment one has to extend this object after creating an instance of the operator (or create a custom derived class).

e.g.

class MyKubernetesPodOperator(KubernetesPodOperator):
    def __init__(self, template_fields=(), **kwargs):
        super().__init__(**kwargs)
        self.template_fields = self.template_fields + template_fields

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Why don't you make a POC for that. I don't quite get what you really want to do, but you seem to be well versed in Python and discussion about actual code is much better than clarifying the issue in discussion.

That could also give you a chance to understanda how it works and see if what you propose is doable at all @shane-lab

[potiuk]

Draft PR would be great. You can even start with draft PR for the templated_fields in constructor and see if you can make it works first.

[uranusjr]

I beleive some parts of the scheduler and task runner code uses template_fields as a class variable, so run-time per-instance template_fields additions will break certain features.

[potiuk]

I converted that into discussion. I think having some POC and addressing the class-level variable issue would be needed to see if this is a viable "feature"

[shane-lab]

Sure, I can submit a PR. I think it's actually not too difficult of a task. However, so far I see setting up the local development on a Windows machine is a pain though😅

I try to see if having the enumerated template_fields accessing runtime variables of the operator task actually is an issue. Seems a bit likely though, but a lot of references do show up with a quick file search.

[uranusjr]

From my (ad-hoc) tests, most breeze commands should work at this point, especially tests and shell, which are the most important for a contributor. Not all pre-commit hooks work properly yet, but most of them do, and you can just SKIP the rest.

[potiuk]

Yeah. There are still some shell pre-commits that should be converted to python, we gradually do so :).

[potiuk]

@shane-lab - any help there is most welcome.

[potiuk]

BTW. Once we complete breeze migration I am thinking about adding a "windows" check in GitHub Actions CI and running some basic breeze commands there (including static checks) - maybe not for all PRs but for "main" builds it is likely a good idea.

GitHub supports free windows runners, so it should be easy.

[shane-lab]

Ah cool! Well, I am definitely not a Windows user. Just happen to have a personal laptop running Windows atm and prefer not to touch my work laptop in my free time 😄 . I have setup a simple virtualbox image instead for now to work out a POC.