Remove Pydanitc models introduced for AIP-44

[potiuk]

The Pudanic models have been used in a number of places still and we are using them also for context passing for PythonVirtualEnv and ExternaPythonOperator - this PR removes all the models and their usages.

Closes: #44436

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

It's not likely this one will be green, but it's likely it will show which parts have to be fixed because they used implicitly Pydantic Models synchronisation.

But generally speaking it is ready to take a first pass of review.

Copilot reviewed 5 out of 12 changed files in this pull request and generated no suggestions.

Files not reviewed (7)

• airflow/serialization/pydantic/init.py: Evaluated as low risk
• airflow/serialization/pydantic/asset.py: Evaluated as low risk
• airflow/serialization/pydantic/dag.py: Evaluated as low risk
• airflow/serialization/pydantic/dag_run.py: Evaluated as low risk
• airflow/serialization/pydantic/job.py: Evaluated as low risk
• airflow/serialization/pydantic/taskinstance.py: Evaluated as low risk
• airflow/serialization/pydantic/tasklog.py: Evaluated as low risk

Comments skipped due to low confidence (1)

airflow/models/taskinstance.py:1

• The function _get_previous_dagrun_success is missing the task_instance and session parameters. It should be defined as def _get_previous_dagrun_success(task_instance, session) -> DagRun | None:.

def _get_previous_dagrun_success() -> DagRun | None:

[potiuk]

BTW. This is the last one in the #44436 series !

[jscheffl]

A small leftover with DagTag - but otherwise looking good!

[pierrejeambrun]

Looking good.

I will not say that I'm glad to see all that gone, but I kind of am :p. Working with DagRun | DagRunPydantic | None and similars wasn't super easy

[dstandish]

happy we're removing use_pydantic_models

[potiuk]

happy we're removing use_pydantic_models

Yeah ... I had no time to look at the failures today ... but tomorrow 🤞 is the day :). I just feel that there will be some left-overs with the context and some TODOs to fix - it's worth to do it anyway :)

[potiuk]

Speaking of which. I think we have to agree on Pydantic strategy here.

There are two reasons why this one is "red"

• as discussed before - we are passing Context in PythonVirtualenv operator to virtualenv when use_airflow_context=True - which is likely not good idea for Standard provider.

So far it was really a "future" feature - only working with Airflow 3 and in Airflow 2 it was only working with AIP-44 enabled (which was never a production option, except a few crazy people (cc: @jscheffl ;). So we could possibly even drop it. Or we can likely serialize it using the same mechanism serialization will happen for TaskSDK (@kaxil @ashb)? What would it be? I looked at the datamodels in task_sdk and have not seen context autogenerated yet (did I look in the right place?) maybe it's somewhere else.
So far context was serializable with assumption that AIP-44 is enabled and the "ORM" objects were serialized to their Pydantic counterparts - the whole contex serialization was added here #38567 by @dstandish and it assumed all the ORM models that were part of the context were serializable (this will stop being true once we remove the `*Pydantic classes' )

What plans do you have for for context serialization ?

• also current Fast API implementation uses DagTagPydantic as serialization layer. I am not sure if we can do it differently @pierrejeambrun but that sounds like there should be an easy way to do so?

[pierrejeambrun]

also current Fast API implementation uses DagTagPydantic as serialization layer. I am not sure if we can do it differently @pierrejeambrun but that sounds like there should be an easy way to do so?

We just need to copy into the api_fastapi/core_api/datamodels the DagTagPydantic. (and rename it into DagTagResponse, those are specific RestAPI serializers). The API has it own serializers, but this one was common so I re-used it, but I'm not sure it was even a good idea in the first place. (Now that all those are going away there's no hesitation anymore 😀)

[potiuk]

So from those two responses:

We haven't tackled context yet, but these models can be removed and we'll add back the ones we need in the execution SDK (some will be very different, some similar) so merge when ever you are happy with this

We just need to copy into the api_fastapi/core_api/datamodels the DagTagPydantic. (and rename it into DagTagResponse, those are specific RestAPI serializers). The API has it own serializers, but this one was common so I re-used it, but I'm not sure it was even a good idea in the first place. (Now that all those are going away there's no hesitation anymore 😀)

Thanks for those answers!

Let me try to complete this one then. I will leave the "use_airflow_context" as a feature of Python Virtualenv that only (will) work in Airflow 3 and leave to DO to add context propagation once we have it done for AIP-72.

And we should be able to close this one.

[pierrejeambrun]

Nice thanks.

[potiuk]

Reopen with full-tests-needed

[potiuk]

Ok. Looks like Pydantic classes removal is ready to merge - any last pass? I moved the DagTag to fast_api models, and added error messages/conditional tests/ left TODOs to bring use_airflow_context when we have serializable context for AIP-72.

If anyone want to have last pass -> feel free.

[jscheffl]

Boom. Let is dissolve :-D

[gopidesupavan]

finally :)