Move Pod*Exceptions to separate module to avoid unnecessary slow imports in CLI #45759
+95
−12
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pod*Exceptions has been moved from core to providers back in de92a81, but kept in
airflow/exceptions.py
under try-case for backwards compatibility and allow for usage of an older k8s provider with a newer core airflow.This resulted in 2 unintended side-effects:
airflow
resulted in importingkubernetes
client. It is effectively the most expensive import out of all CLI does for any command, even though it is used by very few commands. Here are the timings of a trivialairflow dag-processor --help
pod_generator.py
, the import fromexceptions.py
failed even if providers were up to date with recursive import. But it was caught by exception handling, resulting infrom airflow.exceptions import PodMutationHookException
andfrom airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookException
pointing to different classes defeating the purpose of the fallback.This PR addresses both by moving PodGenerator exceptions to the separate module that only import its base class. This keeps imports backwards compatible, doesn't attempt to load k8s modules and fixes the divergence of exception classes in
airflow.providers.cncf.kubernetes.pod_generator
andairflow.exceptions
.It is worth noting, that all usages of those exceptions in core Airflow also has been cleaned up since then, so if you believe that we need to remove those exceptions from
airflow/exceptions.py
, please let me know.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.