Does it really matter if you import numpy when it will be in sys.modules after the first parse?

[karlkovaciny]

Talking about https://airflow.apache.org/docs/apache-airflow/2.7.0/best-practices.html#top-level-python-code, which says

import numpy as np  # <-- THIS IS A VERY BAD IDEA! DON'T DO THAT!

I would like to be able to import modules at the top level so I can define my functions with type hints. Could I potentially just extend the timeout the first time I ran the dag import, and then every subsequent parse would just fetch numpy out of sys.modules? Or does the scheduler make a brand new environment every AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT seconds so it can catch dynamic changes to the dags?

[potiuk]

You can use

if typing.TYPE_CHECKING:
    import numpy as np
...

def my_func(a: 'np.np_type'):
    ....

This is a standard way of adding imports that are expensive but needed for type hints:

See PEP specification https://docs.python.org/3/library/typing.html#constant

[potiuk]

You can also add from __future__ import annotations to avoid having to use quotes around the type specified (but this is all explained in the docs).

[karlkovaciny]

I appreciate the answer to what was annoying me, thanks. I'm still curious about the cost impact of repeated imports, since normally the second import of numpy costs nothing.

[potiuk]

Every DAG in Airflow is parsed in a separately forked process. This is in order to achieve a) isolation b) reloading classes every time import is made to make sure we load latest version of imported files c) in order to not crash whole DAG file processor in case - for whatever reason the import will fail (even with errors like SIGSEGV) - only the forked process will get killed and main process will continue forking processes to parse DAG files.

This means that whatever is imported in DAG parsed by DAG file processor is only cached in that forked process and is discarded once parsing of that individual file completes (the processes exit after the DAG is serialized to json form and saveed to database).

So no - we do not effectively cache such imports (and this is by design - we can't cache imports, for the reasons described above).

[karlkovaciny]

Thanks so much for the knowledgeable and sensemaking answer!