Increase timeout to read in DAGs

[waardd]

If i look at the Airflow logging for a certain DAG i see that it loads this DAG into the system every 30 seconds

2023-08-16 08:18:41 | [2023-08-16T08:18:41.590+0200] {processor.py:153} INFO - Started process (PID=221050) to work on /appl/mca/airflow/dags/TimescapeCommoditiesImport.py
-- | --
2023-08-16 08:18:41 | [2023-08-16T08:18:41.591+0200] {processor.py:743} INFO - Processing file /appl/mca/airflow/dags/TimescapeCommoditiesImport.py for tasks to queue
2023-08-16 08:18:41 | [2023-08-16T08:18:41.592+0200] {logging_mixin.py:137} INFO - [2023-08-16T08:18:41.592+0200] {dagbag.py:538} INFO - Filling up the DagBag from /appl/mca/airflow/dags/TimescapeCommoditiesImport.py
2023-08-16 08:18:41 | [2023-08-16T08:18:41.623+0200] {processor.py:753} INFO - DAG(s) dict_keys(['TimescapeCommodities']) retrieved from /appl/mca/airflow/dags/TimescapeCommoditiesImport.py
2023-08-16 08:18:41 | [2023-08-16T08:18:41.720+0200] {logging_mixin.py:137} INFO - [2023-08-16T08:18:41.720+0200] {dag.py:2690} INFO - Sync 1 DAGs
2023-08-16 08:18:41 | [2023-08-16T08:18:41.745+0200] {logging_mixin.py:137} INFO - [2023-08-16T08:18:41.745+0200] {dag.py:3437} INFO - Setting next_dagrun for TimescapeCommodities to 2023-08-17T06:00:00+00:00, run_after=2023-08-17T06:00:00+00:00
2023-08-16 08:18:41 | [2023-08-16T08:18:41.793+0200] {processor.py:175} INFO - Processing /appl/mca/airflow/dags/TimescapeCommoditiesImport.py took 0.206 seconds

Is there a way to increase this timeout to for example 10 minutes to increase performance?

[Taragolis]

In general for reduce usage of resources or improve time to appear new DAGs and changes you should adjust selected parameters

• AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: by default Airflow scan for new files only once per 5 minutes
• AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: This is the setting you are looking for. number of seconds after which a DAG file is parsed, default: 30
• AIRFLOW__CORE__MIN_SERIALIZED_DAG_FETCH_INTERVAL - Fetching serialized DAG, default to 10
• AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL - Updating serialized DAG, default to 30

[potiuk]

Also we have quite comprehensive documentation on Fine tuning your scheduler performance that not only mentions various parameters but gives a little bit of context of why we have so many parameters and generally tells you what "knobs" you can turn to what effect .