Option to include subdir in trigger_dag to not make scheduler scan the whole dag folder

[GagandeepS]

Description

We have a design where 10s of child dags gets created every few minutes and gets triggered from a parent dag. While triggering each of the child dag, I believe scheduler searches the whole dag bag which is making the whole process slower. We have put a while loop to run TriggerDagRunOperator/trigger_dagy, if it is successful in triggering then it exits the loop otherwise trigger it again.

I believe that to decrease the load from the scheduler, there should be a provision to supply subdir to TriggerDagRunOperator so that scheduler only searched for the dag_id inside of that folder instead of whole dagbag.

Use case/motivation

We have a parent dag trigger multiple child dags and we want to decrease the time to let scheduler discover the child dag faster.

Related issues

Discussion: #19547

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

As I can see in the code, this operator checks the path to DAG File in the database and only loads one file. I don't know what you would like to optimize here

airflow/airflow/api/common/experimental/trigger_dag.py

Line 117 in 37a12e9

dagbag = DagBag(dag_folder=dag_model.fileloc, read_dags_from_db=True)

[GagandeepS]

So we have a use case where multiple dynamic dags are getting added to the dagbag and I believe there will always be a latency between dropping a new dag into the dagbag folder and operator checking if the path/record of that new dag exists in the table or not using trigger_dag.

So, to let scheduler take as much time it needs to insert the record into the table, we trigger the new dag and
check if it gets triggered without error or not. If there is an error (usually 'Dag xxx does not exists') then it retries again in some time. So far so good, except when there is a peak load (10s of DAGs are getting generated dynamically and getting saved in the DAG bag). In this case scheduler gets slow coz it needs to insert multiple record and hence trigger_dag (coz of retry) takes 3-10min. I want to minimize this 3-10min.

Proposed solution: Potentially, either add a table in airflow backend data model or use an index or bulk insert or similar so that the performance of scheduler, while inserting the new record, does not gets hampered and searching of the new dag gets faster.

Just a thought: May be if we can have the provision to change the type of DB of Airflow so that instead of postgres, we can change it to a NoSQL with index matching that in postgres right now (I am hoping) so that inserting and searching gets faster and one can manage a not-so-fast update.

[potiuk]

I think that was a series of misunderstanding how Ariflow works. You need to have DAGs in the dag folder before you trigger them and Airflow needs to schedule them.

Scheduler works continuously. Triggering a DAG is instantaneous as long as DAG is already parsed by scheduler. I think you need to re-work your case - your proposal about changgn DB to NoSQL are out of the blue and I am not sure where the ideas are from. Please take a closer look at airflow design and implementation and if you want to make proposal that changes architecture and DB, you need to write Airflow Improvement Proposal https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals and before it discuss it on the devlist - but I suggest to be better prepared if you want to make proposals.

When you want to discuss improviements in some behaviours instead of proposing solutions that have absolute no ground in how Airflow work but are wild (and incorrect guesses) just start discussin on the observation and what you do (code examples would be good) and what you observe rather than proposing (wrong) solutions based on bad understanding how Airflow worksl.

I have a feeling you are doing something strange (for example mentioning DagBag indicates that you are trying to do something using internal airflow classes whch is most likely wrong). Please describe your case, what you observe, what you want to improve in enough level of details that we can try to take a look and see what you want to achieve in better way (if you are doing it wrongly) or indeed work out a agood way of improving things.