structure DAG

[sonnn1505]

I have 20 tables need to ETL parallel from MySQL to Postgres SQL
should I create a DAG combine all 20 table ?which best way for person maintain ?

[enchant3dmango]

I'll try to rephrase your question first: I have 20 tables that need to be extracted, transform, and loaded (ETL) parallelly from MySQL to PostgreSQL. Should I create a DAG to combine all 20 tables? Or what is the best way to create it that is easier to maintain?

Using a DAG to represent your ETL processes for each table is a solid approach, especially when you need to process tables in parallel. However, how you structure your DAG, modularize your code, manage dependencies, and handle errors will determine how maintainable and efficient your ETL pipeline is in the long run.

Another option, you may create only 1 DAG. But, you need to create subdag for each ETL process. In your case, means that you need to create 20 subdags.