AIP-66: Add support for parsing DAG bundles #45532
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:API
jedcunningham
added
the
full tests needed
jedcunningham
force-pushed
the
bundle-parsing
branch
from
1e74bff to
37c41a2
Compare
Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).
ephraimbuddy
approved these changes
Jan 13, 2025
agupta01
pushed a commit
to agupta01/airflow
that referenced
this pull request
Jan 13, 2025
* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <[email protected]> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <[email protected]>
karenbraganz
pushed a commit
to karenbraganz/airflow
that referenced
this pull request
Jan 13, 2025
* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <[email protected]> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <[email protected]>
HariGS-DB
pushed a commit
to HariGS-DB/airflow
that referenced
this pull request
Jan 16, 2025
* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <[email protected]> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is #45371, which had to be reverted.
Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.
This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).