[Depends on #65958] Refactor dag discovery to respect coordinator#66392
Draft
jason810496 wants to merge 49 commits intoapache:mainfrom
Draft
[Depends on #65958] Refactor dag discovery to respect coordinator#66392jason810496 wants to merge 49 commits intoapache:mainfrom
jason810496 wants to merge 49 commits intoapache:mainfrom
Conversation
- Introduced the `apache-airflow-providers-languages-java` package with version 0.1.0. - Added Java-specific task coordinators and DAG file processors. - Created documentation including README, changelog, and installation instructions. - Implemented provider info retrieval and commit tracking. - Established testing framework with initial unit tests for Java provider components.
…ng for improved performance and reliability
…e related components
- Renamed all instances of "process coordinators" to "runtime coordinators" in the codebase. - Updated the ProvidersManager and ProvidersManagerTaskRuntime classes to handle runtime coordinators. - Modified the DagFileProcessorManager to collect file extensions from runtime coordinators. - Adjusted the Java provider to implement the new runtime coordinator structure. - Updated tests to reflect changes from process to runtime coordinators.
…_runtime_mapping] config
Tweak coordinator class names, attribute names, and method names to be shorter and avoid the term 'runtime'.
- Remove Java SDK setup in Dockerfile - add multi-language extras documentation - Update TaskInstanceDTO description, and adjust API version in generated files
jason810496
requested review from
hussein-awala,
o-nikolas and
pierrejeambrun
as code owners
boring-cyborg
Bot
added
area:ConfigTemplates
area:DAG-processing
area:dev-tools
area:Executors-core
jason810496
removed
area:providers
area:Executors-core
jason810496
added
AIP-108: Coordinator
jason810496
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DAG discovery hard-coded the
.py/ ZIP heuristic and only delegated to coordinators via a flat list of file extensions. That prevented per-file decisions and let coordinator-claimed ZIPs be double-enumerated by the generic ZIP path.What
airflow.utils.file.list_py_file_pathswithairflow.dag_processing.manager.discover_dag_file_paths..py→ coordinator (can_handle_dag_file(bundle_name, path)) → generic ZIP. Coordinator wins over ZIP, so a claimed.jar/archive isn't also scanned as a dag-zip.bundle_nameso coordinators can scope per bundle._runtime_file_extensionscache;_get_observed_filelocsnow asks coordinators directly.Was generative AI tooling used to co-author this PR?