Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-82 Handle trigger serialization #45562

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

vincbeck
Copy link
Contributor

@vincbeck vincbeck commented Jan 10, 2025

Handle serialization of triggers in DAGs as part of AIP-82. With AIP-82 (External event driven scheduling in Airflow) you can define DAGs as below:

trigger = FileTrigger(....)
asset = Asset("<my_queue>", watchers=[AssetWatcher(name="my_file_watcher", trigger=trigger)])
 
with DAG(
    dag_id=DAG_ID,
    schedule=asset,
):
    empty_task = EmptyOperator(task_id="empty_task")
 
    chain(empty_task)

Triggers can be instantiated as part of a DAG. As such, we need to serialize them.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@vincbeck vincbeck force-pushed the vincbeck/aip-82-serialize branch 4 times, most recently from fa9f975 to 5f2a621 Compare January 11, 2025 00:10
airflow/triggers/base.py Outdated Show resolved Hide resolved
airflow/triggers/base.py Outdated Show resolved Hide resolved
@vincbeck
Copy link
Contributor Author

vincbeck commented Jan 17, 2025

I addressed your comments but I also added something new. It was actually a suggestion from @dstandish that I think is a great idea! I introduced the concept of AssetWatcher that is a thin wrapper around trigger. Explanations:

Instead of having:

trigger = FileTrigger(....)
asset = Asset("<my_queue>", watchers=[trigger])

Now it is:

trigger = FileTrigger(....)
asset = Asset("<my_queue>", watchers=[AssetWatcher(name="my_queue_watcher", trigger=trigger)])

AssetWatcher does not add much besides just a name, but I think it is important because, in the future, we might want to surface the relations between assets and triggers in the graph view in the UI. Representing a trigger in the graph can be tricky because triggers have 2 pieces of information: classpath and kwargs. None of them are very suitable for displaying purposes. classpath is debatable but it is anyway not representable of the trigger. Example: you might have an asset with 2 watchers: each one uses the trigger FileTrigger to monitor a different file. Having only the classpath displayed will not help the user to understand which one is which. Hence the name.

@vincbeck vincbeck force-pushed the vincbeck/aip-82-serialize branch from f64e72d to 29b9834 Compare January 17, 2025 14:46
@vincbeck
Copy link
Contributor Author

@uranusjr @ashb what do you guys think?

@ashb
Copy link
Member

ashb commented Jan 21, 2025

I'll take another look today or tomorrow.

Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks much better thanks, but a few changes I'd like you to make please.

airflow/dag_processing/collection.py Outdated Show resolved Hide resolved
airflow/example_dags/example_asset_with_watchers.py Outdated Show resolved Hide resolved
airflow/serialization/serialized_objects.py Outdated Show resolved Hide resolved
task_sdk/src/airflow/sdk/definitions/asset/__init__.py Outdated Show resolved Hide resolved
@vincbeck vincbeck force-pushed the vincbeck/aip-82-serialize branch from 3859d7a to 021c863 Compare January 22, 2025 20:25
@vincbeck
Copy link
Contributor Author

Other questions/concerns? :)

@vincbeck
Copy link
Contributor Author

@ashb Are you okay with these changes? I'll make the changes related to the discussion we had on Slack (creating the interface BaseEventTrigger) in a separate PR.

@ashb
Copy link
Member

ashb commented Jan 31, 2025

All good, thanks @vincbeck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants