AIP-76: Add PartitionAtRuntime authoring API to Task SDK#65447
AIP-76: Add PartitionAtRuntime authoring API to Task SDK#65447anishgirianish wants to merge 12 commits intoapache:mainfrom
Conversation
38335fb to
e71d7e3
Compare
939a95e to
08d1a66
Compare
08d1a66 to
ee3ee08
Compare
Lee-W
left a comment
There was a problem hiding this comment.
I think we also need airflow-core counter part for PartitionAtRuntime
|
@anishgirianish Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush. Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
2a7a684 to
5685717
Compare
|
Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item: Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project. Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
Sure, thank you. |
5685717 to
a9ceef5
Compare
Done, added Thank you |
|
@Lee-W Thank you so much for the review. addressed all three comments, ready for another look whenever you get a chance, thanks. |
927f68b to
7573eef
Compare
Lee-W
left a comment
There was a problem hiding this comment.
If I'm understanding it correctly, we're only adding the interface but not actually making it work yet? (since we did not touch the code in scheduler and use partitioned_at_runtime)
e833c4a to
b110249
Compare
856f352 to
73a1da7
Compare
@Lee-W Thank you much for the review! Yep this PR is interface only; the scheduler wiring + example DAG + docs will land together in a follow-up PR. I have addressed feebacks in the latest push.Would like to request you for a another look when you whenever you get a chance? |
Lee-W
left a comment
There was a problem hiding this comment.
mostly good from my end and would need @uranusjr 's help for another round.
also please take a look at airflow-core/src/airflow/api_fastapi/execution_api/versions/v2026_04_06.py and contributing-docs/19_execution_api_versioning.rst. it's the last missing piece from me :)
73a1da7 to
93b5805
Compare
93b5805 to
6e9a635
Compare
Was generative AI tooling used to co-author this PR?
First of three PRs decomposing draft #65300 for AIP-76, per the design agreed in #44146.
Adds the Task SDK producer surface for runtime partition keys:
PartitionAtRuntime()marker timetablePartitionKey(key, extra)for per-event metadataoutlet_events[asset].partition_keys = [...]/.add_partition(...)self.partition_keys = [...]inside@assetvia_AssetSelfProxyoutlet_eventskwarg injection in@asset/@asset.multiNo server-side consumption yet, that lands in PR 2 (core behavior + fan-out) along with PR 3 (example DAG + docs).
cc: @Lee-W
related: #44146
related: #65300
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.