AIP-76: Add PartitionAtRuntime authoring API to Task SDK#65447
AIP-76: Add PartitionAtRuntime authoring API to Task SDK#65447anishgirianish wants to merge 13 commits into
Conversation
anishgirianish
requested review from
amoghrajesh,
ashb and
kaxil
as code owners
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
from
38335fb to
e71d7e3
Compare
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
from
939a95e to
08d1a66
Compare
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
from
08d1a66 to
ee3ee08
Compare
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
I think we also need airflow-core counter part for PartitionAtRuntime
|
@anishgirianish Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush. Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
3 times, most recently
from
2a7a684 to
5685717
Compare
|
Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item: Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project. Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
Sure, thank you. |
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
from
5685717 to
a9ceef5
Compare
Done, added Thank you |
|
@Lee-W Thank you so much for the review. addressed all three comments, ready for another look whenever you get a chance, thanks. |
potiuk
added
the
ready for maintainer review
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
from
927f68b to
7573eef
Compare
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
If I'm understanding it correctly, we're only adding the interface but not actually making it work yet? (since we did not touch the code in scheduler and use partitioned_at_runtime)
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
mostly good from my end and would need @uranusjr 's help for another round.
also please take a look at airflow-core/src/airflow/api_fastapi/execution_api/versions/v2026_04_06.py and contributing-docs/19_execution_api_versioning.rst. it's the last missing piece from me :)
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
2 times, most recently
from
93b5805 to
6e9a635
Compare
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
2 times, most recently
from
30b21ec to
7587a39
Compare
anishgirianish
force-pushed
the
aip-76-partition-at-runtime-authoring-api
branch
from
7587a39 to
4318045
Compare
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
LGTM. Maybe we can start work on the scheduler side, based on the current PR design, on a separate branch.
4 participants
Was generative AI tooling used to co-author this PR?
First of three PRs decomposing draft #65300 for AIP-76, per the design agreed in #44146.
Adds the Task SDK producer surface for runtime partition keys:
PartitionAtRuntime()marker timetable, with a newpartitioned_at_runtime: boolflag onBaseTimetableso downstream code can detect the marker without isinstance checks.outlet_eventskwarg injection inside@asset/@asset.multi— the parameter is populated with the runtimeOutletEventAccessors, andself,context, andoutlet_eventsare reserved parameter names that are not treated as inlet asset references.OutletEventAccessor.partition_keys: set[str]plusaccessor.add_partitions(key_or_keys)for recording one or more partition keys against an emitted asset event. Inside an@assetfunction the idiomatic call isoutlet_events[self].add_partitions("us")(oroutlet_events[self].partition_keys = {"us", "eu"});selfhere is a plainAsset._serialize_outlet_eventsnow fans the recordedpartition_keysout into one asset-event payload perpartition_key(singular) on the existing column. Duplicate keys recorded within the same task instance collapse to a single event.The server-side join, partition-aware consumption / fan-in by the scheduler — lands in PR 2, along with the example DAG and end-user docs in PR 3.
cc: @Lee-W
related: #44146
related: #65300
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.