Fix branch not skipping mapped tasks with custom XCom backend#68954
Draft
darshil929 wants to merge 2 commits into
Draft
Fix branch not skipping mapped tasks with custom XCom backend#68954darshil929 wants to merge 2 commits into
darshil929 wants to merge 2 commits into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
Contributor
|
Please provide evidance of before and after images of a running dag with the setup of custom backend. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a custom XCom backend stores values externally (e.g. in S3), a branch operator stops skipping some of its downstream tasks on Airflow 3 — specifically dynamically-mapped tasks and tasks that are cleared and re-run.
Why: the branch records its skip decision as an XCom, and that XCom is pushed through your custom backend, which only persists a pointer to the real value. The scheduler reads that XCom as plain JSON to decide what to skip, can't interpret the pointer, and lets the task run instead of skipping it. It's the same root cause as the original 2.10.5
NoCredentialsError— on Airflow 3 the scheduler no longer calls the backend, so the crash turned into a silent skip failure.Fix: keep the skip decision out of the custom backend. It's now stored in the same plain form Airflow uses by default, so the scheduler can always read it. Nothing changes when no custom backend is configured, and there's no Execution API change.
Added a regression test that runs a real branch operator with an externalizing backend configured — it fails before this change and passes after.
Separate from #52219, which was a different skip bug already fixed in #53455.
closes: #50491
Was generative AI tooling used to co-author this PR?