Skip to content

Fix branch not skipping mapped tasks with custom XCom backend#68954

Draft
darshil929 wants to merge 2 commits into
apache:mainfrom
darshil929:fix/50491-branch-skip-custom-xcom-backend
Draft

Fix branch not skipping mapped tasks with custom XCom backend#68954
darshil929 wants to merge 2 commits into
apache:mainfrom
darshil929:fix/50491-branch-skip-custom-xcom-backend

Conversation

@darshil929

Copy link
Copy Markdown

When a custom XCom backend stores values externally (e.g. in S3), a branch operator stops skipping some of its downstream tasks on Airflow 3 — specifically dynamically-mapped tasks and tasks that are cleared and re-run.

Why: the branch records its skip decision as an XCom, and that XCom is pushed through your custom backend, which only persists a pointer to the real value. The scheduler reads that XCom as plain JSON to decide what to skip, can't interpret the pointer, and lets the task run instead of skipping it. It's the same root cause as the original 2.10.5 NoCredentialsError — on Airflow 3 the scheduler no longer calls the backend, so the crash turned into a silent skip failure.

Fix: keep the skip decision out of the custom backend. It's now stored in the same plain form Airflow uses by default, so the scheduler can always read it. Nothing changes when no custom backend is configured, and there's no Execution API change.

Added a regression test that runs a real branch operator with an externalizing backend configured — it fails before this change and passes after.

Separate from #52219, which was a different skip bug already fixed in #53455.

closes: #50491


Was generative AI tooling used to co-author this PR?
  • Yes — claude sonnet 4.6

@boring-cyborg

boring-cyborg Bot commented Jun 24, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@eladkal

eladkal commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Please provide evidance of before and after images of a running dag with the setup of custom backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BranchingOperator fails when using custom backend xcom deserialization with NoCredentialsError

2 participants