Skip to content

Fix in-process Execution API secrets routing in client contexts#65587

Open
henry3260 wants to merge 3 commits intoapache:mainfrom
henry3260:fix-infinite-waiting-get
Open

Fix in-process Execution API secrets routing in client contexts#65587
henry3260 wants to merge 3 commits intoapache:mainfrom
henry3260:fix-infinite-waiting-get

Conversation

@henry3260
Copy link
Copy Markdown
Contributor

@henry3260 henry3260 commented Apr 21, 2026

Why

Execution API requests can run inside in-process paths where SUPERVISOR_COMMS is present, which may incorrectly classify server-side code as client-side Task SDK execution.
When thathappens, Variable or Connectionreads can route back into Task SDK paths instead of staying server-side, creating a recursive chain and potential hangs.
The previous context signal was primarily environment-based, which is not always safe for request-scoped behavior in threaded in-process execution.

Because of that, Variable and Connection operations triggered while serving in-process Execution API
requests could still be routed back through Task SDK APIs instead of using the server-side secrets/backend
path, leading to recursive self-calls and hanging lookups.
closes: #65482

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:task-sdk labels Apr 21, 2026
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch from 6758f3a to c8ece7f Compare April 21, 2026 07:22
@henry3260 henry3260 changed the title fix infinite loop for Variable.get Fix in-process Execution API secrets routing in client contexts Apr 21, 2026
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch from c8ece7f to 6dc8178 Compare April 21, 2026 14:14
@henry3260 henry3260 marked this pull request as ready for review April 21, 2026 16:59
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 23, 2026
@henry3260
Copy link
Copy Markdown
Contributor Author

Could someone please take a look at this PR when they have a chance? Thanks!

@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch from d68c027 to 73528bb Compare April 27, 2026 16:05
@jason810496 jason810496 self-requested a review April 29, 2026 15:28
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch 3 times, most recently from 8fe3872 to ca97e7f Compare May 6, 2026 18:38
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch from ca97e7f to 5fc80b5 Compare May 7, 2026 02:57
Copy link
Copy Markdown
Member

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the fix.

Additionally, could we add test for the scenario mentioned in the issue (#65482) to make sure the expected behavior?

Thanks.

"_AIRFLOW_PROCESS_CONTEXT_OVERRIDE",
default=None,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem like a good place for these utilities. Shouldn't models/* be something related to "Data Model" (or a Table in Metadata DB more specific).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem like a good place for these utilities. Shouldn't models/* be something related to "Data Model" (or a Table in Metadata DB more specific).

Good point on the directory structure. I'm also concerned that models/ isn't the right fit since these aren't DB models. Where would you suggest moving them? I previously thought about utils/, but the CI checks didn't allow that, and putting them in _shared might be overkill.

Comment thread airflow-core/src/airflow/models/variable.py Outdated
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch from 5fc80b5 to 532cba8 Compare May 7, 2026 08:41
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch 2 times, most recently from d23eeef to b21962b Compare May 7, 2026 17:46
@henry3260 henry3260 force-pushed the fix-infinite-waiting-get branch from b21962b to 0a6ed36 Compare May 7, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PythonVirtualenvOperator hangs indefinitely on Variable.get() (Task SDK) under airflow dags test and dag.test() — no error, no timeout

3 participants