Fix in-process Execution API secrets routing in client contexts#65587
Fix in-process Execution API secrets routing in client contexts#65587henry3260 wants to merge 3 commits intoapache:mainfrom
Conversation
6758f3a to
c8ece7f
Compare
c8ece7f to
6dc8178
Compare
|
Could someone please take a look at this PR when they have a chance? Thanks! |
d68c027 to
73528bb
Compare
8fe3872 to
ca97e7f
Compare
ca97e7f to
5fc80b5
Compare
jason810496
left a comment
There was a problem hiding this comment.
Nice! Thanks for the fix.
Additionally, could we add test for the scenario mentioned in the issue (#65482) to make sure the expected behavior?
Thanks.
| "_AIRFLOW_PROCESS_CONTEXT_OVERRIDE", | ||
| default=None, | ||
| ) | ||
|
|
There was a problem hiding this comment.
It doesn't seem like a good place for these utilities. Shouldn't models/* be something related to "Data Model" (or a Table in Metadata DB more specific).
There was a problem hiding this comment.
It doesn't seem like a good place for these utilities. Shouldn't
models/*be something related to "Data Model" (or a Table in Metadata DB more specific).
Good point on the directory structure. I'm also concerned that models/ isn't the right fit since these aren't DB models. Where would you suggest moving them? I previously thought about utils/, but the CI checks didn't allow that, and putting them in _shared might be overkill.
5fc80b5 to
532cba8
Compare
d23eeef to
b21962b
Compare
b21962b to
0a6ed36
Compare
Why
Execution API requests can run inside in-process paths where
SUPERVISOR_COMMSis present, which may incorrectly classify server-side code as client-side Task SDK execution.When that
happens, Variable or Connectionreads can route back into Task SDK paths instead of staying server-side, creating a recursive chain and potential hangs.The previous context signal was primarily environment-based, which is not always safe for request-scoped behavior in threaded in-process execution.
Because of that,
VariableandConnectionoperations triggered while serving in-process Execution APIrequests could still be routed back through Task SDK APIs instead of using the server-side secrets/backend
path, leading to recursive self-calls and hanging lookups.
closes: #65482
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.