VertexAI handle streaming requests#3331
Conversation
github-actions
Bot
assigned
alizenhom,
codefromthecrypt,
gyliu513,
karthikscale3,
lmolkova,
lzchen and
nirga
github-actions
Bot
requested review from
alizenhom,
codefromthecrypt,
gyliu513,
karthikscale3,
lmolkova,
lzchen and
nirga
| get_genai_response_attributes(response) | ||
| ) | ||
|
|
||
| for event in response_to_events( |
There was a problem hiding this comment.
I'm not sure this is the case for vertexai too but in other instrumentations we pass the span explicitly to the code sending events. This is because at the time we evaluate the stream response the span has been already closed
There was a problem hiding this comment.
This is because at the time we evaluate the stream response the span has been already closed
Isn't the span supposed to remain un-finished until the last response is received?
Anyways, calls to this function happen under the yield stmt on L148 before the "with" block from L117 exits. So the span should be accessible in Context even if it were ended.
|
Putting this into draft since I think we are blocked on data model a bit
|
WIP using shared context manager Properly implement uninstrument Shared code with a contextmanager tmp
|
This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment. |
|
This PR has been closed due to inactivity. Please reopen if you would like to continue working on it. |
9 participants
Fixes #3297
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Does This PR Require a Core Repo Change?
Checklist:
See contributing.md for styleguide, changelog guidelines, and more.