feat(instrumentation-aws-sdk): add bedrock extension to apply gen ai conventions #2700

anuraaga · 2025-02-06T04:10:55Z

Which problem is this PR solving?

AWS SDK spans for bedrock should be gen AI spans but are currently generic SDK spans

Short description of the changes

Add a service extension for bedrock, applying gen AI conventions
- This initial PR is to get the general infrastructure and setup so applies minimally, only Converse with span attributes. Future PRs will add other bedrock APIs and other gen AI conventions such as events and metrics
Update AWS SDK deps to sync up with the now newer required SDK version

…y-js-contrib into aws-bedrock-extension

anuraaga · 2025-02-06T04:16:55Z

Hi @AmanAgarwal041 - just wanted to share this PR adding gen ai instrumentation of bedrock. Notably, for tests it takes a pattern of recording real LLM responses using nock back, which I see most gen ai instrumentation outside of opentelemetry taking due to the complexity of the models. It may be a good approach for #2402 (if you need help with that PR, let me know). Cheers.

codecov · 2025-02-06T04:22:37Z

Codecov Report

Attention: Patch coverage is 96.55172% with 2 lines in your changes missing coverage. Please review.

Project coverage is 92.60%. Comparing base (92106ff) to head (f768a7d).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...entelemetry-instrumentation-aws-sdk/src/aws-sdk.ts	0.00%	1 Missing ⚠️
...umentation-aws-sdk/src/services/bedrock-runtime.ts	97.67%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2700      +/-   ##
==========================================
+ Coverage   90.96%   92.60%   +1.64%     
==========================================
  Files         171      168       -3     
  Lines        8133     7889     -244     
  Branches     1649     1621      -28     
==========================================
- Hits         7398     7306      -92     
+ Misses        735      583     -152

Files with missing lines	Coverage Δ
...entelemetry-instrumentation-aws-sdk/src/semconv.ts	`100.00% <100.00%> (ø)`
...ntation-aws-sdk/src/services/ServicesExtensions.ts	`96.55% <100.00%> (+0.25%)`	⬆️
...entelemetry-instrumentation-aws-sdk/src/aws-sdk.ts	`94.67% <0.00%> (-0.39%)`	⬇️
...umentation-aws-sdk/src/services/bedrock-runtime.ts	`97.67% <97.67%> (ø)`

... and 9 files with indirect coverage changes

anuraaga · 2025-02-06T04:25:37Z

@trentm Oh I remembered now that the latest version of the AWS SDK doesn't support Node 14, which is causing the unit tests to fail after anuraaga#1 (comment). Let me know what's a good way forward, I think either dropping Node 14 support for this package (not sure it's allowed) or downgrading again.

codefromthecrypt

Surprisingly concise change to add the toehold here. I know there's more to do, but good work!

jj22ee · 2025-02-07T20:03:54Z

Hi @anuraaga, thanks for this. Just to let you know, the ADOT team from AWS also had vested interest in Gen AI support in AWS SDK instrumentation, and planning to contribute in the future as well.

There are Bedrock Service Extension implementations in multiple languages in the ADOT's downstream of the auto-instrumentations (JS, Python), which are similar but might not be fully 1:1 with your changes. I was wondering if you could also consider ADOT's implementations in this PR? I'll also spend time to compare the 2 implementations for any subtle differences.

codefromthecrypt · 2025-02-07T22:07:11Z

@jj22ee drive-by comment, but one thing to give heads up about is that the toehold here is based on the latest genai semantic conventions, and I don't expect this to want to do too much in one PR. So, one thing to think about is what happens in this PR vs a follow-up to add more features or things that are defined beyond the otel specs. Doing it like that might result in the same or similar end, but faster vs trying to do too much in the first of many PRs. food for thought!

https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-spans.md

jj22ee · 2025-02-07T22:21:46Z

Thanks for the heads up @codefromthecrypt. I agree to start smaller. Just wanted to ensure there are no conflicts, but the latest Gen AI conventions will take priority, so I have no issue with that.

jj22ee · 2025-02-08T00:07:48Z

plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/bedrock-runtime.ts

+    const { inputTokens, outputTokens } = usage;
+    if (inputTokens !== undefined) {
+      span.setAttribute(ATTR_GEN_AI_USAGE_INPUT_TOKENS, inputTokens);
+    }
+    if (outputTokens !== undefined) {
+      span.setAttribute(ATTR_GEN_AI_USAGE_OUTPUT_TOKENS, outputTokens);
+    }


In case usage is undefined:

Suggested change

const { inputTokens, outputTokens } = usage;

if (inputTokens !== undefined) {

span.setAttribute(ATTR_GEN_AI_USAGE_INPUT_TOKENS, inputTokens);

}

if (outputTokens !== undefined) {

span.setAttribute(ATTR_GEN_AI_USAGE_OUTPUT_TOKENS, outputTokens);

}

if (usage) {

const { inputTokens, outputTokens } = usage;

if (inputTokens !== undefined) {

span.setAttribute(ATTR_GEN_AI_USAGE_INPUT_TOKENS, inputTokens);

}

if (outputTokens !== undefined) {

span.setAttribute(ATTR_GEN_AI_USAGE_OUTPUT_TOKENS, outputTokens);

}

}

jj22ee · 2025-02-08T00:19:15Z

plugins/node/opentelemetry-instrumentation-aws-sdk/src/semconv.ts

+ * limitations under the License.
+ */
+
+// Gen AI conventions


Can you also help copy over comments for the copied ATTR_* attribute keys from the semantic-conventions?

Example from a resource-detector package

I have a commit that re-generates the src/semconv.ts using a coming script for this (from #2669), if you are okay with me pushing to your branch, @anuraaga.
It also updates the semconv dep to ^1.29.0 because you are using semconv attributes defined in that version (GEN_AI_SYSTEM_VALUE_AWS_BEDROCK) -- though technically because a copy (in src/semconv.ts) is being used, there isn't really the runtime dep.

Thanks @trentm - feel free to push anything you need to this branch

karthikscale3 · 2025-02-09T05:23:32Z

nice work! looks good for the most part.

…y-js-contrib into aws-bedrock-extension

anuraaga · 2025-02-10T01:46:10Z

Thanks @jj22ee - I think the implementation pattern itself is similar/same as almost all the work is done by the service extension anyways. Technically there isn't any overlap yet since AFAIU, ADOT currently only instruments InvokeModel while I started with Converse here, though when adding InvokeModel support here I think it will be very similar to ADOT. Initially, this wasn't obvious so I reworked the implementation to make it clearer for this PR it is Converse-only, while hopefully making the extension to InvokeModel obvious. Skimming through the attributes, I didn't see any conflicts in semconv either yet (it's still gen_ai.system, for now...)

The bigger difference is the test infrastructure using nock-back, which we also adopted in upstream python

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-botocore/tests/conftest.py#L71

so I think if we get this in for the initial infrastructure we'll be able to smoothly add more features over subsequent PRs by only focusing on business logic instead of build infra.

jj22ee · 2025-02-11T02:52:36Z

Thank you, the changes lgtm!

As for the suddenly failing unit tests, the latest commit shows 15 tests failing, but I'm pretty sure 14 of them are unrelated to your changes. I suppose there is an issue with the recently added "does not currently add genai conventions" test which is causing the other 14 to fail. Likely InvokeModel isn't intercepted by nock-back?

I recall I had a similar issue when using nock for Kinesis Client, but needed to enforce NodeHttpHandler requestHandler since I found that the default handler was HTTP2 which was not supported by nock. Not sure if this issue is the same.

anuraaga · 2025-02-11T04:57:55Z

Sorry @jj22ee I missed the test failures behind the known node 14 ones I need advice on. Just forgot to git add the recording will do it soon

trentm · 2025-02-14T04:58:46Z

@anuraaga I have an idea for dealing with the node v14 test breakage. It may end up looking a little gross. I still have to write it up, and I may break it into a separate PR: a PR that (a) updates the existing @aws-sdk/client-* deps to recent versions; (b) changes the npm test for this package to just skip running with Node.js v14 and v16; and (c) uses the "test-all-versions" tests (i.e. .tav.yml) to handle running tests with node 14 and 16.

trentm · 2025-02-19T01:09:30Z

I have an idea

#2722 and #2723 for this.

Assuming these get in, I can resync this PR to main and add my semconv updates that I mentioned above at #2700 (comment)

anuraaga added 6 commits January 30, 2025 15:55

feat(instrumentation-aws-sdk): add bedrock extension

6cb00fa

Use minimum bedrock that supports Converse

765c280

Cleanups

94d7fe4

Update deps

36ca8fd

Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…

fc8fd69

…y-js-contrib into aws-bedrock-extension

git add

5bf7306

anuraaga requested a review from a team as a code owner February 6, 2025 04:10

github-actions bot assigned blumamir, jj22ee and trivikr Feb 6, 2025

github-actions bot requested a review from blumamir February 6, 2025 04:11

github-actions bot added the pkg:instrumentation-aws-sdk label Feb 6, 2025

github-actions bot requested review from jj22ee and trivikr February 6, 2025 04:11

anuraaga changed the title ~~Aws bedrock extension~~ feat(instrumentation-aws-sdk): add bedrock extension to apply gen ai conventions Feb 6, 2025

codefromthecrypt approved these changes Feb 6, 2025

View reviewed changes

jj22ee reviewed Feb 8, 2025

View reviewed changes

anuraaga added 4 commits February 10, 2025 10:08

Cleanup

e2366aa

Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…

018cb5c

…y-js-contrib into aws-bedrock-extension

Extract converse handler

b8e3c16

Add pre-implementation test case of InvokeModel

c33aaaa

git add

f768a7d

trentm mentioned this pull request Feb 19, 2025

discussion: how to npm test with modern versions of instrumented packages? #2722

Open

anuraaga mentioned this pull request Feb 20, 2025

Add instrumentation of AWS Bedrock to use gen_ai conventions open-telemetry/opentelemetry-java-instrumentation#13355

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(instrumentation-aws-sdk): add bedrock extension to apply gen ai conventions #2700

feat(instrumentation-aws-sdk): add bedrock extension to apply gen ai conventions #2700

anuraaga commented Feb 6, 2025

anuraaga commented Feb 6, 2025 •

edited

Loading

codecov bot commented Feb 6, 2025 •

edited

Loading

anuraaga commented Feb 6, 2025

codefromthecrypt left a comment

jj22ee commented Feb 7, 2025 •

edited

Loading

codefromthecrypt commented Feb 7, 2025

jj22ee commented Feb 7, 2025

jj22ee Feb 8, 2025

jj22ee Feb 8, 2025

trentm Feb 12, 2025

anuraaga Feb 12, 2025

karthikscale3 commented Feb 9, 2025

anuraaga commented Feb 10, 2025 •

edited

Loading

jj22ee commented Feb 11, 2025

anuraaga commented Feb 11, 2025

trentm commented Feb 14, 2025

trentm commented Feb 19, 2025 •

edited

Loading

feat(instrumentation-aws-sdk): add bedrock extension to apply gen ai conventions #2700

Are you sure you want to change the base?

feat(instrumentation-aws-sdk): add bedrock extension to apply gen ai conventions #2700

Conversation

anuraaga commented Feb 6, 2025

Which problem is this PR solving?

Short description of the changes

anuraaga commented Feb 6, 2025 • edited Loading

codecov bot commented Feb 6, 2025 • edited Loading

Codecov Report

anuraaga commented Feb 6, 2025

codefromthecrypt left a comment

Choose a reason for hiding this comment

jj22ee commented Feb 7, 2025 • edited Loading

codefromthecrypt commented Feb 7, 2025

jj22ee commented Feb 7, 2025

jj22ee Feb 8, 2025

Choose a reason for hiding this comment

jj22ee Feb 8, 2025

Choose a reason for hiding this comment

trentm Feb 12, 2025

Choose a reason for hiding this comment

anuraaga Feb 12, 2025

Choose a reason for hiding this comment

karthikscale3 commented Feb 9, 2025

anuraaga commented Feb 10, 2025 • edited Loading

jj22ee commented Feb 11, 2025

anuraaga commented Feb 11, 2025

trentm commented Feb 14, 2025

trentm commented Feb 19, 2025 • edited Loading

anuraaga commented Feb 6, 2025 •

edited

Loading

codecov bot commented Feb 6, 2025 •

edited

Loading

jj22ee commented Feb 7, 2025 •

edited

Loading

anuraaga commented Feb 10, 2025 •

edited

Loading

trentm commented Feb 19, 2025 •

edited

Loading