feat(instrumentation-aws-sdk): add Bedrock InvokeModelWithResponseStream instrumentation #2845

yuliia-fryshko · 2025-05-22T16:25:21Z

Which problem is this PR solving?

Adds instrumentation of the InvokeModelWithResponseStreamCommand in the AWS Bedrock SDK.

Short description of the changes

Captures GenAI semantic span attributes for Amazon Titan, Anthropic Claude, Amazon Nova models.
Ensures span.end() is deferred until the full stream is consumed.
instrumentAsyncIterable is used to inspect streamed chunks in real time and extract relevant telemetry.
Includes unit tests for Titan, Claude and Nova models.

trentm · 2025-05-27T19:12:23Z

plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/bedrock-runtime.ts

      case 'InvokeModel':
        return this.requestPreSpanHookInvokeModel(request, config, diag);
+      case 'InvokeModelWithResponseStream':
+        return this.requestPreSpanHookInvokeModelWithResponseStream(


Is there a reason to not re-use requestPreSpanHookInvokeModel: add a 4th isStream argument and pass in false for 'InvokeModel', true for 'InvokeModelWithResponseStream', and then make the minor update to the implementation? This is how it was done for 'Converse' and 'ConverseStream'.

It looks to me like the requestPreSpanHookInvokeModel and requestPreSpanHookInvokeModelWithResponseStream functions are almost identical ... except that the latter doesn't have blocks for 'meta.llama', 'cohere.*', and 'mistral'.

Thanks for the suggestion, @trentm ! You are absolutely right! I've updated the code to consolidate requestPreSpanHookInvokeModel and requestPreSpanHookInvokeModelWithResponseStream into a single method using isStream parameter as you suggested.

codecov · 2025-05-27T19:22:05Z

Codecov Report

❌ Patch coverage is 88.09524% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.60%. Comparing base (fd9e262) to head (efda112).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...umentation-aws-sdk/src/services/bedrock-runtime.ts	88.09%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2845      +/-   ##
==========================================
- Coverage   91.44%   86.60%   -4.85%     
==========================================
  Files         146       17     -129     
  Lines        8192      933    -7259     
  Branches     1846      258    -1588     
==========================================
- Hits         7491      808    -6683     
+ Misses        701      125     -576

Flag	Coverage Δ
instrumentation-amqplib	`?`
instrumentation-aws-lambda	`?`
instrumentation-aws-sdk	`86.60% <88.09%> (+0.11%)`	⬆️
instrumentation-bunyan	`?`
instrumentation-cassandra-driver	`?`
instrumentation-connect	`?`
instrumentation-cucumber	`?`
instrumentation-dataloader	`?`
instrumentation-dns	`?`
instrumentation-express	`?`
instrumentation-fastify	`?`
instrumentation-generic-pool	`?`
instrumentation-graphql	`?`
instrumentation-hapi	`?`
instrumentation-ioredis	`?`
instrumentation-kafkajs	`?`
instrumentation-knex	`?`
instrumentation-koa	`?`
instrumentation-lru-memoizer	`?`
instrumentation-memcached	`?`
instrumentation-mongodb	`?`
instrumentation-mongoose	`?`
instrumentation-mysql	`?`
instrumentation-mysql2	`?`
instrumentation-nestjs-core	`?`
instrumentation-net	`?`
instrumentation-openai	`?`
instrumentation-oracledb	`?`
instrumentation-pg	`?`
instrumentation-pino	`?`
instrumentation-redis	`?`
instrumentation-restify	`?`
instrumentation-router	`?`
instrumentation-runtime-node	`?`
instrumentation-socket.io	`?`
instrumentation-tedious	`?`
instrumentation-typeorm	`?`
instrumentation-undici	`?`
instrumentation-winston	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...umentation-aws-sdk/src/services/bedrock-runtime.ts	`95.72% <88.09%> (-3.27%)`	⬇️

... and 129 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jj22ee · 2025-06-13T01:22:06Z

plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/bedrock-runtime.ts

+  ): Promise<any> {
+    const stream = response.data?.body;
+    const modelId = response.request.commandInput?.modelId;
+    if (!stream || !span.isRecording()) return;


Suggested change

if (!stream || !span.isRecording()) return;

if (!stream) return;

!span.isRecording() is already checked before responseHookInvokeModelWithResponseStream() is called

You are right, @jj22ee! I removed this unnecessary check

jj22ee · 2025-06-13T01:38:56Z

plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/bedrock-runtime.ts

+    return response.data;
+
+    // Tap into the stream at the chunk level without modifying the chunk itself.
+    function instrumentAsyncIterable<T>(


Wouldn't it be more efficient to declare these following functions outside of this member function scope?

Thank you, @jj22ee , for the review and good catches! Yes, these helper functions could be declared outside the method, at the class level, instead of being re-created every time the method is called.
Changed!

JacksonWeber · 2025-07-02T04:34:43Z

Update this to pull in the main branch and I can take another look over. Thanks!

trentm · 2025-07-02T22:25:08Z

plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/bedrock-runtime.ts

+              parsedChunk,
+              span
+            );
+          }


The non-stream responseHookInvokeModel() is adding some attributes for a few more models: 'meta.llama', 'cohere.command-r', etc.
Should this function also add relevant attributes for those models? Or is it possible streaming is not supported for those models?

Also, perhaps the recordNovaAttributes(), recordClaudeAttributes(), etc. methods could be used by both responseHookInvokeModel and responseHookInvokeModelWithResponseStream. Or perhaps all the record*Attribute() methods could be moved to one setInvokeModelResponseAttributes() that is used by both responseHookInvokeModel*() methods.

plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/bedrock-runtime.ts

pichlermarc · 2025-07-30T16:50:06Z

@yuliia-fryshko do you have bandwidth to work on this? Looks like there are some unaddressed comments that are blocking this from getting merged.

yuliia-fryshko · 2025-08-07T14:22:20Z

@yuliia-fryshko do you have bandwidth to work on this? Looks like there are some unaddressed comments that are blocking this from getting merged.

Hi @pichlermarc ! Sorry for the long reply, I just back from the vacation today and I will try to address all comments today

…h isStream flag

…d mistral

pichlermarc · 2025-08-27T16:45:13Z

Hi @trivikr @blumamir @jj22ee, please take a look. Thanks! :)

trentm

I'm good with this PR now. Sorry, I realize this has dragged on for a long while.

Some notes:

I have a nit about error handling (see below).
At this point, because gen_ai semconv moves quickly enough, some of the semconv vars here are outdated. E.g. gen_ai.system is now replaced with gen_ai.provider.name. However, this instrumentation was already using the older gen-ai semconv version and the current gen-ai semconv docs (https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) suggest that existing instrumentations should stick to their current version by default:

SHOULD NOT change the version of the GenAI conventions that they emit by default. Conventions include, but are not limited to, attributes, metric, span and event names, span kind and unit of measure.
SHOULD introduce an environment variable OTEL_SEMCONV_STABILITY_OPT_IN ...

So any updates of semconv version could be done in a separate PR.

trentm · 2025-09-15T21:46:14Z

packages/instrumentation-aws-sdk/src/services/bedrock-runtime.ts

+        for await (const chunk of stream) {
+          const parsedChunk = this.parseChunk(chunk?.chunk?.bytes);
+
+          if (!parsedChunk) return;


@yuliia-fryshko

nit: If, for whatever reason, a chunk cannot be parsed for adding otel data, should we perhaps continue to yield the chunks and eventually span.end() rather than return; here?

I agree, it will be better to close the span in this case. Done

I think your change in commit 80d93e3 is not correct. It would result in span.end() being called and then an attempt to call BedrockRuntimeServiceExtension.record*Attributes(...) which would likely break.

I've change it to what I think it should be in commit ee6f51e.
@yuliia-fryshko Please let me know if my change looks inappropriate.

yuliia-fryshko · 2025-10-07T10:18:36Z

I'm good with this PR now. Sorry, I realize this has dragged on for a long while.

Some notes:

I have a nit about error handling (see below).

At this point, because gen_ai semconv moves quickly enough, some of the semconv vars here are outdated. E.g. gen_ai.system is now replaced with gen_ai.provider.name. However, this instrumentation was already using the older gen-ai semconv version and the current gen-ai semconv docs (https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) suggest that existing instrumentations should stick to their current version by default:

SHOULD NOT change the version of the GenAI conventions that they emit by default. Conventions include, but are not limited to, attributes, metric, span and event names, span kind and unit of measure.
SHOULD introduce an environment variable OTEL_SEMCONV_STABILITY_OPT_IN ...

So any updates of semconv version could be done in a separate PR.

Thank you, @trentm !

I created an issue to review and upgrade all fields with latest semantic convention. Happy to take it after this MR will be merged.

…emetry attributes, but yield the chunk as if there were no telemetry

trentm · 2025-10-08T22:35:54Z

plugins/node/instrumentation-cucumber/test/current.feature

Just noticed that this file got added accidentally at some point.

removed in commit 35143a4

otelbot · 2025-10-08T23:25:13Z

Thank you for your contribution @yuliia-fryshko! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey.

yuliia-fryshko requested a review from a team as a code owner May 22, 2025 16:25

github-actions bot assigned blumamir and jj22ee May 22, 2025

github-actions bot added the pkg:instrumentation-aws-sdk label May 22, 2025

github-actions bot assigned trivikr May 22, 2025

github-actions bot requested review from blumamir, jj22ee and trivikr May 22, 2025 16:25

yuliia-fryshko force-pushed the bedrock-invoke-model-stream branch 2 times, most recently from 09f9777 to eeb1e84 Compare May 23, 2025 09:56

trentm reviewed May 27, 2025

View reviewed changes

yuliia-fryshko force-pushed the bedrock-invoke-model-stream branch from eeb1e84 to 0d197f0 Compare June 2, 2025 08:36

jj22ee reviewed Jun 13, 2025

View reviewed changes

yuliia-fryshko force-pushed the bedrock-invoke-model-stream branch from fe8416b to ff1cb2d Compare June 27, 2025 18:54

pichlermarc assigned trentm, trivikr, jj22ee and blumamir and unassigned trivikr, blumamir and jj22ee Jul 2, 2025

trentm reviewed Jul 2, 2025

View reviewed changes

yuliia-fryshko force-pushed the bedrock-invoke-model-stream branch from 0e2fef3 to dcd6b28 Compare August 7, 2025 14:18

yuliia-fryshko force-pushed the bedrock-invoke-model-stream branch from 1941501 to d86f003 Compare August 11, 2025 07:48

yuliia-fryshko added 2 commits August 12, 2025 23:24

feat: add Bedrock InvokeModelWithResponseStream instrumentation

e9ae8b8

refactor: unify InvokeModel pre-span hooks into a single function wit…

da2cbfa

…h isStream flag

yuliia-fryshko added 3 commits August 12, 2025 23:24

Moved static functions out of the method

0a80410

added instrumentation for Bedrock Response Stream of llama, cohere an…

ff59f72

…d mistral

added tests for cohere, cohere-r and mistral

f7bbec1

yuliia-fryshko force-pushed the bedrock-invoke-model-stream branch from 71c7a6e to f7bbec1 Compare August 12, 2025 21:24

yuliia-fryshko added 3 commits August 14, 2025 10:05

Merge branch 'main' into bedrock-invoke-model-stream

44a3dec

Merge branch 'main' into bedrock-invoke-model-stream

4e8b39e

Merge branch 'main' into bedrock-invoke-model-stream

945df2c

yuliia-fryshko added 3 commits August 28, 2025 00:34

Merge branch 'main' into bedrock-invoke-model-stream

451c95d

Merge branch 'main' into bedrock-invoke-model-stream

0fa5a4a

Merge branch 'main' into bedrock-invoke-model-stream

efda112

trentm approved these changes Sep 15, 2025

View reviewed changes

pichlermarc and others added 2 commits September 24, 2025 18:38

Merge branch 'main' into bedrock-invoke-model-stream

6eb6eca

close span if a chunk cannot be parsed

80d93e3

yuliia-fryshko mentioned this pull request Oct 7, 2025

Replace gen_ai.system with gen_ai.provider.name in AWS SDK instrumentation #3152

Open

yuliia-fryshko and others added 3 commits October 7, 2025 12:18

Merge branch 'main' into bedrock-invoke-model-stream

74736da

if cannot 'parseChunk' for telemetry processing, then skip adding tel…

ee6f51e

…emetry attributes, but yield the chunk as if there were no telemetry

Merge branch 'main' into bedrock-invoke-model-stream

5ac99f3

trentm reviewed Oct 8, 2025

View reviewed changes

this file was accidentally added in this feature branch

35143a4

trentm changed the title ~~feat: add Bedrock InvokeModelWithResponseStream instrumentation~~ feat(instrumentation-aws-sdk): add Bedrock InvokeModelWithResponseStream instrumentation Oct 8, 2025

trentm merged commit 84e3330 into open-telemetry:main Oct 8, 2025
19 checks passed

otelbot-js-contrib bot mentioned this pull request Oct 8, 2025

chore: release main #3151

Merged

	if (!stream \|\| !span.isRecording()) return;
	if (!stream) return;

feat(instrumentation-aws-sdk): add Bedrock InvokeModelWithResponseStream instrumentation #2845

feat(instrumentation-aws-sdk): add Bedrock InvokeModelWithResponseStream instrumentation #2845

Uh oh!

Conversation

yuliia-fryshko commented May 22, 2025

Which problem is this PR solving?

Short description of the changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JacksonWeber commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pichlermarc commented Jul 30, 2025

Uh oh!

yuliia-fryshko commented Aug 7, 2025

Uh oh!

pichlermarc commented Aug 27, 2025

Uh oh!

trentm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuliia-fryshko commented Oct 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

otelbot bot commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov bot commented May 27, 2025 •

edited

Loading