OpenAI Responses API Observability

[gyliu513]

🚀 Describe the new functionality needed

Add observability support for the OpenAI Responses API in llama-stack. This involves integrating OpenTelemetry instrumentation so that calls made through the Responses API (both sync and streaming) produce proper traces and spans, consistent with the existing chat completions observability.

This feature depends on upstream work in the OpenTelemetry Python contrib repository. The following PRs need to be merged and released first:

• Implement OpenAI Responses API instrumentation and examples Implement OpenAI Responses API instrumentation and examples open-telemetry/opentelemetry-python-contrib#4166
• Add response wrappers for OpenAI Responses API streams Add response wrappers for OpenAI Responses API streams. open-telemetry/opentelemetry-python-contrib#4280
• feat: OpenAI responses extractors feat: OpenAI responses extractors open-telemetry/opentelemetry-python-contrib#4337

Once these upstream PRs are merged and a new opentelemetry-instrumentation-openai-v2 release is available, llama-stack can integrate the updated package to enable Responses API observability.

This is a sub task of #2596

@leseb @iamemilio @cdoern ^^

💡 Why is this needed? What if we don't build it?

Llama-stack already supports the OpenAI Responses API, but there is currently no telemetry coverage for it. Without this, users have no visibility into Responses API call latency, token usage, errors, or streaming behavior through their observability stack (e.g., Jaeger, Grafana). This makes debugging and performance monitoring significantly harder for anyone using the Responses API.

Other thoughts

• This is blocked on upstream OTel contrib PRs, no implementation work should start until those are merged and released, but we can do some early test with patched code.
• Once available, integration should be straightforward: bump the opentelemetry-instrumentation-openai-v2 dependency version and verify traces are emitted for Responses API calls.

[iamemilio]

What additional traces did we want to capture in Llama Stack specifically? The issue in OTEL python contrib enables OTEL auto instrumentation to capture OpenAI client calls to Responses API automatically, which is important for us so we can have complete distributed traces from agents using responses API, but we don't make any client calls in Llama Stack using the responses API, only completions.

[gyliu513]

Good point, Llama Stack doesn't make outbound Responses API calls itself. But there's a case we need to cover: an OpenAI client calling the Llama Stack server via the Responses API.

Since Llama Stack exposes an OpenAI-compatible API and already supports the Responses API endpoint, users can use the OpenAI SDK's client.responses.create() to interact with Llama Stack. The client-side instrumentation from the OTel contrib PR (open-telemetry/opentelemetry-python-contrib#4166) will capture the outbound call, covering things like request handling latency, token usage, and errors on the Llama Stack side.

Without server-side instrumentation for the Responses API, we'd have a gap in the trace: the client span exists but there's no corresponding server span in Llama Stack, making it harder to debug end-to-end.

[iamemilio]

We do have server side instrumentation in llama stack, that gets injected with auto instrumentation and propagated from all httpx requests and responses.

[gyliu513]

Yes, I open this mainly for integration test with llamastack and documentation update for llamastack observability.

[iamemilio]

Ah ok, I misunderstood. Thanks for clarifying

[iamemilio]

OTEL + streaming tool calls: workaround and upstream bug

While investigating streaming tool-call failures under opentelemetry-instrument (#5200), we found a bug in opentelemetry-instrumentation-openai-v2 (v2.3b0). Their BaseStreamWrapper accumulates tool-call arguments across chunks and serializes them with "".join(buffer) during cleanup. When any provider sends arguments=None on a tool-call delta (common with vLLM, TGI, etc.), this crashes with TypeError, killing the stream and causing response.failed in llama-stack.

PR #5200 works around this by normalizing arguments=None to "" on streaming chunks before yielding, matching the OpenAI spec. The PR also includes an xfail(strict=True) canary test against the real OTEL ChoiceBuffer — when the upstream bug is fixed and the package is updated, this test will break the build, signaling that our workaround can be removed.

We should open an issue in opentelemetry-python-contrib for this. The fix is a one-line null check in ChoiceBuffer.append_tool_call(). Ideally it ships before or alongside the Responses API instrumentation PRs (#4166, #4280, #4337) since they use the same BaseStreamWrapper code path.

Upstream Issue: open-telemetry/opentelemetry-python-contrib#4344