Incompatibility between openllmetry and langfuse OpenAI wrapper

[pamelafox]

I'm not super surprised that I had an error when using both langfuse and openllmetry, but I figured I'd report it for others to find, and on the off chance it's an easy fix. I haven't looked into how each of you monkey patch OpenAI.

Specifically, the incompatibility is when I have both:
from langfuse.openai import AsyncAzureOpenAI, AsyncOpenAI

and:
OpenAIInstrumentor().instrument()

The error happens when I try to handle the response from the streaming chat routine:

Traceback (most recent call last):
  File "/tmp/8dc3d27ca286fca/app.py", line 154, in format_as_ndjson
    async for event in r:
  File "/tmp/8dc3d27ca286fca/approaches/chatapproach.py", line 170, in run_with_streaming
    async for event_chunk in await chat_coroutine:
                             ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/8dc3d27ca286fca/antenv/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 73, in achat_wrapper
    _handle_response(response, span)
  File "/tmp/8dc3d27ca286fca/antenv/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 96, in _handle_response
    _set_completions(span, response_dict.get("choices"))
                           ^^^^^^^^^^^^^^^^^
AttributeError: 'async_generator' object has no attribute 'get'

The error goes away when I comment out the instrument call.

[marcklingen]

Thanks for reporting, that's good to know! Love openllmetry for interop with Otel. Langfuse will add an alternative decorator based Python integration soon which could resolve this by replacing the monkey patched import.

[antoniomdk]

I'd love to see interoperability between OpenLLMetry and Langfuse. I think Langfuse is great for analyzing traces, adding annotations, and managing datasets. But I think it'd be really useful to have a single trace ID that could help us go from errors detected at the APM level (e.g. NewRelic) to detailed traces in Langfuse.

One solution could be that the traces generated via OpenLLMetric are ingested into Langfuse and the general-purpose observability platform. My main concern is how well those OpenLLMetry traces would translate into Langfuse traces.

[marcklingen]

Agree on all of the above, I think the way to go about this is to wait for stable semantic conventions for LLM related spans in OTel and then add an OTel collector to Langfuse to consume these traces.

This discussion is kind of related: https://www.reddit.com/r/LangChain/comments/1bnkvtv/comment/kwmb1dk/

[antoniomdk]

That makes a lot of sense to me. Hopefully, the OTEL LLM conventions will be published in Q2 or Q3 this year. I'll keep an eye to this PR: open-telemetry/semantic-conventions#825