how to return trace_id in openai completion response when using langfuse as a proxy in litellm?

[octadion]

Hi everyone,

I’m using the OpenAI Completion API and using langfuse for the callback as a proxy. i have included a trace_id in my request. However, I’m not seeing the trace_id in the response. I expected the trace_id to be echoed back in the response for tracking purposes

Here’s a summary of what I’ve done:

Request: I’m passing the trace_id as part of the request payload.
Response: The response does not include the trace_id.
Could someone please confirm if the trace_id is supposed to be included in the response? If it is, could there be any issues or additional steps I need to consider?

Thanks in advance for your help!

[dosubot[bot]]

Yes, the trace_id is supposed to be included in the response when using the OpenAI Completion API with Langfuse as a proxy in LiteLLM. To ensure it is returned, you need to set the trace_id in the metadata parameter when making the completion call.

Here is an example of how to do this:

from litellm import completion

# set custom langfuse trace params and generation params
response = completion(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ],
  metadata={
      "trace_id": "trace-id",  # set langfuse Trace ID
  },
)

print(response)

Additionally, ensure that you are using the observeOpenAI function to create the client and handle the response properly. This function helps in observing and managing the trace data effectively. Here is an example in TypeScript from the Langfuse JS/TS SDK:

const client = observeOpenAI(openai, { generationName: "your-generation-name" });
const res = await client.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [{ role: "user", content: "Explain how to assemble a PC" }],
  metadata: { trace_id: "trace-id" },
  user: "langfuse-user@gmail.com",
  max_tokens: 300,
});

await client.flushAsync();
const response = await getGeneration("your-generation-name");
const traceId = response.data.data[0].traceId;
console.log(traceId);

To further ensure the trace_id is included and properly managed, you can refer to the Langfuse Python SDK tests. For example, the test_openai_chat_completion_with_trace function demonstrates how to set and verify the trace_id:

def test_openai_chat_completion_with_trace():
    api = get_api()
    generation_name = create_uuid()
    trace_id = create_uuid()
    langfuse = Langfuse()

    langfuse.trace(id=trace_id)

    chat_func(
        name=generation_name,
        model="gpt-3.5-turbo",
        trace_id=trace_id,
        messages=[{"role": "user", "content": "1 + 1 = "}],
        temperature=0,
        metadata={"someKey": "someResponse"},
    )

    openai.flush_langfuse()

    generation = api.observations.get_many(name=generation_name, type="GENERATION")

    assert len(generation.data) != 0
    assert generation.data[0].name == generation_name
    assert generation.data[0].trace_id == trace_id

This will ensure that the trace_id is included in the response and properly managed ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[octadion]

I’ve already been passing the trace_id in my request, specifically within the metadata parameter. Here’s how I’m setting it:

if 'metadata' not in extra_body:
    extra_body['metadata'] = {}
extra_body['metadata']['trace_id'] = "test"  # Add trace_id to metadata

print("Modified extra_body:", extra_body)
response = await self.async_client.chat.completions.create(
    model=model,
    messages=messages,
    stream=streaming,
    extra_body=extra_body,
    **kwargs,
)

Despite setting the trace_id in this way, it doesn’t seem to be included in the response. Could there be anything I’m missing or another step I should take to ensure the trace_id is returned?
@dosu

[marcklingen]

I'm unsure whether I understand your problem. Do you use LiteLLM proxy or how do you exactly integrate with Langfuse? Please provide some code. Happy to have a look.

[octadion]

Thank you for the reply!

To clarify, I’m using LiteLLM as a proxy and have integrated Langfuse as a callback. Here’s a bit more detail about my setup:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: gpt-4o-mini
  - model_name: text-embedding-3-small
    litellm_params:
      model: text-embedding-3-small
litellm_settings:
  success_callback: ["langfuse"]

I’m running LiteLLM on port 4000, and I use the following code to create a ChatOpenAI instance:

llm = ChatOpenAI(
    api_key=api_key,
    model=llm_model,
    api_type=OpenaiApiType.OpenAI,
    max_retries=20,
    api_base=litellm_api_base,
    extra_body={"metadata": {"trace_user_id": botId}}
)

I’ve configured Langfuse as a callback, so all the ChatOpenAI processes are logged in Langfuse. My goal is to set a trace_id by sending it as part of the metadata, hoping that it will be included in the response so I can retrieve it.

However, even after setting the trace_id in the metadata, it’s not being returned in the response. I’m wondering if there’s anything I’m missing in this setup or if there’s another approach I should take.

Thank you for your help! @marcklingen

[marcklingen]

Have a look here for the different options that are available: https://github.com/BerriAI/litellm/blob/main/litellm/integrations/langfuse.py

I think it is trace_id and existing_trace_id