How to emulate same trace format in experiment run in sdk as prompt experiment in ui.

[savitha-suresh]

Describe your question

When i run a datasetrun from ui. The trace shows user prompt, system prompt. but when i run an experiment from sdk, the trace shows only the dataset input and not the system and user prompt sent to llm. Can someone please tell how to emulate traces and the exact flow of prompt experiment in ui using sdk?
https://langfuse.com/docs/evaluation/experiments/experiments-via-ui

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The key difference is that Prompt Experiments via UI automatically execute prompts from Prompt Management and create detailed traces showing the system and user prompts(1). When using Experiments via SDK, you define your own task function and control what gets traced(2).

To emulate the same trace format as UI Prompt Experiments in SDK experiments, you need to:

1. Use Langfuse's OpenAI integration in your task function to automatically capture the full prompt structure(2):

from langfuse import get_client
from langfuse.openai import OpenAI

langfuse = get_client()

def my_task(*, item, **kwargs):
    question = item["input"]
    response = OpenAI().chat.completions.create(
        model="gpt-4.1", 
        messages=[{"role": "user", "content": question}]
    )
    
    return response.choices[0].message.content

(2)

2. Include your system and user prompts explicitly in the messages array when calling the LLM, just as they appear in your Prompt Management prompts(2).

The langfuse.openai.OpenAI() wrapper automatically creates generations in your traces that show the full message structure including system and user prompts(2). This mirrors what happens when UI Prompt Experiments execute prompts from Prompt Management(1).

If you're using a different LLM provider or framework, ensure you're using Langfuse's integration for that provider (like langfuse.openai.AsyncOpenAI for async calls) to get automatic tracing of the complete prompt structure(2).

📚 Sources:

1. Experiments via UI (Prompt Experiments)
2. Experiments via SDK

---

Have another question? Just tag @inkeep.

[savitha-suresh]

@inkeep def task(*, item, **kwargs):
# Get the prompt
prompt = self.langfuse.get_prompt(prompt_name)

        # Get input data
        input_data = item.input
        print("Input Data:", input_data)
        
        # Compile prompt with input data
        compiled_prompt = prompt.compile(**input_data)
        
        # Get optional parameters from prompt config
        prompt_config = prompt.config or {}
        print("Compiled prompt:", compiled_prompt)

        # Call OpenAI API using the shared client
        response = self.openai_client.chat.completions.create(
            model=self.model_name,
            messages=compiled_prompt,
            langfuse_prompt=prompt
        )
        print(response)
        return response.choices[0].message.content

I am already doing that still i can only dataset input and not the prompt and user messages

[inkeep-ai-assistant[bot]]

The issue is that you're using self.openai_client instead of the Langfuse-wrapped OpenAI client. To get the full trace with system and user prompts visible, you need to use langfuse.openai.OpenAI() or import the wrapped client(1)(2).

Replace your OpenAI client initialization with:

from langfuse.openai import OpenAI

(1)

Then in your task function, use:

def task(*, item, **kwargs):
    # Get the prompt
    prompt = self.langfuse.get_prompt(prompt_name)

    # Get input data
    input_data = item.input
    
    # Compile prompt with input data
    compiled_prompt = prompt.compile(**input_data)
    
    # Call OpenAI API using Langfuse-wrapped client
    response = OpenAI().chat.completions.create(
        model=self.model_name,
        messages=compiled_prompt,
        langfuse_prompt=prompt
    )
    
    return response.choices[0].message.content

(1)

The key difference is that langfuse.openai.OpenAI() automatically creates generation observations in your traces that capture the full message structure including system and user prompts(1). Using a standard OpenAI client (self.openai_client) won't automatically trace the prompt details.

Additionally, ensure you have proper OpenTelemetry setup if you're using the Python SDK v3, as tracing relies on it(2). Make sure to call langfuse.flush() at the end of your experiment to ensure all traces are sent(1)(2).

📚 Sources:

1. Experiments via SDK
2. Langfuse SDKs

---

Have another question? Just tag @inkeep.

[savitha-suresh]

from langfuse.openai import OpenAI already using that

[savitha-suresh]

@inkeep from langfuse.openai import OpenAI

I am already doing that

[jannikmaierhoefer]

Hi @savitha-suresh, the LLM call in the task function you are passing to the experiment method needs to be instrumented. Is this the case? Could you share how how task function looks like?