Experiments via UI for Agent Server?

[ToanLyHoa]

Describe your question

Can I connect to an agent with A2A protocol and experiment directly on UI? Or for complex task/agent I have to use langfuse sdk for experiment

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

For complex tasks and agents, you need to use the Langfuse SDK for experiments(1). The UI-based experiment feature you're referencing (with the webhook configuration for "Custom Experiment") is designed for triggering SDK-based experiments remotely, not for directly connecting to an agent server via A2A protocol(1).

The workflow is:

1. Configure a webhook URL in the Langfuse UI that points to your external evaluation service(1)
2. When you trigger an experiment from the UI, Langfuse sends the dataset metadata (ID and name) along with custom configuration to your webhook(1)
3. Your webhook receives the request, fetches the dataset from Langfuse, runs your application (including your agent) against the dataset items, evaluates the results, and ingests the scores back into Langfuse as a new experiment run(1)

The SDK approach is necessary because experiments require defining a task function that executes your agent logic, along with evaluators to assess the outputs(1)(2). The SDK handles concurrent execution, automatic tracing, and error isolation during the experiment run(1).

📚 Sources:

1. Experiments via SDK
2. Systematic Evaluation of AI Agents - Langfuse Blog

---

Have another question? Just tag @inkeep.

[ToanLyHoa]

I need an answer from maintainer or dev actually.

[jannikmaierhoefer]

Hi @ToanLyHoa, for UI experiments, you could expose your application in OpenAI API schema format to add it as an LLM connection.

Alternatively, you can use the SDK experiment methods for this.

You can also set up remote experiment triggers to enable non-technical people in your team to trigger SDK experiments via the UI: https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk#optional-trigger-sdk-experiment-from-ui

[ToanLyHoa]

• Thank you, I got it.