Allow triggering Prompt Experiments programmatically (via public API)

[elevemonkey]

Describe the feature or potential improvement

Feature Request: Allow triggering Prompt Experiments programmatically (via public API)

Hi Langfuse team 👋,

Thanks for the amazing work on Langfuse — it's been a game changer for managing prompt evaluations!

Context

I’m currently building an automated evaluation pipeline where:

• Prompt templates are version-controlled and updated frequently.
• Every time a prompt is updated, we want to automatically trigger a new Prompt Experiment (i.e. Native Dataset Run) on a selected dataset, using a specific LLM configuration and optionally a custom evaluator.

This is essential for running automated regression tests and maintaining prompt quality at scale.

Current Limitation

Right now, Prompt Experiments can only be triggered manually through the UI, which is not ideal for automation.
I noticed that the Langfuse UI calls the internal endpoint:

POST /api/trpc/experiments.createExperiment

However, this endpoint:

• Requires a user session (cookie), and
• Is not exposed or documented as part of the public API or SDK.

What I'm Requesting

Could you please consider:

1. Officially exposing a public API or SDK method to programmatically trigger Prompt Experiments, or
2. Allowing authenticated access to trpc/experiments.createExperiment, with documented input parameters, or
3. Providing a webhook or CI-compatible trigger mechanism for initiating experiments on prompt updates

Why It Matters

Having a way to programmatically trigger experiments would:

• Enable automated CI workflows for prompt development
• Reduce human error from manual UI interaction
• Make it easier to enforce prompt quality gates before deployment

This would be a huge boost for any team doing continuous prompt tuning and evaluation.

Let me know if there's a current workaround (e.g. experimental API, remote evaluation trigger), or if something is already on the roadmap!

Thanks again 🙏

Additional information

No response

[marcklingen]

Thanks for sharing. Currently this is not available, let's track interest for this here.

The recommended approach for regression testing would be to run an experiment via the SDKs, thereby you have full control and can run this in your own environment: https://langfuse.com/docs/evaluation/dataset-runs/remote-run

The trpc routes are as you mentioned not meant to be used directly as they are UI-only.

[elevemonkey]

langfuse CLI not available after installing langfuse package (no langfuse command found)

Problem

I am trying to use the langfuse eval run command to run a dataset evaluation with a custom script.

langfuse eval run --dataset-name "<dataset_name>" --path ./evaluate.py

However, after installing the langfuse package (v3.3.4) via pip, the langfuse CLI command is not available on my system.

Environment

• OS: macOS (Apple Silicon)
• Python: 3.12.4
• Installed package: langfuse==3.3.4
• Installed using: pip install langfuse and also tried pip install 'langfuse[cli]' (but got a warning: "langfuse 3.3.4 does not provide the extra 'cli'")

Error

$ langfuse --help
zsh: command not found: langfuse

Also checked:

$ ls .venv/bin/langfuse
ls: .venv/bin/langfuse: No such file or directory

Question

Is the langfuse CLI a separate package?
If so, how can I install it?
The documentation mentions langfuse eval run, but no instructions are given on how to install the CLI binary.

Would love to get clarification on how to properly set up the CLI command to use dataset evaluation.

Thanks!

[marcklingen]

Where did you get the langfuse eval run from? Please follow the guide mentioned above

[mherbauxMLT]

Hello,
@marcklingen , we (as a company, not related to OP) use langfuse as a single place to store our prompts and evaluators.
I think having the ability to either trigger an experiment from the SDK/API or provide a way to retrieve an evaluator (prompt/config) would be helpful.
The first option is straightforward: we run/create the experiment and wait for experiement to end.
The second option allows us to recreate with the SDK what Langfuse implemented server side.
Kind regards

[jannikmaierhoefer]

Thanks @mherbauxMLT! We have managing LLM-as-a-Judge via API on our roadmap here: https://github.com/orgs/langfuse/discussions/8241. Feel free to subscribe to this post to get updates on this and add any additional thoughts you might have.