Run prompt experiments via the SDKs/API

[hspak]

Describe the feature or potential improvement

If the prompt experiment creation was exposed via the SDK, it could make automated eval runs pretty straightforward.

I was thinking something along the lines of:

1. Datasets and evals are pre-configured.
2. The prompt experiment could be programmatically kicked off (e.g by CI) to proactively catch regressions.

Let me know if there's an alternate approach I should consider as well.

Additional information

No response

[marcklingen]

Thanks for sharing! There is no near-term plan to extend the in-ui prompt experiments to the SDKs as these kinds of tests can be run via the Langfuse SDK as well in a few lines of code (happy to help figure this out if you have questions): https://langfuse.com/docs/datasets/get-started

Prompt Experiments are a way of doing everything in the UI, it is basically a fully-managed experiment run that would otherwise need to be created via the SDK.

[Whadup]

@marcklingen We would like to run our prompt experiments repeatedly on a schedule. To this end, either a UI option for setting up schedules or a SDK option to start the prompt experiment would help a lot.
With UI Scheduling support, we could do everything we need without code and additional infrastructure. Perfect for our less technical users.
With SDK support we wouldn't need to have a runner that executes the experiment in python, but only triggers the runner in the langfuse instance with a single post request. Less maintenance work, less infrastructure work.

[savitha-suresh]

Hello,
Was this introduced ? We would be needing this feature as well, would appreciate alternative suggestions if any.
Thanks