A/B testing

[boxabirds]

Problem: with such wild variability in output based on not only the LLMs but the prompts, small changes can result in quite significant differences.

Solution: ability to specify a list of prompt variations and a list of different LLMs to try.

You could use Optuna for efficient evaluation (cf DSPy), along with argilla the human evaluation.

[sysradium]

@boxabirds have you got an API in mind?