Experiment via UI fails

[stikoson]

Describe your question

After setting up an experiment with UI in an Azure self-hosted environment and an litellm proxy using azure-gpt-4o I get the following error when running the experiment:

In the GUI´s "Experiment run details window":

{7 Items
error: "Langfuse in-app experiments can only be run with prompt and model configurations in metadata."
model: "azure-gpt-4o"
provider: "litellm-dev"
prompt_id: "b01cb2b6-bedc-4c8c-8900-4507de83d986"
model_params: {0 Items
}
experiment_name: "Prompt AA Acceptance-v2 on dataset TestTest"
experiment_run_name: "Prompt AA Acceptance-v2 on dataset TestTest - 2026-01-19T12:39:29.488Z"
}

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

3.10.0

If Langfuse Cloud

No response

SDK and integration versions

Package Version Editable project location

---

agno 1.8.1
aio-pika 9.5.7
aiohappyeyeballs 2.6.1
aiohttp 3.12.15
aiormq 6.9.0
aiosignal 1.4.0
alembic 1.16.5
annotated-doc 0.0.4
annotated-types 0.7.0
anthropic 0.66.0
anyio 4.12.1
argon2-cffi 25.1.0
argon2-cffi-bindings 25.1.0
arrow 1.4.0
asgiref 3.9.1
asttokens 3.0.0
async-lru 2.0.5
asyncpg 0.30.0
attrs 25.3.0
authlib 1.6.3
azure-common 1.1.28
azure-core 1.35.0
azure-identity 1.24.0
azure-search-documents 11.5.3
azure-servicebus 7.14.2
babel 2.17.0
backoff 2.2.1
beautifulsoup4 4.14.3
bleach 6.3.0
boto3 1.40.23
botocore 1.40.23
certifi 2026.1.4
cffi 1.17.1
cfgv 3.4.0
charset-normalizer 3.4.4
click 8.2.1
colorama 0.4.6
comm 0.2.3
coverage 7.10.6
cryptography 45.0.7
cyclopts 3.23.1
debugpy 1.8.16
decorator 5.2.1
defusedxml 0.7.1
distlib 0.4.0
distro 1.9.0
dnspython 2.7.0
docstring-parser 0.17.0
docutils 0.22
email-validator 2.3.0
et-xmlfile 2.0.0
events 0.5
exceptiongroup 1.3.0
executing 2.2.1
faker 37.6.0
fastapi 0.124.4
fastjsonschema 2.21.2
fastmcp 2.12.2
fastuuid 0.12.0
filelock 3.19.1
fqdn 1.5.1
frozenlist 1.7.0
fsspec 2025.9.0
geographiclib 2.1
geopy 2.4.1
gitdb 4.0.12
gitpython 3.1.45
googleapis-common-protos 1.72.0
greenlet 3.2.4
griffe 1.13.0
grpcio 1.74.0
h11 0.16.0
httpcore 1.0.9
httpx 0.28.1
httpx-sse 0.4.1
huggingface-hub 0.34.4
identify 2.6.13
idna 3.11
importlib-metadata 8.7.1
iniconfig 2.1.0
ipykernel 6.30.1
ipython 9.5.0
ipython-pygments-lexers 1.1.1
ipywidgets 8.1.8
isodate 0.7.2
isoduration 20.11.0
jedi 0.19.2
jinja2 3.1.6
jiter 0.12.0
jmespath 1.0.1
json5 0.12.1
jsonpatch 1.33
jsonpointer 3.0.0
jsonschema 4.25.1
jsonschema-path 0.3.4
jsonschema-specifications 2025.4.1
jupyter 1.1.1
jupyter-client 8.6.3
jupyter-console 6.6.3
jupyter-core 5.8.1
jupyter-events 0.12.0
jupyter-lsp 2.3.0
jupyter-server 2.17.0
jupyter-server-terminals 0.5.3
jupyterlab 4.5.0
jupyterlab-pygments 0.3.0
jupyterlab-server 2.28.0
jupyterlab-widgets 3.0.16
langchain 0.3.27
langchain-core 0.3.75
langchain-text-splitters 0.3.11
langfuse 3.10.0
langsmith 0.4.23
lark 1.3.1
lazy-object-proxy 1.12.0
litellm 1.76.2
loguru 0.7.3
mako 1.3.10
markdown-it-py 4.0.0
markupsafe 3.0.2
matplotlib-inline 0.1.7
mcp 1.13.1
mdurl 0.1.2
mistune 3.1.4
more-itertools 10.8.0
msal 1.33.0
msal-extensions 1.3.1
multidict 6.6.4
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest-asyncio 1.6.0
nodeenv 1.9.1
notebook 7.5.0
notebook-shim 0.2.4
numpy 2.3.2
openai 2.15.0
openai-agents 0.2.11
openapi-core 0.19.5
openapi-pydantic 0.5.1
openapi-schema-validator 0.6.3
openapi-spec-validator 0.7.2
openinference-instrumentation 0.1.38
openinference-instrumentation-agno 0.1.14
openinference-instrumentation-openai 0.1.31
openinference-semantic-conventions 0.1.21
openlit 1.35.1
openpyxl 3.1.5
opensearch-py 3.0.0
opentelemetry-api 1.39.1
opentelemetry-exporter-otlp 1.36.0
opentelemetry-exporter-otlp-proto-common 1.39.1
opentelemetry-exporter-otlp-proto-grpc 1.36.0
opentelemetry-exporter-otlp-proto-http 1.39.1
opentelemetry-instrumentation 0.57b0
opentelemetry-instrumentation-aiohttp-client 0.57b0
opentelemetry-instrumentation-asgi 0.57b0
opentelemetry-instrumentation-django 0.57b0
opentelemetry-instrumentation-falcon 0.57b0
opentelemetry-instrumentation-fastapi 0.57b0
opentelemetry-instrumentation-flask 0.57b0
opentelemetry-instrumentation-httpx 0.57b0
opentelemetry-instrumentation-pyramid 0.57b0
opentelemetry-instrumentation-requests 0.57b0
opentelemetry-instrumentation-starlette 0.57b0
opentelemetry-instrumentation-tornado 0.57b0
opentelemetry-instrumentation-urllib 0.57b0
opentelemetry-instrumentation-urllib3 0.57b0
opentelemetry-instrumentation-wsgi 0.57b0
opentelemetry-proto 1.39.1
opentelemetry-sdk 1.39.1
opentelemetry-semantic-conventions 0.60b1
opentelemetry-util-http 0.57b0
orjson 3.11.3
packaging 25.0
pamqp 3.3.0
pandas 2.3.2
pandocfilters 1.5.1
parse 1.20.2
parso 0.8.5
pathable 0.4.4
pgvector 0.4.1
platformdirs 4.4.0
pluggy 1.6.0
pre-commit 4.3.0
prometheus-client 0.23.1
prompt-toolkit 3.0.52
propcache 0.3.2
protobuf 6.33.4
psutil 7.0.0
psycopg 3.2.9
psycopg-binary 3.2.9
psycopg-pool 3.2.6
pure-eval 0.2.3
pycparser 2.22
pydantic 2.12.5
pydantic-core 2.41.5
pydantic-settings 2.10.1
pygments 2.19.2
pyjwt 2.10.1
pyperclip 1.9.0
pytest 8.4.2
pytest-asyncio 1.1.0
pytest-cov 6.2.1
pytest-mock 3.14.1
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-json-logger 4.0.0
python-multipart 0.0.20
pytz 2025.2
pywin32 311
pywinpty 3.0.2
pyyaml 6.0.2
pyzmq 27.0.2
redis 6.4.0
referencing 0.36.2
regex 2025.9.1
requests 2.32.5
requests-toolbelt 1.0.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rfc3987-syntax 1.1.0
rich 14.1.0
rich-rst 1.3.1
rpds-py 0.27.1
ruff 0.12.11
s3transfer 0.13.1
schedule 1.2.2
send2trash 1.8.3
setuptools 80.9.0
shellingham 1.5.4
six 1.17.0
smmap 5.0.2
sniffio 1.3.1
soupsieve 2.8
sqlalchemy 2.0.43
sse-starlette 3.0.2
stack-data 0.6.3
starlette 0.47.3
tenacity 9.1.2
terminado 0.18.1
tiktoken 0.11.0
tinycss2 1.4.0
tokenizers 0.22.0
tomli 2.2.1
tornado 6.5.2
tqdm 4.67.1
traitlets 5.14.3
typer 0.17.3
types-requests 2.32.4.20250809
typing-extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
uri-template 1.3.0
urllib3 2.6.3
uvicorn 0.35.0
virtualenv 20.34.0
wcwidth 0.2.13
webcolors 25.10.0
webencodings 0.5.1
websocket-client 1.9.0
werkzeug 3.1.1
widgetsnbextension 4.0.15
win32-setctime 1.2.0
wrapt 1.17.3
xmltodict 0.14.2
yarl 1.20.1
zipp 3.23.0
zstandard 0.24.0

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The error "Langfuse in-app experiments can only be run with prompt and model configurations in metadata" indicates that your experiment setup is missing required configuration in the LLM connection metadata.(1)

When running Experiments via UI with a custom LiteLLM proxy, you need to configure an LLM connection in your project settings.(1) The LLM connection should include both the prompt and model configurations in its metadata to work properly with in-app experiments.(1)

To resolve this:

1. Navigate to your project settings and configure your LLM connection properly(1)
2. Ensure the LLM connection includes the necessary prompt and model configuration metadata for your azure-gpt-4o model through the litellm-dev provider
3. Verify that your LiteLLM proxy configuration is correctly set up to work with Azure OpenAI

The experiment setup requires:

• A usable prompt with variables matching your dataset item keys(1)
• A properly configured LLM connection(1)
• A usable dataset with JSON objects as input(1)

Since you're using a self-hosted environment (version 3.10.0), make sure your LLM connection configuration in the project settings contains all the required metadata fields for the experiment to execute successfully.(1)

📚 Sources:

1. Experiments via UI (Prompt Experiments)

---

Have another question? Just tag @inkeep.