How to add LLM connection, as I am using LLM models from litellm.

[avalsonigara15]

Describe your question

How can I connect my litellm models in the LLM connection section, as in attempt to doing so I am having connection error and unable to figure out what is the error about.
This is how added my llm connection in the UI. please let me know if I have configured it wrong.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

https://cloud.langfuse.com/project/cmkzdd3bm0438ad07hp998w1n/settings/llm-connections

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on your screenshot and the common issues with LiteLLM connections, here are the likely problems:

API Base URL Configuration

Your API Base URL should end with /v1 and not include /api(1). The correct format should be:

http://your-litellm-host:port/v1

Not:

http://your-litellm-host/api/

Network Accessibility Issues

If you're running LiteLLM and Langfuse in Docker or Kubernetes, using localhost may not work(1)(2). From the Langfuse container's perspective, you may need to use:

• host.docker.internal (for Docker on most systems)(2)
• The internal network hostname of your LiteLLM service(1)

OpenAI API Schema Compliance

Langfuse expects the backend to fully implement the OpenAI API schema(1). Your LiteLLM Proxy must:

• Return a 200 status from the /v1/models endpoint(1)
• Provide a valid OpenAI-compliant response(1)

Message Role Validation

Langfuse sends a test payload with a "system" message followed by a "user" message during connection setup(1). Some OpenAI-compatible backends, including certain LiteLLM setups, require strict alternation between "user" and "assistant" roles and may reject this sequence(1).

Recommended Configuration

Based on your setup, try:

1. Set API Base URL to: http://host.docker.internal:8000/v1 (adjust port as needed)(2)
2. Ensure your LiteLLM /v1/models endpoint is accessible and returns valid responses(1)
3. Verify your API key is correct(1)
4. Check that no firewall rules block traffic between Langfuse and LiteLLM(1)

If you're still experiencing issues, you can also use the API to manage LLM connections programmatically via PUT /api/public/llm-connections(3)(4).

📚 Sources:

1. getting connection error when ading custom models using LLM connection
2. bug: create new llm connection error
3. LLM Connections
4. PUT /api/public/llm-connections

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @avalsonigara15, I think you need to make sure to use the OpenAI API compatible LiteLLM endpoint to use the models as LLM connections: https://docs.litellm.ai/docs/providers/openai