LiteLLM + Ollama + lobe-chat implementation issues.

[adan89lion]

Hi,

I'm trying to use lobe-chat as a frontend for Ollama, proxied via LiteLLM. However, I've been encountering 500 internal server error.

The following are my docker-compose and config files
LiteLLM Docker Compose File

version: "3.9"
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "1234:8000"
    volumes:
      - ./config.yml:/app/config.yaml
    command: [ "--config", "/app/config.yaml", "--port", "8000", "--num_workers", "8" ]

LiteLLM Config File

model_list:
  - model_name: mistral-v2-7b-uncensored
    litellm_params:
      model: ollama/dolphin-mistral:7b-v2-q5_K_S
      api_base: http://192.168.0.161:11434

lobe-chat Docker Compose File

version: '3.8'

services:
  lobe-chat:
    image: lobehub/lobe-chat
    container_name: lobe-chat
    restart: unless-stopped
    ports:
      - '3210:3210'
    environment:
      OPENAI_API_KEY: sk-xxxx
      OPENAI_PROXY_URL: http://192.168.0.161:1234/
      ACCESS_CODE: lobe66

Accessing model list produces the following error for LiteLLM

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/litellm/proxy/proxy_server.py", line 1430, in chat_completion
    response = await litellm.acompletion(**data)
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2349, in wrapper_async
    raise e
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2241, in wrapper_async
    result = await original_function(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 227, in acompletion
    raise exception_type(
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 6534, in exception_type
    raise e
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 5556, in exception_type
    raise APIError(
litellm.exceptions.APIError: OpenAIException - The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
INFO:     192.168.148.1:35271 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

And lobe-chat returned the following error

{
  "endpoint": "http://192.168.***.161:****/",
  "error": {
    "headers": {
      "content-length": "2244",
      "content-type": "application/json",
      "date": "Sat, 06 Jan 2024 17:01:43 GMT",
      "server": "uvicorn"
    },
    "stack": "Error: 500 status code (no body)\n    at W.generate (/app/.next/server/app/api/openai/chat/route.js:22:1839)\n    at tL.makeStatusError (/app/.next/server/app/api/openai/chat/route.js:22:13170)\n    at tL.makeRequest (/app/.next/server/app/api/openai/chat/route.js:22:13900)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async k (/app/.next/server/app/api/openai/chat/route.js:2:2906)\n    at async /app/.next/server/app/api/openai/chat/route.js:7:64510",
    "status": 500
  }
}```

Am I missing something for my configuration? Thanks in advance.