Ollama not working with librechat

[SamDc73]

When sending a message to ollama using librechat I get this back:

Something went wrong. Here's the specific error message we encountered: Error connecting to server, try refreshing the page.

but when running

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "phi3:3.8b",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

everything runs as expected, and the command line ollama is running fine ollama run phi3:3.8b

my librechat.yaml file

    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://localhost:11434/v1/chat/completions"
      models:
        default: [
          "qwen2:0.5b",
          "gemma:7b",
          "gemma:2b",
          "phi3:3.8b"
          ]
        fetch: false # fetching list of models is not supported
      titleConvo: true
      titleModel: "current_model"
      modelDisplayLabel: "Ollama"

what I tried:

• changed baseurl to "http://localhost:11434/v1" and "http://0.0.0.0:11434/v1/chat/completions" and "http://127.0.0.1:11434/v1/chat/completions" with no luck
• librechat is installed using docker and is up-to-date.

additional information:

• I have multiple other endpoints that are working perfectly
• I'm running Linux Fedora
• when Ollama is installed using docker it works fine with librechat but with no GPU acceleration

[danny-avila]

when Ollama is installed using docker it works fine with librechat but with no GPU acceleration

Depending on your graphics card, you can add the resources as shown here:
https://www.librechat.ai/blog/2024-03-02_ollama

# docker-compose.override.yml
version: '3.4'
 
services:
# USE LIBRECHAT CONFIG FILE
  api:
    volumes:
    - type: bind
      source: ./librechat.yaml
      target: /app/librechat.yaml
 
# ADD OLLAMA
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [compute, utility]
    ports:
      - "11434:11434"
    volumes:
      - ./ollama:/root/.ollama

[danny-avila]

[SamDc73]

Cool, I'm using docker now and GPU acceleration is working!
ty!

[monahk]

The LibreChat service is trying to access ollama using a host or IP address relative to the docker container it's running in. In this case localhost is the container and not your host machine. You need to reference the host machine by referring to it as:
baseURL: "http://host.docker.internal:11434/v1/"

In this case avoid installing ollama in a container and make use of the actual host for acceleration.