Can't fetch models from local Ollama API

[ai212983]

What happened?

With local Ollama being set up as a custom endpoint in librechat.yaml, I see the following error in the console:

LibreChat         | 2025-01-26 18:42:30 error: Failed to fetch models from Ollama API. If you are not using Ollama directly, and instead, through some aggregator or reverse proxy that handles fetching via OpenAI spec, ensure the name of the endpoint doesn't start with `ollama` (case-insensitive). Cannot read properties of undefined (reading 'status')

And the following json in the log files:

{
  "level": "error",
  "message": "Failed to fetch models from Ollama API. If you are not using Ollama directly, and instead, through some aggregator or reverse proxy that handles fetching via OpenAI spec, ensure the name of the endpoint doesn't start with `ollama` (case-insensitive). Cannot read properties of undefined (reading 'status')",
  "stack": "TypeError: Cannot read properties of undefined (reading 'status')\n    at /app/packages/data-provider/dist/index.js:1:158656\n    at /app/packages/data-provider/dist/index.js:1:2003\n    at Object.next (/app/packages/data-provider/dist/index.js:1:2108)\n    at /app/packages/data-provider/dist/index.js:1:951\n    at new Promise (<anonymous>)\n    at v (/app/packages/data-provider/dist/index.js:1:696)\n    at /app/packages/data-provider/dist/index.js:1:158526\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Axios.request (/app/node_modules/axios/dist/node/axios.cjs:4257:14)\n    at async OllamaClient.fetchModels (/app/api/app/clients/OllamaClient.js:63:24)\n    at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:4262:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async OllamaClient.fetchModels (/app/api/app/clients/OllamaClient.js:63:24)\n    at async fetchModels (/app/api/server/services/ModelService.js:46:12)\n    at async Promise.all (index 0)\n    at async loadConfigModels (/app/api/server/services/Config/loadConfigModels.js:94:23)\n    at async loadModels (/app/api/server/controllers/ModelController.js:30:30)\n    at async modelController (/app/api/server/controllers/ModelController.js:39:23)"
}

curl -s http://127.0.0.1:11434/api/tags works fine:

{
  "models": [
    {
      "name": "deepseek-r1:32b",
      "model": "deepseek-r1:32b",
      "modified_at": "2025-01-26T17:26:50.446264368+01:00",
      "size": 19851337640,
      "digest": "38056bbcbb2d068501ecb2d5ea9cea9dd4847465f1ab88c4d4a412a9f7792717",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "qwen2",
        "families": [
          "qwen2"
        ],
        "parameter_size": "32.8B",
        "quantization_level": "Q4_K_M"
      }
    },
    {
      "name": "mistral-nemo:latest",
      "model": "mistral-nemo:latest",
      "modified_at": "2024-10-23T20:07:09.56710404+02:00",
      "size": 7071713232,
      "digest": "994f3b8b78011aa6d578b0c889cbb89a64b778f80d73b8d991a8db1f1e710ace",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "12.2B",
        "quantization_level": "Q4_0"
      }
    },
    {
      "name": "phi3.5:latest",
      "model": "phi3.5:latest",
      "modified_at": "2024-10-21T15:16:38.384678442+02:00",
      "size": 2176178843,
      "digest": "61819fb370a3c1a9be6694869331e5f85f867a079e9271d66cb223acb81d04ba",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "phi3",
        "families": [
          "phi3"
        ],
        "parameter_size": "3.8B",
        "quantization_level": "Q4_0"
      }
    },
    {
      "name": "mxbai-embed-large:latest",
      "model": "mxbai-embed-large:latest",
      "modified_at": "2024-10-21T13:37:01.179155837+02:00",
      "size": 669615493,
      "digest": "468836162de7f81e041c43663fedbbba921dcea9b9fefea135685a39b2d83dd8",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "bert",
        "families": [
          "bert"
        ],
        "parameter_size": "334M",
        "quantization_level": "F16"
      }
    },
    {
      "name": "nomic-embed-text:latest",
      "model": "nomic-embed-text:latest",
      "modified_at": "2024-10-21T12:59:08.265131818+02:00",
      "size": 274302450,
      "digest": "0a109f422b47e3a30ba2b10eca18548e944e8a23073ee3f3e947efcf3c45e59f",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "nomic-bert",
        "families": [
          "nomic-bert"
        ],
        "parameter_size": "137M",
        "quantization_level": "F16"
      }
    },
    {
      "name": "mistral:latest",
      "model": "mistral:latest",
      "modified_at": "2024-10-21T09:03:58.048470126+02:00",
      "size": 4113301824,
      "digest": "f974a74358d62a017b37c6f424fcdf2744ca02926c4f952513ddf474b2fa5091",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "7.2B",
        "quantization_level": "Q4_0"
      }
    },
    {
      "name": "llama3.2:latest",
      "model": "llama3.2:latest",
      "modified_at": "2024-10-17T18:12:18.451237082+02:00",
      "size": 2019393189,
      "digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "3.2B",
        "quantization_level": "Q4_K_M"
      }
    }
  ]
}

Steps to Reproduce

1. Set up LibreChat to work with librechat.yaml.
2. Install and start local Ollama
3. Configure a custom external endpoint for Ollama in librechat.yaml:

  custom:           
    - name: "Ollama"
      apiKey: "ollama"                                                                                                                                       
      baseURL: "http://127.0.0.0:11434/api"   # slug doesn't matter, LibreChat will strip it                                                                                    
      models:                                                                                                                                                
        default: [                                                                                                                                           
          "llama3:latest"                                                                                                                                    
        ]                                                                                                                                                    
        fetch: true                                                                                                                                          
      titleConvo: true                                                                                                                                       
      titleModel: "current_model"                                                                                                                            
      summarize: false                                                                                                                                       
      summaryModel: "current_model"                                                                                                                          
      forcePrompt: false                                                                                                                                     
      modelDisplayLabel: "ollama"                                                                                                                            
      capabilities: ["retrieval", "actions", "tools", "image_vision", "code_interpreter"]

4. Start LibreChat via docker compose up (non-daemon mode to see errors and messages)
5. Open the LibreChat page with a new chat
6. Observe the error in the log, "Failed to fetch models from Ollama API.".

What browsers are you seeing the problem on?

No response

Relevant log output

LibreChat         | 2025-01-26 18:42:30 error: Failed to fetch models from Ollama API. If you are not using Ollama directly, and instead, through some aggregator or reverse proxy that handles fetching via OpenAI spec, ensure the name of the endpoint doesn't start with `ollama` (case-insensitive). Cannot read properties of undefined (reading 'status')

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[dane-git]

Same issue, can't curl from the docker to the local hosted ollama on port: 11434. Some docker network bridge problem maybe?

librechat.yaml effectively the same as ai212983.

added OLLAMA_HOST env var to local. Also added this line to docker compose.yml , to no avail:

environment:
 - OLLAMA_HOST=http://host.docker.internal:11434 # added

same problem:
/app/api/logs/errror*.logs

"code": "ECONNABORTED",
    "config": {
      "adapter": [
        "xhr",
        "http",
        "fetch"
      ],
      "env": {},
      "headers": {
        "Accept": "application/json, text/plain, */*",
        "Accept-Encoding": "gzip, compress, deflate, br",
        "User-Agent": "axios/1.7.7"
      },
      "maxBodyLength": -1,
      "maxContentLength": -1,
      "method": "get",
      "timeout": 5000,
      "transformRequest": [
        null
      ],
      "transformResponse": [
        null
      ],
      "transitional": {
        "clarifyTimeoutError": false,
        "forcedJSONParsing": true,
        "silentJSONParsing": true
      },
      "url": "http://host.docker.internal:11434/api/tags",
      "xsrfCookieName": "XSRF-TOKEN",
      "xsrfHeaderName": "X-XSRF-TOKEN"
    },
    "level": "error",
    "message": "Failed to fetch models from Ollama API. If you are not using Ollama directly, and instead, through some aggregator or reverse proxy that handles fetching via OpenAI spec, ensure the name of the endpoint doesn't start with `ollama` (case-insensitive). timeout of 5000ms exceeded",

[ai212983]

@dane-git I guess this is not the same issue. The meaningful part of the error message is at the end of it. In my case it is Cannot read properties of undefined (reading 'status'), which imho points to the code, in your case it is timeout of 5000ms exceeded.

[danny-avila]

Make sure Ollama is configured for network access: https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server

[itoubixi]

environment:
 - OLLAMA_HOST=http://host.docker.internal:11434 # added

Not sure if you are using WSL. If yes, have you tried your WSL 2 IP in case?

environment:
      - OLLAMA_HOST=http://x.x.x.x:11434

You can get your WSL 2 IP with this command :

ip route | grep default | awk '{print $3}'

For me, that was the way I fixed it. I am using WSL with Windows 11. I am not using Ollama as a container. I have it as an exe.

[F1zzyD]

I'm having this same issue. Libre refuses to see, connect, or have anything to do with Ollama even though Ollama is set up correctly, exposed for Libre to see, and shows models ready for use. Seems to be a Libre issue.

[limdingwen]

I'm running Ollama on my Windows natively.

What worked for me is commenting out this part in docker-compose.yml:

services:
  api:
    container_name: LibreChat
    ports:
      - "${PORT}:${PORT}"
    depends_on:
      - mongodb
      - rag_api
    image: ghcr.io/danny-avila/librechat-dev:latest
    restart: always
    user: "${UID}:${GID}"
    # extra_hosts:
    #   - "host.docker.internal:host-gateway"

Remember that your ollama URL needs to use host.docker.internal!

  custom:
    - name: "Ollama"
      apiKey: "ollama"
      # use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
      baseURL: "http://host.docker.internal:11434/"
      models:
        default:
          ["dolphin-phi:2.7b"]
        # fetching list of models is supported but the `name` field must start
        # with `ollama` (case-insensitive), as it does in this example.
        fetch: true
      titleConvo: false
      titleModel: "dolphin-phi:2.7b"
      summarize: false
      summaryModel: "dolphin-phi:2.7b"
      forcePrompt: false
      modelDisplayLabel: "Ollama"

I think host.docker.internal is already enabled in Windows, and aliasing it to host-gateway was screwing up something. Hope this helps!