MCP tools not working for LibreChat + LiteLLM + Ollama

[snowstopxt]

I am using litellm proxy with these configs

and my librechat with these configs

I am trying to use the mcp tools defined in my fastapi_mcp server but am always met with an incorrect response and tools not being run correctly. Agents are also met with the same problem.

I am running everything in deploy-compose (including my mcp server) Any help will be appreciated, as I have been working on this for hours and am still not able to get it to work.

What I have tried:

• Using ollama_chat -> returns error: json: cannot unmarshal array into Go struct field .tools.function.parameters.properties.type of type string
• Changing mcp server to the litellm exposed mcp url -> error: Connection failed

[wagnerjt]

What transport does your MCP server use? Right now litellm only supports SSE.

You can also check out the Litellm proxy UI to inspect and run the tool from there at the /ui

One other thing you can try is to use the docker-compose identifier as the network like http://docker_identifier:PORT/mcp instead of http://host.docker.internal:PORT/mcp.

[snowstopxt]

I am using the SSE endpoint exposed by fastapi_mcp. Using swagger ui lists the tools correctly. Docker-compose identifier failed to list the tools.

However, I could not use /ui to list the mcp tools
2025-06-02 09:53:05 01:53:05 - LiteLLM:ERROR: server.py:295 - Error getting tools from 5d2efd77-774d-4a0b-923b-eb6c1adaa06f: unhandled errors in a TaskGroup (1 sub-exception)
2025-06-02 09:53:05 Traceback (most recent call last):
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/mcp/client/sse.py", line 47, in sse_client
2025-06-02 09:53:05 async with aconnect_sse(
2025-06-02 09:53:05 ~~~~~~~~~~~~^
2025-06-02 09:53:05 client,
2025-06-02 09:53:05 ^^^^^^^
2025-06-02 09:53:05 ...<2 lines>...
2025-06-02 09:53:05 timeout=httpx.Timeout(timeout, read=sse_read_timeout),
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 ) as event_source:
2025-06-02 09:53:05 ^
2025-06-02 09:53:05 File "/usr/lib/python3.13/contextlib.py", line 214, in aenter
2025-06-02 09:53:05 return await anext(self.gen)
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx_sse/_api.py", line 69, in aconnect_sse
2025-06-02 09:53:05 async with client.stream(method, url, headers=headers, **kwargs) as response:
2025-06-02 09:53:05 ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/contextlib.py", line 214, in aenter
2025-06-02 09:53:05 return await anext(self.gen)
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1617, in stream
2025-06-02 09:53:05 response = await self.send(
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 ...<4 lines>...
2025-06-02 09:53:05 )
2025-06-02 09:53:05 ^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1661, in send
2025-06-02 09:53:05 response = await self._send_handling_auth(
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 ...<4 lines>...
2025-06-02 09:53:05 )
2025-06-02 09:53:05 ^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
2025-06-02 09:53:05 response = await self._send_handling_redirects(
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 ...<3 lines>...
2025-06-02 09:53:05 )
2025-06-02 09:53:05 ^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
2025-06-02 09:53:05 response = await self._send_single_request(request)
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1763, in _send_single_request
2025-06-02 09:53:05 response = await transport.handle_async_request(request)
2025-06-02 09:53:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 372, in handle_async_request
2025-06-02 09:53:05 with map_httpcore_exceptions():
2025-06-02 09:53:05 ~~~~~~~~~~~~~~~~~~~~~~~^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/contextlib.py", line 162, in exit
2025-06-02 09:53:05 self.gen.throw(value)
2025-06-02 09:53:05 ~~~~~~~~~~~~~~^^^^^^^
2025-06-02 09:53:05 File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
2025-06-02 09:53:05 raise mapped_exc(message) from exc
2025-06-02 09:53:05 httpx.ConnectError: All connection attempts failed
2025-06-02 09:53:05
2025-06-02 09:53:05 During handling of the above exception, another exception occurred:
2025-06-02 09:53:05
2025-06-02 09:53:05 + Exception Group Traceback (most recent call last):
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/litellm/proxy/_experimental/mcp_server/server.py", line 284, in list_tool_rest_api
2025-06-02 09:53:05 | tools = await global_mcp_server_manager._get_tools_from_server(server)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/litellm/proxy/_experimental/mcp_server/mcp_server_manager.py", line 163, in _get_tools_from_server
2025-06-02 09:53:05 | async with sse_client(url=server.url) as (read, write):
2025-06-02 09:53:05 | ~~~~~~~~~~^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/contextlib.py", line 214, in aenter
2025-06-02 09:53:05 | return await anext(self.gen)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/mcp/client/sse.py", line 43, in sse_client
2025-06-02 09:53:05 | async with anyio.create_task_group() as tg:
2025-06-02 09:53:05 | ~~~~~~~~~~~~~~~~~~~~~~~^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 685, in aexit
2025-06-02 09:53:05 | raise BaseExceptionGroup(
2025-06-02 09:53:05 | "unhandled errors in a TaskGroup", self._exceptions
2025-06-02 09:53:05 | )
2025-06-02 09:53:05 | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2025-06-02 09:53:05 +-+---------------- 1 ----------------
2025-06-02 09:53:05 | Traceback (most recent call last):
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
2025-06-02 09:53:05 | yield
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
2025-06-02 09:53:05 | resp = await self._pool.handle_async_request(req)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
2025-06-02 09:53:05 | raise exc from None
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
2025-06-02 09:53:05 | response = await connection.handle_async_request(
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | pool_request.request
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | )
2025-06-02 09:53:05 | ^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
2025-06-02 09:53:05 | raise exc
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
2025-06-02 09:53:05 | stream = await self._connect(request)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 124, in _connect
2025-06-02 09:53:05 | stream = await self._network_backend.connect_tcp(**kwargs)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
2025-06-02 09:53:05 | return await self._backend.connect_tcp(
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | ...<5 lines>...
2025-06-02 09:53:05 | )
2025-06-02 09:53:05 | ^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
2025-06-02 09:53:05 | with map_exceptions(exc_map):
2025-06-02 09:53:05 | ~~~~~~~~~~~~~~^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/contextlib.py", line 162, in exit
2025-06-02 09:53:05 | self.gen.throw(value)
2025-06-02 09:53:05 | ~~~~~~~~~~~~~~^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
2025-06-02 09:53:05 | raise to_exc(exc) from exc
2025-06-02 09:53:05 | httpcore.ConnectError: All connection attempts failed
2025-06-02 09:53:05 |
2025-06-02 09:53:05 | The above exception was the direct cause of the following exception:
2025-06-02 09:53:05 |
2025-06-02 09:53:05 | Traceback (most recent call last):
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/mcp/client/sse.py", line 47, in sse_client
2025-06-02 09:53:05 | async with aconnect_sse(
2025-06-02 09:53:05 | ~~~~~~~~~~~~^
2025-06-02 09:53:05 | client,
2025-06-02 09:53:05 | ^^^^^^^
2025-06-02 09:53:05 | ...<2 lines>...
2025-06-02 09:53:05 | timeout=httpx.Timeout(timeout, read=sse_read_timeout),
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | ) as event_source:
2025-06-02 09:53:05 | ^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/contextlib.py", line 214, in aenter
2025-06-02 09:53:05 | return await anext(self.gen)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx_sse/_api.py", line 69, in aconnect_sse
2025-06-02 09:53:05 | async with client.stream(method, url, headers=headers, **kwargs) as response:
2025-06-02 09:53:05 | ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/contextlib.py", line 214, in aenter
2025-06-02 09:53:05 | return await anext(self.gen)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1617, in stream
2025-06-02 09:53:05 | response = await self.send(
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | ...<4 lines>...
2025-06-02 09:53:05 | )
2025-06-02 09:53:05 | ^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1661, in send
2025-06-02 09:53:05 | response = await self._send_handling_auth(
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | ...<4 lines>...
2025-06-02 09:53:05 | )
2025-06-02 09:53:05 | ^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
2025-06-02 09:53:05 | response = await self._send_handling_redirects(
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | ...<3 lines>...
2025-06-02 09:53:05 | )
2025-06-02 09:53:05 | ^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
2025-06-02 09:53:05 | response = await self._send_single_request(request)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1763, in _send_single_request
2025-06-02 09:53:05 | response = await transport.handle_async_request(request)
2025-06-02 09:53:05 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 372, in handle_async_request
2025-06-02 09:53:05 | with map_httpcore_exceptions():
2025-06-02 09:53:05 | ~~~~~~~~~~~~~~~~~~~~~~~^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/contextlib.py", line 162, in exit
2025-06-02 09:53:05 | self.gen.throw(value)
2025-06-02 09:53:05 | ~~~~~~~~~~~~~~^^^^^^^
2025-06-02 09:53:05 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
2025-06-02 09:53:05 | raise mapped_exc(message) from exc
2025-06-02 09:53:05 | httpx.ConnectError: All connection attempts failed
2025-06-02 09:53:05 +------------------------------------

Is litellm proxy acting as the MCP client, or do I need to create a new MCP client?

[wagnerjt]

The litellm proxy should create a MCP client by default over SSE which is what is happening and connecting in the list tools call.
Kind of interesting the proxy is not responding as well as the UI. Maybe it has something to do with the last commit that was merged. Let me take a look

[wagnerjt]

@ishaan-jaff looks like there is a double server_id being created for the config MCP servers which is causing this issue.

[snowstopxt]

I see, I have also managed to get the MCP tool call to work in LibreChat thanks to this discussion danny-avila/LibreChat#2215. I needed to use openai prefix for the model and add /v1 to the api_base in litellm-config.yaml.

[wagnerjt]

Glad to hear it works! Made the pr fix here