How do I get embeddings to work with LiteLLM?

[rcdailey]

At the moment I'm using LiteLLM as a centralized hub for my third party chat services configuration. At the moment I use it with Anthropic and OpenAI for chat based models.

Sometimes I want to be able to attach PDF files and ask GPT or Claude to explain those files. However, this isn't working at the moment.

It looks like a feature called "embeddings" is used when you attach PDF files. I don't know a lot about them, but I tried to set things up the best I could. On the LiteLLM side, I added a model for embeddings:

  - model_name: gpt-embedding
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

And in LibreChat, I set these in .env:

RAG_OPENAI_BASEURL=http://litellm:4000
# RAG_OPENAI_API_KEY=
EMBEDDINGS_PROVIDER=openai
EMBEDDINGS_MODEL=gpt-embedding

And I have the following in my librechat.yaml:

    - name: LiteLLM
      apiKey: snip
      baseURL: http://litellm:4000/v1
      models:
        default: ["claude-3.5"]
        fetch: true
      titleConvo: true
      titleModel: gpt-4o-mini
      summarize: false
      summaryModel: gpt-4o-mini
      forcePrompt: false
      modelDisplayLabel: "LiteLLM"

At this point I:

1. Start a new chat
2. Attach my PDF file, which seems to upload fine
3. Add the prompt "explain this" and submit. I get an error. Logs from docker compose logs -f api rag_api below.

What am I doing wrong? How do I set this up correctly?

rag_api-1  | 2025-01-04 17:42:48,678 - root - INFO - Request POST http://rag_api:8000/embed - 200
rag_api-1  | 2025-01-04 17:42:51,169 - openai._base_client - INFO - Retrying request to /embeddings in 0.450418 seconds
rag_api-1  | 2025-01-04 17:42:51,623 - openai._base_client - INFO - Retrying request to /embeddings in 0.810049 seconds
rag_api-1  | 2025-01-04 17:42:52,440 - root - ERROR - Error in query embeddings | File ID: 4187bf14-9fa8-4681-b2b0-f3cbf42daad0 | Query: explain | Error: Connection error. | Traceback: Traceback (most recent call last):
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
rag_api-1  |     yield
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 250, in handle_request
rag_api-1  |     resp = self._pool.handle_request(req)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
rag_api-1  |     raise exc from None
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
rag_api-1  |     response = connection.handle_request(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
rag_api-1  |     raise exc
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
rag_api-1  |     stream = self._connect(request)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 124, in _connect
rag_api-1  |     stream = self._network_backend.connect_tcp(**kwargs)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
rag_api-1  |     with map_exceptions(exc_map):
rag_api-1  |   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
rag_api-1  |     self.gen.throw(typ, value, traceback)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
rag_api-1  |     raise to_exc(exc) from exc
rag_api-1  | httpcore.ConnectError: [Errno -2] Name or service not known
rag_api-1  | 
rag_api-1  | The above exception was the direct cause of the following exception:
rag_api-1  | 
rag_api-1  | Traceback (most recent call last):
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 993, in _request
rag_api-1  |     response = self._client.send(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
rag_api-1  |     response = self._send_handling_auth(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
rag_api-1  |     response = self._send_handling_redirects(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
rag_api-1  |     response = self._send_single_request(request)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1014, in _send_single_request
rag_api-1  |     response = transport.handle_request(request)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 249, in handle_request
rag_api-1  |     with map_httpcore_exceptions():
rag_api-1  |   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
rag_api-1  |     self.gen.throw(typ, value, traceback)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
rag_api-1  |     raise mapped_exc(message) from exc
rag_api-1  | httpx.ConnectError: [Errno -2] Name or service not known
rag_api-1  | 
rag_api-1  | The above exception was the direct cause of the following exception:
rag_api-1  | 
rag_api-1  | Traceback (most recent call last):
rag_api-1  |   File "/app/main.py", line 232, in query_embeddings_by_file_id
rag_api-1  |     embedding = vector_store.embedding_function.embed_query(body.query)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 629, in embed_query
rag_api-1  |     return self.embed_documents([text])[0]
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 588, in embed_documents
rag_api-1  |     return self._get_len_safe_embeddings(texts, engine=engine)
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 483, in _get_len_safe_embeddings
rag_api-1  |     response = self.client.create(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/resources/embeddings.py", line 124, in create
rag_api-1  |     return self._post(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1280, in post
rag_api-1  |     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 957, in request
rag_api-1  |     return self._request(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1017, in _request
rag_api-1  |     return self._retry_request(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1095, in _retry_request
rag_api-1  |     return self._request(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1017, in _request
rag_api-1  |     return self._retry_request(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1095, in _retry_request
rag_api-1  |     return self._request(
rag_api-1  |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1027, in _request
rag_api-1  |     raise APIConnectionError(request=request) from err
rag_api-1  | openai.APIConnectionError: Connection error.
rag_api-1  | 
rag_api-1  | 2025-01-04 17:42:52,441 - root - INFO - Request POST http://rag_api:8000/query - 500
api-1      | 2025-01-04 17:42:52 error: Error creating context: Request failed with status code 500
api-1      | 2025-01-04 17:42:52 error: [handleAbortError] AI response error; aborting request: Request failed with status code 500

[Vinnie-Singleton-NN]

I'd also like to know. @danny-avila is this a capability or something that would need to be added?

[Vinnie-Singleton-NN]

@rcdailey We were able to get this working using these params in the config

EMBEDDINGS_MODEL=text-embedding-3-small
EMBEDDINGS_PROVIDER=openai
RAG_API_URL=LITELLM-BASE-ENDPOINT/v1/embeddings # e.g. https://my-litellm-endpoint.com/v1/embeddings
RAG_OPENAI_BASEURL=LITELLM-BASE-ENDPOINT
RAG_OPENAI_API_KEY=LITELLM-API-KEY