[Question]: Why am I getting an "Error processing file" message when trying to upload a PDF?

[fmys2010]

What is your question?

I deploy a website on a Linux system. Images can be uploaded, but once a PDF or the like is uploaded, it cannot be uploaded. "An error occurred while processing the file" will be displayed.

More Details

I followed the tutorial and I didn’t know the details.

What is the main subject of your question?

No response

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

See the docs here: https://www.librechat.ai/docs/configuration/rag_api

[fmys2010]

Can I use other services instead of opanai RAG?

[fuegovic]

Try with huggingface instead of OpenAI. I already left you the link for the huggingface guide a couple of post above.

[fmys2010]

I think what I set up doesn't work, it still can't process the file

[unitedrule]

It seems like the OPENAI RAG API only works if the "RAG_OPENAI_API_KEY=sk-your-openai-api-key-example" key is explicitly defined in the .env file, otherwise I get the same file processing error!

Would it be possible to make the API key user_provided, such as "RAG_OPENAI_API_KEY=user_provided", or make it use the user_provided "OPENAI_API_KEY".

I would like the users to provide their own keys, that is very important.

[danny-avila]

Would it be possible to make the API key user_provided, such as "RAG_OPENAI_API_KEY=user_provided", or make it use the user_provided "OPENAI_API_KEY".

Interesting I will think about how to add this

[jowin202]

I have the same problem, but I would like to upload pdfs to ollama (local installation, no openai or any other remote service)

is there a (simple) tutorial for allow uploading pdf files?

[matthewknill]

I can't get this to work, I also tried explicitly defining the RAG_OPENAI_API_KEY...

[danny-avila]

PDFs are supported, it could be a configuration or specific file issue:

I have the same problem, but I would like to upload pdfs to ollama (local installation, no openai or any other remote service)

Sorry, there is not an in-depth guide for this but there is an example docker override file for this:

https://github.com/danny-avila/rag_api/blob/main/docker-compose.override.yml.example

More on docker overrides: https://www.librechat.ai/docs/configuration/docker_override

[OPerepadia]

I have the same problem. I followed instructions from here and added both RAG_API_URL and RAG_OPENAI_API_KEY.

And from the logs, it seems RAG API is reachable, but it still fails when uploading any PDFs

LibreChat         | 2024-08-08 09:57:41 info: RAG API is running and reachable at http://host.docker.internal:8000.
LibreChat         | 2024-08-08 09:57:41 info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it
LibreChat         | 2024-08-08 09:57:48 error: Error embedding file Request failed with status code 404
LibreChat         | 2024-08-08 09:57:48 error: [/files] Error processing file: Request failed with status code 404

I also tried adding DEBUG_RAG_API=True, but it didn't give any more information.

P.S. I'm able to upload image files without issues.

[OPerepadia]

Apparently I was missing RAG_PORT and RAG_API_URL in docker-compose.yml. Added these and it started working

services:
  api:
    environment:
      - RAG_PORT=${RAG_PORT:-8000}
      - RAG_API_URL=http://rag_api:${RAG_PORT:-8000}

It seems like the message RAG API is running and reachable was misleading.

[danny-avila]

@OPerepadia sorry for the nuisance this caused you, since the message is generated from a successful connection, it needs a bit more refinement to handle this edge case

[myron-z-zhang]

I also encountered the same problem. I have tried all the methods discussed here, but they still did not work. My log file shows this:

{"level":"error","message":"Error embedding file File embedding failed.","stack":"Error: File embedding failed.\n    at uploadVectors (/app/api/server/services/Files/VectorDB/crud.js:84:13)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async processFileUpload (/app/api/server/services/Files/process.js:348:7)\n    at async /app/api/server/routes/files/files.js:157:5"}
{"level":"error","message":"[/files] Error processing file: File embedding failed.","stack":"Error: File embedding failed.\n    at uploadVectors (/app/api/server/services/Files/VectorDB/crud.js:95:11)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async processFileUpload (/app/api/server/services/Files/process.js:348:7)\n    at async /app/api/server/routes/files/files.js:157:5"}

The version is LibreChat v0.7.4. Have any good suggestions?

PS: Update: Below is the debug output.

2024-08-30T04:01:41.484Z info: RAG API is running and reachable at http://rag_api:8000.
2024-08-30T04:01:41.509Z info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it
2024-08-30T04:01:45.308Z debug: [validateJson] files
["\"Ai_PDF.json\"","\"BrowserOp.json\"","\"Dr_Thoths_Tarot.json\"","\"DreamInterpreter.json\"","\"VoxScript.json\"","\"askyourpdf.json\"","\"drink_maestro.json\"","\"earthImagesAndVisualizations.json\"","\"image_prompt_enhancer.json\"","\"qrCodes.json\"","\"scholarai.json\"","\"uberchord.json\"","\"web_search.json\""]
2024-08-30T04:02:06.428Z debug: [validateJson] files
["\"Ai_PDF.json\"","\"BrowserOp.json\"","\"Dr_Thoths_Tarot.json\"","\"DreamInterpreter.json\"","\"VoxScript.json\"","\"askyourpdf.json\"","\"drink_maestro.json\"","\"earthImagesAndVisualizations.json\"","\"image_prompt_enhancer.json\"","\"qrCodes.json\"","\"scholarai.json\"","\"uberchord.json\"","\"web_search.json\""]
2024-08-30T04:02:07.792Z error: Failed to fetch models from Mistral API
The request either timed out or was unsuccessful. Error message:
 Cannot read properties of undefined (reading... [truncated]
2024-08-30T04:02:28.989Z debug: Connection error.
{
  status: false,
  file_id: "e42ae590-1d44-4410-bc50-2013f12aae89",
  filename: "LightSpeed_Guidelines.pdf",
  known_type: true,
}
2024-08-30T04:02:28.991Z error: Error embedding file File embedding failed.
2024-08-30T04:02:28.992Z error: [/files] Error processing file: File embedding failed.

.env configuration

#==================================================#
#                        RAG                       #
#==================================================#
# More info: https://www.librechat.ai/docs/configuration/rag_api

RAG_OPENAI_BASEURL=http://host.docker.internal:8000
RAG_OPENAI_API_KEY=sk-*******0117
EMBEDDINGS_PROVIDER=openai
EMBEDDINGS_MODEL=text-embedding-3-small

[myron-z-zhang]

The problem has been solved.
The .env file uses this configuration to modify:

RAG_API_URL=http://host.docker.internal:8000
EMBEDDINGS_PROVIDER=huggingface
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxx

The RAG library in deploy-compose.yml and docker-compose.yml, and the standard library has been installed.

Local embeddings are supported by changing the image used by the default compose file, from ghcr.io/danny-avila/librechat-rag-api-dev-lite:latest to ghcr.io/danny-avila/librechat-rag-api-dev:latest.

Using OpenAI's method doesn't work, and it prompts a key error. I'm using a proxy key, and I'm not sure if this is causing the issue.

[brunovanstaveren]

@myron-z-zhang THANKYOU!!!!!

[koosoli]

for me it is not even working on the librechat demo page

[carlostinc]

I can't make this work out. My .env file is:

RAG_API_URL=http://host.docker.internal:8000

EMBEDDINGS_PROVIDER=huggingface

HF_TOKEN=[MY HF ACCESS CODE]

The docker-compose.override.yml:

version: '3.4'
 
services:
  rag_api:
    image: ghcr.io/danny-avila/librechat-rag-api-dev:latest

I did the "docker compose up -d" gave this:

time="2025-01-24T12:48:36-03:00" level=warning msg="The \"UID\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"GID\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"PORT\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"PORT\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"UID\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"GID\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"UID\" variable is not set. Defaulting to a blank string."
time="2025-01-24T12:48:36-03:00" level=warning msg="The \"GID\" variable is not set. Defaulting to a blank string."
1 error(s) decoding:

* error decoding 'ports': No port specified: :<empty>

Dont know if that's it's fine. But logs says:

2025-01-24 15:41:49 info: [deleteNullOrEmptyConversations] Deleted 0 conversations and 0 messages
2025-01-24 15:41:49 info: RAG API is running and reachable at http://rag_api:8000.
2025-01-24 15:41:49 info: No changes needed for 'USER' role permissions
2025-01-24 15:41:49 info: No changes needed for 'ADMIN' role permissions
2025-01-24 15:41:49 info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it

So i guess RAG API it's fine, but when i upload a csv or PDF files it gives "error processing file". What i'm missing here?

[jquiros2]

Also check the rag_api logs.
Make sure whatever you have set in "EMBEDDINGS_MODEL=" is accessible.
I had not allowed access in openai to text-embedding-3-small.

[marlonka]

ERROR Processing files happens if your PDF files are scanned and not machine readable even if you have configured RAG API