Programmatic Data Ingestion for RAG API

[MisterHarrow]

Goal

I am trying to programmatically upload documents for ingestion into the RAG system using a Python script. The objective is to automate the batch processing of a large number of local files, bypassing the need for manual UI uploads.

Core Problem

Automated, script-based file ingestion appears to be broken for me, regardless of the method used. I have identified two primary paths for uploading files, and both fail in different ways:

1. Via the LibreChat Application (/api/files): When my script sends files to the main LibreChat application, it receives a 200 OK success status for every file. However, the request is never processed further. The files do not appear in the vector database, and there is zero log activity in either my librechat pod or my librechat-rag-api pod, even with debug logging enabled. It's a "silent failure" where the request effectively disappears.

2. Directly to the RAG API (/embed-upload): When my script sends files directly to the RAG API (bypassing the main LibreChat app), the request fails with a 500 Internal Server Error. My RAG API logs show a RuntimeError: Stream consumed, which points to a request validation error happening before the application's business logic is executed.

My Environment

• Deployment: Kubernetes (GKE) cluster
• Components: LibreChat, librechat-rag-api, a KServe embedding model, and a CNPG-managed PostgreSQL/pgvector database.
• Authentication: My script uses a valid JWT Bearer token copied from an active browser session.

Debugging Steps & Findings

I have exhaustively debugged my infrastructure and have confirmed the issue appears to lie within the applications themselves.

---

Path 1: The "Silent Failure" via LibreChat (/api/files)

• Confirmed 200 OK: My script successfully uploads files to the librechat pod and receives a success code.
• Enabled Debug Logging: I edited the librechat-librechat-config ConfigMap to set logLevel: debug. This did not produce any new logs related to file uploads.
• Ruled out Sidecars: kubectl describe pod confirmed my librechat pod has only one container.
• Confirmed Internal Networking: I used kubectl exec to get a shell inside the librechat pod and successfully used wget and nc to connect to the librechat-rag-api service on its internal cluster URL. This proves the two pods can communicate with each other.

My Conclusion for Path 1: The librechat application is accepting the HTTP request but has an internal bug that prevents it from forwarding the file to the RAG API for processing. This failure is completely silent.

---

Path 2: The "Validation Error" via RAG API (/embed-upload)

• Confirmed 500 Internal Server Error: My script fails with a server error, and the logs show a RuntimeError: Stream consumed.
• Inspected Source Code: I found the function signature in the rag-api source: async def embed_upload(files: List[UploadFile], user_id: str = Form(...)).
• Attempted Correct Payload: I built a Python script (attached) using requests that sends a multipart/form-data payload matching this signature exactly (a list of files under the key files and the user ID as a simple form field).
• Failure Persists: Despite matching the function signature, the request still results in the same validation error.

My Conclusion for Path 2: The RAG API's /embed-upload endpoint has a bug/intended feature that prevents it from correctly validating multipart/form-data requests coming from a non-browser client like a Python script.

Given these findings, could you please clarify:

• Is this behavior intended, and is programmatic ingestion via a script simply not a supported feature?
• Is there a different, recommended approach for batch-uploading documents that I should be using?
• Or are these indeed bugs in the respective API endpoints?

Version Information

docker images | grep librechat

ghcr.io/danny-avila/librechat latest 6966cd37fc2d 4 weeks ago 1.01GB

Steps to Reproduce

Steps to Reproduce

Attempt 1: Upload via LibreChat (/api/files)

1. Ensure a port-forward is active for the main LibreChat service (e.g., kubectl port-forward svc/librechat-librechat 8080:3080).
2. Use the Python script below to upload files from a local directory (e.g., ./data-samples).
3. Update the AUTH_TOKEN variable with a fresh token.
4. Run the script.

ingest_through_librechat.py

import os
import requests
import logging
import sys
import mimetypes

# --- Setup ---
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
# --- Config ---
AUTH_TOKEN = "PASTE_YOUR_FRESH_AUTH_TOKEN_HERE"
LIBRECHAT_BASE_URL = "http://localhost:8080"
DATA_SAMPLES_DIR = "./data-samples"

def get_all_files(directory):
    return [os.path.join(directory, f) for f in os.listdir(directory) if os.path.isfile(os.path.join(directory, f))]

def upload_file(file_path, base_url, headers):
    endpoint = f"{base_url}/api/files"
    logging.info(f"Uploading {os.path.basename(file_path)}...")
    mime_type, _ = mimetypes.guess_type(file_path)
    if mime_type is None:
        mime_type = "application/octet-stream"
    try:
        with open(file_path, "rb") as f:
            files = {"file": (os.path.basename(file_path), f, mime_type)}
            data = {"endpoint": "rag_api"}
            response = requests.post(endpoint, headers=headers, files=files, data=data, timeout=300)
            response.raise_for_status()
            logging.info(f"-> Success for {os.path.basename(file_path)}. Status: {response.status_code}")
    except requests.exceptions.RequestException as e:
        logging.error(f"-> Error for {os.path.basename(file_path)}: {e}")

# --- Main ---
headers = {"Authorization": f"Bearer {AUTH_TOKEN}"}
for file_path in get_all_files(DATA_SAMPLES_DIR):
    upload_file(file_path, LIBRECHAT_BASE_URL, headers)
logging.info("Done.")

• Actual Result: The script logs 200 OK success for every file, but no logs appear in the librechat or rag-api pods, and no data is ingested.
• Expected Result: Files are successfully processed and ingested by the RAG API.

---

Attempt 2: Upload Directly to RAG API (/embed-upload)

1. Ensure a port-forward is active for the RAG API service (e.g., kubectl port-forward svc/librechat-rag-api-librechat-rag-api 8001:8000).
2. Use the Python script below.
3. Update the AUTH_TOKEN and USER_ID variables.
4. Run the script.

direct_ingest.py

import os
import requests
import logging

# --- Setup ---
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
# --- Config ---
AUTH_TOKEN = "PASTE_YOUR_FRESH_AUTH_TOKEN_HERE"
USER_ID = "PASTE_YOUR_USER_ID_FROM_THE_TOKEN_HERE"
RAG_API_BASE_URL = "http://localhost:8001"
DATA_SAMPLES_DIR = "./data-samples"

def get_all_files(directory):
    return [os.path.join(directory, f) for f in os.listdir(directory) if os.path.isfile(os.path.join(directory, f))]

def upload_files(file_paths, base_url, headers, user_id):
    files_to_upload = []
    opened_files = []
    for file_path in file_paths:
        f = open(file_path, "rb")
        opened_files.append(f)
        files_to_upload.append(("files", (os.path.basename(file_path), f)))
    try:
        data = {"user_id": user_id}
        logging.info(f"Uploading {len(files_to_upload)} files...")
        response = requests.post(f"{base_url}/embed-upload", headers=headers, files=files_to_upload, data=data, timeout=300)
        response.raise_for_status()
        logging.info(f"-> Success: {response.json()}")
    except requests.exceptions.RequestException as e:
        logging.error(f"-> Error: {e}")
        if e.response:
            logging.error(f"   Response Body: {e.response.text}")
    finally:
        for f in opened_files:
            f.close()

# --- Main ---
headers = {"Authorization": f"Bearer {AUTH_TOKEN}"}
all_files = get_all_files(DATA_SAMPLES_DIR)
if all_files:
    upload_files(all_files, RAG_API_BASE_URL, headers, USER_ID)
logging.info("Done.")

• Actual Result: The script fails with a 500 Internal Server Error, and the rag-api pod logs a RuntimeError: Stream consumed.
• Expected Result: Files are successfully ingested.

---

Questions

Given these findings, could you please clarify:

• Is this behavior intended, and is programmatic ingestion via a script simply not a supported feature?
• Is there a different, recommended approach for batch-uploading documents that I should be using?
• Or are these indeed bugs in the respective API endpoints?

Any guidance you could provide would be greatly appreciated. Thank you!

What browsers are you seeing the problem on?

No response

Relevant log output

**Server-Side Librechat pod log**
# The last log message seen. Nothing new appears during the upload.
    info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it.

**Server-Side librechat-rag-api pod error log**
ERROR: Exception in ASGI application
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
        result = await app(  # type: ignore[func-returns-value]
      File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
        return await self.app(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
        await super().__call__(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
        await self.middleware_stack(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
        raise exc
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
        await self.app(scope, receive, _send)
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 173, in __call__
        with recv_stream, send_stream, collapse_excgroups():
      File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/usr/local/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
        raise exc
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 175, in __call__
        response = await self.dispatch_func(request, call_next)
      File "/app/app/middleware.py", line 21, in security_middleware
        return await next_middleware_call()
      File "/app/app/middleware.py", line 13, in next_middleware_call
        return await call_next(request)
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 153, in call_next
        raise app_exc
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 140, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
        raise exc
      File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
        await app(scope, receive, sender)
      File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 714, in __call__
        await self.middleware_stack(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 734, in app
        await route.handle(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
        await self.app(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
      File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 59, in wrapped_app
        response = await handler(conn, exc)
      File "/app/main.py", line 52, in validation_exception_handler
        body = await request.body()
      File "/usr/local/lib/python3.10/site-packages/starlette/requests.py", line 241, in body
        async for chunk in self.stream():
      File "/usr/local/lib/python3.10/site-packages/starlette/requests.py", line 224, in stream
        raise RuntimeError("Stream consumed")
    RuntimeError: Stream consumed

**Client-Side Script Output**
INFO: Uploading 6 files...
    ERROR: -> Error: 500 Server Error: Internal Server Error for url: http://localhost:8001/embed-upload
    ERROR:    Response Body: {"detail":"Internal Server Error"}

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Moved to Discussions since the RAG API is not tested or developed to be used like this outside of its LibreChat integration.

You are better off inspecting the RAG API codebase to understand it better, and possibly contribute something that would make this work more effectively: https://github.com/danny-avila/rag_api

In your .env, you should also set DEBUG_RAG_API=True for more details when using the RAG API, in general.

[pedrotinoco]

I wanted to follow up and ask if there have been any updates on this issue. I'm encountering the same scenario and would appreciate any insights.