unexpected OOM.

[tip-dteller]

What happened?

We have been using Librechat for quite a while, we have around 100 users on a daily basis.
For an unknown reason at around 2AM, We saw that librechat is aborting requests to OpenAI and subsequently crashed.
during that hour, there were made only 2 requests to Librechat, they lasted for 1.5 seconds and a full minute after that, librechat started crashing
Note: we are aware redis is experimental, but we're on K8s and we must provide a redundancy for our users, so w can simply keep 1 pod alive.
we are running with 3 pods.

I found these logs:

log:(node:25) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 21 error listeners added to [KeyvRedis]. MaxListeners is 20. Use emitter.setMaxListeners() to increase limit

until at some point we saw OOM errors:

log:FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
log:[24:0x7fd2ba17a650] 56204 ms: Mark-Compact 949.6 (1023.9) -> 942.9 (1024.4) MB, 872.14 / 0.01 ms (average mu = 0.143, current mu = 0.068) allocation failure; scavenge might not succeed
log:[25:0x7f5b2932f650] 43519 ms: Mark-Compact 954.0 (1024.4) -> 947.2 (1027.6) MB, 954.29 / 0.01 ms (average mu = 0.161, current mu = 0.178) allocation failure; scavenge might not succeed

I have 3 replicas running with each 6GB and by the looks of it, it could grow.

As there are no monitoring metrics apart from what i can get natively in k8s and no emitted traces, this proves very hard to debug any further.

Version Information

0.7.7-rc

Steps to Reproduce

havent found a way to reproduce.

What browsers are you seeing the problem on?

No response

Relevant log output

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

You can set those listeners for Redis to 0 if that helps.

https://www.librechat.ai/docs/configuration/dotenv#redis

although I don’t think that would be cause for a memory issue, since it’s just a potential warning.

check to see what those user messages were too.

[d-teller]

Nothing that appears to be heavy.
Since there were only 2 requests.
One for a comparison between 2 short texts.
One for analyzing a small pdf.

[d-teller]

@danny-avila this keeps happening quite a lot now. I believe its something at Node level because it's not a k8s oom event, its application

[danny-avila]

I run several instances 24/7 and never have this issue, I also cannot reproduce it. I think the issue may be that there is a low default setting for the JavaScript heap, especially considering the logs that were shared, showing the heap limit to be at 1 GB (so not fully using their 6 GB per node).

You could try adding this in the .env file

NODE_OPTIONS="--max-old-space-size=4096" # or adjust according to reasonable space proportionate to your resources

[peeeteeer]

where are you hosting?

[d-teller]

AWS.
Using elasticache and librechat on EKS.

[peeeteeer]

hmmm - ok, had something similar last night - but I'm on Azure... not sure what it was in the end...

[pedrojreis]

Similar issue here :)

For some context We have 5 pods with --max-old-space-size=6144 and 8GB memory per pod.
One of our pods went from 500Mb to 6.4Gb memory used a timespan of 2minutes, died with similar log as OP.

We have our REDIS_MAX_LISTENERS defined as 0.
We also disabled Meilisearch to reduce the load in the pods.

Our suspition comes from the RAG API that also had a memory spike around the same time and also went OOM.

[maxesse]

This has started happening to me too now, on azure container apps. The app just keeps on dying as soon as it starts running out of memory. Not sure what to do tbh? have you found the root cause?

[maxesse]

Ok for now I fixed it by removing the limit on redis listeners, increasing the replica memory and more aggressive autoscaling. Traffic had gone up from 30-40 concurrent users to 150, so it wasn't staying up. Hope it lasts...

[danny-avila]

Can you clarify what you did to "remove" the redis listener limit? Also are you on latest?

[maxesse]

I set REDIS_MAX_LISTENERS to 0 as you suggested further up in the thread, the docs say that it effectively removes the limit? I'm on a build from a couple of weeks ago in prod, will push a new build as soon as I deploy code interpreter.

[danny-avila]

Thanks, interesting how that helped, may need to update some dependencies involved in the cache handling.

[proschowsky]

We have experienced similar behavior. Librechat has been running consistently for months with around 500 daily users and 5000 messages. We only use one instance and memory consumption has been stable between 500-600 MB. In February we updated to latest code base and deployed agents by default for all users. Now we see a steep increase in memory usage and frequent crashes when we reach the heap limit. We are currently working on enabling Redis, adding more heap and more instances to stabilize it.

[danny-avila]

Images still had issues from blob storage here, but resolved this now: #6852

[proschowsky]

I did a few test, but get mixed results. Will look more into it tomorrow

[danny-avila]

You shouldn't see conversations being "duplicated", if you are let me know how you are filtering the heap. I think there's still room for improvement, but that was a major area addressed that would lead to quick filling of memory especially with image data in payloads.

Also found a memory leak in official openai SDK causing AbortControllers to persist, which can accumulate in memory

https://github.com/openai/openai-node/blob/e9ca1a07691976f41492e3652e1cccea33a9b70b/src/core.ts#L583

[proschowsky]

Latest commit (64bd373) works well, and I am no longer able to trigger the memory leak in our code base.

Thanks for addressing this issue and for providing such an excellent solution

[danny-avila]

Great to hear! Yes, I fixed the issue in the openai library and a persisting signal from KeyvMongo, I need to submit a PR there :)

[Odrec]

@danny-avila

We have the same out of heap memory error. I don't know if it still helps but this is our docker compose file. We're using dev-latest right now.

services:
  api:
    image: ghcr.io/danny-avila/librechat:{{librechat_version}}
    container_name: LibreChat
    ports:
      - "${PORT}:${PORT}"
    depends_on:
      - mongodb
      - rag_api
    restart: always
    user: "${UID}:${GID}"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - HOST=0.0.0.0
      - MONGO_URI=mongodb://mongodb:27017/LibreChat
      - MEILI_HOST=http://meilisearch:7700
      - RAG_PORT=${RAG_PORT:-8000}
      - RAG_API_URL=http://rag_api:${RAG_PORT:-8000}
      - NODE_OPTIONS="--max-old-space-size=8192"
    volumes:
      - type: bind
        source: ./.env
        target: /app/.env
      - ./images:/app/client/public/images
      - ./uploads:/app/uploads
      - ./logs:/app/api/logs
      - /opt/librechat/librechat.yml:/app/librechat.yaml
    networks:
      - librechat
  mongodb:
    container_name: chat-mongodb
    image: mongo
    restart: always
    user: "${UID}:${GID}"
    volumes:
      - ./data-node:/data/db
      - /opt/librechat/set-balance.js:/scripts/set-balance.js
    command: mongod --noauth
    networks:
      - librechat
  meilisearch:
    container_name: chat-meilisearch
    image: getmeili/meilisearch:{{meilisearch_version}}
    restart: always
    user: "${UID}:${GID}"
    environment:
      - MEILI_HOST=http://meilisearch:7700
      - MEILI_NO_ANALYTICS=true
      - MEILI_MAX_INDEXING_THREADS=8
    volumes:
      - ./{{ meilisearch_data_dir }}:/meili_data
    networks:
      - librechat
  vectordb:
    image: ankane/pgvector:latest
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
    restart: always
    volumes:
      - pgdata2:/var/lib/postgresql/data
    networks:
      - librechat
  rag_api:
    image: ghcr.io/danny-avila/librechat-rag-api-dev:latest
    environment:
      - DB_HOST=vectordb
      - RAG_PORT=${RAG_PORT:-8000}
    restart: always
    depends_on:
      - vectordb
    env_file:
      - .env
    networks:
      - librechat
  metrics:
    image: ghcr.io/virtuos/librechat_exporter:{{exporter_version}}
    networks:
      - librechat
    depends_on:
      - mongodb
    restart: always
  # caddy is used as a reverse proxy
  caddy:
    image: docker.io/library/caddy:{{caddy_version}}
    container_name: caddy
    restart: always
    volumes:
      - /opt/librechat/Caddyfile:/etc/caddy/Caddyfile  # Caddy configuration
      - /opt/librechat/static:/var/www/html  # Static files (e.g. datenschutz.html, impressum.html)
      - /opt/librechat/assets:/var/www/html/assets  # Images (e.g. favicon-16x16.png, logo.svg)
      - caddy_data:/data  # Caddy data (e.g. certificates)
      - caddy_config:/config
    ports:
        - "80:80"     # HTTP
        - "443:443"   # HTTPS
    networks:
      - librechat
  node_exporter:
    image: docker.io/prom/node-exporter:v1.8.2
    container_name: node_exporter
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.ignored-mount-points="^(/rootfs|/host|)/(sys|proc|dev|host|etc)($$|/)"'
      - '--collector.filesystem.ignored-fs-types="^(sys|proc|auto|cgroup|devpts|ns|au|fuse\.lxc|mqueue)(fs|)$$"'
    networks:
      - librechat

volumes:
  pgdata2:
  caddy_data:
  caddy_config:

networks:
  librechat:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 192.168.201.0/24

[danny-avila]

I'm working on a fix: #6170 (reply in thread)