Enhancement: Disable mongodb and meilisearch sync on LibreChat startsup and expose sync functionality via a cron job

[cloud-25683]

What features would you like to see added?

Saw following issue recently, I think this is due to the amount of chat records in our librechat system, issue was fixed by increasing the heap size, however, I think it would be good to have following features:

1. expose an Env variable to disable the Sync when librechat starts up (this helps reducing unnecessary allocated cpu/memory for librechat, I noticed I can provision much smaller pod when librechat is running, but to not let it fail during startsup, I have to provision much higher cpu/memory and heap size)
2. Instead expose a one time job/script to do this periodically outside of LibreChat NodeJs server

More details

this is what I saw when librechat crashed due to heap size exceeded

<--- Last few GCs --->

[1:0x7f2b622da650]    30065 ms: Scavenge (reduce) 2041.3 (2080.0) -> 2040.7 (2080.0) MB, 9.73 / 0.00 ms  (average mu = 0.423, current mu = 0.462) task; 
[1:0x7f2b622da650]    30303 ms: Mark-Compact (reduce) 2045.2 (2083.6) -> 2044.7 (2084.3) MB, 223.02 / 0.01 ms  (average mu = 0.323, current mu = 0.225) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Which components are impacted by your request?

No response

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

This can also be mediated by some specific Meili environment variables:

    environment:
      - MEILI_HOST=http://meilisearch:7700
      - MEILI_NO_ANALYTICS=true
      - MEILI_MAX_INDEXING_MEMORY=500MB # 1/4 of my 2 GB RAM VPS
      - MEILI_MAX_INDEXING_THREADS=2
      - MEILI_EXPERIMENTAL_MAX_NUMBER_OF_BATCHED_TASKS=5

The last 2, for indexing and max number of batched tasks, I might add to the default compose files to avoid this issue

[cloud-25683]

Thank you Danny for such prompt response, regarding MEILI_MAX_INDEXING_MEMORY, if this is set too low, will meilisearch crash instead? I'm not sure how meilisearch handles large amounts of concurrent indexing requests? does it do any rate limiting and have an internal queue?

[danny-avila]

The best thing to do is to try what works for you in your environment. Those settings worked well for me after some trial and error.