100 second timeout for long waiting respond probably due to cloudflare http request

[genchmot]

What happened?

When using reasoning models like gpt o1, o3 mini. Or long response model like perplexity deep research. Users need to wait for very long. Typically around a bit more than 100 seconds.
It will return a message "error connecting to server, try refreshing the page"

I found when I use it locally it, it won't happen.
I also tried not using my cloudflare domain, using ip directly from the digital ocean server. It works as well. It's able to generate response even after couple minutes.

It seems like the issue is due to cloudflare. I did some research and found it has a 100 second time out for http response, and to increase the time out limit I need a enterprise plan which is super expensive.
I found other proxy provider has this limit as well typically. Most not going to be long enough for long reasoning or deep research.

Version Information

Using via Docker on Ubuntu 22.04 remote hosting on digital ocean

ghcr.io/danny-avila/librechat-dev-api latest cc3a32f30b70 23 hours ago
ghcr.io/danny-avila/librechat-dev latest 81d75551748c 4 months ago
ghcr.io/danny-avila/librechat-rag-api-dev-lite latest 1acf95644082 4 months ago

Steps to Reproduce

step 1: use a proxy service like cloudflare. use librechat through domain

step 2: use a model that requires long waiting time to generate response. perplexity new deep research api seems to be easiest to reproduce. gpt o1, o3 on hard task can as well.

step 3: wait more than 100 second, the timeout error appears

What browsers are you seeing the problem on?

No response

Relevant log output

[OpenAIClient.chatCompletion][finalChatCompletion] Aborted Message
[OpenAIClient.chatCompletion][finalChatCompletion] API error

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[genchmot]

maybe can add a ping/keepalive mechanism to prevent the timeout?

[danny-avila]

It seems like the issue is due to cloudflare. I did some research and found it has a 100 second time out for http response, and to increase the time out limit I need a enterprise plan which is super expensive.
I found other proxy provider has this limit as well typically. Most not going to be long enough for long reasoning or deep research.
maybe can add a ping/keepalive mechanism to prevent the timeout?

If Cloudflare is severing the connection at the network level, there's unfortunately no client-side code that can prevent this

[genchmot]

is it possible to make a server side rather than client side in LibreChat's API server code to have a keep alive mechanism to maintain long connections through Cloudflare?

[danny-avila]

something like this might be better, especially for longer reasoning tasks:

#5968 (comment)

[genchmot]

yes that should solve this issue.
before this is implemented, the best solution now is switching to another proxy like Traefik instead of cloudflare?

[KiGamji]

something like this might be better, especially for longer reasoning tasks:

#5968 (comment)

keepalive "heartbeat" system as comments (starting with ";") in chunked data should do the trick, but it needs to be properly tested before ensuring that it would work as intended. I have seen this workaround in some proxies