Agent's and LLM's timeout

[humbertaco]

Lately, I have been playing around with the agent's timeout parameter. Initially, I thought that the timeout was meant for waiting for a response from the underlying LLM. However, I found out that the timeout affects the time taken to generate the whole, final answer of the agent. For example, if we set this parameter to 10 seconds, and the LLM generates a long response, the timeout will likely trigger and throw an error.

I wonder whether this is the most logical way of thinking of such timeouts in LangChain. Maybe the use case that I have in mind is not the most common one, but I find myself some times seeing how a request to OpenAI hangs on for quite some time and results eventually in an error (I guess because of an overload in their end? not sure about this).

Wouldn't it make more sense to have a timeout that triggers only when we haven't received any response at all from the underlying LLM? This way, we could have a safeguard if the request did not reach OpenAI for whatever reason and we could inform the end user accordingly.

I guess that this can be achieved programmatically using the callbacks, but I was curious to know others' opinions on this matter.