[ML] Improve error messages for 3rd party APIs in _inference

[maxhniebergall]

Elasticsearch Version

all

Installed Plugins

No response

Java Version

bundled

OS Version

na

Problem Description

Hitting API limits for 3rd party services causes confusing error messages such as Request timed out waiting to be sent after [30s]. We should update our error messages to be clearer, and we should take this opportunity to review the other error messages in _inference.

Steps to Reproduce

send too many API requests

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[maxhniebergall]

Possibly caused by:

1. Items in the queue are failing
2. The failing items are blocking the queue/available network connections
3. The remaining items in the queue are not able to be tried, and eventually time out
4. The failure message on the timed out items doesn't contain any information about what caused the timeout

[jonathan-buttner]

This PR: #123638 might have improved the situation.

[davidkyle]

Closed by #123638