Set a total retry limit on streams, or enforce progress

[findleyr]

Right now, the streamable client connection retries broken resumable connections up to StreamableClientTransport.MaxRetries, but once a connection has been re-established, the counter resets.

This can lead to a failure mode where a server keeps terminating the connection with status 200. We should either:

1. Set a global limit on the number of retries across a single logical stream.
2. Only reset the retry counter when the Last-Event-ID actually makes progress.

I'm leaning toward 2: if the stream is active and long-running, a global limit could be problematic. On the other hand, (2) doesn't have that problem, and still avoids the degenerate failure mode.

CC @jba @markus-kusano

[findleyr]

@markus-kusano this one might be an interesting and smallish bug to pick up. LMK if you're interested.

[markus-kusano]

I agree with your analysis, (2) seems to be the way to go.

Here's my thoughts:

Using (1), global limit, has the issue that the limit needs to be reset occasionally for long running connections. This suggests at a high-level we need a form of short-term quota, i.e., i.e., the connection fails if it continuously fails within a short time window, and we reset the quota outside that time window. Using (2) to monitor Last-Event-ID give us that short-term quota property without introducing new variables to track it (the event ID updates if the stream makes progress, IIUC)

[findleyr]

This could be a good starter bug (also, an agent could probably knock this out pretty easily).

[jba]

Agreed, 2 is better.