[Feature request] Retry Strategy - limited time/attempts

[lanwen]

Hello,

we found that it might be useful to escape the indefinite retry with some configurable strategy, to be able to decide on the spring boot level or even on the superior level what to do next.

For instance, if the configuration changes, affecting channel - there might be no way to automatically recover from currently loaded obsolete configuration - that always requires manual intervention.

As alternative, some way to expose the channel health events/state might be useful.

Would you have any suggestions how to tackle any of the above? Thank you for any pointers

[bsideup]

Thank you @lanwen for the report! After playing with it a bit, I have the following ideas:

1. Simple: just add another IS_TRANSIENT Predicate<Throwable> option, to test the exception before retrying it. Small simple change that gives "just enough" flexibility
2. Forward looking: extract RetryStrategy interface and allow providing a custom implementation. This way, you could perform the test in it, and even throw an exception to propagate to gRPC's internal error handling that might potentially result in the termination of the whole channel (good if you want to fail fast). Just slightly bigger change but gives you full control of the retries and allows for exotic retry scenarios (including waiting for an external event to happen before retrying).

Either one isn't hard to implement but would be curious to hear your preference before I implement it.

[lanwen]

Thank you so much for looking into it!

I experimented myself with retry strategy and that looks better from the library reuse point of view - all the logic to decide on retry stays explicit and can be tampered with on the service level.

I also can see how full blown strategy can keep the state to implement timeout based retry alongside ignoring retry for some types of errors (like authentication errors).

However, as far as I understand, retry handler in the current form can't have a state, same as other parts of the handler until it's not instantiated per pipeline. But I didn't get exactly where would be the best place for that.

One more thing I struggled with was the proper place where we can react on closed channel - could you help with guidance there as well?

[bsideup]

@lanwen this is helpful, thanks! I have a local experiment with stateful retry strategies, will submit a draft PR soon for your feedback 👍

[bsideup]

Okay, I went down the rabbit hole and found the following:

1. gRPC-java Server abstraction does not propagate underlying server errors after starting
2. awaitTermination would only throw if it was interrupted.
3. Technically, I can make .start() throw an error if it failed to establish the first call

Given all of this, I wonder how much value would it add (other than not spamming with warnings, but it can also be a good trigger to look at, e.g. when "5 minutes of reconnect warning" alert triggers) to interrupt the retries given that the server will continue running anyways 😅