Allow retries on services to fetch the host dynamically

[daneah]

Is your feature request related to a problem? Please describe.

Services that work through service discovery mechanisms like Netflix's Eureka have multiple hosts. When a consumer calls that service, it determines which host to call by asking the service discovery mechanism for it. Sometimes a host becomes unavailable before being removed from service discovery, and calls to it will start receiving connection timeouts. In this event, the consumer should use one of the other hosts on the next try.

Describe the solution you'd like

When a call fails with a connection timeout and the retry specification still allows for more retries, choose another host from the service's available hosts instead of retrying the same one. Explicitly choosing a host that wasn't already used is ideal, but randomly choosing a new host from all available hosts is also acceptable recognizing that the same failing host may be tried multiple times in the worst case.

Describe alternatives you've considered

N/A

Additional context

We're currently relying on the elegance of urllib3's Retry class (here) to handle all retry logic for us instead of managing the complexity of our own retrying logic and looping. The Retry machinery is hooked up to the request life cycle right at the start (here), and therefore doesn't seem to offer any seam at which we can dynamically generate a new host for the next request when a retry is necessary. This means we may need to unroll all this into our own retry logic.