Fix: Respect IReconnectRetryPolicy in disconnect loops (#2853)

Fixes the case where we loop from `Connecting` -> `BeginConnectAsync` -> `OnDisconnected` and the next heartbeat triggers a reconnect again due to the backoff thresholds resetting. This ultimately causes the backoff to not be respected.

Here, we're doing 2 things:
- Upping the max exponential backoff window to 60 seconds which better handlers clients at scale and mass DDoS cases.
- Only resets the reconnect counter on the first disconnect, not doing so again until we've successfully reconnected and heartbeated again, so backoffs are properly respected.

Author: NickCraver

docs/ReleaseNotes.md

@@ -7,7 +7,8 @@ Current package versions:
 | [![StackExchange.Redis](https://img.shields.io/nuget/v/StackExchange.Redis.svg)](https://www.nuget.org/packages/StackExchange.Redis/) | [![StackExchange.Redis](https://img.shields.io/nuget/vpre/StackExchange.Redis.svg)](https://www.nuget.org/packages/StackExchange.Redis/) | [![StackExchange.Redis MyGet](https://img.shields.io/myget/stackoverflow/vpre/StackExchange.Redis.svg)](https://www.myget.org/feed/stackoverflow/package/nuget/StackExchange.Redis) |
 
 ## Unreleased
-No pending unreleased changes.
+- Fix: Respect `IReconnectRetryPolicy` timing in the case that a node that was present disconnects indefinitely ([#2853 by NickCraver](https://github.com/StackExchange/StackExchange.Redis/pull/2853))
+- Changes max default retry policy backoff to 60 seconds ([#2853 by NickCraver](https://github.com/StackExchange/StackExchange.Redis/pull/2853))
 
 ## 2.8.24
 

src/StackExchange.Redis/ExponentialRetry.cs

@@ -8,7 +8,7 @@ namespace StackExchange.Redis
     public class ExponentialRetry : IReconnectRetryPolicy
     {
         private readonly int deltaBackOffMilliseconds;
-        private readonly int maxDeltaBackOffMilliseconds = (int)TimeSpan.FromSeconds(10).TotalMilliseconds;
+        private readonly int maxDeltaBackOffMilliseconds = (int)TimeSpan.FromSeconds(60).TotalMilliseconds;
         [ThreadStatic]
         private static Random? r;
 

src/StackExchange.Redis/PhysicalBridge.cs

@@ -47,6 +47,7 @@ internal sealed class PhysicalBridge : IDisposable
         private int beating;
         private int failConnectCount = 0;
         private volatile bool isDisposed;
+        private volatile bool shouldResetConnectionRetryCount;
         private long nonPreferredEndpointCount;
 
         // private volatile int missedHeartbeats;
@@ -597,6 +598,8 @@ internal void OnHeartbeat(bool ifConnectedOnly)
                         // We need to time that out and cleanup the PhysicalConnection if needed, otherwise that reader and socket will remain open
                         // for the lifetime of the application due to being orphaned, yet still referenced by the active task doing the pipe read.
                     case (int)State.ConnectedEstablished:
+                        // Track that we should reset the count on the next disconnect, but not do so in a loop
+                        shouldResetConnectionRetryCount = true;
                         var tmp = physical;
                         if (tmp != null)
                         {
@@ -668,7 +671,14 @@ internal void OnHeartbeat(bool ifConnectedOnly)
                         }
                         break;
                     case (int)State.Disconnected:
-                        Interlocked.Exchange(ref connectTimeoutRetryCount, 0);
+                        // Only if we should reset the connection count
+                        // This should only happen after a successful reconnection, and not every time we bounce from BeginConnectAsync -> Disconnected
+                        // in a failure loop case that happens when a node goes missing forever.
+                        if (shouldResetConnectionRetryCount)
+                        {
+                            shouldResetConnectionRetryCount = false;
+                            Interlocked.Exchange(ref connectTimeoutRetryCount, 0);
+                        }
                         if (!ifConnectedOnly)
                         {
                             Multiplexer.Trace("Resurrecting " + ToString());