Authorization Renewal Jitter (#310)

If a lot of WebSocket connections were severed at the same time, they'll all
reconnect and reauthorize around the same time. Although clients may (and
should!) implement some jitter on their side, it won't be very large because
commonly real-time features need to be available and working.

The problem is, once that first domino of a thundering herd falls, subsequent
dominoes keep falling, precisely in `authorization_renewal_period` intervals.

This PR allows one to add (actually remove) some jitter (in seconds), which
will spread the authorizations around. Those are typically less time-sensitive
than that first reconnection & reauthorization, so they can use a larger jitter.

Author: wojcikstefan

README.rst

@@ -428,6 +428,14 @@ e.g.:
     "error": "Unauthorized."
   }
 
+
+To avoid a thundering herd problem where a lot of authorizations happened at
+the same time *and they continue renewing the authorization at the same time,
+too*, add ``authorization_renewal_jitter`` as a number of seconds. This will
+schedule the upcoming authorization renewal for
+``authorization_renewal_period`` seconds from now *minus* anywhere between 0
+and ``authorization_renewal_jitter`` seconds.
+
 Before subscribe
 ~~~~~~~~~~~~~~~~
 

example_config.py

@@ -42,6 +42,10 @@
         # Fields returned by the authorizer callback that are passed to all
         # subsequent service callbacks.
         'authorizer_fields': ['capabilities'],
+        # Periodically reauthorize subscriptions and add some jitter to avoid
+        # thundering herds.
+        'authorization_renewal_period': 300,
+        'authorization_renewal_jitter': 30,
         # If this service requires extra fields to fulfill a subscription,
         # you may provide them here. They are passed to all URL callbacks.
         'extra_fields': ['organization_id'],

socketshark/subscription.py

@@ -152,13 +152,16 @@ async def authorize_subscription(self):
 
     async def periodic_authorizer(self):
         period = self.service_config['authorization_renewal_period']
+        jitter = self.service_config.get('authorization_renewal_jitter', 0)
         self.session.trace_log.debug(
             'initializing periodic authorizer',
             subscription=self.name,
             period=period,
+            jitter=jitter,
         )
         while True:
-            await asyncio.sleep(period)
+            sleep_duration = period - (random.random() * jitter)
+            await asyncio.sleep(sleep_duration)
             try:
                 self.session.log.debug(
                     'verifying authorization', subscription=self.name