Consider "burst" support for rate limits

[Mr0grog]

This is low priority! It’s a bit complex and the use cases where it’s valuable are few. I’ve mainly filed this issue just to get the idea written down.

We currently implement rate limits by evenly spacing out every HTTP request. However, it would be extremely useful to make paginated CDX index requests or redirect requests for Mementos more quickly when possible.

To address this, we could add some sort of “burst” capability to RateLimit objects that bypasses the rate limit as long as doing so still fits within the overall limit for some larger timeframe and adds additional delay to the next normal request to make up for it. For example:

# Make a rate limit of 1 request/s, but burstable over a 60-second window:
limit = RateLimit(per_second=1, burst_window=60)

# No wait the first time, same as today:
limit.wait()

# Sleeps for 1 second, same as today:
limit.wait()

# Does not sleep since we haven’t exceeded 1 r/s over the last 60 seconds:
limit.burst()

# Same as above, no wait:
limit.burst()

# Sleeps for 3 seconds to even out from the two bursts:
limit.wait()

# Does not sleep since we haven't exceeded 1 r/s over the last 60 seconds:
for i in range(55):
    limit.burst()

# Sleeps for 57 seconds to get back under 1 r/s average over 60 seconds:
limit.burst()  # OR `limit.wait()`; the results would be the same here

# Sleeps for 1 second to stay under 1 r/s average over 60 seconds:
limit.burst()  # OR `limit.wait()`; the results would be the same here

IA folks said “currently we are looking at requests/minute over a 5 minute period,” so the correct default window here is probably 300 seconds.

I don’t think this is super important to do right now. It could be pretty complex to implement correctly (need to consider things like some threads calling burst() while others call wait() at nearly the same time), but the value we get out of it is probably low (nicer performance if you are just making a handful of requests and not likely to hit the limits, but no added value if you are making a lot of requests or using several threads — you’re going to hit the limits and take the same time overall regardless). But I did want to get the concept written down so it doesn’t get lost.