Retry requests rejected with HTTP status 429

[mvdbeek]

Public Galaxy servers limit how many API requests a user may make in a given period of time, and reject the requests made over that limit with HTTP status 429 (Too Many Requests). BioBlend did not handle those in any way: the request simply failed with a ConnectionError.

Requests are now made through a requests session with a retrying adapter mounted, which honours the Retry-After header.

What is retried

All methods, including POST: a 429 response means that the request was rejected before being processed, so replaying it cannot duplicate anything on the server.

Two kinds of requests are deliberately not retried:

• Multipart uploads. Neither urllib3 nor requests rewind the request body between attempts (requests only does it when resolving redirects), so a replayed multipart request would send a truncated body and then block waiting on its own Content-Length. make_post_request(files_attached=True) therefore uses a non-retrying session.
• Connection and read errors, which may have reached the server. Only the status counter of the retry policy is used, so these keep raising exactly the same exceptions as before.

How retrying is bounded

By the total time spent waiting rather than by a number of attempts, so that a rate-limited call fails in a predictable amount of time instead of blocking for as long as the server asks for. urllib3 honours Retry-After but puts no upper bound on it, so both the individual waits and their total are capped:

Property	Default
max_total_retry_delay	60.0	total wait budget; 0 disables retrying
max_retry_after	30.0	cap on a single Retry-After
max_429_retries	10	backstop for when the waits are all zero

Once the budget is spent, the usual ConnectionError is raised with the 429 status code and response body, so the existing contract is unchanged. Worst case is max_total_retry_delay + max_retry_after, as the budget is checked after each wait.

Client._get() no longer retries a 429 response, since it has already been retried while honouring Retry-After; trying again would only add load to an overloaded server. This only affects users who set max_get_attempts to a value greater than 1.

Connection reuse

Since a session is created anyway, it can also be kept open to reuse connections across requests. This is opt-in through the new use_session property (plus close() and context manager support), because a shared session should not be used from multiple threads. By default a session is created and closed per request, which is exactly what requests.get() and friends did before, so there is no change for existing scripts.

Tests

bioblend/_tests/TestGalaxyRateLimit.py adds 20 tests which do not need a Galaxy instance. They run against a minimal http.server-based server rather than using an HTTP mocking library, because the retrying happens inside urllib3, below the layer at which those libraries replace HTTPAdapter.send — mocking would silently bypass the whole feature.

Note

Uploads through the tus endpoint (get_tus_uploader()) are not covered, as they are handled by tusclient, which has its own retry settings. Happy to wire that up too if wanted.

[nsoranzo]

Needs a rebase, sorry!

[nsoranzo]

I think I'd prefer to have use_session just be an boolean instance attribute, and _session_ctx() would store a newly created session only if use_session is True.
Or even, is there a use case for setting use_session to False? Should we just always store a session?

[mvdbeek]

Threads may share a session and that isn't safe. I think we'd want to explicitly delineate what goes into a session? Hence the context manager.