dependency incompatibility: requests (latest:2.30.0), urllib3>=2.0.0

[jayaddison]

Describe the bug

Currently the requests library, as of version 2.30.0 (self-correction: this should have been 'as of version 2.29.0'; I had not read some of the updates in the linked thread), does not support urllib3 versions greater than 2.0.0 - including the most recent urllib3 releases on PyPi.

Sphinx doesn't have any explicit dependencies on urllib3, so it's pulled in by default during pip install and uses the latest-available-version.

How to Reproduce

The test failures (request timeouts) in #11392 appear to be related to this incompatibility.

Placing an upper-bound constraint on the urllib3 version to avoid 2.x package installs (for example, urllib3<2.0.0) resolves the failures; I'll push a demonstration commit to the pull request branch to illustrate that.

Environment Information

Platform:              linux; (Linux-6.1.0-9-amd64-x86_64-with-glibc2.36)
Python version:        3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0])
Python implementation: CPython
Sphinx version:        7.0.1+/824202e99
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.15.1

Sphinx extensions

N/A

Additional context

No response

[AA-Turner]

From what I read on the linked issue, .29 did not support urllib3 version 2, but .30 now does, as the upper-bound was raised to <3.

Is your understanding different?

A

[jayaddison]

Thanks - you read correctly, and I missed that detail.

In practice I find that requests v2.30.0 in combination with urllib3 v2.0.2 produces the unit test failures encountered in #11392 (I can replicate this locally).

I had misattributed that to the compatibility mismatch - but in fact it could be a regression in urllib3 (I've found the relevant urllib3 commit, and will ask about the change-in-behaviour upstream).

[jayaddison]

I've got a near-minimal repro case, but I'm beginning to wonder whether this change could in fact mean that there's a bug / unusual behaviour in the sphinx codebase, and not a regression in the urllib3/requests combination. I'll spend some more time investigating.

[jayaddison]

Repro case:

from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread

import requests


class TestHandler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self):
        if self.path == "/":
            self.send_response(200, "OK")
            self.send_header("Content-Length", 0)
            self.end_headers()
        else:
            self.send_response(404, "Not Found")
            self.send_header("Content-Length", "0")
            self.end_headers()


class TestServer(Thread):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.server = HTTPServer(("localhost", 8080), TestHandler)

    def run(self):
        self.server.serve_forever()

    def join(self):
        self.server.shutdown()
        self.server.server_close()
        super().join()


@contextmanager
def server_context():
    server_thread = TestServer()
    server_thread.start()
    yield server_thread
    server_thread.join()


def test():
    with server_context():
        response = requests.get('http://localhost:8080/start')
        requests.get('http://localhost:8080/', timeout=0.05)

(passes with requests==2.30.0 and urllib3==1.26.15 ; fails with requests==2.30.0 and urllib3==2.0.2)

The relevant urllib3 commit is: a80c248c34a77e106551bf997e726457b03f42b5

(it could be that it fixes a genuine threading bug that exposes a problem in sphinx)

Edit: further simplified the repro case
Edit: further simplifications
Edit: further simplifications

[jayaddison]

Ok: I think that this is a result of the use of non-threaded HTTP webservers during unit testing; not a problem within the sphinx codebase, and not a requests/urllib3 compatibility issue. Apologies for the noise.

(updating the repro case provided previously to swap-out the single-threaded HTTPServer with ThreadingHTTPServer allows the tests to pass when using urllib3 2.0.2)