http.client._MAXHEADERS = 100 limit no longer sufficient #131724
Description
Bug report
Bug description:
This hard-coded sanity check for HTTP response headers is no longer sufficient to fetch a Microsoft 365 page.
Line 112 in 0a91456
I do have a case open with Microsoft Support (TrackingID#2503130010002871), but it's not getting much traction as it's not causing problems with full web browsers.
Steps to reproduce:
import http.client
con = http.client.HTTPSConnection('outlook.office365.com')
con.request("GET", "/owa/example.edu") # any domain seems to trigger
r = con.getresponse()And that throws a HTTPException
>>> con = http.client.HTTPSConnection('outlook.office365.com')
>>> con.request("GET", "/owa/foo.bar")
>>> r = con.getresponse()
Traceback (most recent call last):
File "<python-input-8>", line 1, in <module>
r = con.getresponse()
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 1428, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 350, in begin
self.headers = self.msg = parse_headers(self.fp)
~~~~~~~~~~~~~^^^^^^^^^
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 248, in parse_headers
headers = _read_headers(fp)
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 226, in _read_headers
raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers
>>>
It seems to be just spilling over with 101 headers. Though, not consistently. Presumably it depends upon which load balancer node is responding.
$ curl --silent -D - 'https://outlook.office365.com/owa/example.edu' | grep -E "^[a-zA-Z-]+: " | wc -lreturns with 96, 99, 101, etc. headers, depending on Microsoft's mood unknown factors.
For background, it's common to use https://outlook.com/example.edu as a domain hint ("smart link") to go directly to a tenant's identity provider and avoid the "Please provide your email address" step. We have a nagios check for that, which broke recently as the number of Set-Cookie: OpenIdConnect.token.[...] variants continues to grow.
CPython versions tested on:
3.13, 3.9, 3.11
Operating systems tested on:
Linux, Windows