Bug: Request sometimes has wrong accept header when running with hypercorn 0.17

[dbrnz]

Description

I've reported this as hypercorn/301 but suspect Litestar could be the underlying cause, hence this issue.

We have a Litestar WSGI server running with hypercorn. This has worked just fine until Python packages were recently upgraded as part of a general update -- after this the server occasionally sends the wrong response to a request.

Our code uses content-negotiation, based on the request's Accept header, to determine the response -- comparing browser logs with a server trace, we can see the client consistently sending Accept: application/json; the server though only sees application/json for about about three out of four requests; at other times, values seen have included */*, image/webp,*/*, and image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8.

Prior to the update we were using hypercorn 0.16.0; the update took this to 0.17.3. We've seen this with all browsers so far tested -- Chrome, Firefox and Safari; the server string reported back to the browser is hypercorn-h11.

I've debugged this by freezing our code and all package versions, apart from hypercorn, using this from a git controlled working directory. Binary searching of commits has found that the accept header is correct in the Litestar request until commit d1c1a23 -- this commit added support for lifespan state. Looking at code changes, this support appears to pass state around without modifying anything.

Debugging with print statements in hypercorn, at the point where a HTTP stream is handled, finds scope['headers'] correctly includes (b'accept', b'application/json') but the headers contained in scope['state'] have 'accept': 'image/webp,*/*',.

Why does a scope's state appear to have details of a prior request? And does Litestar prioritise headers in a scope's state over those in the HTTP request?

{
    'type': 'http',
    'http_version': '1.1',
    'asgi': {'spec_version': '2.1',
    'version': '3.0'},
    'method': 'GET',
    'scheme': 'http',
    'path': '/flatmap/f95a494d-8001-578e-8d5b-491ae3968656/',
    'raw_path': b'/flatmap/f95a494d-8001-578e-8d5b-491ae3968656/',
    'query_string': b'',
    'root_path': '',
    'headers': [
        (b'host', b'localhost:8000'), 
        (b'connection', b'keep-alive'), 
        (b'pragma', b'no-cache'), 
        (b'cache-control', b'no-cache'), 
        (b'sec-ch-ua-platform', b'"macOS"'), 
        (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36'), 
        (b'accept', b'application/json'), 
        (b'sec-ch-ua', b'"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"'), 
        (b'sec-ch-ua-mobile', b'?0'), 
        (b'origin', b'http://localhost:3000'), 
        (b'sec-fetch-site', b'same-site'), 
        (b'sec-fetch-mode', b'cors'), 
        (b'sec-fetch-dest', b'empty'), 
        (b'referer', b'http://localhost:3000/'), 
        (b'accept-encoding', b'gzip, deflate, br, zstd'),
        (b'accept-language', b'en-GB,en-US;q=0.9,en;q=0.8')
    ],
    'client': ('127.0.0.1', 58709),
    'server': ('127.0.0.1', 8000),
    'state': {'_ls_connection_state': ScopeState(
        accept=<_EmptyEnum.EMPTY: 0>, 
        base_url=<_EmptyEnum.EMPTY: 0>, 
        body=<_EmptyEnum.EMPTY: 0>, 
        content_type=<_EmptyEnum.EMPTY: 0>, 
        cookies=<_EmptyEnum.EMPTY: 0>, 
        csrf_token=<_EmptyEnum.EMPTY: 0>, 
        dependency_cache=<_EmptyEnum.EMPTY: 0>, 
        do_cache=<_EmptyEnum.EMPTY: 0>, 
        exception_handlers={}, 
        form=<_EmptyEnum.EMPTY: 0>, 
        flash_messages=[], 
        headers=<Headers(
            'host': 'localhost:8000',
            'connection': 'keep-alive',
            'pragma': 'no-cache',
            'cache-control': 'no-cache',
            'sec-ch-ua-platform': '"macOS"',
            'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36',
            'accept': 'image/webp,*/*',
            'sec-ch-ua': '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
            'sec-ch-ua-mobile': '?0',
            'origin': 'http://localhost:3000',
            'sec-fetch-site': 'same-site',
            'sec-fetch-mode': 'cors',
            'sec-fetch-dest': 'empty',
            'referer': 'http://localhost:3000/',
            'accept-encoding': 'gzip, deflate, br, zstd',
            'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8'
        )>,
        is_cached=<_EmptyEnum.EMPTY: 0>, 
        json=<_EmptyEnum.EMPTY: 0>, 
        log_context={}, 
        msgpack=<_EmptyEnum.EMPTY: 0>, 
        parsed_query=<_EmptyEnum.EMPTY: 0>, 
        response_compressed=<_EmptyEnum.EMPTY: 0>, 
        response_started=True, 
        session_id=<_EmptyEnum.EMPTY: 0>, 
        url=<_EmptyEnum.EMPTY: 0>, 
        _compat_ns={})
    },
    'extensions': {}
}

URL to code causing the issue

No response

MCVE

Steps to reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Screenshots

No response

Logs

Litestar Version

2.15.2

Platform

• Linux
• Mac
• Windows
• Other (Please specify in the description above)

[dbrnz]

Also it's interesting to note that I see the following -- why is accept from state of the last request image/webp,*/* and not */*? Is it because the immediately prior content was application/octet-stream??:

      .
      .
      .
headers accept: image/webp,*/*
state headers accept: image/webp,*/*
response content type: image/png

headers accept: image/webp,*/*
state headers accept: image/webp,*/*
response content type: image/png

headers accept: */*
state headers accept: image/webp,*/*
response content type: application/octet-stream

headers accept: */*
state headers accept: image/webp,*/*
response content type: application/octet-stream

headers accept: */*
state headers accept: image/webp,*/*
response content type: application/octet-stream

headers accept: application/json
state headers accept: image/webp,*/*
response content type: image/svg+xml

[provinzkraut]

This is a known bug in hypercorn. See pgjones/hypercorn#274.

Why does a scope's state appear to have details of a prior request?

Essentially, hypercorn, instead of associating the request's state with the request, it associates it with the underlying connection, which itself can be associated with more than one request.

There's unfortunately not a lot that can be done from Litestar's side, although a while ago I did propose a fix for the bug in hypercorn.

[provinzkraut]

And does Litestar prioritise headers in a scope's state over those in the HTTP request?

To answer this: Yes it does, as it assumes the scope["state"] is per request, as defined in the ASGI spec, and caches the normalised header values there.

So what happens is that request_1<connection 1> comes in, and Litestar caches its in scope["state"]. Then request_2<connection 1> comes in, Litestar checks if cached headers are in scope["state"], finds them, and associates them with request_2. But since hypercorn has sent the same scope["state"] for request_2 as for request_1, Litestar now uses the headers for request_1 for request_2.

[provinzkraut]

For an quick fix, I'd recommend using a different server such as uvicorn or granian.

[dbrnz]

Thanks for your response and explanation -- it all makes sense (this is the first time that I've looked at the details of ASGI). I wasn't aware of granian -- it looks as if will more than do the job (and is actively maintained!), so will be switching to it.