Invalid character in chunk size

[datamuc]

I have an IOT device that measures temperatures and provides an http api. Accessed with curl it looks like that:

$ curl -v 127.0.0.88:8080/Single; echo
*   Trying 127.0.0.88:8080...
* Connected to 127.0.0.88 (127.0.0.88) port 8080 (#0)
> GET /Single HTTP/1.1
> Host: 127.0.0.88:8080
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 08 Sep 2023 16:18:09 UTC
< Last-Modified: Fri, 08 Sep 2023 16:18:08 UTC
< ETag: 73696E676C65-7A78-64FB49416A1CB
< Expires: Sat, 01 Jul 2017 15:42:26 UTC
< Transfer-Encoding: chunked
< Content-Type: text/plain; charset=iso-8859-1
< Connection: keep-alive
< 
* Connection #0 to host 127.0.0.88 left intact
34,6C;22,9C;26,8C;18,0C;28,9C;----C;----C;----C

Accessing that worked with aiohttp 3.8.4 but with 3.8.5 this happens:

[...]
  Invalid character in chunk size:
                                   
    b'37 \r'                                                          
        ^                                                             
[...]
aiohttp.client_exceptions.ClientResponseError: 400, message="Invalid character in chunk size:\n\n  b'37 \\r'\n      ^", url=URL('http://127.0.0.88:8080/Single')

here is the hexdump of wireshark of the session:

00000000  47 45 54 20 2f 53 69 6e  67 6c 65 20 48 54 54 50   GET /Sin gle HTTP
00000010  2f 31 2e 31 0d 0a 48 6f  73 74 3a 20 31 32 37 2e   /1.1..Ho st: 127.
00000020  30 2e 30 2e 38 38 3a 38  30 38 30 0d 0a 41 63 63   0.0.88:8 080..Acc
00000030  65 70 74 3a 20 2a 2f 2a  0d 0a 41 63 63 65 70 74   ept: */* ..Accept
00000040  2d 45 6e 63 6f 64 69 6e  67 3a 20 67 7a 69 70 2c   -Encodin g: gzip,
00000050  20 64 65 66 6c 61 74 65  0d 0a 55 73 65 72 2d 41    deflate ..User-A
00000060  67 65 6e 74 3a 20 50 79  74 68 6f 6e 2f 33 2e 39   gent: Py thon/3.9
00000070  20 61 69 6f 68 74 74 70  2f 33 2e 38 2e 35 0d 0a    aiohttp /3.8.5..
00000080  0d 0a                                              ..
    00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d   HTTP/1.1  200 OK.
    00000010  0a 44 61 74 65 3a 20 46  72 69 2c 20 30 38 20 53   .Date: F ri, 08 S
    00000020  65 70 20 32 30 32 33 20  31 35 3a 34 39 3a 30 31   ep 2023  15:49:01
    00000030  20 55 54 43 0d 0a 4c 61  73 74 2d 4d 6f 64 69 66    UTC..La st-Modif
    00000040  69 65 64 3a 20 46 72 69  2c 20 30 38 20 53 65 70   ied: Fri , 08 Sep
    00000050  20 32 30 32 33 20 31 35  3a 34 39 3a 30 30 20 55    2023 15 :49:00 U
    00000060  54 43 0d 0a 45 54 61 67  3a 20 37 33 36 39 36 45   TC..ETag : 73696E
    00000070  36 37 36 43 36 35 2d 37  41 37 38 2d 36 34 46 42   676C65-7 A78-64FB
    00000080  34 32 36 44 44 31 41 43  46 0d 0a 45 78 70 69 72   426DD1AC F..Expir
    00000090  65 73 3a 20 53 61 74 2c  20 30 31 20 4a 75 6c 20   es: Sat,  01 Jul 
    000000A0  32 30 31 37 20 31 35 3a  34 32 3a 32 36 20 55 54   2017 15: 42:26 UT
    000000B0  43 0d 0a 54 72 61 6e 73  66 65 72 2d 45 6e 63 6f   C..Trans fer-Enco
    000000C0  64 69 6e 67 3a 20 63 68  75 6e 6b 65 64 0d 0a 43   ding: ch unked..C
    000000D0  6f 6e 74 65 6e 74 2d 54  79 70 65 3a 20 74 65 78   ontent-T ype: tex
    000000E0  74 2f 70 6c 61 69 6e 3b  20 63 68 61 72 73 65 74   t/plain;  charset
    000000F0  3d 69 73 6f 2d 38 38 35  39 2d 31 0d 0a 43 6f 6e   =iso-885 9-1..Con
    00000100  6e 65 63 74 69 6f 6e 3a  20 6b 65 65 70 2d 61 6c   nection:  keep-al
    00000110  69 76 65 0d 0a 0d 0a 33  37 20 0d 0a 33 34 2c 35   ive....3 7 ..34,5
                                          ^^
    00000120  b0 43 3b 32 32 2c 39 b0  43 3b 32 36 2c 38 b0 43   .C;22,9. C;26,8.C
    00000130  3b 31 38 2c 31 b0 43 3b  32 39 2c 31 b0 43 3b 2d   ;18,1.C; 29,1.C;-
    00000140  2d 2d 2d b0 43 3b 2d 2d  2d 2d b0 43 3b 2d 2d 2d   ---.C;-- --.C;---
    00000150  2d b0 43 0d 0a 30 0d 0a  0d 0a                     -.C..0.. ..

As you can see in 0000010A there is a space after the chunk-size. I took a look into the RFCs and indeed it looks like that there must be no space except when a chunk-ext thing is coming.

But every other client I tried so far was pretty relaxed about that space. (firefox, python-requests, curl, perls lwp-request, chrome, and aiohttp up to version 3.8.4)

Is it worth opening an issue for that?

[Dreamsorcerer]

We use llhttp by default, so an issue would need to be opened there.
Let me test out a couple of the lenient options over the weekend first though, and see if one of those allows it. e.g. maybe https://github.com/nodejs/llhttp/#void-llhttp_set_lenient_transfer_encodingllhttp_t-parser-int-enabled

[Dreamsorcerer]

Nope, none of the options seem to help. You'll need to raise a feature request with llhttp.

[Dreamsorcerer]

They've added a new option for this, so it'll be allowed in the next release (when not using dev mode).

[datamuc]

Nice.

In the meantime I will just use

async with aiohttp.ClientSession(version=aiohttp.HttpVersion10, raise_for_status=True) as session:
    ...