Files retrieved with urllib over https are truncated

[electricworry]

Bug report

Bug description:

I am finding that some files downloaded with urllib are always truncated. I have a demonstration file which is 187527168 bytes of NULs.

If I download with wget it always is retrieved ok:

root@697bf25b6113:~# wget https://electricworry-public.s3.eu-west-1.amazonaws.com/test -O test-wget
--2025-01-24 14:41:27--  https://electricworry-public.s3.eu-west-1.amazonaws.com/test
Resolving electricworry-public.s3.eu-west-1.amazonaws.com (electricworry-public.s3.eu-west-1.amazonaws.com)... 52.218.90.80, 52.218.108.120, 3.5.72.214, ...
Connecting to electricworry-public.s3.eu-west-1.amazonaws.com (electricworry-public.s3.eu-west-1.amazonaws.com)|52.218.90.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 187527168 (179M) [binary/octet-stream]
Saving to: 'test-wget'

test-wget                                                   100%[=========================================================================================================================================>] 178.84M  5.57MB/s    in 33s     

2025-01-24 14:42:01 (5.38 MB/s) - 'test-wget' saved [187527168/187527168]

root@697bf25b6113:~# ls -l
total 183132
-rw-r--r-- 1 root root 187527168 Jan 24 14:31 test-wget

If I attempt the following python3 code I end up with a slightly truncated file:

import urllib.request
import shutil
request = urllib.request.Request("https://electricworry-public.s3.eu-west-1.amazonaws.com/test")
r = urllib.request.urlopen(request, None, 1000)
f = open("test-python", "wb")
shutil.copyfileobj(r, f)
f.close()

Here's what I end up with:

root@697bf25b6113:~# python3
Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> import shutil
>>> request = urllib.request.Request("https://electricworry-public.s3.eu-west-1.amazonaws.com/test")
>>> r = urllib.request.urlopen(request, None, 1000)
>>> f = open("test-python", "wb")
>>> shutil.copyfileobj(r, f)
>>> f.close()
>>> 
root@697bf25b6113:~# ls -l
total 363136
-rw-r--r-- 1 root root 184313073 Jan 24 14:43 test-python
-rw-r--r-- 1 root root 187527168 Jan 24 14:31 test-wget

I've tried this on several computers:

• Physical host Dell XPS 13 running Ubuntu 24.04
• Physical own-build workstation running Linux Mint 22.1 Xia
• Docker container running debian:bookworm

A wireshark packet capture seems to indicate that the remote side completes and closes the connection (FIN, PSH, ACK) which it should as urllib by default sends "Connection: close" in the headers.

Is this a known problem? The problem doesn't happen when I switch from https to http.

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Linux

[picnixz]

shutil.copyfile says:

Note that if the current file position of the fsrc object is not 0, only the contents from the current file position to the end of the file will be copied

Is the file position of the response file-like object at 0 or not? In addition, urlopen returns in this case a modified HTTPResponse object which is a BufferedIO object and has an underlying fp attribute. Could you perhaps check that this the case?

Does it also happen for files that have non-NUL bytes or is it for files that only have NULs? it might happen that it's the OS that is actually truncating the file itself, so you might also want to check that the buffer that was retrieved has the appropriate size (namely, the result size and the actual size on the disk may be different due to some optimized copyfileobj, but I don't know if this is the case).

[electricworry]

Thanks for the fast response.

Does it also happen for files that have non-NUL bytes or is it for files that only have NULs? it might happen that it's the OS that is actually truncating the file itself, so you might also want to check that the buffer that was retrieved has the appropriate size (namely, the result size and the actual size on the disk may be different due to some optimized copyfileobj, but I don't know if this is the case).

It does happen with non-NUL files. The problem manifested itself when I was trying to install my own snap packages using Ansible. First I ruled out Ansible as the problem (with this script) and then I tested a file of same length with NULs to eliminate the content as a factor.

shutil.copyfile says:

Note that if the current file position of the fsrc object is not 0, only the contents from the current file position to the end of the file will be copied

Is the file position of the response file-like object at 0 or not? In addition, urlopen returns in this case a modified HTTPResponse object which is a BufferedIO object and has an underlying fp attribute. Could you perhaps check that this the case?

I don't seem to be able to query r.tell() or r.fp.tell() - I get an UnsupportedOperation error - but I can deduce that it starts at zero because the (non-NUL) file always starts correctly and then ends with some varying amount of data omitted. This was confirmed with Beyond Compare in Hex Comparison. The file is byte-for-byte identical until the one retrieved in Python stops early. There is no other corruption before the truncation.

I'm really bewildered. My next step will be to use a proxy so I can read inside the TLS connection, but I'll have to come back to that.

[ronaldoussoren]

FWIW: The script works for me (macOS, python 3.12 using the python.org installer), giving the same result as 'curl -O' for the URL.

[vadmium]

Check the response header to see what kind of response it is. Check for Content-Length and Transfer-Encoding fields. In my case on an older Python with correct file apparently received:

>>> print(r.headers)
x-amz-id-2: eeLzjyXFqvH1+xLS401aIGUwu1aSVCgsVkDoZH5//QIGDCi54HLUpHR/TTfhWv7izKg0K5XAHDrrKd1zy6qpkSGtB4UQLqzJtTDHFLGHUvA=
x-amz-request-id: 96ZBWZVGSWAQKRGZ
Date: Mon, 27 Jan 2025 02:33:10 GMT
Last-Modified: Fri, 24 Jan 2025 14:31:32 GMT
ETag: "7fd748cb6861a39b13525537ccd3293d-11"
x-amz-server-side-encryption: AES256
x-amz-version-id: yljzBue_PWs52Ozr0ybXOAyS6YEnk419
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 187527168
Server: AmazonS3
Connection: close

I would try bypassing the higher-level shutil, urllib and http modules. Assuming the response header is close enough to above, the function below should do the request using the socket and ssl modules. You should be able to paste it directly into the interpreter and run it with “do_request()”.

def do_request(host="electricworry-public.s3.eu-west-1.amazonaws.com", port=443, target="/test", length=187527168):
    import socket, ssl
    context = ssl.create_default_context()
    with socket.create_connection((host, port)) as tcp_conn, \
            context.wrap_socket(tcp_conn, suppress_ragged_eofs=False, server_hostname=host) as ssl_conn:
        request = f'GET {target} HTTP/1.1\r\n' \
            f'Host: {host}\r\n' \
            '\r\n'
        ssl_conn.sendall(request.encode('ascii'))
        with ssl_conn.makefile('rb') as reader:
            header = reader.read(10000)
            [header, body] = header.split(b'\r\n\r\n', 1)
            print(repr(header))
            header = header.lower().decode('ascii')
            assert f'\r\ncontent-length: {length}\r\n' in header
            assert '\ntransfer-encoding:' not in header
            body += reader.read(length - len(body))
    print(len(body))
    assert len(body) == length

[electricworry]

Check the response header to see what kind of response it is. Check for Content-Length and Transfer-Encoding fields. In my case on an older Python with correct file apparently received:

Mine is the same.

I would try bypassing the higher-level shutil, urllib and http modules. Assuming the response header is close enough to above, the function below should do the request using the socket and ssl modules. You should be able to paste it directly into the interpreter and run it with “do_request()”.

Yep, that works:

b'HTTP/1.1 200 OK\r\nx-amz-id-2: C2nl/StjnxVJf96hmCb67h4i089TeBVLSmY/R/tniU0BkcdU5doqYFm/Undapm2RcsOSl26zM8k=\r\nx-amz-request-id: QKAWDT8Z287M2899\r\nDate: Thu, 30 Jan 2025 16:45:34 GMT\r\nLast-Modified: Fri, 24 Jan 2025 14:31:32 GMT\r\nETag: "7fd748cb6861a39b13525537ccd3293d-11"\r\nx-amz-server-side-encryption: AES256\r\nx-amz-version-id: yljzBue_PWs52Ozr0ybXOAyS6YEnk419\r\nAccept-Ranges: bytes\r\nContent-Type: binary/octet-stream\r\nContent-Length: 187527168\r\nServer: AmazonS3'
187527168

Looks like I'll need to look into those modules and see what's going wrong. Thanks!

[electricworry]

I had a bit of time to push the issue further. I build python from source and I modified my test script to remove shutil, but still using urllib. It is as follows:

#!./python
import urllib.request
request = urllib.request.Request("https://electricworry-public.s3.eu-west-1.amazonaws.com/test")
r = urllib.request.urlopen(request, None, 1000)
print(r.headers)
count = 0
while True:
    data = r.read()
    if len(data) == 0:
        break
    print(f"Bytes read: {len(data)}")
r.close()

When I run it I get a IncompleteRead exception:

x-amz-id-2: Fp7tMEDx3IG/AfiVJ5DxXU5ofu7s/hQC2y5tIyoYFGK8aAwy9YxtcbItC/qcgWIWrrCQkSb3r+4=
x-amz-request-id: 652CG6T6W55CV35Z
Date: Thu, 30 Jan 2025 19:14:13 GMT
Last-Modified: Fri, 24 Jan 2025 14:31:32 GMT
ETag: "7fd748cb6861a39b13525537ccd3293d-11"
x-amz-server-side-encryption: AES256
x-amz-version-id: yljzBue_PWs52Ozr0ybXOAyS6YEnk419
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 187527168
Server: AmazonS3
Connection: close


Traceback (most recent call last):
  File "/workspaces/cpython/./test.py", line 8, in <module>
    data = r.read()
  File "/workspaces/cpython/Lib/http/client.py", line 495, in read
    s = self._safe_read(self.length)
  File "/workspaces/cpython/Lib/http/client.py", line 645, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(180116929 bytes read, 7410239 more expected)

This is exactly what we'd expect if the data is being truncated. HOWEVER, if I comment out the following line in Lib/urllib/request.py then it works:

headers["Connection"] = "close"

Here's the output then:

x-amz-id-2: aSxMLWuDGSnebJEvJ/AKkpH0lmJ4zbLz5InKBRJCwHWSymrVFwd/2eTa3qqYyCzDxW/MK7u4w5A=
x-amz-request-id: E52TV5PYSQ664NNW
Date: Thu, 30 Jan 2025 19:26:31 GMT
Last-Modified: Fri, 24 Jan 2025 14:31:32 GMT
ETag: "7fd748cb6861a39b13525537ccd3293d-11"
x-amz-server-side-encryption: AES256
x-amz-version-id: yljzBue_PWs52Ozr0ybXOAyS6YEnk419
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 187527168
Server: AmazonS3


Bytes read: 187527168

[electricworry]

I'd like to crack open the TLS to see what's going on in a packet capture. Is there an easy way to set the max TLS version and get the keys like Firefox allows with SSLKEYLOGFILE?

[electricworry]

Ok, I have the module sslkeylog now and I can break open the TLS connection. I've run out of time today but I'll report back with what I discover. It's all very odd.

[electricworry]

Update: Issue happens with curl too, with a "Connection: close" header.

[root@5f43e9d4d767 cpython]# curl -H "Connection: close" https://electricworry-public.s3.eu-west-1.amazonaws.com/test -o test-curl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 96  178M   96  171M    0     0  4812k      0  0:00:38  0:00:36  0:00:02 5107k
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0

Obviously not a python issue then. Closing issue. Thanks for everyone's input.

[vadmium]

FWIW there is a long-standing report in Issue #57058 about problems caused by Python sending “Connection: close”, including triggering various problems with server responses.

The difference of the IncompleteRead exception is related to Issue #115997. Copyfileobj calls HTTPResponse.read(amt) specifying the read size, which doesn’t raise any exception despite the HTTP transport being broken. But the unlimited read() call does raise the exception.

[electricworry]

I took my laptop into work and tested there and - as all of you probably suspected - the problem is entirely fixed to my home. My hypothesis is that my ISP is doing some kind of TCP connection management where they buffer the data being sent to me, and (due to a bad implementation) when the server terminates the connection it discards the buffer and closes my stream early. That may sound implausible but my Internet access is served over 5G mobile, and it seems likely to me they would have some kind of buffering and ACKing on my behalf to improve performance. I'll test that hypothesis by further scenarios where I can get a packet capture at both ends.

Anyway, sorry for the non-Python digression.

@vadmium said:

FWIW there is a long-standing report in Issue #57058 about problems caused by Python sending “Connection: close”, including triggering various problems with server responses.

That's interesting, though probably not pertinent here; both sides are behaving to spec, as I tested this morning at work.

The difference of the IncompleteRead exception is related to Issue #115997. Copyfileobj calls HTTPResponse.read(amt) specifying the read size, which doesn’t raise any exception despite the HTTP transport being broken. But the unlimited read() call does raise the exception.

Yes, this is a problem. I don't know enough about the Python internals to know why method X raises an exception and method Y doesn't. There could be good reasons for having an throwing and a non-throwing version. But it is completely broken that a high-level HTTP library would quietly allow you to keep a truncated file transfer when the intended length is known.

What do you think needs solved?

• That urllib.request should use an exception throwing read method?
• That the non-throwing method should actually throw an exception?
• The web is fundamentally broken and we should go back to FTP?

[vadmium]

curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0

This indicates the connection was shut down at the TCP level (OS recv returned 0) against TLS protocol. In theory it could be a TLS truncation attack, but in my experience more often a buggy server. The server side is not behaving to the TLS or HTTP spec.

IMO read and related calls should raise an exception if the Content-Length cannot be met, regardless of the method parameters. There is already a comment in the code from 2013 #59838 (comment):

# Ideally, we would raise IncompleteRead if the content-length
# wasn't satisfied, but it might break compatibility.