Tracking download progress

[robpats]

How to track the download progress of (gzip/deflate/br/zstd-)compressed data similar to httpx.Response.num_bytes_downloaded?
Some searches show aiohttp.ClientResponse.content.total_bytes does that, but it is not documented, and from my testing it is the size of decompressed data.

[Dreamsorcerer]

Problem is that .content is reading the stream data, this has already been parsed (and therefore decompressed). With the structure of the code, I'm not sure there's an obvious way to extract the compressed bytes.

[robpats]

Those attributes don't exist in the latest version and on the master branch. Do those exist in some development/debug builds?

I have another way to achieve this.
aiohttp.http_parser.DeflateBuffer holds a reference to aiohttp.streams.StreamReader and feeds the decompressed data to it.
DeflateBuffer can be modified to update an attribute in StreamReader to track the compressed download progress like robpats/aiohttp@30c1691
However this only works when decompression takes place. Non-compressed data and auto_decompress=False wouldn't trigger the update.

[Dreamsorcerer]

I was just guessing from the code. I was trying to get you to this attribute in DeflateBuffer:

aiohttp/aiohttp/http_parser.py

Line 938 in acfc2e6

self.size = 0

However this only works when decompression takes place. Non-compressed data and auto_decompress=False wouldn't trigger the update.

Yes, I guess that's not ideal if we want to expose this somehow.

[robpats]

Maybe just adding another helper property?
robpats/aiohttp@617832b

    @property
    def total_raw_bytes(self) -> int:
        if self.total_compressed_bytes is None:
            return self.total_bytes
        return self.total_compressed_bytes

This would fallback to StreamReader.total_bytes when data has not been decompressed.

[Dreamsorcerer]

That looks plausible. If you can create a PR with a couple of tests demonstrating it works as expected, I think that could probably go in.

[robpats]

I created PR #11483. Please take a look.