Skip to content

Conversation

@ali90h
Copy link

@ali90h ali90h commented Aug 7, 2025

Fix: Respect Content-Length when Content-Encoding is present

Problem Summary

HTTPie's --download flag incorrectly reports "Incomplete download" errors when receiving responses with Content-Encoding: gzip (or other encodings), even when the download is actually complete.

Example error:

http: error: Incomplete download: size=5084527; downloaded=42846965

This occurs because HTTPie was comparing the Content-Length header (which represents compressed size) against the decompressed content size.

Root Cause

The issue violates RFC 9110 § 8.6, which states:

"The Content-Length field indicates the number of octets in the body of the message after any content codings have been applied."

When Content-Encoding: gzip is present:

  • Content-Length: 5084527 = size of compressed data
  • Downloaded content: 42,846,965 bytes = size after decompression
  • HTTPie was incorrectly comparing these mismatched values

Fix Overview

  • Modified download validation logic to respect encoded vs. decoded sizes
  • When Content-Encoding is present, validate against compressed payload size
  • Without Content-Encoding, behavior remains unchanged
  • Added proper tracking of raw bytes received vs. decompressed content

Key Changes

  1. Size Validation: Compare Content-Length against raw bytes received when content encoding is present
  2. Progress Reporting: Show progress based on encoded bytes for encoded responses
  3. Encoding Detection: Check for all standard encodings (gzip, deflate, br, compress)

Testing

  • Added comprehensive regression test with local HTTP server serving gzip content
  • Test verifies download completes without false "Incomplete download" errors
  • Validates downloaded gzip file integrity
  • Ensures non-encoded downloads continue working as before

Backward Compatibility

  • ✅ Fully backward compatible
  • ✅ No breaking changes to API or behavior
  • ✅ Existing functionality unchanged for non-encoded responses
  • ✅ Progress display and validation work correctly in both cases

Related Issues

Fixes #1642

Acknowledgments

Thanks to @kohnalex for the detailed bug report and reproduction steps, which made this fix straightforward to implement and test.

@ali90h
Copy link
Author

ali90h commented Aug 7, 2025

@httpie/maintainers This PR fixes the Content-Length validation bug (#1642) that was causing false "Incomplete download" errors with gzipped responses.

Would appreciate a review when you have time. Thanks!

@ali90h
Copy link
Author

ali90h commented Aug 11, 2025

I opened a minimal, focused PR in #NNNN that only includes the actual fix in httpie/downloads.py plus a dedicated regression test for the gzip + Content-Length case.

It supersedes #1653, which bundled some unrelated CI/docs changes. The new PR keeps the scope tight: when Content-Encoding is present, progress relies on received bytes rather than Content-Length, preventing false “Incomplete download” and >100% overshoot.

If you prefer, I’m happy to close this PR in favor of #1654. Thanks for your time!

@ali90h ali90h closed this by deleting the head repository Aug 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report: http --download misinterprets Content-Length when Content-Encoding: gzip is set

1 participant