S3 with gzip compression

[Callicrate]

Hi!

Long time Cyberduck user, first time Mountduck user :D

I'm trying to access some files in S3 that have content-encoding: gzip and they're being truncated when using Mountainduck. When downloading the files via Cyberduck they're fine which led me to eventually find an old Cyberduck Issue #8263

My symptoms match but since Cyberduck was patched in 2015 I'm hoping that maybe I'm just doing something wrong with Mountainduck. Is there a configuration/setting in Mountainduck that would let me use compressed files in S3?

[tatze96]

Which connection mode (Smart Synchronization or Online) do you use?

Please send us the logging output with enabled debug logging right after the issue occurs again to [email protected] with the ticket references GH-15626 for further investigation.

[Callicrate]

I'm using the online connection mode

I spent some time reproducing the issue, here's a general idea

• validate that you have text file stored as a compressed S3 object
  • the size displayed in the AWS S3 UI will be the compressed size
  • the size after downloading the file via the AWS S3 UI will be the decompressed size
• mount the bucket with Mountainduck
• navigate to the mount point in a terminal and wc -c [filename] which will show a truncated file size.

I noticed something odd when I was testing. In one terminal I ran while true; do wc -c [filename]; sleep 1; done and in another I manually ran cat [filename] a few times. every time I would cated the file the wc output would toggle between being truncated to the size of the compressed file and double that size (which also is only about 1/4 of the actual file size)

I just sent the debug log to your support email, hopefully, you can help figure out what's going on!

[dkocher]

This relates to ¹.

Footnotes

1. Crash and hang downloading a file with Content-Encoding: gzip #15642 ↩

[dkocher]

… which led me to eventually find an old Cyberduck Issue #8263

This was reverted in #11662.