Azure Blob Storage sink issues with compression

[danelkotev]

When using Azure Blob Storage sink with gzip compression, there are issues with Azure - when downloading the file to my local machine, it seems like the file is corrupted.
There are many issues with Azure around it from what I see online, however when removing CONTENT-ENCODING all seems OK, any change I can override the headers added by vector.dev although keep using gzip compression? any other suggestion will gladly accepted...

[jszwedko]

@danelkotev how are you downloading the file? It may be being transparently decompressed by your browser. This gotcha is noted here: https://vector.dev/docs/reference/configuration/sinks/azure_blob/#compression

[danelko]

when downloading it from azure browser to my mac it fails to decompress, claiming the file compression method is unknown, but the issue is that even in azure blob storage it has issue

[jszwedko]

Are you sure the downloaded file was compressed? The browser will transparently decompress when downloading so you may have ended up with the uncompressed file on your local filesystem. You can use the file command locally to verify the type of the file.

[danelko]

seems like the browser decompress them and leave the extension as-is.

can we think of a feature that allows us to uploaded compressed files without the headers or without the extension?

[pront]

I think you can achieve this by setting https://vector.dev/docs/reference/configuration/sinks/azure_blob/#compression to none.

[danelko]

But I do want to use compression, I do want to store uncompressed files

[pront]

I see. You probably want something like this: #21795 (comment)