Please add support for compression (gzip) for the out_azure_blob plugin

[luisfeliz79]

Is your feature request related to a problem? Please describe.
Currently, the out_azure_blob plugin does not support gzip compression when ingesting records.

Describe the solution you'd like
When I'm ingesting records to out_azure_blob, I would like the resulting blobs to be gzip compressed

Describe alternatives you've considered
None

Additional context

Looking to reduce the size of the blobs ingested into the Azure Blob storage service.

For example, there should be a compression option in the out_azure_blob plugin that would gzip compress records and ingest as a gzip compressed blob.

[OUTPUT]
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
shared_key YOUR_SHARED_KEY
path kubernetes
container_name logs
auto_create_container on
tls on
compression = gzip

[kforeverisback]

@luisfeliz79 looking at the code, it seems the out_azure_blob plugin supports gzip compression out of the box.
Check out:
v2.0.0: https://github.com/fluent/fluent-bit/blob/v2.0.0/plugins/out_azure_blob/azure_blob.c#L539
master: https://github.com/fluent/fluent-bit/blob/master/plugins/out_azure_blob/azure_blob.c#L1175

Set compress to gzip, to enable compression during HTTP payload transfer.
If you want the actual blob to be gzip compressed you can set compress_blob to true.

Note: append_blob is not compatible with compress_blob option.

It's just not officially documented.

@patrick-stephens Maybe we should open an issue in the fluentbit-docs repo?

[patrick-stephens]

If it is supported sure although easier just to send a PR :)

[luisfeliz79]

Thank you for the quick response and information, it is great to read that it is already built in!

I tested the compress = gzip attribute and it does exactly what I need (in Append mode) - and it also compresses the payload, so thank you. This issue can be closed.

P.S. compress_blob appears to be an alternative, that works only with blockmode as you pointed out, but compress=gzip seems to be good enough.

[kforeverisback]

@patrick-stephens I've created a PR in the fluent-bit-docs repo documenting these two options.
fluent/fluent-bit-docs#1625