buffer params explanation

[mirisu2]

Hi,
From documentation the chunk size is approximately 2MB.
The Tail plugin supports the following configuration parameters:
Buffer_Chunk_Size and Buffer_Max_Size
I'm trying to understand what this params are? What for? How these parameters correlate with mem_buf_limit and how many files the plugin reads?

mem_buf_limit is a limit of memory for all files the input plugin reads.
But what about Buffer_Chunk_Size and Buffer_Max_Size ?

storage.max_chunks_pause - Specifies if file storage is to be paused when reaching the chunk limit. reaching the chunk limit - what does it mean?

From documentation:
So, if we impose a limit of 10MB for the input plugins and consider the worse case scenario of the output plugin consuming 20MB extra, as a minimum we need (30MB x 1.2) = 36MB.
1.2 - what this multiplier is?
Why to sum a limit of input plugin with output plugin? Mem_Buf_Limit is limit for input plugin.

Can anyone to explain?

[hsingli20]

https://docs.fluentbit.io/manual/pipeline/inputs/tail.
The document is not clear to set the parameters, like Buffer_Chunk_Size and Buffer_Max_Size.
Issues:
log missings about 60% when Buffer_Chunk_Size is 32K, the size of one line log is about 4K. After change Buffer_Chunk_Size to 1M, the log missing is 0%.
Could you give more indicative methods to set these parameters?
when set it too small, what's the impact? once set it too large, what's the impact?

[emmacz]

Any update on this pls.?

[zi1611061206]

I also faced headaches when considering setting these parameters; the issue is that the documentation doesn’t clearly explain the correlation between them.

[pbraun9]

(as of Feb 2025) it is somehow documented, not only on the inputs/tail page but also on the inputs/syslog and outputs/elasticsearch pages

https://docs.fluentbit.io/manual/pipeline/inputs/syslog#configuration-parameters
https://docs.fluentbit.io/manual/pipeline/inputs/syslog#considerations
https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#configuration-parameters

it needs to be increased for archive processing
while it probably needs to be decreased for heavy duty and small logs

[edsiper]

In the meanwhile we update the docs adding for clarity around different buffers:

Tail, buffers and Chunks

When a file is open to be monitored, Tail plugin allocates a buffer in memory of buffer_chunk_size bytes (defaults to 32kb). If a single record (line in this is case) is longer than buffer_chunk_size it won't fit so that buffer will grow up to buffer_max_size. We keep this value of buffer_chunk_size very small since it fits most of use cases, however if your file aims to have very long lines this needs to be adjusted. Note that if you are monitoring a high number of files, a single buffer exists for each one and that might generate a higher memory usage.

Inside each buffer, multiple lines/records might exists, then the final buffer composed with these records processed it compose another buffer in msgpack format (binary serialization), that msgpack is appended to what we call a Chunk: multiple serialized records that belongs to the same Tag.

While we have a soft-limit of chunks for 2MB, input plugins like Tail who ingest data into the pipeline might generate a msgpack buffer which is greater than 2MB, and the final chunk where this is appended can have a higher size (exceed the 2MB soft-limit).

As of Fluent Bit v4.0 we don't have the functionality to enforce the Chunk size, however we already started some work on that area which we want to ship as part of v4.1 release targeted for Aug 2025.

Mem_buf_limit

If Fluent Bit has not been configured to use filesystem buffering, it needs some mechanisms to protect high consume of memory in case there is backpressure e.g: destination endpoints are down, network issues... Fluent Bit will retry sending these chunks, but also more data is being ingested, this is where mem_buf_limit comes to help to restrict how much memory in Chunks an input plugin can have.

As mentioned, this works differently when filesystem buffering is enabled. More details here:

• https://docs.fluentbit.io/manual/administration/buffering-and-storage

[edsiper]

FYI: I have added the similar content in https://docs.fluentbit.io/manual/pipeline/inputs/tail#buffers-and-memory-management