rotate_size not working as expected

[carylewis]

Describe the bug

After adding the following to the fluentd.conf file:

<system>
  <log>
    rotate_age 7
    rotate_size 8192
  </log>
</system>
<source>
  @type tail
  path /var/log/pods/*/*/*.log
  exclude_path ["/var/log/pods/*/*fluentd*/*.log"]
  pos_file /var/log/td-agent/container.log.pos
  tag omera.*
<parse>
    @type none
</parse>
</source>
 
<filter names.**>
  @type record_transformer
  enable_ruby
<record>
    file_name ${tag.split('.')[5]}
</record>
</filter>
 
<match fluentd.**>
  @type null
</match>
 
<match names.**>
  @type file
  append true
  path /fluentd/log/${file_name}.%Y%m%d.log
<buffer file_name,time>
    timekey 1d
    timekey_use_utc true
    timekey_wait 5s
    chunk_limit_size 8192
    flush_at_shutdown true
    flush_interval 1s
    flush_mode interval
</buffer>
<format>
    @type out_file
    time_slice_format %Y%m%d
    time_format %Y-%m-%dT%H:%M:%S%z
</format>
</match>

My output log files continue to grow until the path forces a rollover at midnight.

This is an example config file, I set it to 8192 to try to see if it would rotate - I tried adjusting the chunk_limit_size to match the rotate_size but it had no effect.

Is this the expected behaviour?

To Reproduce

Run fluentd with the supplied config file.

Expected behavior

I was hoping that fluentd would rotate the logs based on the specified rotate_size.

Your Environment

- Fluentd version: 1.16.5

Your Configuration

fluented is running as a k8s pod

Your Error Log

I see it is picking up the rotate_size

2024-05-25 14:35:47 +0000 [info]: #0 init worker0 logger path=nil rotate_age=7 rotate_size=8192

Additional context

No response

[daipom]

@carylewis
I have transferred this to Q&A as it doesn't look like a bug.

rotate_size and rotate_age are options for the internal log files in which Fluentd outputs its own operation logs.

• https://docs.fluentd.org/deployment/logging#output-to-log-file

So, they have nothing to do with out_file's output.
You need to adjust the options of out_file and its buffer.

I tried adjusting the chunk_limit_size

It would not limit the file size as long as you set append true because new buffer chunks could be appended to existing files.
It is the expected behavior.

out_file does not have the feature to rotate the output file.
You can limit the file size by adjusting append and buffer options to manage how files are output.
If you use append true, it would be difficult to strictly limit the size, but you can adjust it by timekey to some extent.
For example, timekey 1h would reduce the file size.

[carylewis]

If there is no native rotation of out_file output files in fluentd, is the recommended approach to configure a logrotate sidecar, or standalone pod to handle the rotation of log files?

[daipom]

I don't think it's supposed to. (Might be possible if done well.)

Although out_file doesn't have the feature to rotate its' files, you can do something similar by adjusting the buffer key as we are talking about here.

You can limit the file size by adjusting append and buffer options to manage how files are output.
If you use append true, it would be difficult to strictly limit the size, but you can adjust it by timekey to some extent.
For example, timekey 1h would reduce the file size.

[lexain12]

@carylewis Have you found the solution to this problem?

[daipom]

You can adjust the following parameters to manage the granularity.

• time format in path (such as %Y%m%d)
• time_key of the buffer (such as 1d)
• append true or append false