Broken "Ignore Older" setting in in_tail plugin

[Lumiond]

Bug Report

Describe the bug
"Ignore older" setting doesn't work at runtime, only at start, after version 2.1.8.

To Reproduce

• Steps to reproduce the problem:

1. Set in_tail plugin to collect logs with Ignore_Older=1h
2. Start fluent-bit

Expected behavior
After 2h of work Fluent-bit keeps open only current file and file from previous hour.

Screenshots

Your Environment

• Version used: 3.0.6
• Configuration: build from source with kafka out plugin
• Operating System and version: windows server 2019

Additional context
I noticed that Fluentbit doesn't release old log files, so I used the "Ignore older" setting to reduce the number of open files, because when I have around 1000 logs per 24h, Fluent-Bit uses a lot of RAM (more than 1 GB, which is a lot for an agent). Unfortunately I can't reduce the number of log files. After v2.1.8, this setting doesn't work at runtime.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[Lumiond]

Not stale

[pmarag]

Issue seems to be replicated also on the latest fluentbit version (3.1.9).

Is this recognized as a bug/issue?

[emmacz]

We see same issue on Linux and Mac with FB version 2.2, 3.X . Anyone from FB development can assign this and fix it?

[emmacz]

Below is example where is seen that FB (version v3.1.9) opened (tailed) 100 files and after 4mins still keeping them opened...

less ../fb.log
[2024/10/24 00:00:33] [debug] [input:tail:tail.0] 100 new files found on path './harvested_folder/*.aud'

[2024/10/24 00:00:33] [debug] [input:tail:tail.0] inode=58807337 file=./harvested_folder/2024-10-24__00-00-04-064_file1.aud promote to TAIL_EVENT

[2024/10/24 00:00:33] [debug] [input:tail:tail.0] inode=58807338 file=./harvested_folder/2024-10-24__00-00-04-083_file2.aud promote to TAIL_EVENT

[2024/10/24 00:00:33] [debug] [input:tail:tail.0] inode=58807339 file=./harvested_folder/2024-10-24__00-00-04-096_file3.aud promote to TAIL_EVENT

---

less ../fb.log
[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-127_file92.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-136_file93.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-148_file94.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-160_file95.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-174_file96.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-185_file97.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-196_file98.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-207_file99.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] 0 new files found on path './harvested_folder/*.aud'`

---

date; lsof -p $(pidof fluent-bit) | grep \.aud | wc -l

Thu Oct 24 00:07:11 CEST 2024
     100

---

date; lsof -p $(pidof fluent-bit) | grep \.aud | tail

Thu Oct 24 00:07:16 CEST 2024

fluent-bi 69133 mlukas  127r     REG               1,18        0 58807563 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-103_file90.aud

fluent-bi 69133 mlukas  128r     REG               1,18        0 58807564 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-115_file91.aud

fluent-bi 69133 mlukas  129r     REG               1,18        0 58807565 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-127_file92.aud

fluent-bi 69133 mlukas  130r     REG               1,18        0 58807566 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-136_file93.aud

fluent-bi 69133 mlukas  131r     REG               1,18        0 58807568 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-148_file94.aud

fluent-bi 69133 mlukas  133r     REG               1,18        0 58807570 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-174_file96.aud

fluent-bi 69133 mlukas  134r     REG               1,18        0 58807571 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-185_file97.aud

fluent-bi 69133 mlukas  135r     REG               1,18        0 58807572 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-196_file98.aud

fluent-bi 69133 mlukas  136r     REG               1,18        0 58807573 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-207_file99.aud

fluent-bi 69133 mlukas  137r     REG               1,18        0 58807574 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-216_file100.aud

FB was started using this command:

fluent-bit -i tail -p path='./harvested_folder/*.aud' -p ignore_older=4m -p parser.conf -l fb.log -v

---

On example of file 2024-10-24__00-00-05-216_file100.aud is seen that file was:

• found
• being tailed (see TAIL_EVENT) for 4 mins (see dismissed)
• then being skipped (ignore_older)
• but FB process still keeps it opened (see lsof command) .

 grep 2024-10-24__00-00-05-216_file100.aud ../fb.log

[2024/10/24 00:00:33] [debug] [input:tail:tail.0] inode=58807574 with offset=0 appended as ./harvested_folder/2024-10-24__00-00-05-216_file100.aud

[2024/10/24 00:00:33] [debug] [input:tail:tail.0] scan_glob add(): ./harvested_folder/2024-10-24__00-00-05-216_file100.aud, inode 58807574

[2024/10/24 00:00:33] [debug] [input:tail:tail.0] inode=58807574 file=./harvested_folder/2024-10-24__00-00-05-216_file100.aud promote to TAIL_EVENT

[2024/10/24 00:01:33] [debug] [input:tail:tail.0] scan_blog add(): dismissed: ./harvested_folder/2024-10-24__00-00-05-216_file100.aud, inode 58807574

[2024/10/24 00:02:33] [debug] [input:tail:tail.0] scan_blog add(): dismissed: ./harvested_folder/2024-10-24__00-00-05-216_file100.aud, inode 58807574

[2024/10/24 00:03:33] [debug] [input:tail:tail.0] scan_blog add(): dismissed: ./harvested_folder/2024-10-24__00-00-05-216_file100.aud, inode 58807574

[2024/10/24 00:04:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:05:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:06:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:07:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:08:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:09:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:10:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:11:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:12:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:13:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:14:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:15:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:16:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

[2024/10/24 00:17:33] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-24__00-00-05-216_file100.aud (ignore_older)

---

lsof -p $(pidof fluent-bit)| grep 2024-10-24__00-00-05-216_file100.aud

fluent-bi 69133 mlukas  137r     REG               1,18        0 58807574 /private/tmp/flb/harvested_folder/2024-10-24__00-00-05-216_file100.aud

[Qubora]

fluent-bit agent (adp-logagent.service) is crashing due this issue because of 'open files' limit (64K files) is crossed.
I all input sections we have following:
ignore_older: 15m

This is /var/log/messages excerpt:
Oct 23 12:24:58 phx00049-nt1fg12 fluent-bit: [/tmp/src/fluent-bit.git/lib/chunkio/src/cio_file_unix.c:410 errno=24] Too many open files
Oct 23 12:24:58 phx00049-nt1fg12 fluent-bit: [log] error opening log file /var/log/logagent/adp-fa-logagent.log. Using stderr.
Oct 23 12:24:58 phx00049-nt1fg12 fluent-bit: [2024/10/23 12:24:58] [error] [input chunk] could not create chunk file: tail.17:104359-1729686298.746564261.flb
Oct 23 12:24:58 phx00049-nt1fg12 fluent-bit: [log] error opening log file /var/log/logagent/adp-fa-logagent.log. Using stderr.
Oct 23 12:24:58 phx00049-nt1fg12 fluent-bit: [2024/10/23 12:24:58] [error] [input chunk] no available chunk
Oct 23 12:25:00 phx00049-nt1fg12 fluent-bit: [2024/10/23 12:25:00] [engine] caught signal (SIGSEGV)
...
Oct 23 12:25:01 phx00049-nt1fg12 systemd: adp-logagent.service: main process exited, code=killed, status=6/ABRT
Oct 23 12:25:01 phx00049-nt1fg12 systemd: Unit adp-logagent.service entered failed state.
Oct 23 12:25:01 phx00049-nt1fg12 systemd: adp-logagent.service failed.
...
Oct 23 12:26:01 phx00049-nt1fg12 systemd: Stopped adp logagent for FA.
Oct 23 12:26:01 phx00049-nt1fg12 systemd: Starting adp logagent for FA...
...

[pmarag]

@patrick-stephens @edsiper Could you please help us understand if this is indeed a bug or a misconfiguration? Problem is that basically fluent-bit stops working (and memory is increasing rapidly until it stop working) in case multiple static files are added on a directory which is monitored.

[Qubora]

@pmarag if we have set e.g. ignore_older: 15m then we expect the any file matching the Path :

• is not being opened for tailing on fluent-bit start when it is older than ignore_older
• is opened for tailing on fluent-bit start when file is younger that ignore_older
• is closed and tailing stopped once the file is not updated within ignore_older time period

Please correct me if I'am wrong.

What we are observing is that fluent-bit process is keeping open files even if they are not updated for longer time. It results in 'Too many open files' error.

[pmarag]

@pmarag if we have set e.g. ignore_older: 15m then we expect the any file matching the Path :

• is not being opened for tailing on fluent-bit start when it is older than ignore_older
• is opened for tailing on fluent-bit start when file is younger that ignore_older
• is closed and tailing stopped once the file is not updated within ignore_older time period

Please correct me if I'am wrong.

What we are observing is that fluent-bit process is keeping open files even if they are not updated for longer time. It results in 'Too many open files' error.

Your expectation is correct and matching also my expectation. Currently it seems that in_tail keep still the old files opened and this leads to "too many open files" issue. There is a workaround/mitigation that we have in place: kill -s SIGHUP <>. This way, fluent-bit reloads the configuration and release files that are older than ignore_older. You may try if something like this works also on your environment/use-case but in the end of the day is just a workaround.

[Qubora]

@pmarag if we have set e.g. ignore_older: 15m then we expect the any file matching the Path :

• is not being opened for tailing on fluent-bit start when it is older than ignore_older
• is opened for tailing on fluent-bit start when file is younger that ignore_older
• is closed and tailing stopped once the file is not updated within ignore_older time period

Please correct me if I'am wrong.
What we are observing is that fluent-bit process is keeping open files even if they are not updated for longer time. It results in 'Too many open files' error.

Your expectation is correct and matching also my expectation. Currently it seems that in_tail keep still the old files opened and this leads to "too many open files" issue. There is a workaround/mitigation that we have in place: kill -s SIGHUP <>. This way, fluent-bit reloads the configuration and release files that are older than ignore_older. You may try if something like this works also on your environment/use-case but in the end of the day is just a workaround.

Tried kill -s SIGHUP fb_pid but it killed the process

service adp-logagent status

Redirecting to /bin/systemctl status adp-logagent.service
● adp-logagent.service - adp logagent for FA
Loaded: loaded (/etc/systemd/system/adp-logagent.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: signal) since Thu 2024-10-24 14:47:41 UTC; 34s ago
Process: 25346 ExecStartPost=/var/lib/oci/scripts/adp-logagent-post-start.sh (code=exited, status=0/SUCCESS)
Process: 25345 ExecStart=/usr/local/bin/adp/fluent-bit -c /usr/local/etc/adp-fluent-bit/fluent-bit.yaml (code=killed, signal=ABRT)

[pmarag]

@pmarag if we have set e.g. ignore_older: 15m then we expect the any file matching the Path :

• is not being opened for tailing on fluent-bit start when it is older than ignore_older
• is opened for tailing on fluent-bit start when file is younger that ignore_older
• is closed and tailing stopped once the file is not updated within ignore_older time period

Please correct me if I'am wrong.
What we are observing is that fluent-bit process is keeping open files even if they are not updated for longer time. It results in 'Too many open files' error.

Your expectation is correct and matching also my expectation. Currently it seems that in_tail keep still the old files opened and this leads to "too many open files" issue. There is a workaround/mitigation that we have in place: kill -s SIGHUP <>. This way, fluent-bit reloads the configuration and release files that are older than ignore_older. You may try if something like this works also on your environment/use-case but in the end of the day is just a workaround.

Tried kill -s SIGHUP fb_pid but it killed the process

service adp-logagent status

Redirecting to /bin/systemctl status adp-logagent.service ● adp-logagent.service - adp logagent for FA Loaded: loaded (/etc/systemd/system/adp-logagent.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: signal) since Thu 2024-10-24 14:47:41 UTC; 34s ago Process: 25346 ExecStartPost=/var/lib/oci/scripts/adp-logagent-post-start.sh (code=exited, status=0/SUCCESS) Process: 25345 ExecStart=/usr/local/bin/adp/fluent-bit -c /usr/local/etc/adp-fluent-bit/fluent-bit.yaml (code=killed, signal=ABRT)

Apologies for this, I think I forgot to mention that you need also to enable the hot reload. There is also a way to trigger the reload through HTTP. You may find more info here:

https://docs.fluentbit.io/manual/administration/hot-reload

Again this is not a solution, just a workaround to mitigate the issue.

[emmacz]

@pmarag @edsiper @lecaros @Lumiond
The w/a with hot-reload works and clearly shows what must be done automatically by Fluent-Bit, it means to release previously tailed files. See below Fluent-Bit output, see tag 'removing file name'.

fb-tester ~$ fluent-bit -i tail -p path='./harvested_folder/*.aud' -p ignore_older=4m -p parser.conf -l fb.log -v -Y
Fluent Bit v3.1.9
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __
|  ___| |                | |   | ___ (_) |         |____ |/  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/10/25 02:30:21] [ info] Configuration:
[2024/10/25 02:30:21] [ info]  flush time     | 1.000000 seconds
[2024/10/25 02:30:21] [ info]  grace          | 5 seconds
[2024/10/25 02:30:21] [ info]  daemon         | 0
[2024/10/25 02:30:21] [ info] ___________
[2024/10/25 02:30:21] [ info]  inputs:
[2024/10/25 02:30:21] [ info]      tail
[2024/10/25 02:30:21] [ info] ___________
[2024/10/25 02:30:21] [ info]  filters:
[2024/10/25 02:30:21] [ info] ___________
[2024/10/25 02:30:21] [ info]  outputs:
[2024/10/25 02:30:21] [ info] ___________
[2024/10/25 02:30:21] [ info]  collectors:
[2024/10/25 02:37:05] [engine] caught signal (SIGHUP)
[2024/10/25 02:37:05] [ info] reloading instance pid=93770 tid=0x1f1740f40
[2024/10/25 02:37:05] [ info] [reload] stop everything of the old context
[2024/10/25 02:37:05] [ info] [input] pausing tail.0
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927386 removing file name ./harvested_folder/2024-10-25__02-20-49-461_file1.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927387 removing file name ./harvested_folder/2024-10-25__02-20-49-473_file2.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927388 removing file name ./harvested_folder/2024-10-25__02-20-49-487_file3.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927389 removing file name ./harvested_folder/2024-10-25__02-20-49-496_file4.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927390 removing file name ./harvested_folder/2024-10-25__02-20-49-507_file5.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927391 removing file name ./harvested_folder/2024-10-25__02-20-49-517_file6.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927392 removing file name ./harvested_folder/2024-10-25__02-20-49-526_file7.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927393 removing file name ./harvested_folder/2024-10-25__02-20-49-536_file8.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927394 removing file name ./harvested_folder/2024-10-25__02-20-49-546_file9.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927395 removing file name ./harvested_folder/2024-10-25__02-20-49-559_file10.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927396 removing file name ./harvested_folder/2024-10-25__02-20-49-572_file11.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927397 removing file name ./harvested_folder/2024-10-25__02-20-49-583_file12.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927398 removing file name ./harvested_folder/2024-10-25__02-20-49-591_file13.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927399 removing file name ./harvested_folder/2024-10-25__02-20-49-601_file14.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927400 removing file name ./harvested_folder/2024-10-25__02-20-49-610_file15.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927401 removing file name ./harvested_folder/2024-10-25__02-20-49-620_file16.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927402 removing file name ./harvested_folder/2024-10-25__02-20-49-630_file17.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927403 removing file name ./harvested_folder/2024-10-25__02-20-49-640_file18.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927404 removing file name ./harvested_folder/2024-10-25__02-20-49-651_file19.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927405 removing file name ./harvested_folder/2024-10-25__02-20-49-662_file20.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927406 removing file name ./harvested_folder/2024-10-25__02-20-49-672_file21.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927407 removing file name ./harvested_folder/2024-10-25__02-20-49-682_file22.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927408 removing file name ./harvested_folder/2024-10-25__02-20-49-691_file23.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927409 removing file name ./harvested_folder/2024-10-25__02-20-49-701_file24.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927410 removing file name ./harvested_folder/2024-10-25__02-20-49-712_file25.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927411 removing file name ./harvested_folder/2024-10-25__02-20-49-724_file26.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927412 removing file name ./harvested_folder/2024-10-25__02-20-49-734_file27.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927413 removing file name ./harvested_folder/2024-10-25__02-20-49-744_file28.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927414 removing file name ./harvested_folder/2024-10-25__02-20-49-755_file29.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927415 removing file name ./harvested_folder/2024-10-25__02-20-49-766_file30.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927416 removing file name ./harvested_folder/2024-10-25__02-20-49-776_file31.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927417 removing file name ./harvested_folder/2024-10-25__02-20-49-787_file32.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927418 removing file name ./harvested_folder/2024-10-25__02-20-49-794_file33.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927419 removing file name ./harvested_folder/2024-10-25__02-20-49-802_file34.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927420 removing file name ./harvested_folder/2024-10-25__02-20-49-812_file35.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927421 removing file name ./harvested_folder/2024-10-25__02-20-49-823_file36.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927422 removing file name ./harvested_folder/2024-10-25__02-20-49-837_file37.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927423 removing file name ./harvested_folder/2024-10-25__02-20-49-848_file38.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927424 removing file name ./harvested_folder/2024-10-25__02-20-49-860_file39.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927425 removing file name ./harvested_folder/2024-10-25__02-20-49-868_file40.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927426 removing file name ./harvested_folder/2024-10-25__02-20-49-875_file41.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927427 removing file name ./harvested_folder/2024-10-25__02-20-49-884_file42.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927428 removing file name ./harvested_folder/2024-10-25__02-20-49-894_file43.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927429 removing file name ./harvested_folder/2024-10-25__02-20-49-904_file44.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927430 removing file name ./harvested_folder/2024-10-25__02-20-49-916_file45.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927431 removing file name ./harvested_folder/2024-10-25__02-20-49-926_file46.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927432 removing file name ./harvested_folder/2024-10-25__02-20-49-935_file47.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927433 removing file name ./harvested_folder/2024-10-25__02-20-49-945_file48.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927434 removing file name ./harvested_folder/2024-10-25__02-20-49-955_file49.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927435 removing file name ./harvested_folder/2024-10-25__02-20-49-964_file50.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927436 removing file name ./harvested_folder/2024-10-25__02-20-49-974_file51.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927437 removing file name ./harvested_folder/2024-10-25__02-20-49-984_file52.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927438 removing file name ./harvested_folder/2024-10-25__02-20-49-995_file53.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927439 removing file name ./harvested_folder/2024-10-25__02-20-50-006_file54.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927440 removing file name ./harvested_folder/2024-10-25__02-20-50-017_file55.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927441 removing file name ./harvested_folder/2024-10-25__02-20-50-028_file56.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927442 removing file name ./harvested_folder/2024-10-25__02-20-50-041_file57.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927443 removing file name ./harvested_folder/2024-10-25__02-20-50-052_file58.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927444 removing file name ./harvested_folder/2024-10-25__02-20-50-062_file59.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927445 removing file name ./harvested_folder/2024-10-25__02-20-50-071_file60.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927446 removing file name ./harvested_folder/2024-10-25__02-20-50-080_file61.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927447 removing file name ./harvested_folder/2024-10-25__02-20-50-091_file62.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927448 removing file name ./harvested_folder/2024-10-25__02-20-50-100_file63.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927449 removing file name ./harvested_folder/2024-10-25__02-20-50-108_file64.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927450 removing file name ./harvested_folder/2024-10-25__02-20-50-115_file65.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927451 removing file name ./harvested_folder/2024-10-25__02-20-50-121_file66.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927452 removing file name ./harvested_folder/2024-10-25__02-20-50-129_file67.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927453 removing file name ./harvested_folder/2024-10-25__02-20-50-136_file68.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927454 removing file name ./harvested_folder/2024-10-25__02-20-50-143_file69.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927455 removing file name ./harvested_folder/2024-10-25__02-20-50-150_file70.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927456 removing file name ./harvested_folder/2024-10-25__02-20-50-159_file71.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927457 removing file name ./harvested_folder/2024-10-25__02-20-50-170_file72.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927458 removing file name ./harvested_folder/2024-10-25__02-20-50-180_file73.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927459 removing file name ./harvested_folder/2024-10-25__02-20-50-188_file74.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927460 removing file name ./harvested_folder/2024-10-25__02-20-50-198_file75.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927461 removing file name ./harvested_folder/2024-10-25__02-20-50-209_file76.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927462 removing file name ./harvested_folder/2024-10-25__02-20-50-219_file77.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927463 removing file name ./harvested_folder/2024-10-25__02-20-50-228_file78.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927464 removing file name ./harvested_folder/2024-10-25__02-20-50-239_file79.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927465 removing file name ./harvested_folder/2024-10-25__02-20-50-258_file80.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927466 removing file name ./harvested_folder/2024-10-25__02-20-50-266_file81.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927467 removing file name ./harvested_folder/2024-10-25__02-20-50-273_file82.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927468 removing file name ./harvested_folder/2024-10-25__02-20-50-280_file83.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927469 removing file name ./harvested_folder/2024-10-25__02-20-50-287_file84.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927470 removing file name ./harvested_folder/2024-10-25__02-20-50-295_file85.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927471 removing file name ./harvested_folder/2024-10-25__02-20-50-305_file86.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927472 removing file name ./harvested_folder/2024-10-25__02-20-50-316_file87.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927473 removing file name ./harvested_folder/2024-10-25__02-20-50-326_file88.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927474 removing file name ./harvested_folder/2024-10-25__02-20-50-342_file89.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927475 removing file name ./harvested_folder/2024-10-25__02-20-50-372_file90.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927476 removing file name ./harvested_folder/2024-10-25__02-20-50-390_file91.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927477 removing file name ./harvested_folder/2024-10-25__02-20-50-408_file92.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927478 removing file name ./harvested_folder/2024-10-25__02-20-50-419_file93.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927479 removing file name ./harvested_folder/2024-10-25__02-20-50-425_file94.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927480 removing file name ./harvested_folder/2024-10-25__02-20-50-432_file95.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927481 removing file name ./harvested_folder/2024-10-25__02-20-50-439_file96.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927482 removing file name ./harvested_folder/2024-10-25__02-20-50-446_file97.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927483 removing file name ./harvested_folder/2024-10-25__02-20-50-452_file98.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927484 removing file name ./harvested_folder/2024-10-25__02-20-50-461_file99.aud
[2024/10/25 02:37:05] [debug] [input:tail:tail.0] inode=58927485 removing file name ./harvested_folder/2024-10-25__02-20-50-467_file100.aud
[2024/10/25 02:37:05] [ info] [reload] start everything

Here is what we did:
Start FB with -Y (hot-reload) option

fluent-bit -i tail -p path='./harvested_folder/*.aud' -p ignore_older=4m -p parser.conf -l fb.log -v -Y

Once we have seen in fb.log that harvested files are being ignored because they became older than 4 minutes limit set, see (ignore_older) , then we checked that FB process still keeps them, which confirms this bug, see lsof command. After that we did hot reload, see kill -s SIGHUP $(pidof fluent-bit) and then we checked again lsof. There were no (zero) previously kept files by the FB process.

tail fb.log
[2024/10/25 02:36:39] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-25__02-20-50-461_file99.aud (ignore_older)
[2024/10/25 02:36:39] [debug] [input:tail:tail.0] excluded=./harvested_folder/2024-10-25__02-20-50-467_file100.aud (ignore_older)
[2024/10/25 02:36:39] [debug] [input:tail:tail.0] 0 new files found on path './harvested_folder/*.aud'

date; lsof -p $(pidof fluent-bit) | grep \.aud | wc -l
Fri Oct 25 02:37:04 CEST 2024
       100

kill -s SIGHUP $(pidof fluent-bit)

date; lsof -p $(pidof fluent-bit) | grep \.aud | wc -l
Fri Oct 25 02:38:39 CEST 2024
       0