Skip to content

Conversation

@EKwongChum
Copy link

@EKwongChum EKwongChum commented Mar 24, 2022

Feature #5170

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • Documentation required for this feature

Doc PR is here770

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Test Part 1
when add new property line_append true in service conf output item

Example configuration file as below:

[service]
    flush               5
    grace               10
    daemon              on
    log_file            /tmp/ekwongchum/flb.log
    log_level           debug

[input]
    Name                 tail
    Tag                  syslog
    Path                 /var/log/syslog
    DB                   /tmp/ekwongchum/syslog.db

[output]
    Name                 file
    Match                syslog
    Path                 /tmp/ekwongchum/
    File                 record.file
    Mkdir                true
    Format               template
    Template             {log}
    line_append          true

debug log output as below:

./fluent-bit -c flb.conf
Fluent Bit v2.0.0
* Git commit: 8efde37ad077f32611ad52f9ce831da413985f57
* Copyright (C) 2015-2021 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/03/25 10:12:41] [ info] Configuration:
[2022/03/25 10:12:41] [ info]  flush time     | 5.000000 seconds
[2022/03/25 10:12:41] [ info]  grace          | 10 seconds
[2022/03/25 10:12:41] [ info]  daemon         | 1
[2022/03/25 10:12:41] [ info] ___________
[2022/03/25 10:12:41] [ info]  inputs:
[2022/03/25 10:12:41] [ info]      tail
[2022/03/25 10:12:41] [ info] ___________
[2022/03/25 10:12:41] [ info]  filters:
[2022/03/25 10:12:41] [ info] ___________
[2022/03/25 10:12:41] [ info]  outputs:
[2022/03/25 10:12:41] [ info]      file.0
[2022/03/25 10:12:41] [ info] ___________
[2022/03/25 10:12:41] [ info]  collectors:
[2022/03/25 10:12:41] [ info] switching to background mode (PID=216399)

and log file show here:

[2022/03/25 10:12:41] [ info] [engine] started (pid=216399)
[2022/03/25 10:12:41] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2022/03/25 10:12:41] [debug] [storage] [cio stream] new stream registered: tail.0
[2022/03/25 10:12:41] [ info] [storage] version=1.1.6, initializing...
[2022/03/25 10:12:41] [ info] [storage] in-memory
[2022/03/25 10:12:41] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/03/25 10:12:41] [ info] [cmetrics] version=0.3.0
[2022/03/25 10:12:41] [debug] [tail:tail.0] created event channels: read=20 write=21
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] inotify watch fd=26
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] scanning path /var/log/syslog
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] inode=36177971 with offset=121979 appended as /var/log/syslog
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] scan_glob add(): /var/log/syslog, inode 36177971
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] 1 new files found on path '/var/log/syslog'
[2022/03/25 10:12:41] [debug] [file:file.0] created event channels: read=31 write=32
[2022/03/25 10:12:41] [debug] [router] match rule tail.0:file.0
[2022/03/25 10:12:41] [ info] [sp] stream processor started
[2022/03/25 10:12:41] [ info] [output:file:file.0] worker #0 started
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] inode=36177971 file=/var/log/syslog promote to TAIL_EVENT
[2022/03/25 10:12:41] [ info] [input:tail:tail.0] inotify_fs_add(): inode=36177971 watch_fd=1 name=/var/log/syslog
[2022/03/25 10:12:41] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2022/03/25 10:13:40] [debug] [input:tail:tail.0] scanning path /var/log/syslog
[2022/03/25 10:13:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/syslog, inode 36177971
[2022/03/25 10:13:40] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/syslog'

and record file is here

Mar 25 10:15:32 ekwong-NUC10 dbus-daemon[748]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.1757' (uid=0 pid=217147 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")Mar 25 10:15:32 ekwong-NUC10 systemd[1]: Starting PackageKit Daemon...Mar 25 10:15:32 ekwong-NUC10 PackageKit: daemon startMar 25 10:15:32 ekwong-NUC10 dbus-daemon[748]: [system] Successfully activated service 'org.freedesktop.PackageKit'Mar 25 10:15:32 ekwong-NUC10 systemd[1]: Started PackageKit Daemon.Mar 25 10:16:39 ekwong-NUC10 dbus-daemon[2999]: [session uid=1000 pid=2999] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=2992 comm="/usr/libexec/tracker-miner-fs " label="unconfined")Mar 25 10:16:39 ekwong-NUC10 systemd[2981]: Starting Tracker metadata database store and lookup manager...Mar 25 10:16:39 ekwong-NUC10 dbus-daemon[2999]: [session uid=1000 pid=2999] Successfully activated service 'org.freedesktop.Tracker1'Mar 25 10:16:39 ekwong-NUC10 systemd[2981]: Started Tracker metadata database store and lookup manager.%

Test Part 2
when add new property line_append false in service conf output item

Example configuration file as below:

[service]
    flush               5
    grace               10
    daemon              on
    log_file            /tmp/ekwongchum/flb.log
    log_level           debug

[input]
    Name                 tail
    Tag                  syslog
    Path                 /var/log/syslog
    DB                   /tmp/ekwongchum/syslog.db

[output]
    Name                 file
    Match                syslog
    Path                 /tmp/ekwongchum/
    File                 record.file
    Mkdir                true
    Format               template
    Template             {log}
    line_append          false

debug log output as below:

./fluent-bit -c flb.conf
Fluent Bit v2.0.0
* Git commit: 8efde37ad077f32611ad52f9ce831da413985f57
* Copyright (C) 2015-2021 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/03/25 10:23:18] [ info] Configuration:
[2022/03/25 10:23:18] [ info]  flush time     | 5.000000 seconds
[2022/03/25 10:23:18] [ info]  grace          | 10 seconds
[2022/03/25 10:23:18] [ info]  daemon         | 1
[2022/03/25 10:23:18] [ info] ___________
[2022/03/25 10:23:18] [ info]  inputs:
[2022/03/25 10:23:18] [ info]      tail
[2022/03/25 10:23:18] [ info] ___________
[2022/03/25 10:23:18] [ info]  filters:
[2022/03/25 10:23:18] [ info] ___________
[2022/03/25 10:23:18] [ info]  outputs:
[2022/03/25 10:23:18] [ info]      file.0
[2022/03/25 10:23:18] [ info] ___________
[2022/03/25 10:23:18] [ info]  collectors:
[2022/03/25 10:23:18] [ info] switching to background mode (PID=217435)

and log file show here:

[2022/03/25 10:23:18] [ info] [engine] started (pid=217435)
[2022/03/25 10:23:18] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2022/03/25 10:23:18] [debug] [storage] [cio stream] new stream registered: tail.0
[2022/03/25 10:23:18] [ info] [storage] version=1.1.6, initializing...
[2022/03/25 10:23:18] [ info] [storage] in-memory
[2022/03/25 10:23:18] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/03/25 10:23:18] [ info] [cmetrics] version=0.3.0
[2022/03/25 10:23:18] [debug] [tail:tail.0] created event channels: read=19 write=20
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] inotify watch fd=26
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] scanning path /var/log/syslog
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] inode=36177971 with offset=124138 appended as /var/log/syslog
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] scan_glob add(): /var/log/syslog, inode 36177971
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] 1 new files found on path '/var/log/syslog'
[2022/03/25 10:23:18] [debug] [file:file.0] created event channels: read=30 write=31
[2022/03/25 10:23:18] [debug] [router] match rule tail.0:file.0
[2022/03/25 10:23:18] [ info] [sp] stream processor started
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] inode=36177971 file=/var/log/syslog promote to TAIL_EVENT
[2022/03/25 10:23:18] [ info] [output:file:file.0] worker #0 started
[2022/03/25 10:23:18] [ info] [input:tail:tail.0] inotify_fs_add(): inode=36177971 watch_fd=1 name=/var/log/syslog
[2022/03/25 10:23:18] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2022/03/25 10:23:20] [debug] [input:tail:tail.0] inode=36177971 events: IN_MODIFY
[2022/03/25 10:23:20] [debug] [input chunk] update output instances with new chunk size diff=72
[2022/03/25 10:23:20] [debug] [input:tail:tail.0] inode=36177971 events: IN_MODIFY
[2022/03/25 10:23:20] [debug] [input chunk] update output instances with new chunk size diff=95
[2022/03/25 10:23:22] [debug] [task] created task=0x7fac6804a6c0 id=0 OK
[2022/03/25 10:23:22] [debug] [output:file:file.0] task_id=0 assigned to thread #0
[2022/03/25 10:23:22] [debug] [out flush] cb_destroy coro_id=0
[2022/03/25 10:23:22] [debug] [task] destroy task=0x7fac6804a6c0 (task_id=0)

and record file is here

Mar 25 10:23:20 ekwong-NUC10 systemd[2981]: tracker-store.service: Succeeded.
Mar 25 10:28:53 ekwong-NUC10 dbus-daemon[748]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.1761' (uid=0 pid=218217 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Mar 25 10:28:53 ekwong-NUC10 systemd[1]: Starting PackageKit Daemon...
Mar 25 10:28:53 ekwong-NUC10 PackageKit: daemon start
Mar 25 10:28:53 ekwong-NUC10 dbus-daemon[748]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Mar 25 10:28:53 ekwong-NUC10 systemd[1]: Started PackageKit Daemon.

Test Part 3
when no property line_append in service conf output item

Example configuration file as below:

[service]
    flush               5
    grace               10
    daemon              on
    log_file            /tmp/ekwongchum/flb.log
    log_level           debug

[input]
    Name                 tail
    Tag                  syslog
    Path                 /var/log/syslog
    DB                   /tmp/ekwongchum/syslog.db

[output]
    Name                 file
    Match                syslog
    Path                 /tmp/ekwongchum/
    File                 record.file
    Mkdir                true
    Format               template
    Template             {log}

debug log output as below:

./fluent-bit -c flb.conf
Fluent Bit v2.0.0
* Git commit: 8efde37ad077f32611ad52f9ce831da413985f57
* Copyright (C) 2015-2021 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/03/25 10:31:26] [ info] Configuration:
[2022/03/25 10:31:26] [ info]  flush time     | 5.000000 seconds
[2022/03/25 10:31:26] [ info]  grace          | 10 seconds
[2022/03/25 10:31:26] [ info]  daemon         | 1
[2022/03/25 10:31:26] [ info] ___________
[2022/03/25 10:31:26] [ info]  inputs:
[2022/03/25 10:31:26] [ info]      tail
[2022/03/25 10:31:26] [ info] ___________
[2022/03/25 10:31:26] [ info]  filters:
[2022/03/25 10:31:26] [ info] ___________
[2022/03/25 10:31:26] [ info]  outputs:
[2022/03/25 10:31:26] [ info]      file.0
[2022/03/25 10:31:26] [ info] ___________
[2022/03/25 10:31:26] [ info]  collectors:
[2022/03/25 10:31:26] [ info] switching to background mode (PID=218374)

and log file show here:

[2022/03/25 10:31:26] [ info] [engine] started (pid=218374)
[2022/03/25 10:31:26] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2022/03/25 10:31:26] [debug] [storage] [cio stream] new stream registered: tail.0
[2022/03/25 10:31:26] [ info] [storage] version=1.1.6, initializing...
[2022/03/25 10:31:26] [ info] [storage] in-memory
[2022/03/25 10:31:26] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/03/25 10:31:26] [ info] [cmetrics] version=0.3.0
[2022/03/25 10:31:26] [debug] [tail:tail.0] created event channels: read=20 write=21
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] inotify watch fd=28
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] scanning path /var/log/syslog
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] inode=36177971 with offset=124847 appended as /var/log/syslog
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] scan_glob add(): /var/log/syslog, inode 36177971
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] 1 new files found on path '/var/log/syslog'
[2022/03/25 10:31:26] [debug] [file:file.0] created event channels: read=30 write=31
[2022/03/25 10:31:26] [debug] [router] match rule tail.0:file.0
[2022/03/25 10:31:26] [ info] [sp] stream processor started
[2022/03/25 10:31:26] [ info] [output:file:file.0] worker #0 started
[2022/03/25 10:31:26] [debug] [input chunk] update output instances with new chunk size diff=977
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] [static files] processed 874b
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] inode=36177971 file=/var/log/syslog promote to TAIL_EVENT
[2022/03/25 10:31:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=36177971 watch_fd=1 name=/var/log/syslog
[2022/03/25 10:31:26] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2022/03/25 10:31:30] [debug] [task] created task=0x7f2e00049b40 id=0 OK
[2022/03/25 10:31:30] [debug] [output:file:file.0] task_id=0 assigned to thread #0
[2022/03/25 10:31:30] [debug] [out flush] cb_destroy coro_id=0
[2022/03/25 10:31:30] [debug] [task] destroy task=0x7f2e00049b40 (task_id=0)
[2022/03/25 10:31:43] [debug] [input:tail:tail.0] inode=36177971 events: IN_MODIFY
[2022/03/25 10:31:43] [debug] [input chunk] update output instances with new chunk size diff=72
[2022/03/25 10:31:43] [debug] [input:tail:tail.0] inode=36177971 events: IN_MODIFY
[2022/03/25 10:31:43] [debug] [input chunk] update output instances with new chunk size diff=95
[2022/03/25 10:31:45] [debug] [task] created task=0x7f2e00049e80 id=0 OK
[2022/03/25 10:31:45] [debug] [output:file:file.0] task_id=0 assigned to thread #0
[2022/03/25 10:31:45] [debug] [out flush] cb_destroy coro_id=1
[2022/03/25 10:31:45] [debug] [task] destroy task=0x7f2e00049e80 (task_id=0)

and record file is here

Mar 25 10:31:43 ekwong-NUC10 tracker-store[218363]: OK
Mar 25 10:31:43 ekwong-NUC10 systemd[2981]: tracker-store.service: Succeeded.
Mar 25 10:32:29 ekwong-NUC10 rtkit-daemon[1276]: Supervising 3 threads of 1 processes of 1 users.
Mar 25 10:32:29 ekwong-NUC10 rtkit-daemon[1276]: Successfully made thread 218394 of process 2990 owned by '1000' RT at priority 5.
Mar 25 10:32:29 ekwong-NUC10 rtkit-daemon[1276]: Supervising 4 threads of 1 processes of 1 users.
Mar 25 10:32:29 ekwong-NUC10 gsd-media-keys[3398]: Unable to get default sink
Mar 25 10:32:29 ekwong-NUC10 gsd-media-keys[3398]: gvc_mixer_card_get_index: assertion 'GVC_IS_MIXER_CARD (card)' failed
Mar 25 10:32:29 ekwong-NUC10 gsd-media-keys[3398]: message repeated 4 times: [ gvc_mixer_card_get_index: assertion 'GVC_IS_MIXER_CARD (card)' failed]
Mar 25 10:32:29 ekwong-NUC10 gnome-shell[3246]: gvc_mixer_card_get_index: assertion 'GVC_IS_MIXER_CARD (card)' failed
Mar 25 10:32:29 ekwong-NUC10 gnome-shell[3246]: message repeated 4 times: [ gvc_mixer_card_get_index: assertion 'GVC_IS_MIXER_CARD (card)' failed]
Mar 25 10:32:29 ekwong-NUC10 dbus-daemon[748]: [system] Activating via systemd: service name='net.reactivated.Fprint' unit='fprintd.service' requested by ':1.91' (uid=1000 pid=3246 comm="/usr/bin/gnome-shell " label="unconfined")
Mar 25 10:32:29 ekwong-NUC10 systemd[1]: Starting Fingerprint Authentication Daemon...
Mar 25 10:32:29 ekwong-NUC10 dbus-daemon[748]: [system] Successfully activated service 'net.reactivated.Fprint'
Mar 25 10:32:29 ekwong-NUC10 systemd[1]: Started Fingerprint Authentication Daemon.

{
FLB_CONFIG_MAP_BOOL, "line_append", "false",
0, FLB_TRUE, offsetof(struct flb_file_conf, line_append),
"Append new data to the end off log in same line. Default value is false"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo here, should be "of" not "off"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I fix it .

@patrick-stephens patrick-stephens changed the title new property to control output file append at the end line out_file: new property to control output file append at the end line Mar 24, 2022
@patrick-stephens
Copy link
Collaborator

Can you fill in the details required in the template? Be good to have example config, plus valgrind etc.
There should be a linked docs PR as well to add the new config option to the docs.
If it should be backported then another PR targeting 1.8 branch as well.

@EKwongChum
Copy link
Author

Hi @patrick-stephens .
Should I create new PR in fluent-bit-docs for this feature ?

@patrick-stephens
Copy link
Collaborator

Hi @patrick-stephens . Should I create new PR in fluent-bit-docs for this feature ?

Yes please and link it here so we know to wait for code merge

Copy link
Collaborator

@nokute78 nokute78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some comments for maintainability.

@EKwongChum
Copy link
Author

Hi @patrick-stephens . Should I create new PR in fluent-bit-docs for this feature ?

Yes please and link it here so we know to wait for code merge

Here is the PR for Doc.

@github-actions
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@patrick-stephens
Copy link
Collaborator

@nokute78 are you happy with this now?

@nokute78
Copy link
Collaborator

nokute78 commented Jul 2, 2022

@patrick-stephens I confirmed they were fixed.
However commit message should be fixed.

@EKwongChum Sorry for late reply.
Could you check commit message style and modify them ?
e.g. out_file: some messages
https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes

@EKwongChum
Copy link
Author

@EKwongChum Sorry for late reply. Could you check commit message style and modify them ? e.g. out_file: some messages https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes

Thanks for your reply and friendly tips about commit message style.

Commit message is updated.

It is very honor since this pull request is reopen.

@patrick-stephens
Copy link
Collaborator

Unfortunately the second commit is incorrect now, can you update it as well to follow the same style?

Make sure the commits pass the DCO check as well: it'll tell you what is wrong if you click on it. Essentially they need to have the Signed-off-by: statement at the end, if you run git commit --signoff this will do it.

> Co-authored-by: Pat <[email protected]>

Signed-off-by: ekwongchum <[email protected]>
@EKwongChum
Copy link
Author

Hi.
Dear admins.
Commit message is updated
Please review

@github-actions
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Dec 29, 2022
@github-actions github-actions bot removed the Stale label Aug 15, 2024
@github-actions
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Nov 26, 2024
@github-actions github-actions bot removed the Stale label Mar 21, 2025
@github-actions
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Aug 27, 2025
@eschabell
Copy link

@patrick-stephens can you review this now that @EKwongChum addressed your changes?

@github-actions github-actions bot removed the Stale label Oct 23, 2025
Copy link
Collaborator

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok now but we need to rebase in order to trigger CI properly and ensure we are up-to-date for merging.

@patrick-stephens patrick-stephens added the ok-package-test Run PR packaging tests label Nov 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-package-test Run PR packaging tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants