out_syslog: handle multivalued structured data #8355

bplessis · 2024-01-06T22:31:46Z

When generating a syslog SD-ELEMENT it is permitted to repeat a SD-PARAM multiple times:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

This is used by loggly to pass tags alongside syslog messages for example https://documentation.solarwinds.com/en/success_center/loggly/content/admin/tags.htm

This proposal convert array-typed object into multiple SD-PARAM elements to be able to satisfy this need.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

bplessis · 2024-01-06T22:33:00Z

Example configuration file for the change:

[INPUT]
  Name  dummy
  Dummy {"hostname":"test.example.org", "appname": "my-application", "procid": "123", "msgid": "ID01", "log": "test log message", "uls@0": {"tag":[ "tag1", "tag2", "tag3", "tag4"], "key": "value" } }

[OUTPUT]
    name                 syslog
    match                *
    host                 127.0.0.1
    port                 5140
    mode                 udp
    syslog_format        rfc5424
    syslog_maxsize       2048
    syslog_hostname_key  hostname
    syslog_appname_key   appname
    syslog_procid_key    procid
    syslog_msgid_key     msgid    
    syslog_sd_key        uls@0
    syslog_message_key   log

bplessis · 2024-01-06T22:36:12Z

Debug run

Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

       \ || /
       = ** =
        /__\ 
       /____\
      /______\
     /________\
    /__________\
   /____________\
  /__Fluent Bit__\
 /__|__v2.2.1_|___\
/___\_________/____\
        |||
        |||
       =====

[2024/01/06 23:35:23] [ info] Configuration:
[2024/01/06 23:35:23] [ info]  flush time     | 1.000000 seconds
[2024/01/06 23:35:23] [ info]  grace          | 5 seconds
[2024/01/06 23:35:23] [ info]  daemon         | 0
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  inputs:
[2024/01/06 23:35:23] [ info]      dummy
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  filters:
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  outputs:
[2024/01/06 23:35:23] [ info]      syslog.0
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  collectors:
[2024/01/06 23:35:23] [ info] [fluent bit] version=2.2.2, commit=f2692ed152, pid=1158178
[2024/01/06 23:35:23] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/01/06 23:35:23] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/01/06 23:35:23] [ info] [cmetrics] version=0.6.6
[2024/01/06 23:35:23] [ info] [ctraces ] version=0.4.0
[2024/01/06 23:35:23] [ info] [input:dummy:dummy.0] initializing
[2024/01/06 23:35:23] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/01/06 23:35:23] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/01/06 23:35:23] [debug] [syslog:syslog.0] created event channels: read=23 write=24
[2024/01/06 23:35:23] [ info] [output:syslog:syslog.0] setup done for 127.0.0.1:5140 (TLS=off)
[2024/01/06 23:35:23] [ info] [sp] stream processor started
[2024/01/06 23:35:24] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:25] [trace] [task 0x7f9ac4018ef0] created (id=0)
[2024/01/06 23:35:25] [debug] [task] created task=0x7f9ac4018ef0 id=0 OK
[2024/01/06 23:35:25] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:25] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:25] [debug] [out flush] cb_destroy coro_id=0
[2024/01/06 23:35:25] [trace] [coro] destroy coroutine=0x7f9ac4019010 data=0x7f9ac4019030
[2024/01/06 23:35:25] [debug] [task] destroy task=0x7f9ac4018ef0 (task_id=0)
[2024/01/06 23:35:26] [trace] [task 0x7f9ac4019300] created (id=0)
[2024/01/06 23:35:26] [debug] [task] created task=0x7f9ac4019300 id=0 OK
[2024/01/06 23:35:26] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:26] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:26] [debug] [out flush] cb_destroy coro_id=1
[2024/01/06 23:35:26] [trace] [coro] destroy coroutine=0x7f9ac4019420 data=0x7f9ac4019440
[2024/01/06 23:35:26] [debug] [task] destroy task=0x7f9ac4019300 (task_id=0)
[2024/01/06 23:35:27] [trace] [task 0x7f9ac40195f0] created (id=0)
[2024/01/06 23:35:27] [debug] [task] created task=0x7f9ac40195f0 id=0 OK
[2024/01/06 23:35:27] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:27] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:27] [debug] [out flush] cb_destroy coro_id=2
[2024/01/06 23:35:27] [trace] [coro] destroy coroutine=0x7f9ac40196e0 data=0x7f9ac4019700
[2024/01/06 23:35:27] [debug] [task] destroy task=0x7f9ac40195f0 (task_id=0)
^C[2024/01/06 23:35:28] [engine] caught signal (SIGINT)
[2024/01/06 23:35:28] [trace] [engine] flush enqueued data
[2024/01/06 23:35:28] [trace] [task 0x7f9ac4019830] created (id=0)
[2024/01/06 23:35:28] [debug] [task] created task=0x7f9ac4019830 id=0 OK
[2024/01/06 23:35:28] [ warn] [engine] service will shutdown in max 5 seconds
[2024/01/06 23:35:28] [ info] [input] pausing dummy.0
[2024/01/06 23:35:28] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:28] [debug] [out flush] cb_destroy coro_id=3
[2024/01/06 23:35:28] [trace] [coro] destroy coroutine=0x7f9ac40191b0 data=0x7f9ac40191d0
[2024/01/06 23:35:28] [debug] [task] destroy task=0x7f9ac4019830 (task_id=0)
[2024/01/06 23:35:28] [ info] [engine] service has stopped (0 pending tasks)
[2024/01/06 23:35:28] [ info] [input] pausing dummy.0

bplessis · 2024-01-06T22:38:56Z

Valgrind run

==1157749== Memcheck, a memory error detector
==1157749== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==1157749== Using Valgrind-3.20.0 and LibVEX; rerun with -h for copyright info
==1157749== Command: bin/fluent-bit -c bin/test.conf
==1157749== 
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

       \ || /
       = ** =
        /__\ 
       /____\
      /______\
     /________\
    /__________\
   /____________\
  /__Fluent Bit__\
 /__|__v2.2.1_|___\
/___\_________/____\
        |||
        |||
       =====

[2024/01/06 23:22:52] [ info] [fluent bit] version=2.2.2, commit=f2692ed152, pid=1157749
[2024/01/06 23:22:52] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/01/06 23:22:52] [ info] [cmetrics] version=0.6.6
[2024/01/06 23:22:52] [ info] [ctraces ] version=0.4.0
[2024/01/06 23:22:52] [ info] [input:dummy:dummy.0] initializing
[2024/01/06 23:22:52] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/01/06 23:22:52] [ info] [output:syslog:syslog.0] setup done for 127.0.0.1:5140 (TLS=off)
[2024/01/06 23:22:52] [ info] [sp] stream processor started
[2024/01/06 23:22:54] [ warn] [engine] failed to flush chunk '1157749-1704579773.514668660.flb', retry in 8 seconds: task_id=0, input=dummy.0 > output=syslog.0 (out_id=0)
[2024/01/06 23:22:56] [ warn] [engine] failed to flush chunk '1157749-1704579775.501811336.flb', retry in 7 seconds: task_id=1, input=dummy.0 > output=syslog.0 (out_id=0)
[2024/01/06 23:23:02] [ info] [engine] flush chunk '1157749-1704579773.514668660.flb' succeeded at retry 1: task_id=0, input=dummy.0 > output=syslog.0 (out_id=0)
[2024/01/06 23:23:03] [ info] [engine] flush chunk '1157749-1704579775.501811336.flb' succeeded at retry 1: task_id=1, input=dummy.0 > output=syslog.0 (out_id=0)
^C[2024/01/06 23:23:07] [engine] caught signal (SIGINT)
[2024/01/06 23:23:07] [ warn] [engine] service will shutdown in max 5 seconds
[2024/01/06 23:23:07] [ info] [input] pausing dummy.0
[2024/01/06 23:23:08] [ info] [engine] service has stopped (0 pending tasks)
[2024/01/06 23:23:08] [ info] [input] pausing dummy.0
==1157749== 
==1157749== HEAP SUMMARY:
==1157749==     in use at exit: 0 bytes in 0 blocks
==1157749==   total heap usage: 2,299 allocs, 2,299 frees, 6,031,600 bytes allocated
==1157749== 
==1157749== All heap blocks were freed -- no leaks are possible
==1157749== 
==1157749== For lists of detected and suppressed errors, rerun with: -s
==1157749== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

When generating a syslog SD-ELEMENT it is permitted to repeat a SD-PARAM multiple times: https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3 This is used by loggly to pass tags alongside syslog messages for example https://documentation.solarwinds.com/en/success_center/loggly/content/admin/tags.htm This proposal convert array-typed object into multiple SD-PARAM elements to be able to satisfy this need. Signed-off-by: Benoit Plessis <[email protected]>

Documentation change for PR fluent/fluent-bit#8355 Signed-off-by: Benoit Plessis <[email protected]>

bplessis-swi · 2024-03-04T16:11:55Z

Do i need to do something more to help this ?
@edsiper @leonardo-albertovich @fujimotos @koleini ?

github-actions · 2024-06-08T01:51:12Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

bplessis-swi · 2024-06-08T08:28:10Z

how can we proceed now ?
@edsiper @leonardo-albertovich @fujimotos @koleini ?

github-actions · 2024-12-14T02:07:37Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

bplessis-swi · 2024-12-27T07:58:10Z

No sure what is missing here ..
how can we proceed now ?
@edsiper @leonardo-albertovich @fujimotos @koleini ?

github-actions · 2025-09-06T02:05:02Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

eschabell · 2025-10-21T14:10:09Z

Docs for this PR available here: fluent/fluent-bit-docs#1292

eschabell · 2025-10-21T14:11:04Z

@edsiper can someone review this PR?

bplessis requested review from edsiper, fujimotos, koleini and leonardo-albertovich as code owners January 6, 2024 22:31

github-actions bot added the docs-required label Jan 6, 2024

bplessis force-pushed the out_syslog_multi_value_sd branch from c962f08 to 560b645 Compare January 7, 2024 16:57

bplessis temporarily deployed to pr January 8, 2024 15:48 — with GitHub Actions Inactive

bplessis temporarily deployed to pr January 8, 2024 16:08 — with GitHub Actions Inactive

bplessis added a commit to bplessis/fluent-bit-docs that referenced this pull request Jan 26, 2024

out_syslog: handle multivalued structured data

6225b23

Documentation change for PR fluent/fluent-bit#8355 Signed-off-by: Benoit Plessis <[email protected]>

bplessis mentioned this pull request Jan 26, 2024

out_syslog: handle multivalued structured data fluent/fluent-bit-docs#1292

Open

github-actions bot added the Stale label Jun 8, 2024

github-actions bot removed the Stale label Jun 11, 2024

github-actions bot added the Stale label Dec 14, 2024

github-actions bot removed the Stale label Dec 28, 2024

github-actions bot added the Stale label Sep 6, 2025

github-actions bot removed the Stale label Oct 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

out_syslog: handle multivalued structured data #8355

out_syslog: handle multivalued structured data #8355

Uh oh!

bplessis commented Jan 6, 2024 •

edited

Loading

Uh oh!

bplessis commented Jan 6, 2024

Uh oh!

bplessis commented Jan 6, 2024

Uh oh!

bplessis commented Jan 6, 2024

Uh oh!

bplessis-swi commented Mar 4, 2024

Uh oh!

github-actions bot commented Jun 8, 2024

Uh oh!

bplessis-swi commented Jun 8, 2024

Uh oh!

github-actions bot commented Dec 14, 2024

Uh oh!

bplessis-swi commented Dec 27, 2024

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

eschabell commented Oct 21, 2025

Uh oh!

eschabell commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

out_syslog: handle multivalued structured data #8355

Are you sure you want to change the base?

out_syslog: handle multivalued structured data #8355

Uh oh!

Conversation

bplessis commented Jan 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bplessis commented Jan 6, 2024

Example configuration file for the change:

Uh oh!

bplessis commented Jan 6, 2024

Debug run

Uh oh!

bplessis commented Jan 6, 2024

Valgrind run

Uh oh!

bplessis-swi commented Mar 4, 2024

Uh oh!

github-actions bot commented Jun 8, 2024

Uh oh!

bplessis-swi commented Jun 8, 2024

Uh oh!

github-actions bot commented Dec 14, 2024

Uh oh!

bplessis-swi commented Dec 27, 2024

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

eschabell commented Oct 21, 2025

Uh oh!

eschabell commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bplessis commented Jan 6, 2024 •

edited

Loading