Config files are included in arbitrary order when using `@INCLUDE` command with a wildcard

[lht]

Bug Report

Describe the bug
Config files are included in arbitrary order when using @INCLUDE command with wildcard characters

To Reproduce

Populate some config files.

cat <<EOF > /tmp/flb-20.conf
[FILTER]
  Name modify
  Alias rename_key
  Match *
  Condition Key_exists service
  rename service svc
EOF

cat <<EOF > /tmp/flb-10.conf
[FILTER]
  Name modify
  Alias add_key
  Match *
  Condition Key_exists tool
  Add service hello
EOF

Confirm the files are listed in the order such as:

# ls --sort=none /tmp | grep ^flb.*conf
flb-20.conf
flb-10.conf

Main configuration file: main.conf

[SERVICE]
  Flush           5
  Daemon          off

[INPUT]
  Name dummy
  Dummy  {"tool": "fluent"}
  Tag dummy

[OUTPUT]
  Name stdout
  Match *

@INCLUDE /tmp/flb-*.conf

Start fluent-bit,

# /fluent-bit/bin/fluent-bit -c main.conf
Fluent Bit v1.9.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/06/22 12:34:50] [ info] [fluent bit] version=1.9.5, commit=9ec43447b6, pid=163
[2022/06/22 12:34:50] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/06/22 12:34:50] [ info] [cmetrics] version=0.3.4
[2022/06/22 12:34:50] [ info] [sp] stream processor started
[2022/06/22 12:34:50] [ info] [output:stdout:stdout.0] worker #0 started
[0] dummy: [1655901290.831520000, {"tool"=>"fluent", "service"=>"hello"}]
...

Expected behavior
Expect flb-10.conf is included before flb-20.conf and a field svc=>hello exists in the final event.

[0] dummy: [1655901294.831573400, {"tool"=>"fluent", "svc"=>"hello"}]

Also in the step of populating the config files, If flb-10.conf is created before flb-20.conf, we get the expected result.

Your Environment

• Version used: 1.9.5
• Operating System and version
  Linux 674dd909c68f 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 GNU/Linux

Additional context

We have multiple configuration files for our Kubernetes clusters and we combine them with Kustomize configmapGenerator. Wildcard is used in @INCLUDE command because different clusters require different combinations of config files. We need certain config files be included in determined order and expect the glob matched files are sorted by name.

GLOB_NOSORT is used when globing the files

fluent-bit/src/config_format/flb_cf_fluentbit.c

Line 158 in 7668f91

ret_glb = glob(glb_path, GLOB_NOSORT, NULL, &glb);

This probably is not needed. According to https://man7.org/linux/man-pages/man3/glob.3.html

   GLOB_NOSORT
          Don't sort the returned pathnames.  The only reason to do
          this is to save processing time.  By default, the returned
          pathnames are sorted.

[patrick-stephens]

I don't think this is a bug but an enhancement request (the current implementation is to not sort them rather than sort them but it is not doing that) - could you submit a PR for the change? We should probably have a docs ones as well to cover both the current behaviour and a linked one to your PR for any change.

I would say that relying on an ordering is not a good idea in general, presumably you are doing it to order filters? Generally I've done that by ensuring any filtering order is within the include file itself and the actual order of includes does not affect the processing pipeline but it can be tricky particularly if you want dynamic file inclusion. You can even nest the includes within another included file to order them that way.

Just trying to understand the use case mainly rather than a critique in any way.

[lht]

@patrick-stephens I think It's fair that this is considered an enhancement. 👍

Yes we'd like to have a guaranteed filters ordering across files included with wildcards. The use case is using Kustomize configmapGenerator to combine various filters in different scenarios. One actual use case is we have a lua filter to inject a field XYZ at the end of the filter pipeline. There is another parser filter with Reserve_Data=False, which may produce the same field XYZ. We want to make sure the lua filter is always executed at last and value of XYZ should be the one injected by the lua filter only.

The idea is we want to reuse individual filter configuration as much as possible and combine them with Kustomize. Putting the lua filter to multiple config files would not be ideal. A workaround is explicitly @includeing the lua filter in the main configuration file. But since the arbitrary ordering behaviour from GLOB_NOSORT have and may cause future hard to debug issues, it'll be great if the glob order is presorted.

I'll prepare a patch.

[patrick-stephens]

Excellent, could you submit a docs PR as well both for current (I can merge straight away) and your upcoming one? fluent/fluent-bit-docs

We may need to indicate this explicitly for anyone currently relying on the no-sort order (I'm not sure how you would!) but it is a change in behaviour - possibly file creation ordering or whatever your OS did by default for no-sort.