No logs are read on systemd 253+ potentially

[ksandrmatveyev]

Hello,
We have an issue that appears after upgrading to Fedora CoreOS 37.20230401.3.0 to 38.20230414.3.0 (later version are affected as well). It might be related to systemd/journald packages upgrade (full list of changes can be found here https://fedoraproject.org/coreos/release-notes/?arch=x86_64&stream=stable)

Details

Distributive: Fedora CoreOS 38.20230414.3.0 and later
Systemd version: systemd 253 (253.7-1.fc38)
Fluentd: v1.16-1 as a container process in systemd unit service
User: example - rootless user (added to the group systemd-journal)

Config

fluentd.service:

[Unit]
Description=Fluentd
Requires=afterburn.service
After=afterburn.service systemd-logind.service network-online.target
[Install]
WantedBy=multi-user.target
[Service]
User=example
Restart=on-failure
EnvironmentFile=/run/metadata/afterburn
RestartSec=30
SyslogIdentifier=fluentd
ExecStartPre=/bin/bash -c '/bin/loginctl enable-linger $(/usr/bin/id -u example)'
ExecStartPre=/usr/bin/podman pull fluent/fluentd:v1.16-1
ExecStartPre=-/usr/bin/podman rm -f fluentd
ExecStart=/usr/bin/podman run --name fluentd --log-opt max-size=10m  --log-opt max-file=3 -p 24224:24224 -p 24231:24231 -v /var/log/journal:/var/log/journal:ro -v /etc/fluentd:/fluentd/etc:ro fluent/fluentdv1.16-1
:v1.16-1

fluentd.conf:

<source>
  @type systems
  @id systemd-example
  tag example
  path /var/log/journal
  read_from_head false
  matches [{ "_SYSTEMD_UNIT": "example.service" }] # replace with any real service
  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<match example>
  @type stdout
  @id systemd-stdout-example
</match>

Issue

No logs are read from systemd. No logs from stdout about any fluentd entry (same as in https://github.com/fluent-plugin-systemd/fluent-plugin-systemd#when-i-look-at-fluentd-logs-everything-looks-fine-but-no-journal-logs-are-read-)

Expected results

Logs are read and send to stdout

Workaround

Use Fedora CoreOS 37.20230401.3.0 and older (systemd 251)

Btw, running of fluent-bit with similar config works correct (same user, similar fluent and systemd config).

[errm]

I could be wrong, but it looks like the container you are using might be alpine based. You would need to use a container with libsystemd installed for this plugin to work!

The fluentd container images recommend using the debain variant for this plugin https://github.com/fluent/fluentd-docker-image/tree/master#debian-included-tag

[kalaksi]

This issue also points to an incompatibility between systemd-versions: fluent/fluent-bit#6581
I'm currently also struggling to get this to work and my setup did work previously, but broke at some point.

The docker hub image fluentd:v1.16.2-debian-1.1 seems to contain version 247.3-7+deb11u4 of libsystemd0 while my host has version 252.