Skip to content
This repository was archived by the owner on Jan 20, 2026. It is now read-only.
This repository was archived by the owner on Jan 20, 2026. It is now read-only.

Unable to collect systemd journal logs #30

Open
#31
Open
Unable to collect systemd journal logs#30
#31
Labels
bugSomething isn't working
@pradeep-hegde

Description

@pradeep-hegde
pradeep-hegde
opened on Jun 1, 2023

Describe the bug

VMware Aria Operations for Logs 8.12

image: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.4.0

Fluent conf:

<source>
  @type systemd
  @id in_systemd_logs
  path "/var/log/journal"
  matches [{"_SYSTEMD_UNIT":"kubelet.service"}]
  tag "kubelet"
  read_from_head true
  <storage>
    @type "local"
    persistent true
    path "/var/log/kubeletlog.pos"
  </storage>
  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  enable_stat_watcher false
  refresh_interval 5
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      keep_time_key true
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

# Enriches records with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  @id filter_kube_metadata
  kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
  verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
  ca_file "#{ENV['KUBERNETES_CA_FILE']}"
  skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
  skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'true'}"
  skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'true'}"
  skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
  watch "#{ENV['FLUENT_KUBERNETES_WATCH'] || 'true'}"
</filter>

# Prevent fluentd from handling records containing its own logs to handle cycles.
<label @FLUENT_LOG>
  <match fluent.**>
    @type null
  </match>
</label>


<filter **>
  @type record_transformer
  <record>
    log_type kubernetes
    cluster <CLUSTERNAME>
  </record>
  watch false
</filter>

<match **>
  @type vmware_loginsight
  scheme https
  ssl_verify false
  host <LOGINSIGHT_SERVER_IP>
  port 9543
  http_method post
  serializer json
  rate_limit_msec 0
  raise_on_error false
  flatten_hashes true
  http_conn_debug false
</match>

Pods ends up in CrashLoopBackOff state with error:

/usr/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library 'libsystemd.so.0': libsystemd.so.0: cannot open shared object file: No such file or directory. (LoadError)
Could not open library 'libsystemd.so': libsystemd.so: cannot open shared object file: No such file or directory.
Could not open library 'libsystemd-id128.so.0': libsystemd-id128.so.0: cannot open shared object file: No such file or directory.
Could not open library 'libsystemd-id128.so': libsystemd-id128.so: cannot open shared object file: No such file or directory
        from /usr/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `map'
        from /usr/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `ffi_lib'
        from /usr/lib/ruby/gems/2.7.0/gems/systemd-journal-1.4.2/lib/systemd/id128.rb:48:in `<module:Native>'
        from /usr/lib/ruby/gems/2.7.0/gems/systemd-journal-1.4.2/lib/systemd/id128.rb:45:in `<module:Id128>'
        from /usr/lib/ruby/gems/2.7.0/gems/systemd-journal-1.4.2/lib/systemd/id128.rb:6:in `<module:Systemd>'
        from /usr/lib/ruby/gems/2.7.0/gems/systemd-journal-1.4.2/lib/systemd/id128.rb:3:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/systemd-journal-1.4.2/lib/systemd/journal/native.rb:1:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:2:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:17:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:110:in `new_input'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:315:in `add_source'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:161:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:155:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:155:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- systemd/journal (LoadError)
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:17:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:110:in `new_input'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:315:in `add_source'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:161:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:155:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:155:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

Reproduction steps

  1. Use systemd source in fluent conf
  2. Run fluent plugin pod with latest release image

Expected behavior

Pods are expected to be in a running state and kubelet logs are to be shown in Aria Operations for Logs Server with given tag for kubelet logs and cluster

Additional context

Related to the fluent plugin for systemd issue, plugin expects libsystemd package to be installed in the docker image. fluent/fluentd-docker-image#49

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions