[in_systemd_docker] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address

[hariskmohd]

Describe the bug

I am using fluentd-kubernetes-daemonset:v1.18-debian-opensearch-1 to send logs from my EKS cluster to opensearch. I am getting the following error in some of my pods

2025-04-01 02:57:55 +0000 [warn]: #0 [in_systemd_docker] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address
2025-04-01 02:57:55 +0000 [warn]: #0 [in_systemd_kubelet] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address
2025-04-01 02:57:55 +0000 [warn]: #0 [in_systemd_bootkube] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address

Below is the output of my fluentd starup log to know more about the configuration details

2025-04-01 08:11:04 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2025-04-01 08:11:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2025-04-01 08:11:05 +0000 [info]: gem 'fluentd' version '1.18.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.16'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.6.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.5'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-systemd' version '1.1.1'
2025-04-01 08:11:06 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type systemd
    @id in_systemd_kubelet
    matches [{"_SYSTEMD_UNIT":"kubelet.service"}]
    read_from_head true
    tag "kubelet"
    <storage>
      @type "local"
      persistent true
      path "/var/log/fluentd-journald-kubelet-cursor.json"
    </storage>
  </source>
  <source>
    @type systemd
    @id in_systemd_docker
    matches [{"_SYSTEMD_UNIT":"docker.service"}]
    read_from_head true
    tag "docker.systemd"
    <storage>
      @type "local"
      persistent true
      path "/var/log/fluentd-journald-docker-cursor.json"
    </storage>
  </source>
  <source>
    @type systemd
    @id in_systemd_bootkube
    matches [{"_SYSTEMD_UNIT":"bootkube.service"}]
    read_from_head true
    tag "bootkube"
    <storage>
      @type "local"
      persistent true
      path "/var/log/fluentd-journald-bootkube-cursor.json"
    </storage>
  </source>
  <source>
    @type prometheus
    @id in_prometheus
    bind "0.0.0.0"
    port 24231
    metrics_path "/metrics"
  </source>
  <source>
    @type prometheus_output_monitor
    @id in_prometheus_output_monitor
  </source>
  <label @FLUENT_LOG>
    <match fluent.**>
      @type null
      @id ignore_fluent_logs
    </match>
  </label>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/fluentd-cluster-autoscaler.log.pos"
    tag "cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    tag "kubernetes.*"
    exclude_path []
    read_from_head true
    <parse>
      @type "cri"
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/fluentd-etcd.log.pos"
    tag "etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/fluentd-glbc.log.pos"
    tag "glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/kube-apiserver-audit.log.pos"
    tag "kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/fluentd-kube-apiserver.log.pos"
    tag "kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/fluentd-kube-controller-manager.log.pos"
    tag "kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/fluentd-kube-proxy.log.pos"
    tag "kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/fluentd-kube-scheduler.log.pos"
    tag "kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/fluentd-kubelet.log.pos"
    tag "kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/fluentd-rescheduler.log.pos"
    tag "rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_minion
    path "/var/log/salt/minion"
    pos_file "/var/log/fluentd-salt.pos"
    tag "salt"
    <parse>
      @type "regexp"
      expression /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
      time_format "%Y-%m-%d %H:%M:%S"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
    kubernetes_url "https://10.100.0.1:443/api"
    verify_ssl true
    ca_file ""
   skip_labels false
    skip_container_metadata false
    skip_master_url false
    skip_namespace_metadata false
    watch true
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    remove_keys $['kubernetes']['labels']['app.kubernetes.io/instance'],$['kubernetes']['labels']['app.kubernetes.io/managed-by'],$['kubernetes']['labels']['app.kubernetes.io/version'],$['kubernetes']['labels']['app.kubernetes.io/name'],$['kubernetes']['labels']['app.kubernetes.io/component'],$['kubernetes']['labels']['app.kubernetes.io/part-of']
  </filter>
  <match **>
    @type opensearch
    @id out_os
    @log_level "error"
    include_tag_key true
    host "*********************************************************"
    port 443
    path ""
    scheme https
    ssl_verify true
    ssl_version TLSv1_2
    user "*******************"
    password xxxxxx
    index_name "logs"
    logstash_dateformat "%Y.%m.%d"
    logstash_format true
    logstash_prefix "logs"
    logstash_prefix_separator "-"
    <buffer>
      flush_thread_count 1
      flush_mode interval
      flush_interval 60s
      chunk_limit_size 8M
      total_limit_size 512M
      retry_max_interval 30
      retry_timeout 72h
      retry_forever false
    </buffer>
    <endpoint>
      url https://vpc-pm-hpl-opensearch-5jlny6in26kkudopanxxbzsi74.us-east-1.es.amazonaws.com
      region "us-east-1"
    </endpoint>
  </match>
</ROOT>
2025-04-01 08:11:06 +0000 [info]: starting fluentd-1.18.0 pid=7 ruby="3.2.6"
2025-04-01 08:11:06 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/opensearch_simple_sniffer.rb", "--under-supervisor"]
2025-04-01 08:11:08 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2025-04-01 08:11:08 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2025-04-01 08:11:08 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2025-04-01 08:11:08 +0000 [info]: adding filter pattern="kubernetes.**" type="record_transformer"
2025-04-01 08:11:08 +0000 [info]: adding match pattern="**" type="opensearch"
2025-04-01 08:11:09 +0000 [info]: adding source type="systemd"
2025-04-01 08:11:09 +0000 [info]: adding source type="systemd"
2025-04-01 08:11:09 +0000 [info]: adding source type="systemd"
2025-04-01 08:11:09 +0000 [info]: adding source type="prometheus"
2025-04-01 08:11:09 +0000 [info]: adding source type="prometheus_output_monitor"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"

When I restart the pod, it recovers automatically, but I want to know what exactly the issue is.

To Reproduce

Not sure how the issue occured

Expected behavior

There shouldn't be any warnings

Your Environment

- Tag of using fluentd-kubernetes-daemonset: fluentd-kubernetes-daemonset:v1.18-debian-opensearch-1

Your Configuration

Mentioned in description

Your Error Log

Mentioned in description

Additional context

No response

[Watson1978]

Hmm, sounds it might be issue which is related to fluent-plugins-nursery/fluent-plugin-systemd#119

I'm guessing It might be occurred when systemd journal was re-opened.

Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address,
is this message continuously outputted?

[Watson1978]

I guess this issue will be solved by ledbettj/systemd-journal#109

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days