Collect journald / systemd logs from Kubernetes nodes

[ghost]

Hello,

We are trying to configure Kubernets nodes logs collecting using JournalD source.

There are almost no full/working examples around and just brief discussions in the Discord and GitHub.

1. Environment - Kubernetes
2. Nodes - openSUSE MicroOS
3. journalctl_path - /usr/bin/journalctl
4. journal_directory - /var/log/journal
5. Docker images - 0.44.0-distroless-libc, 0.44.0-distroless-static, 0.44.0-debian, 0.44.0-alpine

values.yaml

# Image
image:
  repository: timberio/vector
  base: debian

# Volumes
extraVolumes:
  - name: log-journalctl
    hostPath:
      path: /var/log/journal

extraVolumeMounts:
  - name: log-journalctl
    mountPath: /var/log/journal
    readOnly: true

# Config
customConfig:
  data_dir: /var/lib/vector

  sources:
    journald-source:
      type: journald
      journal_directory: /var/log/journal

  transforms:
    journald-transform-elasticsearch:
      inputs:
      - journald-source
      source: |
        .index_name = "journald"
      type: remap

  sinks:
    journald-sink-elasticsearch:
      type: elasticsearch
      inputs:
        - journald-transform-elasticsearch
      encoding:
        except_fields: ["index_name"]
      endpoints:
        - https://elastic-es-http:9200
      tls:
        verify_certificate: false
        verify_hostname: false
      api_version: auto
      compression: gzip
      batch:
        max_bytes: 1000000
        timeout_secs: 5
      mode: bulk
      bulk:
        action: index
        index: '{{ printf "{{ index_name }}-%%Y.%%m.%%d" }}'

Issues

1. We need more working examples to not check different sources, trying to understand how to configure it properly and all possible drawbacks.

2. A single way which was working is to use 0.44.0-debian image, without additional volumes mounting.

   # Volumes
   extraVolumes:
     - name: log-pods
       hostPath:
         path: /var/log/pods
     - name: log-containers
       hostPath:
         path: /var/log/containers
     - name: log-journalctl
       hostPath:
         path: /var/log/journal
     # - name: journalctl-binary
     #   hostPath:
     #     path: /usr/bin/journalctl
     # - name: journalctl-libs
     #   hostPath:
     #     path: /usr/lib64/systemd
     # - name: journalctl-extralibs
     #   hostPath:
     #     path: /usr/lib64/libacl.so.1s
   
     extraVolumeMounts:
       - name: log-pods
         mountPath: /var/log/pods
         readOnly: true
       - name: log-containers
         mountPath: /var/log/containers
         readOnly: true
       - name: log-journalctl
         mountPath: /var/log/journal
         readOnly: true
       # - name: journalctl-binary
       #   mountPath: /usr/bin/journalctl
       #   readOnly: true
       # - name: journalctl-libs
       #   mountPath: /usr/lib64/systemd
       #   readOnly: true
       # - name: journalctl-extralibs
       #   mountPath: /usr/lib64/libacl.so.1s
       #   readOnly: true

3. When we are using default image, it is required to mount to Pod additionally journalctl binary and if required set journalctl_path.

4. That doesn't work because we miss libraries

   /usr/lib64/systemd/libsystemd-core-257.so
   /usr/lib64/systemd/libsystemd-shared-257.so

   We can mount /usr/lib64/systemd --> /usr/lib64/systemd, but we miss more libraries - the next one was /usr/lib64/libacl.so.1s

   ldd /usr/bin/journalctl 
   
       linux-vdso.so.1 (0x00007fa1a08ce000)
       libsystemd-shared-257.so => /usr/lib64/systemd/libsystemd-shared-257.so (0x00007fa1a0400000)
       libc.so.6 => /lib64/libc.so.6 (0x00007fa1a0000000)
       libacl.so.1 => /lib64/libacl.so.1 (0x00007fa1a08a1000)
       libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fa1a0863000)
       libcap.so.2 => /lib64/libcap.so.2 (0x00007fa1a0857000)
       libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fa1a03c5000)
       libmount.so.1 => /lib64/libmount.so.1 (0x00007fa1a0378000)
       libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fa19fa00000)
       libpam.so.0 => /lib64/libpam.so.0 (0x00007fa1a0841000)
       libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007fa1a0349000)
       libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fa1a031a000)
       libm.so.6 => /lib64/libm.so.6 (0x00007fa1a022f000)
       /lib64/ld-linux-x86-64.so.2 (0x00007fa1a08d0000)
       libeconf.so.0 => /lib64/libeconf.so.0 (0x00007fa1a0221000)
       libz.so.1 => /lib64/libz.so.1 (0x00007fa1a0207000)
       libjitterentropy.so.3 => /lib64/libjitterentropy.so.3 (0x00007fa19fff4000)
       libaudit.so.1 => /lib64/libaudit.so.1 (0x00007fa19ffc9000)
       libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fa19f950000)
   

5. We can try to mount whole /usr/lib64 --> /usr/lib64 but got another error related to the gclibc or so.

6. Setting LD_LIBRARY_PATH is a wrong way?