Parsing logs from fluent bit daemon set

[miguelvr]

I just had an incident where fluent-bit was generating tons of error logs, but none of them were parsed.

I can't seem to find the correct way to set-up fluent-bit to output structured logs from its own daemon set.

My current config:

  fluent-bit.conf: |
    [SERVICE]
      Flush         1
      Log_Level     info
      Daemon        off
      Parsers_File  parsers.conf
      HTTP_Server   On
      HTTP_Listen   0.0.0.0
      HTTP_Port     2020

    @INCLUDE input.conf
    @INCLUDE filter.conf
    @INCLUDE output.conf

  input.conf: |
    [INPUT]
      Name              tail
      Tag               kube.*
      Path              /var/log/containers/*.log
      Parser            cri
      DB                /var/log/flb_kube.db
      Mem_Buf_Limit     20MB
      Skip_Long_Lines   On
      Refresh_Interval  10
      Buffer_Chunk_Size 256k
      Buffer_Max_Size   10240k

  parsers.conf: |
    [PARSER]
      Name          cri
      Format        regex
      Regex         ^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$
      Time_Key      time
      Time_Format   %Y-%m-%dT%H:%M:%S.%L%z

  filter.conf: |
    [FILTER]
      Name                kubernetes
      Match               kube.*
      Kube_URL            https://kubernetes.default.svc:443
      Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
      Merge_Log           On
      Keep_Log            Off
      Labels              On
      Annotations         Off
      K8S-Logging.Parser  Off
      K8S-Logging.Exclude Off

    [FILTER]
      Name          modify
      Match         kube.*
      Hard_rename   log message

    [FILTER]
      Name          modify
      Match         kube.*
      Hard_rename   msg message

    [FILTER]
      Name          modify
      Match         kube.*
      Hard_rename   level severity

    [FILTER]
      Name          modify
      Match         kube.*
      Hard_rename   trace         logging.googleapis.com/trace
      Hard_rename   spanId        logging.googleapis.com/spanId
      Hard_rename   traceSampled  logging.googleapis.com/trace_sampled

  output.conf: |
    [OUTPUT]
      name        stdout
      match       kube.*

[patrick-stephens]

You might want to have a look at #5712