FluentBit Tail Plugin DB on Azure Files.

[MalcolmPereira]

I am trying to set up Fluent Bit Pipeline on Azure AKS and Azure Files as Storage using DB parameter for tail input plugin.

I am using helm to install, simple tail input plugin and a file output plugin works great, however if I try to use SQLLite DB settings for tail plugin I get database locked error.

.....
[INPUT]
name tail
path /var/log/containers/server*.log
tag kube.*
multiline.parser docker,cri
mem_buf_limit 5MB
skip_long_lines On
#db /fluent-bit/logcollector/fluent-bit-storage/${HOSTNAME}_fluentbit.db
storage.type filesystem
...

commenting out #db /fluent-bit/logcollector/fluent-bit-storage/${HOSTNAME}_fluentbit.db works

uncommenting db /fluent-bit/logcollector/fluent-bit-storage/${HOSTNAME}_fluentbit.db

[2022/11/02 01:22:09] [ info] [storage] backlog input plugin: storage_backlog.1
[2022/11/02 01:22:09] [ info] [cmetrics] version=0.5.4
[2022/11/02 01:22:09] [ info] [ctraces ] version=0.2.5
[2022/11/02 01:22:09] [ info] [input:tail:tail.0] initializing
[2022/11/02 01:22:09] [ info] [input:tail:tail.0] storage_strategy='filesystem' (memory + filesystem)
[2022/11/02 01:22:09] [debug] [tail:tail.0] created event channels: read=21 write=22
[2022/11/02 01:22:09] [error] [sqldb] error=database is locked
[2022/11/02 01:22:09] [error] [input:tail:tail.0] db: could not create 'in_tail_files' table
[2022/11/02 01:22:09] [error] [input:tail:tail.0] could not open/create database
[2022/11/02 01:22:09] [error] failed initialize input tail.0
[2022/11/02 01:22:09] [error] [lib] backend failed
Using Fluent Bit v2.0.3

Not sure if its a limitation trying to create SQLLite DB on Azure Files share or some timing issue in Fluent Bit. I see db created on Azure
Files but 0 bytes. (edited)

Any pointers from the community - Thank you.

---

Complete Helm Values file:

kind: DaemonSet

replicaCount: 1

image:
repository: cr.fluentbit.io/fluent/fluent-bit
tag: "latest-debug"
pullPolicy: Always

extraVolumes:

• name: fluent-bit-volume
  persistentVolumeClaim:
  claimName: fluent-bit-file

extraVolumeMounts:

• name: fluent-bit-volume
  mountPath: /fluent-bit/logcollector

serviceAccount:
create: true
annotations: {}
name:

rbac:
create: true
nodeAccess: false

dnsPolicy: ClusterFirst

resources:
requests:
cpu: 250m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi

labels:
appname: server-fluentbit
aadpodidbinding: server

config:
service: |
[SERVICE]
daemon Off
flush {{ .Values.flush }}
log_level debug
parsers_file parsers.conf
parsers_file custom_parsers.conf
http_server On
http_listen 0.0.0.0
http_port {{ .Values.metricsPort }}
health_check On
storage.path /fluent-bit/logcollector/fluent-bit-storage

inputs: |
[INPUT]
name tail
path /var/log/containers/server*.log
tag kube.*
multiline.parser docker,cri
mem_buf_limit 5MB
skip_long_lines On
#db /fluent-bit/logcollector/fluent-bit-storage/${HOSTNAME}_serverfluentbit.db
storage.type filesystem

filters: |
[FILTER]
name multiline
match kube.*
multiline.key_content log
multiline.parser multiline-neo-parser

[FILTER]
    name kubernetes
    match kube.*
    merge_log On
    merge_log_key log_processed

[FILTER]
    name parser
    parser neo-parser
    match kube.*
    key_name log
    preserve_key true
    reserve_data true
    match kube.*

[FILTER]
    name lua
    match kube.*
    call getTag
    code function getTag(tag, timestamp, record) timevalue = record["time"]  kubeValue = record["kubernetes"] podName = kubeValue["pod_name"] containerName = kubeValue["container_name"] logessageValue = record["log"] logessageValue  = string.gsub(logessageValue, "\n", "",2) new_record = record new_record["log"] = logessageValue dateValue = string.sub(timevalue, 1, 10) hourValue = string.sub(timevalue, 12, 13) new_record["folderPath"] = "/"..dateValue.."/"..containerName.."/"..hourValue.."-00".."_"..podName return 1, timestamp, record end

[FILTER]
    name rewrite_tag
    match kube.*
    rule $time ^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\.(\d+)Z$ $folderPath false

upstream: {}

outputs: |
[OUTPUT]
name stdout
match *

[OUTPUT]
    name file
    match *
    mkdir true
    path /fluent-bit/logcollector
    format template
    template {log}

customParsers: |
[PARSER]
name neo-parser
format regex
regex ^[(?[A-Z ]+)],[(?\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}-\d+)],[(?\d+)],[TID=(?[a-zA-Z0-9])],[(?[a-z.A-Z]+)],(?.)|(?.*)

[MULTILINE_PARSER]
  name multiline-neo-parser
  type regex
  rule "start_state" "^\[.*"  "cont"
  rule "cont"        "^[^\[].*" "cont"
  flush_timeout 1000
  decode_field_as  escaped_utf8 log

extraFiles: {}

logLevel: debug

---

Complete Azure Files PVC/PV

apiVersion: v1
kind: PersistentVolume
metadata:
name: server-fluent-bit
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: file.csi.azure.com
readOnly: false
volumeHandle: server-fluent-bit-volumeid
volumeAttributes:
shareName: fluent-bit
nodeStageSecretRef:
name: serversecret
namespace: server
mountOptions:
- dir_mode=0777
- file_mode=0777

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fluent-bit-file
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 5Gi

If I comment out #db /fluent-bit/logcollector/fluent-bit-storage/${HOSTNAME}_fluentbit.db everything works fine

else get

[2022/11/02 01:22:09] [error] [sqldb] error=database is locked
[2022/11/02 01:22:09] [error] [input:tail:tail.0] db: could not create 'in_tail_files' table
[2022/11/02 01:22:09] [error] [input:tail:tail.0] could not open/create database
[2022/11/02 01:22:09] [error] failed initialize input tail.0
[2022/11/02 01:22:09] [error] [lib] backend failed

Complete Error:

Fluent Bit v2.0.3

• Copyright (C) 2015-2022 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

[2022/11/02 01:22:09] [ info] Configuration:
[2022/11/02 01:22:09] [ info] flush time | 1.000000 seconds
[2022/11/02 01:22:09] [ info] grace | 5 seconds
[2022/11/02 01:22:09] [ info] daemon | 0
[2022/11/02 01:22:09] [ info] ___________
[2022/11/02 01:22:09] [ info] inputs:
[2022/11/02 01:22:09] [ info] tail
[2022/11/02 01:22:09] [ info] ___________
[2022/11/02 01:22:09] [ info] filters:
[2022/11/02 01:22:09] [ info] multiline.0
[2022/11/02 01:22:09] [ info] kubernetes.1
[2022/11/02 01:22:09] [ info] parser.2
[2022/11/02 01:22:09] [ info] lua.3
[2022/11/02 01:22:09] [ info] rewrite_tag.4
[2022/11/02 01:22:09] [ info] ___________
[2022/11/02 01:22:09] [ info] outputs:
[2022/11/02 01:22:09] [ info] stdout.0
[2022/11/02 01:22:09] [ info] ___________
[2022/11/02 01:22:09] [ info] collectors:
[2022/11/02 01:22:09] [ info] [fluent bit] version=2.0.3, commit=86250fa62d, pid=1
[2022/11/02 01:22:09] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2022/11/02 01:22:09] [ info] [storage] ver=1.3.0, type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=128
[2022/11/02 01:22:09] [ info] [storage] backlog input plugin: storage_backlog.1
[2022/11/02 01:22:09] [ info] [cmetrics] version=0.5.4
[2022/11/02 01:22:09] [ info] [ctraces ] version=0.2.5
[2022/11/02 01:22:09] [ info] [input:tail:tail.0] initializing
[2022/11/02 01:22:09] [ info] [input:tail:tail.0] storage_strategy='filesystem' (memory + filesystem)
[2022/11/02 01:22:09] [debug] [tail:tail.0] created event channels: read=21 write=22
[2022/11/02 01:22:09] [error] [sqldb] error=database is locked
[2022/11/02 01:22:09] [error] [input:tail:tail.0] db: could not create 'in_tail_files' table
[2022/11/02 01:22:09] [error] [input:tail:tail.0] could not open/create database
[2022/11/02 01:22:09] [error] failed initialize input tail.0
[2022/11/02 01:22:09] [error] [lib] backend failed
[2022/11/02 01:22:09] [error] [engine] input initialization failed

Thank you

[agup006]

Currently we have limitations on NFS which Azure Files would fall under for the DB file. Do you have the ability to have the DB file on local storage?