Fluentd drainer job cannot complete when old chunks with no associated config are present

[aslafy-z]

Describe the bug:
leftover buffers with no associated configs (flow / output does not exist anymore) are never drained, so drain-watch does not kill fluentd and buffer-volume-sidecar and the drainer jobs finishes with error because of timeout.

Expected behaviour:
Drainer job skips old chunks with no associated config.
Better: Drainer job is executed on config update so no orphan chunks stays in the buffers.

Steps to reproduce the bug:

• Delete an output with chunks still in the pipe.
• Once Fluentd is reloaded, these chunks (in /buffers) are now unmatched and will never be sent to the destination and stay forever in the folder.
• If you scale down this fluentd, the drainer job will stay stuck forever as the script since this condition never becomes true https://github.com/banzaicloud/logging-operator/blob/76533182e425e00ade40d8d9d8b3ffadda6c4548/drain-watch-image/drain-watch.sh#L23

Workaround:

for each errored drainer pods (with associated logging-operator-logging-fluentd-XX pod):

• remove the drainer pod and wait for it to be recreated
• exec to the drainer pod and empty the /buffers directory

Environment details:

• Kubernetes version (e.g. v1.15.2): v1.20.9
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): RKE
• logging-operator version (e.g. 2.1.1): 3.17.9
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs): Let me know if you need something
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data: N/A

/kind bug

[siliconbrain]

TL;DR: This is by design and the suggested solution is what you mention as workaround. The buffer dropping flag could be implemented as an enhancement iff it's feasible.

---

By default, it's better to call operators' attention to the situation (by the way of job failure) than to lose logs silently. There could be a configuration option to drop orphaned buffers, but I'm not sure how complex it would be to determine whether a buffer is orphaned.

Also, I'm not sure how practical it would be to run the drain job on config update, because it might lengthen the config update or even make it fail e.g. if the update was done to get rid of an obsolete/erroneous output that won't accept logs anyway.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

We've been discussing a possible solution where we would create configuration snapshots and drainer pods would be tied to a snapshot instead of a moving target.

In my opinion this should be an opt-in mechanism that would work in an immutable manner.

That means configuration changes wouldn't automatically trigger a change in the existing deployment, but rather spin up a new cluster with the new config and redirect traffic there.

Once the new cluster is up and running the existing cluster can be scaled down with the original config with the drainer as usual.