Once rabbitmq is restarted, fluentBit pods are unable to connect to it #2
Description
We are using this utility to read few component log files and filter for alarms and send the processed alarm entry to rabbitmq which will then be picked up by Alarm management system.
Observation :
- RabbitMQ (deployed as Microservice) was down and restarted..
- After rabbitMQ is up, a new component alarm got generated.
- This generated alarm logs are not pushed to Rabbitmq.
And notice following error
2021/11/24 13:21:15 Couldn't publish record: : Exception (504) Reason: "channel/connection is not open"
NOTE: This utility - fluent-bit-rabbitmq-output - is deployed as a daemonSet. Only after restart of the daemonSet, logs are forwarded to rabbitmq.
On further checking, we see that in output_rabbitmq.go following function is responsible for rabbitMQ connection -> func FLBPluginInit
And as per the comment in the code, this func is called only once during the instance startup.
// Gets called only once for each instance you have configured.
Could you please suggest a workaround for this issue ?