Certain Syslogs Not Arriving

[jdonovan-itcnp]

Version: 2.3.260
On-Prem
Ubuntu 20.04

Seeing an issue where syslogs from specific devices only are not populating into elastic indices. Issue started around the same time another issue occurred which knocked log aggregation offline for a little while, resolved by various reboots and increasing logstash memory allocations. Several devices are still logging successfully, but others are not.

As far as I can tell no other changes were made. I see packets arriving on port 514 and I can read the contained syslog messages, but they do not appear to be getting as far as filebeat in order for ingestion to occur. I am wondering if the syslog module or other component of filebeat was damaged during prior issues.

Is there anywhere I can look for more information? Filebeat logs don't tell me much about incoming syslog or other errors. I do see errors about failing to get tcp and udp stats, but those seem to pre-date this issue.

Is there any way to pull a fresh container image in case corruption has occurred?

[cm-ops]

Do you see any issues in /opt/so/log/logstash/logstash.log? You could docker stop and docker rm the so-filebeat container then salt-call state.apply filebeat to start a new container.

[jdonovan-itcnp]

Really appreciate this information, good to know there is a way to reset to a fresh container!

Turns out this issue was on the other side - device logs must have been malformed in some way. Issue was resolved and working now.