Monitoring SO servers with SNMP or similar

[rosswakelin]

We want to be able to monitor our SO servers using our network and systems management platform so that we get alerted when something stops working. Our platform can use basic ping etc. tests to see if the host is up, but also allows for more comprehensive monitoring using SNMP.
Is it possible to configure the SNMP agents on the SO servers (is it event included in the image?) and if so, will the configs be overwritten each time the platform is upgraded/updated?
Or has anyone else any recommendations about how to do enterprise level platform monitoring of a SO installation?

thanks

Ross

[dougburks]

Most folks monitor their Security Onion deployments using the included Grafana as it's already configured to monitor the most important health metrics:
https://docs.securityonion.net/en/2.3/grafana.html

If you feel the need to monitor uptime externally you could certainly augment with the basic ping tests from your external system.

[rosswakelin]

Thanks for that. We don't have the staff/time to have someone look at Grafana for this sort of stuff, but we do have people dedicated to responding to network and systems management events 24/7 through the centralised management platform. We want to get an alert if a node goes offline for some reason (that in itself may be an indication of a breach) or if the suricata service stops on a sensor node, but the node itself is still responding to pings.
DebianGuru suggested manually adding the snmpd package to the servers, but our concerns around that included:

• ISO updates (we are airgapped) might (!) remove that package
• updates/salt might(!) remove the config files
  Can we get a statement about support for these sort of 3rd party packages inside the SO deployments, or an indication if they might be supported in a future release?

Thanks

[dougburks]

We can't provide any guarantees or support for those 3rd party packages.

[DebianGuru]

You have two steps. First is to install and configure snmpd (If you're monitoring other Linux based machines, you probably know how to do that. If not, you'll have to google it. It's fairly easy and a simple google should return some tutorials on it.
The second step, and more tricky is to get the firewall to allow traffic into the SO box. Unfortunately, there's no easy "so-allow" that will work for you. Of course, the firewall is managed by salt (what isn't in SO?). Read through https://docs.securityonion.net/en/2.3/firewall.html. You may need to read it a few times to understand it. I'm no expert, but I got this working by reading that page (especially the "Allow hosts to send syslog to a sensor node" section. Here's a summary of how I did it.
"sudo so-firewall ?" will show various options and syntax.

1. sudo so-firewall addportgroup snmpports (You could call the groupname whatever you want.)
2. sudo so-firesall addport snmp udp 161
3. sudo so-firewall addhostgroup snmpgroup (Again, call the group whatever make sense to you.)
4. sudo so-firewall includehost snmpgroup www.xxx.yyy.zzz (include the IP of your SNMP polling machine.)

So, you've made a port group called snmpports, and added UDP 161 to that. Then you added a group called snmpgroup and added the IP of your polling machine to that group.

The last step is to make add the entry into your minion.sls file, so the changes will be made to the firewall.
It is just like in the example on the website, except I realized that the entry had to be made into the INPUT chain (if you've used iptables before you may understand that the INPUT chain applies to traffic addressed to the machine).

In my minion.sls file, I added this:
firewall:
assigned_hostgroups:
chain:
INPUT:
hostgroups
snmpgroup:
portgroups:

• portgroups.snmpports
  (Be careful in the above portion. Spacing is very important. Each line adds two blank spaces before the text. And make sure to only use spaces, no tabs).

I hope that helps. Like you, I'm just muddling my way through, but the people on here, especially the SO staff are usually very helpful and I can't speak highly enough of their company.

[IzacFrank]

Hello @DebianGuru

I'm followed your instructions but I get time out error when check with snmpwalk or snmpget.

I created a user like the one below. Could you please share some screenshots from your config (minions, snmpd.conf)?

sudo net-snmp-config --create-snmpv3-user -ro -A PASSWORD-X privpass -a MD5 -x AES USER

THank you very much in advance.
Cheers,
Isac

[DebianGuru]

Just to clarify for you and others posting on this thread, Yes, I know you can use Grafana to monitor. I use it too for detailed info on SO. But I also use Nagios as a central solution to monitor all my network equipment and sending alerts. I want to alerted if the SO box gets short on resources, or maybe high temperature, etc.

IzakFrank,
You didn't specify what machine you were running this command from. I'm assuming it's on another machine.
I think I'd start by doing this:

1. First, see if SNMPd is running "sudo ps aux |grep snmp". Maybe SNMPd isn't even running or installed correctly. You should see something like this:

Debian-+ 572978 0.1 0.0 32864 5608 ? Ss Aug31 32:50 /usr/sbin/snmpd -LOw -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f -p /run/snmpd.pid`.

2. Try running an SNMP query from the SO box to itself (You may have to install snmp first. If using the Ubuntu, apt-get install snmp. Not sure how any Redhat releases do package managment). Run "snmpwalk -c YourString -v1 -H 127.0.0.1". I use version 1 because it's the easiest. It's RO mode and SNMPd only allows one host to access it. If you want to fiddle with SNMP v3, that's fine, but I'd try getting v1 to work first. When I run it, I get something like this returned (just posting the last few lines.)

defPrivLocalizedKey      string
defVersion               1|2c|3
defSecurityLevel         noAuthNoPriv|authNoPriv|authPriv

In snmpapp.conf and snmpapp.local.conf:
defDomain application domain
defTarget application domain target
sourceFilterType none|whitelist|blacklist
sourceFilterAddress host
engineID string
engineIDType num
engineIDNic string

3. My minions.sls file contains (at the end).

firewall:
. . assigned_hostgroups:
. . . . chain:
. . . . . . INPUT:
. . . . . . . . hostgroups:
. . . . . . . . . . snmp:
. . . . . . . . . . . . portgroups:
. . . . . . . . . . . . . . - portgroups.snmp

Pay close attention to the spacing of each line! the "firewall:" is at position 0 (left most) and each following line is indented two spaces for each line. Because this website keeps removing the leading spaces, I'm going to show them with a ". ")

You can run the following command which should look to see if you're allowing SNMP traffic through your firewall:

"sudo iptables -L |grep snmp"

ACCEPT udp -- AllowedMachine.domain.com anywhere udp dpt:snmp

"AllowedMachine.domain.com" is the machine you want to monitor your SO box. It may be a name or IP, depending on if you have a reverse DNS entry for it in your domain.

"sudo iptables -l |less" should show that stanza is indeed in the Chain INPUT.

If you can get a simple snmpwalk from the SO box to work, then try running one from the IP you allowed in the snmpd.conf file (I attached a sample based off of mine). If it works on the SO box, but not the other box, then it's likely a firewall issue.

Here's an example of my minions.sls file.
SNMPExample.txt

If you're having issues, with SNMPv3 specifically, I probably can't help. Let me know how it goes. Good luck and God bless you.

[IzacFrank]

Thank you so much @DebianGuru
I'm really thankful for your post and the time you spent on this. It took time to confirm it is working because another company was adding it to their monitoring servers.

Here is my SNMP V3 config for anyone needing for SO CENTOS box. It is standalone so I made the changes on global.SLS and not on the host SLS file.

$ yum install net-snmp net-snmp-utils net-snmp-devel -y
$ sudo systemclt stop snmpd
$ sudo systemclt status snmpd

$ sudo so-firewall addportgroup snmpports
$ sudo so-firewall addport snmpports udp 161
$ sudo so-firewall addhostgroup snmpgroup
$ sudo so-firewall includehost snmpgroup MyMonitoringServer1-IP
$ sudo so-firewall includehost snmpgroup MyMonitoringServer2-IP

$ sudo net-snmp-create-v3-user -ro -A MyV3Password -X privpass -a MD5 -x AES Myv3User
$ sudo vi /opt/so/saltstack/local/pillar/global.sls

$ sudo systemctl start snmpd
$ sudo systemclt status snmpd
$ sudo salt-call state.apply firewall
$ sudo salt '*' state.highstate # I did in any case, maybe not needed.
$ snmpwalk -v3 -u Myv3User -l authNoPriv -a MD5 -A MyV3Password localhost

Thanks again @DebianGuru

[TOoSmOotH]

https://www.youtube.com/watch?v=8FmZ4MRe8Uk

You don't need to actively monitor anything you can have Grafana send you an email if there is an issue.