2.3.260: Suricata not generating Alerts on 1 FW-node

[Jaap79]

Hi,

We're running a grid of 90+ sensor nodes. Everything seems to be in order, however, one recently added sensor node is not generating Suricata Alerts.

I do get a ton of Zeek logs (HTTP, SMB-files, DNS, the whole shabang), but 0 Suricata alerts even after waiting for a week. I can manually trigger a curl https://testmynids.org/uid/index.html on the sensor node and that will generate a Suricata Alert in the Alerts overview as well as the Dashboards. But if I trigger the same thing on a workstation within the network covered by this node, nothing happens. I do however see the connection to that domain, but no Alert is triggered.

Kind of last as to troubleshooting this. Suricata seems to work, the /nsm/suricata on the sensor creates (empty) logs but will list the manually triggered Alert when I run this from the sensor node itself. So I'd say communication and ingesting still works.

The docker logs so-suricata shows nothing particular:

26/10/2023 -- 18:32:12 - - This is Suricata version 6.0.13 RELEASE running in SYSTEM mode
26/10/2023 -- 18:32:15 - - all 2 packet processing threads, 4 management threads initialized, engine started.

(These are the only 2 lines it'll display right now).

'top' also shows suricata-main running and using some CPU (anywhere between 0.5%, and 5.0%-ish). so-status is fully green et cetera.

Since Zeek is running fine and a manually triggered alert from the sensor node works fine, it would seem Suricata is doing it's thing. Just not when it's supposed to look through the actual captured data??

Anyone have any idea on how to proceed/what to look at next ?

[reyesj2]

Does that particular sensor have a different $HOMENET than globally set?

Check & compare the values from the manager node

grep hnmanager /opt/so/saltstack/local/pillar/global.sls && \
grep hnsensor /opt/so/saltstack/local/pillar/minions/$SENSORNAME_sensor.sls

[Jaap79]

They weren't set the same - I've adjusted it, restarted the sensor node, made no difference though :(

[Jaap79]

I can't reproduce the same thing anymore from the sensor node itself either by the way. Suricata is running fine (it says) but nothing is happening. Zeek and other logs are working just fine.

tail -50 /opt/so/log/suricata/stats.log:

app_layer.tx.ikev2 | Total | 0
app_layer.flow.krb5_tcp | Total | 0
app_layer.tx.krb5_tcp | Total | 0
app_layer.flow.dhcp | Total | 28
app_layer.tx.dhcp | Total | 129
app_layer.flow.snmp | Total | 342
app_layer.tx.snmp | Total | 684
app_layer.flow.sip | Total | 0
app_layer.tx.sip | Total | 0
app_layer.flow.rfb | Total | 0
app_layer.tx.rfb | Total | 0
app_layer.flow.rdp | Total | 0
app_layer.tx.rdp | Total | 0
app_layer.flow.failed_tcp | Total | 0
app_layer.flow.dcerpc_udp | Total | 0
app_layer.tx.dcerpc_udp | Total | 0
app_layer.flow.dns_udp | Total | 1392
app_layer.tx.dns_udp | Total | 3798
app_layer.flow.nfs_udp | Total | 0
app_layer.tx.nfs_udp | Total | 0
app_layer.flow.krb5_udp | Total | 0
app_layer.tx.krb5_udp | Total | 0
app_layer.flow.failed_udp | Total | 3336
flow.mgr.full_hash_pass | Total | 15
flow.mgr.closed_pruned | Total | 0
flow.mgr.new_pruned | Total | 0
flow.mgr.est_pruned | Total | 0
flow.mgr.bypassed_pruned | Total | 0
flow.spare | Total | 9501
flow.emerg_mode_entered | Total | 0
flow.emerg_mode_over | Total | 0
flow.mgr.rows_maxlen | Total | 2
flow.mgr.flows_checked | Total | 8317
flow.mgr.flows_notimeout | Total | 2775
flow.mgr.flows_timeout | Total | 5542
flow.mgr.flows_timeout_inuse | Total | 0
flow.mgr.flows_evicted | Total | 5542
flow.mgr.flows_evicted_needs_work | Total | 336
flow_bypassed.closed | Total | 0
flow_bypassed.pkts | Total | 0
flow_bypassed.bytes | Total | 0
tcp.memuse | Total | 607584
tcp.reassembly_memuse | Total | 212992
http.memuse | Total | 0
http.memcap | Total | 0
ftp.memuse | Total | 0
ftp.memcap | Total | 0
app_layer.expectations | Total | 0
file_store.open_files | Total | 0
flow.memuse | Total | 7394304

and the last part of the /opt/so/log/suricata/suricata.log:

28/10/2023 -- 07:15:31 - - This is Suricata version 6.0.13 RELEASE running in SYSTEM mode
28/10/2023 -- 07:15:31 - - CPUs/cores online: 4
28/10/2023 -- 07:15:31 - - Setting engine mode to IDS mode by default
28/10/2023 -- 07:15:31 - - dropped the caps for main thread
28/10/2023 -- 07:15:31 - - eve-log output device (regular) initialized: /nsm/eve-%Y-%m-%d-%H:%M.json
28/10/2023 -- 07:15:31 - - stats output device (regular) initialized: stats.log
28/10/2023 -- 07:15:31 - - Running in live mode, activating unix socket
28/10/2023 -- 07:15:32 - - 1 rule files processed. 35146 rules successfully loaded, 0 rules failed
28/10/2023 -- 07:15:32 - - Threshold config parsed: 0 rule(s) found
28/10/2023 -- 07:15:32 - - 35149 signatures processed. 1240 are IP-only rules, 5300 are inspecting packet payload, 28582 inspect application layer, 0 are decoder event only
28/10/2023 -- 07:15:34 - - Going to use 1 thread(s)
28/10/2023 -- 07:15:34 - - Running in live mode, activating unix socket
28/10/2023 -- 07:15:34 - - Using unix socket file '/var/run/suricata/suricata-command.socket'
28/10/2023 -- 07:15:34 - - all 1 packet processing threads, 4 management threads initialized, engine started.
28/10/2023 -- 07:15:34 - - Using BPF 'not port 9993' on iface 'bond0'
28/10/2023 -- 07:15:34 - - All AFP capture threads are running.

(this was since the last reboot, no new lines were added after that).

There is an 'eve-2023-10-28-07:15.json log in /nsm/suricata but it's 0 bytes, just like the ones from the past 2 days (the last log with an actual entry was the testmynids.org/uid/index.html that I fired from the sensor node)

The ONE thing I haven't checked (or wasn't able to):

The network segment that the node is connected to runs at 172.17.8.x/23 ... so there's a 23-bit subnet. The docker0 interface runs at 172.17.0.1/24 .. that shouldn't interfere, right ? All the other logs are processed just fine...

[reyesj2]

If you're getting zeek logs, but not suricata alerts its likely your homenet variable. Suricata isn't triggering alerts for networks outside its 'homenet'. Check out https://docs.securityonion.net/en/2.3/homenet.html?highlight=homenet#homenet and ensure that the homenet for that sensor is correctly configured