System periodically unresponsive

[sblanch]

Home Network (on-prem)
Intel i5 Processor (4 core)
Security Onion 2.3.100 Standard Install from ISO (then upgrade applied)
1 Node
CentOS 7 Install
16GB RAM (unfortunately the max the system can hold)
1TB SSD System Disk
2TB SSD /nsm (Only have 1TB partitioned currently as I copied my old 1TB spinning disk)
Avg 1.8Mb/s on Monitor Interface over 24 hours
Avg RAM usage (92%) over 24 hours
Avg CPU 22% over 24 hours
All services (so-status) green

The last time I posted (Playbook impact on performance #6837), I believe the issue was an IOWait issue of a failing system disk (which I replaced with an SSD).

After the system is up for over a week, I start getting times when the system interface is unresponsive for 5+ minutes, then returns to normal. Below is the Grafana dashboard. I'm trying to figure out what kicks off to cause this issue. I have verified that neither disk (both new) are showing any issues.

Any suggestions on what I should investigate would be appreciated.

Regards,
Scott

[dougburks]

Are you able to share Grafana graphs from a 15 minute window around the 5 minute unresponsiveness (5 minutes before the unresponsiveness began and 5 minutes after it stopped)?

Have you checked the logs in /opt/so/log/ for additional clues?

[sblanch]

Here are some more graphs.

Happy to add any other graphs that would be useful.

I'll start digging into the logs. Any suggestions on which you would search first as there are a bunch?

Thanks!
Scott

[dougburks]

Is this happening on a regular basis?

Do you have playbook enabled?

If so, do you have any experimental plays enabled (specifically Malicious Nishang PowerShell Commandlets)?

[sblanch]

In general, it seems to regularly happen once I get over 7 days of uptime.
I have one Playbook enabled, #478 Registry Persistence Mechanisms.

Given I only have 16GB RAM (and run on average at 93% memory utilization) should I have swap enabled since my disks are SSD. I don't know if it becomes memory starved at certain moments, and then whatever is happening causes a cascading effect.
Ironically, it appears memory usage drops when the problem occurs (I'm guessing something must crash/restart?).

Looking at the Elasticsearch logs, I do see it throw a java error just before one of the spikes. I can grab more of any of the logs that may help.

[2022-03-07T07:13:51,846][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.so-v2.seeds] from [[]] to [["127.0.0.1:9300"]]
[2022-03-07T07:13:54,981][WARN ][rest.suppressed ] path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, track_total_hits=true} org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:927) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:761) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:397) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:485) [elasticsearch-7.16.3.jar:7.16.3] ...
Happy to provide any additional log detail as necessary.

Regards,
Scott

[dougburks]

Yes, please try enabling swap. From https://docs.securityonion.net/en/2.3/hardware.html#standalone-deployments:

You’ll need at minimum 16GB RAM, 4 CPU cores, and 200GB storage. At the bare minimum of 16GB RAM, you would most likely need swap space to avoid issues.

[sblanch]

Ok, I have created a 4GB Swapfile, and I'll monitor to see how it affects the system.
Thanks!