High Server load

[Crimson1110]

Hello community,

I am a newby for Security Onion. I am testing Security Onion 2.4.30 under single Proxmox host with 2x E5-2430L v3 (20M Cache, 1.80 GHz), 256Gb RAM and Raid5 SSD Crucial MX500 with 5 disks. The Manager Node & Search node has 12 cores / 64 Gb Ram. Both have java heapsize of 16Gb (InfluxDB shows the use percent flows under 12Gb). Logstash has 4Gb of heap. The sensor node has 6 cores / 16Gb. All have 400Gb of hardisk space.

I added to fleet around 20 linux machines which I use for hosting my personal websites / mail / Proxmox etc. The linux VMs other than SO are very idle in terms of load and network. InfluxDB shows iodelay as 0.0%. Playbook is on.

I did different combinations of heap size, CPU, total memory etc. but I am not able to lower the server load increasing to 16-18 within 2 days after I turn on the SO VMs . The SO VMs seems to utilize only 50-60% CPU on average.

What am I missing?

Thank you in advance.

[dougburks]

I am not able to lower the server load increasing to 16-18 within 2 days after I turn on the SO VMs

Which server load are you referring to? The Proxmox host?

[Crimson1110]

There is no swapping in any hosts or Proxmox.

There are no pinned VMs.

The idea to move the search node to entirely another server with new set of SSD disks was to eliminate that as well. But just to follow you I created a new RAID1 with 2 new SSD's and moved the Manager Node disk to the new LVM. Nothing changed. Also it seems its averaging around 0.5Mb write which is nothing and if there was a problem with disks I would expect I/O in Influxdb would show up something.

The problem could be related to Garbage Collection "young_collection_time_in_millis" as it keeps on growing (600k!). I do not know, it is getting complicated for me...

[dougburks]

The problem could be related to Garbage Collection "young_collection_time_in_millis" as it keeps on growing (600k!). I do not know, it is getting complicated for me...

This isn't a setting that we commonly change. Which component are you referring to?

There are no pinned VMs.

You might try pinning VMs to a particular socket to see if that makes any difference. In particular, you will most likely want to pin your sensor node to the same CPU that services your sniffing NIC:
https://docs.securityonion.net/en/2.4/zeek.html?highlight=numa#performance
https://docs.securityonion.net/en/2.4/suricata.html?highlight=numa#performance

Here's another idea to try. It seems like you're not monitoring that much network traffic and you're not consuming a huge amount of endpoint logs. So if you haven't already, you might try turning off all your existing Security Onion VMs and then creating a single VM, installing Security Onion in Standalone mode with default settings. See how that runs for a while and then try pinning that VM to the same CPU that services your sniffing NIC.

[Crimson1110]

This isn't a setting that we commonly change. Which component are you referring to?

I was able to check "ParallelGCThreads" & "NewRatio" which seems ok to me.

Here's another idea to try. It seems like you're not monitoring that much network traffic and you're not consuming a huge amount of endpoint logs. So if you haven't already, you might try turning off all your existing Security Onion VMs and then creating a single VM, installing Security Onion in Standalone mode with default settings. See how that runs for a while and then try pinning that VM to the same CPU that services your sniffing NIC.

I was using standalone and decided to test the distributed version as I was hoping to introduce SO to some clients at some time.

In the end, disabling the Playbook, where I enabled lots of rules, "kinda" did the trick. The search node was around 2.5 load on the dedicated Proxmox host and on the other host with all the VMs + SO Manager + SO Sensor, the load dropped down to 5. I moved the search node back to the host where others VMs are and the load is now around 9.4 - 9.8. I hope it will stay like this.

Perhaps I will try to turn the Playbook again with fever rules enabled in the future. :(

Thank you very much for your help.

[dougburks]

In the end, disabling the Playbook, where I enabled lots of rules, "kinda" did the trick. The search node was around 2.5 load on the dedicated Proxmox host and on the other host with all the VMs + SO Manager + SO Sensor, the load dropped down to 5. I moved the search node back to the host where others VMs are and the load is now around 9.4 - 9.8. I hope it will stay like this.

Yes, enabling lots of plays in Playbook will generate lots of Elasticsearch queries. Specifically, did you enable the Malicious Nishang PowerShell Commandlets play or any experimental plays?

From https://docs.securityonion.net/en/2.4/playbook.html#putting-a-play-into-production:
Performance testing is still ongoing. We recommend avoiding the Malicious Nishang PowerShell Commandlets play as it can cause serious performance problems. You may also want to avoid others with a status of experimental.

[Crimson1110]

In the light of this conversation I suggest an edit to the "Read before posting" part:

• Did you read manual?
• Yes, at least 3 times.
• Read it again.

Thanks a lot. :)

I will enable only the rules that matter, and will be very careful about the ones that i'll write..

[dougburks]

I've updated the documentation and highlighted this as a WARNING: