First time user - documentation about nb of processors to allocate Zeek and Surricata

[b-u-g-s]

Hi,

I am doing a basic installation of Security onion 2.3.140.
On a VM with 4 processors allocated to it.
It is setup as a standalone installation running all the default settings/services.
In the initial setup/installation, towards the end the installer asks how many processors we want to allocate to Zeek and Suricata.
the default is 1
I couldn't find in your documentation how much processors should be allocated depending on how much you have for the Security Onion server (VM or not).

The install process does not seem to automatically detect how much processors there is and adapt it's default value (of 1).
Should I allocate 2 for each? as I have 4?
Or should I leave it to the default of 1?

I have always allocated 4 processors for my SO VM and kept the default settings but thought I better ask in case I am "wasting" 2x available processors which could help speed up Zeek/Suricata's work

Thanks,
Bugs.

[isaacgolding]

Security Onion is quite CPU Intensive. You wont find a lot of advanced instruction for virtual machines because its not fully supported by the team. However having said that use all 4 CPUs for your install.

copied from the docs

Standalone Deployments
In a standalone deployment, the manager components and the sensor components all run on a single box, therefore, your hardware requirements will reflect that. You’ll need at minimum 16GB RAM, 4 CPU cores, and 200GB storage. At the bare minimum of 16GB RAM, you would most likely need swap space to avoid issues.

This deployment type is recommended for evaluation purposes, POCs (proof-of-concept) and small to medium size single sensor deployments. Although you can deploy Security Onion in this manner, it is recommended that you separate the backend components and sensor components.

CPU: Used to parse incoming events, index incoming events, search metatadata, capture PCAP, analyze packets, and run the frontend components. As data and event consumption increases, a greater amount of CPU will be required.
RAM: Used for Logstash, Elasticsearch, disk cache for Lucene, Suricata, Zeek, etc. The amount of available RAM will directly impact search speeds and reliability, as well as ability to process and capture traffic.
Disk: Used for storage of indexed metadata. A larger amount of storage allows for a longer retention period. It is typically recommended to retain no more than 30 days of hot ES indices.
Please refer to the Architecture section for detailed deployment scenarios.

[xfaith]

Well thats wierd, I have my file open right now and have the zeek_pins and suripins.
I would validate you are looking at the right file. I am also on a .140 VM assigned 2 CPU, 2 cores.
I know way under powered, its just for playing.

[b-u-g-s]

So it looks like a similar setup, yup, it is not there. I am looking at the only .sls file in the following directory:
/opt/so/saltstack/local/pillar/minions

Maybe it is because it is a brand new install from a .140 image.
Could it be that you upgraded from lower version?

that's what I would have done if my setup didn't have an issue upgrading to .130, somehow it didn't like the transition to Case. But as this is more or less a test system I don't mind reinstalling everything!

[xfaith]

Mine is a new install .140 standalone VM.

[b-u-g-s]

Ok, I checked again and it is not there. Maybe some gremlins in the machine...
Can you please tell me if you can see zeek_lbprocs and suriprocs under the "sensor:" section and what number do you have there.
I wonder what this is for...

[b-u-g-s]

Actually, it is in the doc, it doesn't say what it is but it needs to be the same number as the number of proc being "pinned"

[dougburks]

I couldn't find in your documentation how much processors should be allocated depending on how much you have for the Security Onion server (VM or not).

https://docs.securityonion.net/en/2.3/hardware.html#cpu

The install process does not seem to automatically detect how much processors there is and adapt it's default value (of 1).

Yes, it does:
https://github.com/Security-Onion-Solutions/securityonion/blob/master/setup/so-functions#L183

If necessary, you can change these values after Setup:
https://docs.securityonion.net/en/2.3/suricata.html#performance
https://docs.securityonion.net/en/2.3/zeek.html#performance

[b-u-g-s]

Hi Doug,

I missed that part, thanks.
So actually, I need to ADD the "suripins:" and "zeek_pins:" sections, and not modify the numbesr on the suriprocs and seek_lbprocs lines correct?

So to be sure I got this correctly, with a CPU with 4 processors I should be adding the following to the .sls file:

suripins:

• <cpu_1>
• <cpu_2>

and

zeek_pins:

• <cpu_3>
• <cpu_4>

Literally, writing "<cpu_1>" and 2, 3, 4

Also, you said the setup automatically adapts to the number of processors, but in my case it never does. I always allocate a 4x processors CPU for SO and it always only offers me 1 processor for Suricata and 1 for Zeek during the install process (Hyper-V or ESX). I understand VM are not supported officially, so maybe it is a "problem" with VMs. When I SSH to SO, I can see 4x processors in TOP.

Cheers,
Bugs.

[dougburks]

Also, you said the setup automatically adapts to the number of processors, but in my case it never does.

Yes, it does. My previous response included a link to the calculation:
https://github.com/Security-Onion-Solutions/securityonion/blob/master/setup/so-functions#L183

In your case, you have 4 cores so 4 / 2 - 1 = 1. So for a machine with only 4 cores, the recommended starting point for Zeek workers is 1 and Suricata workers is 1.

The next question is: how much traffic are you actually monitoring?

From https://docs.securityonion.net/en/2.3/hardware.html#cpu:
A very rough ballpark estimate would be 200Mbps per Suricata worker or Zeek worker. So if you have a fully saturated 1Gbps link and are running Suricata and Zeek, then you’ll want at least 5 Suricata instances and 5 Zeek workers, which means you’ll need at least 10 CPU cores for Suricata and Zeek with additional CPU cores for Stenographer and/or other services.

So if you're monitoring a small amount of traffic (less than 200Mbps), then 1 Zeek worker and 1 Suricata worker may be entirely sufficient.

If they are not sufficient and you end up dropping traffic, then you may need to follow https://docs.securityonion.net/en/2.3/suricata.html#performance to increase suriprocs to 2 and/or https://docs.securityonion.net/en/2.3/zeek.html#performance to increase zeek_lbprocs to 2.

With only 4 cores, you probably don't want to go above 2 Suricata workers and 2 Zeek workers and you most likely don't need to worry about pinning at all.

[b-u-g-s]

Oh ok, sorry! I stand corrected then!
And it isn't much traffic, I don't usually get drops so that's indeed enough. I just wanted to tune the settings as bets as possible.
Now I know it will automatically detect the best config for the number of processors from the installation process.

Thanks.