storage management

[caplam]

Version

2.4.170

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

32G

Storage for /

278G

Storage for /nsm

568G

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

hello,

I'm trying to better understand S.O. I have some difficulties with storage management.

As i understand it /nsm is the directory where almost all collected and processed data are.

/nsm/suripcap for packet captures
/nsm/elasticsearch for all data related to elastic search
/nsm/zeek for zeek logs

What i find difficult is that /nsm storage space is used concurently by pcap, elasticsearch and zeek
Keep in mind that for now i tried so only in standalone mode.

I read about ILM but it only concerns elastic search data.
afaik the hot, warm and cold indices are managed by node: e.g you affect a node as warm, hot or cold data tier.
For S.O i suppose these nodes are search nodes.
In my case (homelab with only 1 physical host with 1 more host soon), i would need to add search node as cold tier.
I think you may have guessed that i'm looking to better use my storage. I can't have all the data on ssd.

Is there a way to have all data in /nsm spread accross hdd or ssd according to how frequent they are accessed?
In order to have a better idea of traffic in my network i recently setup zabbix with an agent on my firewall (opnsense). I will have insight on inter vlan traffic.
The monitor i use until now is lan inbound & outbound, vlans inbound & outbound. This traffic flows through 3 2,5gbe interfaces. The monitor is on a 10gbps interface

My precedent S.O standalone instance barely reach 6 days of pcap retention with a 1Tb vdisk and ends up with problems in elastic search indices.
For pcap i restricted capture to 1Mb.

Wouldn't it be better to have the ability to configure nsm more granularly for all kind of data S.O produce and access?
My next install will have more resources (192Gb ram, 8Tb ssd, 24 cores on a proxmox host) but still spinning disks could be better used if you can use them on the same host that use ssd.

How people plan storage for their S.O setup? I would prefer a standalone installation but if distributed is mandatory i can do that.
Should i have 2 search node for a physical host : 1 to use ssd and the second with hdd ?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

From https://docs.securityonion.net/en/2.4/architecture.html#standalone This type of deployment is typically used for testing, labs, POCs, or very low-throughput environments. If you are monitoring a good amount of throughput then you would want to separate the nodes. You could run a distributed managersearch and sensor if you have resources for the managersearch.

For Elastic data, the longer your retention goal the more storage. If you have two search nodes, one with SSD, one with regular HDD, you could run them in data tiers. Hot data (most recent and searched) on the SSD search node and warm/cold data on the HDD search node. I have seen managersearch + search node + sensor node grids, but if you want to separate all the node roles you would go fully distributed.

[caplam]

thank you for your answer.

If i understand correctly there is no way to use hdd and ssd in the same node as the only mount point known by S.O is /nsm.
i'd like to be able to have the choice for. each type of data (eg: elasticsearch, suripcap, zeek ) a storage path for recent and old files.
For now my /nsm partition looks like this:
`12G ./docker-registry

3.5G ./repo
3.0G ./elastic-fleet
36M ./kratos
118M ./rules
359M ./influxdb
8.4G ./backup
0 ./soc
174G ./suripcap
0 ./pcap
0 ./import
0 ./pcapout
98G ./elasticsearch
4.0K ./logstash
4.0K ./redis
4.3M ./suricata
56G ./zeek
57M ./strelka
0 ./custom-mappings
279M ./securityonion-resources
0 ./airgap-resources
353G .`

In my case suripcap takes quite some space. It could make sense to have a sensor node. But then no way to choose between hdd and ssd.
Is it realistic to have a sensor node with hdd? According to zabbix my average throughput is around 15Mbps with spikes to 2Gbps. 15Mbps on hdd is realistic but during spikes i could have a huge capture loss ratio. I guess i may have to switch to zfs with caching.
I could also have a search node with ssd and another one with ssd.
At least with a standalone node to begin i could see how storage is distributed among the different type of data.

[caplam]

I feel a bit dumb as i wasn't aware of lvm-cache.
When my new device arrive I will reinstall SO on proxmox with 1 vdisk on proxmox datastore for system and hdd & ssd passthrough which i will organize with lvm as lvm volume with dm-cache on the ssd.
Does it sound go to you?
I don't think my traffic volume needs a distributed setup.
I still need to figure out how to export backups to my nas.

[reyesj2]

I would still lean towards distributed setup. It can be as simple as a managersearch + sensor. That way there is separation between your PCAP / raw network traffic logs and elasticsearch. It also gives you much more flexibility for down the line if you want to expand your grid.