10Gbps Forward Nodes hardware

[KimkimCyber]

I'd like to build a Proof of concept with Security Onion, starting with 10Gbps, but due to budgetary restrictions I'm forced to save money wherever possible.
One of the goals is to store full pcaps with a retention of a few days. At 10Gbps, one day will already exceed 80TB.

My question is, can I use SAS HDD instead of SSD for the forward nodes?
In other words, will HDDs bottleneck and cause packet drops? And if so, would adding Smart NICs like Napatech or Accolade overcome this problem?

Especially when the analysts will query for Pcaps during Hunts.

I'd like to use RAID 6 over RAID 10 as the latter would cut the total storage in half.
Of course, if the only solution in order to use HDD is using RAID10 I will use this configuration.
The forward nodes will have to process Zeek, Suricata and Strelka as well.

I will probably use older gen hardware and was looking at E5-2699AV4 and/or AMD EPYC 7763.

Are there any known issues running SO 2.4 with these CPU's and related motherboards?

All inputs are much appreciated :-)

Thanks!

[InfosecGoon]

You're anticipating sustained traffic of 10Gb/sec?

[TOoSmOotH]

Unless you are going to load balance over multiple sensors you can rule out spinny. You would need to use large capacity U.2 NVME drives to get the appropriate speeds to cap that on a single box.

[petiepooo]

If you're really ingesting at 10Gbps, and writing to two disks for RAID10, that's already 20Gbps of writes. RAID6 increases that to 30Gbps and adds a bunch of reads as well to be able to calculate parity. Since most disk controller cards don't have more than 4 channels on the cards themselves, they use SAS expanders to handle multiple HDDs per channel, meaning even RAID10 could exceed one channel's 12Gbps bandwidth on the card unless you're really careful about which drives you bind as mirrors. An option would be to forego disk redundancy entirely and rebuild on failure, or have server-level redundancy instead.

Last I checked, even SOS says they don't support PCAP at 10Gbps with their beefiest appliance. It's simply unrealistic to do what you want on a low budget, I think.

I suspect your best bet is using a network packet broker (NPB) to split your traffic into multiple forward nodes handling no more than 2-3Gbps each, but brokers with SFP28s ingress ports are far from cheap. There's been VSS (now Netscout) and Gigamon for ages, and a few others, plus Garland is a newer player with their PacketMAX line.

If I had the budget, I'd start with a NPB feeding four or more forward nodes each with 4x20TB+ LFF HDDs (60TB+ in RAID5), a manager node and separate receiver node, and three search nodes using SSDs or NVMe. You could stick with 1RU servers like the Dell R6515 but you're still talking about 9-10RU and close to $100k even before the NPB... Or... If you have that to spend, enlist SOS support directly and buy their appliances.

Please let us all know what you come up with and how it works!

[petiepooo]

Going through your question again, a couple other notes:

I've run Security Onion on both Intel E5-series and AMD EPYC with no issues. As long as they're x86_64, you're fine.

It's possible you could forego a NPB by using an intelligent NIC on each forward node instead, having them filter out all but their portion of the IP address space. I have no experience doing that, but know you would want to have the NIC apply BPF before it sends it to the OS as you wouldn't want all 10Gbps traversing the kernel to zeek, suricata and stenographer before a filter is applied. You also wouldn't get the dynamic balancing and failover features a NPB could offer as you would need to balance the load manually between each node.

That's just a different way to split it up, which you would still need to do.

[KimkimCyber]

@petiepooo Thanks for sharing your technical experties. I will use a packetbroker for load balancing over 2 or 3 forward nodes.
I will also deploy a few receiver nodes and several search nodes.
I will update from time to time how things are going :)

[petiepooo]

Best of luck. Sounds like a fun project!

FWIW, another advantage to the broker is that you can dial down the bandwidth you're sending the nodes and ramp it up as you see they can take it. That would allow you to start smaller and ramp with additional nodes as you see the need. I haven't built a grid with separate search or receiver nodes yet, so have no clue how much they scale and how many you would need except that 3 search nodes is the minimum to form an HA cluster. The manager is a receiver as well, so the separate receiver is only for redundancy and additional throughput.

[Acewiza]

Just curious, but who or what's the perspective on this God-like telco-level view into a chunk 'o fiber? Sounds like really interesting state-sponsored or Big Tech-level stuff...

[KimkimCyber]

Unless you are going to load balance over multiple sensors you can rule out spinny. You would need to use large capacity U.2 NVME drives to get the appropriate speeds to cap that on a single box.

Forgot to say I will use 2 (or even 3 if not enough cores to handle zeek and suricata) servers as forward nodes. But yeah, guess NVMe is still the safest bet

[TOoSmOotH]

We do have appliances that are capable of doing 10Gbit sustained PCAP that use U.2 NVME drives. That being said I always recommend a packet broker with more sensors that have spinny disk so that you can get longer retention. The only real difference between those 2 appliances is the disk... everything else is the same.

[petiepooo]

I stand corrected. It's been a while since I looked.