Storage and Proxmox

[Gramarin]

I plan on installing Security Onion in a VM on Proxmox. I'll be installing a bunch of other VMs also. The whole thing will be my main and only server. I've never used Proxmox or Security Onion before and I'm currently buying components for the server and I need some help figuring out what hardware to get (specifically how to structure my storage needs). I obviously can't play around with any of this because the server is incomplete, so I'm not entirely sure how all this will function.

In the manual it says "Use local storage and avoid NFS, NAS, iSCSI, etc." so that's why I'm trying to achieve.

The simplest solution is to simply get two large NVMe SSDs, mirror them and install Proxmox and VMs onto them including Security Onion. The problem with this is the whole wearing out of drives issue. When the SSDs eventually wear out (I'm aware of Enterprise drives but those wear out also), I'll lose everything and will need to re-install everything from scratch. So I think a better idea would be to get two small NVMe SSDs for Proxmox and VMs and then somehow connect an additional either 2 large RAID 1 NVMe (on one of those PCIe expansion cards) or 4 RAID 10 SSDs (using motherboard SATA ports) and use those for Security Onion storage. x4 NVMe or x4 SSDs would server the same purpose, just two options I'm considering.

Only I'm not sure how that will work in terms of making them "local". Having never used Proxmox or Security Onion, I'm not sure how to structure everything for it to all play together nicely. Do I need to do PCIe pass-through for a the dual NVMe? Or do I simply create a partition in Proxmox for these two drives and use that? Are SSDs connected to motherboard SATA ports "local"? Will this even work?

[dougburks]

I've never used Proxmox or Security Onion before

Some folks find it difficult to learn both virtualization and Security Onion at the same time. I don't know if that will be an issue for you, this is just something that I've observed over the years. So you might want to consider learning Proxmox first and and asking Proxmox questions on the Proxmox forum (https://forum.proxmox.com/).

Once you're comfortable with Proxmox, storage, and networking, then implementing Security Onion on top of Proxmox will be much easier.

[isaacgolding]

So running SO on top of Proxmox will present you with some interesting logistical issues that you will have to deal with.

#1 you will take a performance hit... its minor but if you are thinking of deploying SO in a real large environment speed matters. If this is more of a school house project and you will be monitoring a small lab environment, then you are generally going to be ok running SO over proxmox.
#2 You should really consider having the OS and the storage spaces on separate drives. If you are just running a school project then consider using spinning drives for storage and SSDs for OS. Direct access to the drives is great... but for my setup I've always used Proxmox VM drives configured to a set size.
#3 Networking with Proxmox requires some special considerations when attempting to run a NIC in promiscuous mode. The normal config for Proxmox doesnt work. Using Proxmox networking to provide you with a TAP is doable but its not a standard out of the box config. So you have to work out things like adding a second NIC into the Proxmox server vs using virtual NICs and all the joys of that system.
#4 When building your Proxmox config u should understand that everything that you allocate to SO will be USED by SO. It's not your run of the mill program where it generally plays nice with others. If you give it 16 Cores 64GB of ram and 6 TB of drive space that will "all" be consumed as there is no such thing as Idle when capturing packets and constantly analyzing them in real time. Other VMs that you though you could share those resources with, will feel the pain.
#5 Just assume you are going to mess it all up. Build your Proxmox cluster/server as simple as you can. I rebuilt my cluster a few times before I eventually found the right mix of features and configuration that worked for my needs. Also consider just buying a couple of old PowerEdge servers from of the bigger refurb companies on the internet. They often come with a bundle of used hard drives. The R720xd blades can be found for 500-600$ on amazon and they work great for their age.
It's definitely doable. There are at least a few of us that linger on the forums here that run on Proxmox. But it is a bigger learning curve than just installing SO directly onto a server and running "bare metal".

[Gramarin]

@dougburks Although I haven’t used SO or Proxmox before, I have devoted a lot of time researching both. I’m pretty close at this point. Well close to beginning my SO journey, from a successful installation point of view.

@isaacgolding
#2 is what I’m trying to understand with my question.

How do you put Security Onion data onto a data drive? I’ve watched the installation process from a few sources and in all of them, there’s no mention of putting the data onto a data drive. I presume PCAP and log files constitute most of the data for SO?

#4 In that case I think I’ll go with standalone option. My network is small and my server although somewhat beastly, I have now realised it’s not as overkill as I first thought when speccing it.

#5 That’s fine, I’m no stranger to this technique.

[Gramarin]

I think I found the answer to my question. I believe the solution lies in the fact that all data is stored in the /nsm partition. So exploring in that direction now. I wonder what nsm stands for?