Security Onion and Proxmox FYI on promisc setup

[isaacgolding]

(SPOILER) This is just a quick and dirty how-to in case you are using proxmox and having issues with traffic volume.

So for the past month since setting up SO (multiple times and configs) I felt like I was never really seeing all of my network traffic. I saw lots of broadcast & multicast and some direct traffic but the volumes never really got over 1 meg a second. The setup is roughly as follows.

• A Dell Poweredge server with a 4 nic card and a 2 nic card+ several TB of raid space+96 GB of ram running Proxmox.
• SO installed in a VM with access to 5 of the 6 interfaces. 1 interface is the mgmt and the other 4 are monitors.
• Each monitor is plugged into a separate switch, each switch setup to SPAN or MIRROR depending on the switch OS.
• 70+ other devices on various subnets, all visible to SO via the various interfaces.

What I discovered after lots and lots of digging (and learning) was that even though SO was setup to be promisc on the monitor interfaces is that Proxmox needs to be setup properly to pass the traffic along to the VM. Its just not a simple bridge interface that needs to be setup.

What I had to do to get the entire system to push the promisc data to the VM was as follows:
For each NIC in proxmox I had to set promisc on: ip link set eno1 promisc on
For each BRIDGE in proxmox I had to change aging and forward delay as follows: brctl setageing vmbr1 0 and brctl setfd vmbr1 0

Once this happened my steady slow drip of data jumped from an average of .5 Mbs to an average of 15 Mbs.

I'm now questioning if I need to have 4 monitor ports... but at this point I can now sit back and watch as the system gets out of 1st gear for the first time since I started working with SO.

[metapolitics]

I'll have to see if this works better than the previous trick I was using from vext: https://vext.info/2018/09/03/cheat-sheet-port-mirroring-ids-data-into-a-proxmox-vm.html

ovs-vsctl clear bridge vmbr4 mirrors

ovs-vsctl -- --id=@p get port tap202i1 \
-- --id=@m create mirror name=span1 select-all=true output-port=@p \
-- set bridge vmbr4 mirrors=@m

[isaacgolding]

What I understand is that proxmox bridges don't mirror everything to everybody, they act as a switch. But by changing the bridge settings as shown you are making the bridge act as an old fashioned HUB instead of a switch. And as most people will understand hubs send "everything" to "everywhere".

The commands I listed prior make the changes immediately in proxmox but I did have to go back and define the changes in /etc/networks/interfaces for them to remain after a reboot.

For clarity: These changes are all being made on the proxmox machine. The Onion VM does not need any special config.
I run 4 sensors from the same machine because I have multiple switches on multiple VLANS and subnets that I want to monitor.
I have 2 network interfaces. The 4 port interface is used for all of the sensors. The 2 port interface is used as my admin interface with 1 port still inactive (for later use).

Here is a copy of my interfaces file.
###########################
auto lo
iface lo inet loopback

iface eno1 inet manual promisc

iface eno2 inet manual promisc

iface eno3 inet manual promisc

iface eno4 inet manual promisc

auto enp6s0
iface enp6s0 inet static
address 192.168.1.236/24
gateway 192.168.1.1
#PVE host IP address

auto enp7s0
iface enp7s0 inet manual
#bridge nic VMBR4

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge_ageing 0
bridge-stp off
bridge-fd 0
#sensor

auto vmbr2
iface vmbr2 inet manual
bridge_ageing 0
bridge-ports eno3
bridge-stp off
bridge-fd 0
#sensor

auto vmbr3
iface vmbr3 inet manual
bridge-ports eno4
bridge_ageing 0
bridge-stp off
bridge-fd 0
#sensor

auto vmbr1
iface vmbr1 inet manual
bridge-ports eno2
bridge_ageing 0
bridge-stp off
bridge-fd 0
#sensor

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp7s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#bridge to enp7s0 for VMs
############################

[pcoreyster]

two questions:

1. The 2nd to last line (marked as a comment) is meant to be enp7s0 rather than emp7s0 correct?

2. Outside of Security onion, Is this proxmox host have other Virtual Guests within this host?

[isaacgolding]

two questions:

1. The 2nd to last line (marked as a comment) is meant to be enp7s0 rather than emp7s0 correct?
2. Outside of Security onion, Is this proxmox host have other Virtual Guests within this host?

Yes you are correct on second to last line. I fixed it.
And yes I have other VMs running on this host. I have a Zabbix installed on a VM running 24x7, the SO IDH installed in a VM running 24x7, A Dell Open Manager is setup on a VM and only brought up as needed and then a couple other VMs for testing and whatnot. Needless to say the server is "never" idle.

[dougburks]

I've added a Proxmox page to our documentation and included this recommendation:
https://docs.securityonion.net/en/2.3/proxmox.html

[nopoxpan]

I may be able to write brief guide on that if can be added to docs, If you would like me to do so, please ping me.

[nopoxpan]

I see good talk here, but let me give more on this as I heavily worked on it long time ago.

So, I found that we have mainly three options:
1- Using Linux bridges:
Setting forward delay and ageing time to 0, so it does behave like a conventional HUB not SWITCH.

2- Using ovs bridges:
Hence you can use the native mirroring feature supported by OpenVSwitch, you need to download its packages before.

3- Pass-through the interfaces directly to SO:
SO will see the actual interface. PCIe pass-through is enabled by default in the latest releases of Proxmox, which is nice.

By experiment, I found out the third method is the most performant "make sense", especially when coupled with "passing through" of CPU, DISK, and RAM resources and pinning it to the VM-if I can say, so you can get performance almost identical to bare-metal.

[chrislawso]

Why VM instead of CT lxc container in proxmox? This year I was looking at standing up SO on Proxmox and assumed linux container was the more performant and efficient method?

[pcoreyster]

As I am reading this thread, I would image that having docker within an LXC presents few more challenges than having docker run within a guest. The LXC has to have specific settings in place such as '... docker will only run with the lxc execution driver and in a unconfined lxc'.

There is probably a security aspect to think about: Linux containers (LXC) can be an attack vector and developers (of software) would need to account for the nuances (when it comes to having privilege or unprivileged settings).

I also found this past post (thread): #1750