Using docker_container.running to do CPU and Memory isolation on NUMA nodes

[novaksam]

Looking at https://docs.saltproject.io/en/latest/ref/states/all/salt.states.docker_container.html and I observed that Cpu and memory pinning look to be options, but I'm not sure how to implement these on my minion state (sorry if using the wrong terms, new to salt). I'm doing CPU isolation on the kernel, but using Numatop is revealing that java processes are using memory on the isolated CPUs and while I don't believe it's a problem for me, I'd like to squeeze as much water of out this rock as possible :)

Before rebuilding with security onion, I had the ELK stack running one Numa node 0, using numactl commands to run them on that node, and suricata was running on node 1.

Any insights into modifying the docker_running in my minion config file would be appreciated!

[dougburks]

If you haven't already, I would start with pinning Suricata and Zeek to the NUMA node that services your sniffing NIC:
https://docs.securityonion.net/en/2.3/suricata.html#performance
https://docs.securityonion.net/en/2.3/zeek.html#performance

[novaksam]

Yup, already doing that :)

  docker_container.running:
    - image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elasticsearch:{{ VERSION }}
    - hostname: elasticsearch
    - name: so-elasticsearch
    - user: elasticsearch
    - cpuset_mems: "0"
    - cpuset_cpus: "1-11,24-35"

I've modified the init.sls's for Elasticsearch, zeek, suricata and logstash, Suricata and zeek on the numa node with the NICs, the ELK/Logstash on the node with the NVMe, and numastat looks significantly better. I also pin the IRQ's for the NICs to two of my isolated cores using a script called set_irq_affinity from Intel (I think it's in the driver source for i40e), and have an /etc/rc.local to set some parameters with ethtool

# Setting the cpumask for other devices
echo 000f,ff000fff > /sys/bus/workqueue/devices/writeback/cpumask
echo 0 > /sys/bus/workqueue/devices/writeback/numa
# Make sure we only have 1 RSS queue
/usr/sbin/ethtool -L ens4f0 combined 1
/usr/sbin/ethtool -L ens4f1 combined 1
/usr/sbin/ethtool -L ens4f2 combined 1
/usr/sbin/ethtool -L ens4f3 combined 1
# Increase the number of interrupts that can be done
/usr/sbin/ethtool -C ens4f0 adaptive-rx off
/usr/sbin/ethtool -C ens4f1 adaptive-rx off
/usr/sbin/ethtool -C ens4f2 adaptive-rx off
/usr/sbin/ethtool -C ens4f3 adaptive-rx off
/usr/sbin/ethtool -C ens4f0 rx-usecs 100
/usr/sbin/ethtool -C ens4f1 rx-usecs 100
/usr/sbin/ethtool -C ens4f2 rx-usecs 100
/usr/sbin/ethtool -C ens4f3 rx-usecs 100
# Bind the hardware interrupts for both NICs to a single core
/root/set_irq_affinity one 12 ens4f0
/root/set_irq_affinity one 13 ens4f1
/root/set_irq_affinity one 12 ens4f2
/root/set_irq_affinity one 13 ens4f3

# 'rx' is removed from these lists because the suricata documentation state
# that rx checksum offloading is fine
for i in tx tso ufo gso gro lro \
    rxhash ntuple rx sg txvlan rxvlan rx-udp_tunnel-port-offload; \
    do /usr/sbin/ethtool -K ens4f0 $i off 2>&1 > /dev/null; done;

for i in tx tso ufo gso gro lro \
    rxhash ntuple rx sg txvlan rxvlan rx-udp_tunnel-port-offload; \
    do /usr/sbin/ethtool -K ens4f1 $i off 2>&1 > /dev/null; done;

for i in tx tso ufo gso gro lro \
    rxhash ntuple rx sg txvlan rxvlan rx-udp_tunnel-port-offload; \
    do /usr/sbin/ethtool -K ens4f2 $i off 2>&1 > /dev/null; done;

for i in tx tso ufo gso gro lro \
    rxhash ntuple rx sg txvlan rxvlan rx-udp_tunnel-port-offload; \
    do /usr/sbin/ethtool -K ens4f3 $i off 2>&1 > /dev/null; done;