Memory (RAM) usage issues

[Kaiketsu-Zorori]

Hello and thank you for your time.
I am currently working on getting a standalone security onion server configured in my environment on the following hardware:

Memory

[administrator@onion etc]$ lsmem
RANGE                                  SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x000000207fffffff  130G online       yes  0-64

Memory block size:         2G
Total online memory:     130G
Total offline memory:      0B

CPU

[administrator@onion etc]$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual       
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0-31
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    CPU family:          6
    Model:               45
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           2
    Stepping:            7
    CPU max MHz:         2800.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            3999.95
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush 
                         dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_ts                         c arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni                          pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca s                         se4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibr                         s ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln 
                         pts md_clear flush_l1d
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                   512 KiB (16 instances)
  L1i:                   512 KiB (16 instances)
  L2:                    4 MiB (16 instances)
  L3:                    40 MiB (2 instances)
NUMA:
  NUMA node(s):          2
  NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
  NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable        
  Mds:                   Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Unknown: No mitigations
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB fillin                         g, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

Problem:
My Issue is that my memory usage peaks out at around 40%-45%.
Despite this, swap will creep up to 100%. As swap usage increases, some issues arise, including failing PCAP retrieval.

I have turned off swap using sudo swapoff -a which did increase memory usage a few percentage points, and made PCAPs accessible again.
However, later (4~6 hours?) swap somehow is enabled again, and starts to creep up, repeating the cycle.

Using free -h I can see that the memory looks to be allocated to something (SO?)

[administrator@onion etc]$ free -h
	total        used        free      shared  buff/cache   available
Mem:	125Gi        54Gi       2.0Gi       593Mi        70Gi        71Gi
Swap:	 0B          0B          0B

Side Notes:

• This may be a separate issue but my PCAP retention is currently at 0.1days, even though I have lots of free recourses available.
• I installed this standalone server using BIOS, instead of UEFI, could this be leading to HW utilization issues...?

[Acewiza]

The same thing cropped up for somebody a year or so ago, but I cannot find the thread right now. It did not get resolved IIRC, or maybe the OP just stopped responding. Starting to look like one of those really rare corner cases, for real. Probably a hardware issue of some sort, would be my guess.

[soad20000]

Have you tried changing the value of "swappiness" that's configured? It's in the administration -> configuration section of the SO web console. I can't recall exactly what the setting is called but it's in there somewhere. The lower the value the less aggressively SO will try to use swap memory over RAM. I also have a standalone deployment and the value defaulted to 10 (max is 100). You could try changing it to 1, or even 0. I personally haven't tested other values but I'd give it a shot.

[Kaiketsu-Zorori]

I had to do some searching, But I found the 'swappiness' setting.
I found the answer here (of course, this is referenced in the documentation)

The command is:

$ sudo sysctl -w vm.swappiness=0

[soad20000]

Could have sworn I saw the setting in the GUI, sorry about that. Let us know if you see any improvement with the swap usage after making that change.

[dougburks]

Problem:
My Issue is that my memory usage peaks out at around 40%-45%.

Can you provide more information about why you perceive this to be a problem?

Despite this, swap will creep up to 100%. As swap usage increases, some issues arise, including failing PCAP retrieval.
I have turned off swap using sudo swapoff -a which did increase memory usage a few percentage points, and made PCAPs accessible again.
However, later (4~6 hours?) swap somehow is enabled again, and starts to creep up, repeating the cycle.

I'm not sure why swap would be re-enabled unless the box were rebooted. If you want to make sure it is permanently disabled, then you can comment out the swap line in /etc/fstab and reboot the box.

Using free -h I can see that the memory looks to be allocated to something (SO?)
[administrator@onion etc]$ free -h
total used free shared buff/cache available
Mem: 125Gi 54Gi 2.0Gi 593Mi 70Gi 71Gi
Swap: 0B 0B 0B
Side Notes:

Are you referring to the 70Gi buff/cache? This is most likely the operating system caching data for Elasticsearch.

This may be a separate issue but my PCAP retention is currently at 0.1days, even though I have lots of free recourses available.

PCAP retention is more related to disk space than RAM.

I installed this standalone server using BIOS, instead of UEFI, could this be leading to HW utilization issues...?

Most likely not.