Elastalert process being killed

[jdecker79]

Recently my SO instance has been getting out of memory errors and killing the elastalert process. I am looking for any direction on what I should look at to figure out the issue. My system has 32 gb of memory and the memory usage in Grafana normally sits around 60%. I just noticed this starting after I updated to version 2.3.190 but I am not convinced that is the issue. I have included a screen shot of the message.

[reyesj2]

Can you give some more information?

From #1720:

Please include the following details where you can:

• Security Onion version as seen in the lower left corner of SOC and in /etc/soversion. For example: 2.3.140
• Is this a cloud deployment or on-prem? If on-prem, do you have Internet access or this an airgap installation?
• Did you install from our Security Onion ISO image or did you perform a network installation?
• If network installation, did you install on CentOS 7 or Ubuntu?
• How many nodes do you have?
• What are the hardware specs of each of those nodes?
• How are each of those nodes configured? (ex. manager with 2 search nodes and 3 forward nodes)

[jdecker79]

SO Version 2.3.190
On prem with internet access
Installed from ISO a year or two ago
ISO CentOS 7
1 stand alone manager node
Hardware Specs i7-4770 CPU 3.40GHz , 32 GB memory, 1 TB Hard Drive.

[reyesj2]

does free -mh show the correct amount of memory 32G?

Also try top -o %MEM and watch for what processes are at the top of the list

[jdecker79]

free -mh shows 31G total / 29G used / 529M free / 91M shared / 1.4G buff-cache / 1.5G available
I attached a screen shot of the %MEM results.

[reyesj2]

What is your esheap size set to? sudo salt '*' pillar.get elasticsearch:esheap

[jdecker79]

It shows 8170m

[cm-ops]

Have you checked the Elastalert log for any issues? /opt/so/log/elastalert/elastalert.log Do you have Playbook plays or Elastalert alerts enabled? If so, how many and which ones?