Zeek post-terminate crash

[zpravaiz]

HI,

I am running the latest SO built ( 2.3.90) . I have recently noticed that my /nsm/zeek/spool/tmp is filling up with crash logs. I have checked one post-terminate-logger-2021-12-16-07-12-30-78-crash/post-terminate.out file its has a message

send-mail: /usr/sbin/sendmail not found

Also many other folders have zeek logs too.

I would appreciate it if someone can help me fix the issue.

Thanks

[astroc0]

On our systems zeek proccess crashes happen quite often with low ram sensors..

[zpravaiz]

I don't think RAM is an issue with my box. I have almost 300Gb

Thanks

[dougburks]

I'm not seeing this issue on any of my systems.

@zpravaiz please provide more information about your system and configuration including:

• What are the hardware specs of your system (CPU, RAM, storage, etc.)?
• What options did you choose in Setup?
• How many Zeek workers do you have?
• How much traffic are you monitoring?

[zpravaiz]

Hi Doug,

• its dell server with 2 X Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (32 cores), 300 Gb RAM, 11 TB Disk
• Its a stand-alone server setup. Packet capture and strelka are disabled. There is no other data being ingested.
• I see zeek_lbprocs: 7 in minion file so 7 workers. Let me know if my understanding is correct.
• I have a 10G interface that monitors approximately 2Gbps data on peak time. I have 2 more 1G but data is hardly 10% on these interfaces.

I am running this box for a very long time. The purpose is just to record & monitor the activity on the network. I am not ingesting anything in elastic too. I store Zeek and Suricata logs and use the command line whenever needed. I notice this issue when the disk is filling up quickly.

Do let me know if you need further information.

I appreciate your help.

Best

Update:
I noticed this error in one to the folder under spool/tmp

root@so-main:/nsm/zeek/spool/tmp# cat post-terminate-logger-2021-12-18-18-12-10-8842/stderr.log
/opt/zeek/share/zeekctl/scripts/run-zeek: line 61: ulimit: core file size: cannot modify limit: Operation not permitted
send-mail: /usr/sbin/sendmail not found
received termination signal

root@so-main:/nsm/zeek/spool/tmp# ls -a post-terminate-logger-2021-12-18-18-12-10-8842/
. .cmdline .rotated.dce_rpc .rotated.known_certs .rotated.packet_filter .rotated.smtp .rotated.tunnel files.2021-12-18-18-00-00.log
.. .env_vars .rotated.dhcp .rotated.known_hosts .rotated.pe .rotated.snmp .rotated.weird post-terminate.out
.archive-log.running.7461.tmp .pid .rotated.dns .rotated.known_services .rotated.radius .rotated.socks .rotated.x509 ssl.2021-12-18-18-00-00.log
.archive-log.running.7497.tmp .rotated.broker .rotated.dpd .rotated.loaded_scripts .rotated.rdp .rotated.software .startup stats.log
.archive-log.running.7529.tmp .rotated.capture_loss .rotated.files .rotated.mysql .rotated.reporter .rotated.ssh .status stderr.log
.archive-log.running.7583.tmp .rotated.cluster .rotated.http .rotated.notice .rotated.sip .rotated.ssl conn-summary.21-12-18_18.00.00.log stdout.log
.archive-log.running.7669.tmp .rotated.conn .rotated.intel .rotated.ntlm .rotated.smb_files .rotated.stats conn.2021-12-18-18-00-00.log syslog.2021-12-18-18-00-00.log
.archive-log.running.7811.tmp .rotated.conn-summary .rotated.kerberos .rotated.ntp .rotated.smb_mapping .rotated.syslog dns.2021-12-18-18-00-00.log x509.2021-12-18-18-00-00.log

Hope this may help.

[zpravaiz]

Hi All,

I would appreciate it if anyone can share some pointers to resolve this issue.

Thanks

[dougburks]

What is the output of df -h?

[zpravaiz]

Hi Doug,

See the below. Space is not an issue. I have 3.5T free space.

root@so-main:/nsm/zeek/spool/tmp# df -h
Filesystem Size Used Avail Use% Mounted on
udev 150G 0 150G 0% /dev
tmpfs 30G 5.0M 30G 1% /run
/dev/mapper/securityonion--vg-root 12T 7.1T 3.5T 68% /
tmpfs 150G 360K 150G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 150G 0 150G 0% /sys/fs/cgroup
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/60e7fa25e26122abd60c730eaa05f5aff360f92abd1481fb07406d18cdd2d962/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/acb4d8f63b51238c6cdf6cddbdf61b94f5af604a92143afd3a098f7d86b561af/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/d2d180d5cbbd553e12cb9ae0833bf48922d62550a6c685661bba5258f8e2d933/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/4aabb4195c7a8fca51a5a668dfe1c60d3617c583c665f952d14e552135499ff3/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/4e9adf5f1f16fe87019cb2d8e0e29cd220ffad97acf6e3e3bee7906dcfa35406/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/c35de93bc12aefbff3f4cece7ae7496307ffb292b35bea450d6f83f64472deef/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/dfb7567a24d54259ae5cf7646a2f06ee48d1e6b5d0a60677a6374c89a026565f/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/fba7caf462c4a141339f0769f8ef36cbef1eb3516abb9d9af0217efa0f972d5f/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/ed5001f7a2de335ac0ab0278eb5c6f3be8b3a6e5e6334cd6e41f41c41db4a8bd/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/4daf84a6089ece2a251462194ece3aee7d1fe446338e6b4112ee4161806d2903/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/4f5b45596ecc1b0057968ef3d2f228501318b28517aa6399eee18679b16853ea/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/90fc492a4297573e733c4c1de7283f8ebabbdfd7b53edc180f4f70579c970c9a/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/d1453cbf7e8a7ecd64d489139cd2a827c14a7923fdf5ba1c9e9631ce451904c6/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/e6ec435fec3dd5cefb3d3b222f94f752c369d84d9fd00f0e9915cb62fe63d185/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/459dea9712cf0a19e6f0240f9f49087973945afd7b4b6aa44f32f1a2cb1c6761/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/a5da2bab73ae883f7d78adace9fcd18bede128bdd9060eeb714f7067a84fe02a/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/aaeb2e12f8544ca20d2c95d216ed22469054dee8099fbad2cc10a5e47c29ac74/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/585356c4eeaa295db31e4e7fce5fdfba1ea7e767fb77f58a26c8e8f7ee2b089f/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/c0e9d7453800a8f27fa0f3ab5a7463610fd40dfc3fb767e840b8d7e1e29b28ad/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/99c67abf9c4213a035a77c6441f473ea7149a5b2ca97bb08b64fe17d767c12fe/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/00c2b051017505d50d062dd9d7d47868f14ac3a32b87e81bac13c66d20fe6699/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/06b8636b80c3e53101560d5f9652313cc416a509cc219e9d00f48b990f4c6f88/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/aea35ae519e3a76c46a3fea298edb53bc2c9096a0ae77c5c0182fd7f6ef85e46/merged
overlay 12T 7.1T 3.5T 68% /var/lib/docker/overlay2/4be75cb34f608611cd86943a9da5bc444e591579cc65aa6e30e994922eb8855f/merged

[dougburks]

If you haven't already, I would go ahead and upgrade to Security Onion 2.3.91 and reboot the box.

If the crashes continue, you might look for any non-standard traffic on your network that might be causing Zeek to crash.

You could also try switching from Zeek metadata to Suricata metadata:
https://docs.securityonion.net/en/2.3/suricata.html#metadata

[zpravaiz]

Doug,

I did so but still, the problem did not resolve. I really like Zeek and my all scrips are built around that. It would be great if this problem gets resolved.

[presianbg]

There was such an issue before - #3869
I've checked and can see some crashes and pilling up of log files on almost all of my sensors.
How to troubleshoot such problem ?

[dougburks]

all scrips are built around that

Please try removing your custom scripts to see if that makes a difference.

[presianbg]

I've just removed all custom zeek scripts and disabled the intel check just to be sure.
I will update this thread with the results.

[zpravaiz]

HI Dough,

an update. I noticed post-terminate-logger crash was happening every hour. This reminds me OTX gets updated every hour by the zeek_otx.py.

I disabled the cron job for /usr/sbin/zeek_otx.py and now the crash is not happening anymore since yesterday. can you please suggest what could be the issue with OTX.dat file?

I would appreciate your help.

Thanks

[presianbg]

I'm using the same script and my wild guess it's buggy or its pulling data from OTX which is not well accepted from zeek.

According to the docs:

Please note that Zeek is very strict about the format of intel.dat. When editing this file, please follow these guidelines:

- no leading spaces or lines
- separate fields with a single literal tab character
- no trailing spaces or lines

The default intel.dat file follows these guidelines so you can reference it as an example of the proper format.

Cheers,
PY

[presianbg]

@zpravaiz have you used this one - https://github.com/weslambert/securityonion-otx ?

Looks like my otx "integration" is not using the custom zeek scripts and relies on the built-in intel functionality.

I will try Wes's script and see if OTX intel causes any troubles.

[zpravaiz]

@presianbg
Yes, I am using https://github.com/weslambert/securityonion-otx. I have 2 boxes and I have the same issue with both. After disabling the cron on both crashes is not happening anymore.

[presianbg]

I'm using slightly modified version and its causing troubles too, so ... looks like the intel data is breaking zeek from time to time.

[dougburks]

We already had a warning at https://docs.securityonion.net/en/2.3/alienvault-otx.html that we don't officially support OTX integration, but I've also added a warning that it may cause Zeek crashes.