Two cron jobs for - so-sensor-clean on a heavy node

[presianbg]

Hi,

We have joined a heavy node to our distributed setup.
Once the /nsm got 90% , the CPU usage skyrocketed, because two processes of so-sensor-clean were racing each other!
The log (/opt/so/log/sensor_clean.log) was flooded with:

Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/pcapout
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/suricata
Tue Dec  7 00:51:07 UTC 2021 - No old Zeek logs available to clean up in /nsm/zeek/logs/
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/wazuh/logs/archives
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/strelka/processed
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/suricata
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/pcapout
Tue Dec  7 00:51:07 UTC 2021 - No old Zeek logs available to clean up in /nsm/zeek/logs/
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/wazuh/logs/archives
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/strelka/processed
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/suricata
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/pcapout
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/wazuh/logs/archives
Tue Dec  7 00:51:07 UTC 2021 - No old Zeek logs available to clean up in /nsm/zeek/logs/
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/strelka/processed
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/pcapout
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/suricata
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/wazuh/logs/archives
Tue Dec  7 00:51:07 UTC 2021 - No old Zeek logs available to clean up in /nsm/zeek/logs/
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/strelka/processed
Tue Dec  7 00:51:07 UTC 2021 - No old files available to clean up in /nsm/pcapout

Examining the crontab , showed 2 jobs for so-sensor-clean running every minute (first and the last job):

# Lines below here are managed by Salt, do not edit
# 
* * * * * /usr/sbin/so-sensor-clean
#  SALT_CRON_IDENTIFIER:/usr/local/bin/sensor-rotate
1 0 * * * /usr/local/bin/sensor-rotate
# SALT_CRON_IDENTIFIER:/usr/local/bin/common-rotate
1 0 * * * /usr/local/bin/common-rotate
# SALT_CRON_IDENTIFIER:/usr/sbin/so-status -q; echo $? > /opt/so/log/sostatus/status.log 2>&1
*/1 * * * * /usr/sbin/so-status -q; echo $? > /opt/so/log/sostatus/status.log 2>&1
# SALT_CRON_IDENTIFIER:/usr/sbin/so-suricata-eve-clean > /dev/null 2>&1
*/5 * * * * /usr/sbin/so-suricata-eve-clean > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:/usr/local/bin/packetloss.sh
*/10 * * * * /usr/local/bin/packetloss.sh
# SALT_CRON_IDENTIFIER:/usr/sbin/so-curator-closed-delete > /opt/so/log/curator/cron-closed-delete.log 2>&1
*/5 * * * * /usr/sbin/so-curator-closed-delete > /opt/so/log/curator/cron-closed-delete.log 2>&1
# SALT_CRON_IDENTIFIER:/usr/sbin/so-curator-close > /opt/so/log/curator/cron-close.log 2>&1
*/5 * * * * /usr/sbin/so-curator-close > /opt/so/log/curator/cron-close.log 2>&1
# SALT_CRON_IDENTIFIER:/usr/sbin/so-curator-delete > /opt/so/log/curator/cron-delete.log 2>&1
*/5 * * * * /usr/sbin/so-curator-delete > /opt/so/log/curator/cron-delete.log 2>&1
# SALT_CRON_IDENTIFIER:so-salt-minion-check
*/5 * * * * /usr/sbin/so-salt-minion-check -q
# SALT_CRON_IDENTIFIER:zeek-extracted-strelka-sync
* * * * * [ -d /nsm/zeek/extracted/complete/ ] && mv /nsm/zeek/extracted/complete/* /nsm/strelka/unprocessed/ > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:/usr/sbin/so-sensor-clean
* * * * * /usr/sbin/so-sensor-clean

This should be a bug, because this is a brand new (2.3.90) installation of the heavy node and the cron wasn't modified by me.

Cheers,
PY

[m0duspwnens]

I was unable to reproduce this issue. Can you provide some details of your install? Is this CentOS or Ubuntu? Is it a network or ISO install? Did you have any issues during install and reinstall the node? Can you provide a sanitized version of sosetup.log?

Thanks.

[presianbg]

Hi, sorry for omitting those details in my original post.
It is ISO (CentOS) install with the initial 2.3.90 image.
If my memory serves me right, the install completed with some errors and after rebooting the system everything worked.
Later I've noticed that the heavy node is in airgap mode and changed the os stanza to use auto update and fixed the yum config (repo).
Not sure why this node was installed in airgap mode, but I had to move on with the configurations and noticed the double cronjob just yesterday.

Here is the sosetup.log:

so-setup-heavy-log.tar.gz

[dougburks]

The airgap issue was a bug in 2.3.90 that was fixed via hotfix:
https://blog.securityonion.net/2021/12/security-onion-2390-airgap-hotfix-now.html

I'm not sure why you have duplicate cron jobs, but I've created an issue to improve so-sensor-clean since it should have prevented multiple instances itself:
#6622