after update from 2.4.100 to the latest, zeek has missing status #14409

011248163264 · 2025-03-20T12:59:24Z

011248163264
Mar 20, 2025

Version

2.4.130

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

56 cpus

RAM

128GB

Storage for /

128GB

Storage for /nsm

2TB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

so-status = so-zeek │ missing │
i run:

salt-call state.highstate
local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 382833 and was started at 2025, Mar 20 12:47:45.104780 with jid 20250320124745104780

grid status has = fault
after few minutes zeek try to start again, and the so-status is:
so-zeek │ running │ Up 3 minutes (health: starting)
but it failed after 5-10min
there are no zeek logs in HUNT section

Guidelines

I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

dougburks · 2025-03-20T13:55:47Z

dougburks
Mar 20, 2025
Maintainer

Have you previously modified your Zeek configuration in any way?

What is the output of the following?

sudo docker logs so-zeek

Is Zeek logging anything in /nsm/zeek/logs/current/?

What is the output of the following?

sudo salt-call state.apply zeek queue=True

0 replies

011248163264 · 2025-03-20T14:21:23Z

011248163264
Mar 20, 2025
Author

i have not chnaged the zeek config
docker-logs-zeek.txt

sudo salt-call state.apply zeek queue=True
[INFO ] Loading fresh modules for state activity
[INFO ] Running state [zeek] at time 14:18:07.090862
[INFO ] Executing state group.present for [zeek]
[INFO ] Group zeek is present and up to date
[INFO ] Completed state [zeek] at time 14:18:07.094467 (duration_in_ms=3.604)
[INFO ] Running state [zeek] at time 14:18:07.095075
[INFO ] Executing state user.present for [zeek]
[INFO ] User zeek is present and up to date
[INFO ] Completed state [zeek] at time 14:18:07.125360 (duration_in_ms=30.285)
[INFO ] Running state [/opt/so/conf/zeek/policy] at time 14:18:07.126772
[INFO ] Executing state file.directory for [/opt/so/conf/zeek/policy]
[INFO ] The directory /opt/so/conf/zeek/policy is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/policy] at time 14:18:07.127982 (duration_in_ms=1.21)
[INFO ] Running state [/nsm/zeek/logs] at time 14:18:07.128155
[INFO ] Executing state file.directory for [/nsm/zeek/logs]
[INFO ] The directory /nsm/zeek/logs is in the correct state
[INFO ] Completed state [/nsm/zeek/logs] at time 14:18:07.129217 (duration_in_ms=1.061)
[INFO ] Running state [/nsm/zeek/spool/manager] at time 14:18:07.129397
[INFO ] Executing state file.directory for [/nsm/zeek/spool/manager]
[INFO ] The directory /nsm/zeek/spool/manager is in the correct state
[INFO ] Completed state [/nsm/zeek/spool/manager] at time 14:18:07.130574 (duration_in_ms=1.177)
[INFO ] Running state [/nsm/zeek/extracted] at time 14:18:07.130745
[INFO ] Executing state file.directory for [/nsm/zeek/extracted]
[INFO ] The directory /nsm/zeek/extracted is in the correct state
[INFO ] Completed state [/nsm/zeek/extracted] at time 14:18:07.131807 (duration_in_ms=1.063)
[INFO ] Running state [/nsm/zeek/extracted/complete] at time 14:18:07.131978
[INFO ] Executing state file.directory for [/nsm/zeek/extracted/complete]
[INFO ] The directory /nsm/zeek/extracted/complete is in the correct state
[INFO ] Completed state [/nsm/zeek/extracted/complete] at time 14:18:07.133130 (duration_in_ms=1.151)
[INFO ] Running state [/opt/so/conf/zeek/policy] at time 14:18:07.133300
[INFO ] Executing state file.recurse for [/opt/so/conf/zeek/policy]
[INFO ] The directory /opt/so/conf/zeek/policy is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/policy] at time 14:18:07.367780 (duration_in_ms=234.48)
[INFO ] Running state [/nsm/zeek/spool] at time 14:18:07.367991
[INFO ] Executing state file.directory for [/nsm/zeek/spool]
[INFO ] The directory /nsm/zeek/spool is in the correct state
[INFO ] Completed state [/nsm/zeek/spool] at time 14:18:07.369145 (duration_in_ms=1.153)
[INFO ] Running state [/nsm/zeek/spool/state.db] at time 14:18:07.369364
[INFO ] Executing state file.managed for [/nsm/zeek/spool/state.db]
[INFO ] File /nsm/zeek/spool/state.db exists with proper permissions. No changes made.
[INFO ] Completed state [/nsm/zeek/spool/state.db] at time 14:18:07.370679 (duration_in_ms=1.315)
[INFO ] Running state [/usr/sbin] at time 14:18:07.370851
[INFO ] Executing state file.recurse for [/usr/sbin]
[INFO ] {'/usr/sbin': {'/usr/sbin': {'user': 939}, 'user': 939}}
[INFO ] Loading fresh modules for state activity
[INFO ] Completed state [/usr/sbin] at time 14:18:07.590977 (duration_in_ms=220.125)
[INFO ] Running state [/opt/so/conf/policy/intel/load.zeek] at time 14:18:07.593006
[INFO ] Executing state file.managed for [/opt/so/conf/policy/intel/load.zeek]
[INFO ] File /opt/so/conf/policy/intel/load.zeek is in the correct state
[INFO ] Completed state [/opt/so/conf/policy/intel/load.zeek] at time 14:18:07.604870 (duration_in_ms=11.864)
[INFO ] Running state [/opt/so/conf/zeek/zeekctl.cfg] at time 14:18:07.605059
[INFO ] Executing state file.managed for [/opt/so/conf/zeek/zeekctl.cfg]
[INFO ] File /opt/so/conf/zeek/zeekctl.cfg is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/zeekctl.cfg] at time 14:18:07.620313 (duration_in_ms=15.255)
[INFO ] Running state [/opt/so/conf/zeek/node.cfg] at time 14:18:07.620507
[INFO ] Executing state file.managed for [/opt/so/conf/zeek/node.cfg]
[INFO ] File /opt/so/conf/zeek/node.cfg is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/node.cfg] at time 14:18:07.632547 (duration_in_ms=12.041)
[INFO ] Running state [/opt/so/conf/zeek/networks.cfg] at time 14:18:07.632729
[INFO ] Executing state file.managed for [/opt/so/conf/zeek/networks.cfg]
[INFO ] File /opt/so/conf/zeek/networks.cfg is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/networks.cfg] at time 14:18:07.643063 (duration_in_ms=10.334)
[INFO ] Running state [/usr/local/bin/packetloss.sh] at time 14:18:07.643248
[INFO ] Executing state file.managed for [/usr/local/bin/packetloss.sh]
[INFO ] File /usr/local/bin/packetloss.sh is in the correct state
[INFO ] Completed state [/usr/local/bin/packetloss.sh] at time 14:18:07.651451 (duration_in_ms=8.202)
[INFO ] Running state [/opt/so/conf/zeek/bpf] at time 14:18:07.651639
[INFO ] Executing state file.managed for [/opt/so/conf/zeek/bpf]
[INFO ] File /opt/so/conf/zeek/bpf is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/bpf] at time 14:18:07.653465 (duration_in_ms=1.826)
[INFO ] Running state [/opt/so/conf/zeek/local.zeek] at time 14:18:07.653642
[INFO ] Executing state file.managed for [/opt/so/conf/zeek/local.zeek]
[INFO ] File /opt/so/conf/zeek/local.zeek is in the correct state
[INFO ] Completed state [/opt/so/conf/zeek/local.zeek] at time 14:18:07.665018 (duration_in_ms=11.377)
[INFO ] Running state [/opt/so/conf/so-status/so-status.conf] at time 14:18:07.665198
[INFO ] Executing state file.append for [/opt/so/conf/so-status/so-status.conf]
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
salt.utils.versions.warn_until(
[INFO ] Executing command git in directory '/root'
[INFO ] Executing command 'grep' in directory '/root'
[INFO ] ['unless condition is true']
[INFO ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 14:18:08.694646 (duration_in_ms=1029.447)
[INFO ] Running state [so-zeek] at time 14:18:08.696471
[INFO ] Executing state docker_container.running for [so-zeek]
[INFO ] Container 'so-zeek' is already configured as specified
[INFO ] Completed state [so-zeek] at time 14:18:10.221804 (duration_in_ms=1525.333)
[INFO ] Running state [/opt/so/conf/so-status/so-status.conf] at time 14:18:10.222025
[INFO ] Executing state file.uncomment for [/opt/so/conf/so-status/so-status.conf]
[INFO ] Pattern already uncommented
[INFO ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 14:18:10.224167 (duration_in_ms=2.142)
[INFO ] Running state [/usr/local/bin/packetloss.sh] at time 14:18:10.224354
[INFO ] Executing state cron.present for [/usr/local/bin/packetloss.sh]
[INFO ] Executing command 'crontab' in directory '/root'
[INFO ] Cron /usr/local/bin/packetloss.sh already present
[INFO ] Completed state [/usr/local/bin/packetloss.sh] at time 14:18:10.228600 (duration_in_ms=4.245)
local:

      ID: zeekgroup
Function: group.present
    Name: zeek
  Result: True
 Comment: Group zeek is present and up to date
 Started: 14:18:07.090863
Duration: 3.604 ms
 Changes:

      ID: zeek
Function: user.present
  Result: True
 Comment: User zeek is present and up to date
 Started: 14:18:07.095075
Duration: 30.285 ms
 Changes:

      ID: zeekpolicydir
Function: file.directory
    Name: /opt/so/conf/zeek/policy
  Result: True
 Comment: The directory /opt/so/conf/zeek/policy is in the correct state
 Started: 14:18:07.126772
Duration: 1.21 ms
 Changes:

      ID: zeeklogdir
Function: file.directory
    Name: /nsm/zeek/logs
  Result: True
 Comment: The directory /nsm/zeek/logs is in the correct state
 Started: 14:18:07.128156
Duration: 1.061 ms
 Changes:

      ID: zeekspooldir
Function: file.directory
    Name: /nsm/zeek/spool/manager
  Result: True
 Comment: The directory /nsm/zeek/spool/manager is in the correct state
 Started: 14:18:07.129397
Duration: 1.177 ms
 Changes:

      ID: zeekextractdir
Function: file.directory
    Name: /nsm/zeek/extracted
  Result: True
 Comment: The directory /nsm/zeek/extracted is in the correct state
 Started: 14:18:07.130744
Duration: 1.063 ms
 Changes:

      ID: zeekextractcompletedir
Function: file.directory
    Name: /nsm/zeek/extracted/complete
  Result: True
 Comment: The directory /nsm/zeek/extracted/complete is in the correct state
 Started: 14:18:07.131979
Duration: 1.151 ms
 Changes:

      ID: zeekpolicysync
Function: file.recurse
    Name: /opt/so/conf/zeek/policy
  Result: True
 Comment: The directory /opt/so/conf/zeek/policy is in the correct state
 Started: 14:18:07.133300
Duration: 234.48 ms
 Changes:

      ID: zeekspoolownership
Function: file.directory
    Name: /nsm/zeek/spool
  Result: True
 Comment: The directory /nsm/zeek/spool is in the correct state
 Started: 14:18:07.367992
Duration: 1.153 ms
 Changes:

      ID: zeekstatedbownership
Function: file.managed
    Name: /nsm/zeek/spool/state.db
  Result: True
 Comment: File /nsm/zeek/spool/state.db exists with proper permissions. No changes made.
 Started: 14:18:07.369364
Duration: 1.315 ms
 Changes:

      ID: zeek_sbin
Function: file.recurse
    Name: /usr/sbin
  Result: True
 Comment: Recursively updated /usr/sbin
 Started: 14:18:07.370852
Duration: 220.125 ms
 Changes:   
          ----------
          /usr/sbin:
              ----------
              /usr/sbin:
                  ----------
                  user:
                      939
              user:
                  939

      ID: zeekintelloadsync
Function: file.managed
    Name: /opt/so/conf/policy/intel/__load__.zeek
  Result: True
 Comment: File /opt/so/conf/policy/intel/__load__.zeek is in the correct state
 Started: 14:18:07.593006
Duration: 11.864 ms
 Changes:

      ID: zeekctlcfg
Function: file.managed
    Name: /opt/so/conf/zeek/zeekctl.cfg
  Result: True
 Comment: File /opt/so/conf/zeek/zeekctl.cfg is in the correct state
 Started: 14:18:07.605058
Duration: 15.255 ms
 Changes:

      ID: nodecfg
Function: file.managed
    Name: /opt/so/conf/zeek/node.cfg
  Result: True
 Comment: File /opt/so/conf/zeek/node.cfg is in the correct state
 Started: 14:18:07.620506
Duration: 12.041 ms
 Changes:

      ID: networkscfg
Function: file.managed
    Name: /opt/so/conf/zeek/networks.cfg
  Result: True
 Comment: File /opt/so/conf/zeek/networks.cfg is in the correct state
 Started: 14:18:07.632729
Duration: 10.334 ms
 Changes:

      ID: plcronscript
Function: file.managed
    Name: /usr/local/bin/packetloss.sh
  Result: True
 Comment: File /usr/local/bin/packetloss.sh is in the correct state
 Started: 14:18:07.643249
Duration: 8.202 ms
 Changes:

      ID: zeekbpf
Function: file.managed
    Name: /opt/so/conf/zeek/bpf
  Result: True
 Comment: File /opt/so/conf/zeek/bpf is in the correct state
 Started: 14:18:07.651639
Duration: 1.826 ms
 Changes:

      ID: localzeek
Function: file.managed
    Name: /opt/so/conf/zeek/local.zeek
  Result: True
 Comment: File /opt/so/conf/zeek/local.zeek is in the correct state
 Started: 14:18:07.653641
Duration: 11.377 ms
 Changes:

      ID: append_so-zeek_so-status.conf
Function: file.append
    Name: /opt/so/conf/so-status/so-status.conf
  Result: True
 Comment: unless condition is true
 Started: 14:18:07.665199
Duration: 1029.447 ms
 Changes:

      ID: so-zeek
Function: docker_container.running
  Result: True
 Comment: Container 'so-zeek' is already configured as specified
 Started: 14:18:08.696471
Duration: 1525.333 ms
 Changes:

      ID: delete_so-zeek_so-status.disabled
Function: file.uncomment
    Name: /opt/so/conf/so-status/so-status.conf
  Result: True
 Comment: Pattern already uncommented
 Started: 14:18:10.222025
Duration: 2.142 ms
 Changes:

      ID: zeekpacketlosscron
Function: cron.present
    Name: /usr/local/bin/packetloss.sh
  Result: True
 Comment: Cron /usr/local/bin/packetloss.sh already present
 Started: 14:18:10.224355
Duration: 4.245 ms
 Changes:

Summary for local

Succeeded: 22 (changed=1)
Failed: 0

Total states run: 22
Total run time: 3.129 s

1 reply

dougburks Mar 20, 2025
Maintainer

I asked this question previously but I don't see a reply, so I'll ask again:
Is Zeek logging anything in /nsm/zeek/logs/current/?

Also, is it possible that Zeek is getting OOM killed by the kernel?

If you're monitoring less than 1Gb of traffic, then you probably don't need 27 Zeek workers. Please try lowering the Zeek worker count to a smaller number like 8 as shown at https://docs.securityonion.net/en/2.4/zeek.html#performance and see if that helps any.

011248163264 · 2025-03-20T16:17:53Z

011248163264
Mar 20, 2025
Author

Is Zeek logging anything in /nsm/zeek/logs/current/? = yes
in the admin section zeek has grid value = false and workers 27
sorry we are monitoring 10Gb traffic
what was oom?

6 replies

011248163264 Mar 20, 2025
Author

OOM stands for Out Of Memory: = oh yes sure, lol
i will check the memory

011248163264 Mar 20, 2025
Author

there is no swap space

dougburks Mar 20, 2025
Maintainer

If you haven't already, please try the recommendation from my previous reply which was to decrease Zeek workers to 20 and see if that allows Zeek to run without crashing.

011248163264 Mar 20, 2025
Author

i have now reduced the worker to 10, but still so-status zeek = missing, now i have reduced to 5 but its not done yet

011248163264 Mar 20, 2025
Author

ok zeek is now running with 5 workers, i will try tomorrow how many soc will start without a crash

011248163264 · 2025-03-20T16:25:25Z

011248163264
Mar 20, 2025
Author

there is a /opt/zeek/share/zeekctl/scripts/run-zeek: line 61: ulimit: core file size: cannot modify limit: Operation not permitted in stderr.log

0 replies

after update from 2.4.100 to the latest, zeek has missing status #14409

Uh oh!

Uh oh!

011248163264 Mar 20, 2025

Version

Installation Method

Description

Installation Type

Location

Hardware Specs

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

Network Traffic Speeds

Status

Salt Status

Logs

Detail

Guidelines

Replies: 4 comments · 7 replies

Uh oh!

dougburks Mar 20, 2025 Maintainer

Uh oh!

011248163264 Mar 20, 2025 Author

Summary for local

Succeeded: 22 (changed=1) Failed: 0

Uh oh!

dougburks Mar 20, 2025 Maintainer

Uh oh!

011248163264 Mar 20, 2025 Author

Uh oh!

011248163264 Mar 20, 2025 Author

Uh oh!

011248163264 Mar 20, 2025 Author

Uh oh!

dougburks Mar 20, 2025 Maintainer

Uh oh!

011248163264 Mar 20, 2025 Author

Uh oh!

011248163264 Mar 20, 2025 Author

Uh oh!

011248163264 Mar 20, 2025 Author

011248163264
Mar 20, 2025

Replies: 4 comments 7 replies

dougburks
Mar 20, 2025
Maintainer

011248163264
Mar 20, 2025
Author

Succeeded: 22 (changed=1)
Failed: 0

dougburks Mar 20, 2025
Maintainer

011248163264
Mar 20, 2025
Author

011248163264 Mar 20, 2025
Author

011248163264 Mar 20, 2025
Author

dougburks Mar 20, 2025
Maintainer

011248163264 Mar 20, 2025
Author

011248163264 Mar 20, 2025
Author

011248163264
Mar 20, 2025
Author