SO Sensor Showing Fault in Grid

[jgiuliano2024]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

2 x Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz

RAM

96.00 GB

Storage for /

1022M

Storage for /nsm

3.4T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have 4 sensors running. 1 shows fault in Grid view no metrics showing and cpu at 100%. The node can reach the master on all ports that are needed. The node is sending data to the master. Everything about the node shows it's working as it should. Netstat shows the following ports have established connections to the master 443, 8220, 4505, 8086, 4506, 5055. I have removed the node from grid and fleet rebooted no change. I removed it from both again and reinstalled the node from scratch no change.

All other sensor nodes are fine and showing metrics for the hardware. The sensor is a bare metal server. Dell R430
If anyone has had this issue and a fix I would love to hear from you. Not sure what else to try at this point?
From the sensor

so-status

                  Security Onion Status
              Container │ Status  │               Details
────────────────────────┼─────────┼───────────────────────
           so-sensoroni │ running │           Up 14 hours
               so-steno │ running │           Up 14 hours
     so-strelka-backend │ running │           Up 14 hours
 so-strelka-coordinator │ running │           Up 14 hours
  so-strelka-filestream │ running │           Up 14 hours
    so-strelka-frontend │ running │           Up 14 hours
  so-strelka-gatekeeper │ running │           Up 14 hours
     so-strelka-manager │ running │           Up 14 hours
            so-suricata │ running │            Up 7 hours
            so-telegraf │ running │           Up 14 hours
                so-zeek │ running │ Up 14 hours (healthy)

In fleet it shows healthy

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

What is the output of the following command on the sensor?

top -n 1

[jgiuliano2024]

Tasks: 417 total,   1 running, 413 sleeping,   0 stopped,   3 zombie
%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.2 hi,  0.2 si,  0.0 st
MiB Mem :  96035.6 total,  68165.3 free,  12581.8 used,  16390.2 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  83453.8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  52488 root      20   0 3118780 185628 101412 S  17.6   0.2   1:04.64 agentbeat
  49641 zeek      20   0  900668 445772  38192 S   5.9   0.5   9:59.21 zeek
  50202 zeek      20   0  553908 370428 171188 S   5.9   0.4   7:37.77 zeek
      1 root      20   0  175688  19528  10912 S   0.0   0.0   2:49.17 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.90 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 slub_flushwq
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.49 kworker/0:1H-events_highpri
     12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
     13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
     15 root      20   0       0      0      0 S   0.0   0.0   0:00.93 ksoftirqd/0
     16 root      20   0       0      0      0 I   0.0   0.0   0:49.62 rcu_sched
     17 root      rt   0       0      0      0 S   0.0   0.0   0:00.21 migration/0
     19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     21 root      rt   0       0      0      0 S   0.0   0.0   0:00.43 migration/1
     22 root      20   0       0      0      0 S   0.0   0.0   0:00.19 ksoftirqd/1
     24 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0H-events_highpri
     26 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2
     27 root      rt   0       0      0      0 S   0.0   0.0   0:00.44 migration/2
     28 root      20   0       0      0      0 S   0.0   0.0   0:00.25 ksoftirqd/2
     30 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/2:0H-events_highpri
     31 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3
     32 root      rt   0       0      0      0 S   0.0   0.0   0:00.44 migration/3
     33 root      20   0       0      0      0 S   0.0   0.0   0:00.12 ksoftirqd/3
     35 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/3:0H-events_highpri
     36 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/4
     37 root      rt   0       0      0      0 S   0.0   0.0   0:00.46 migration/4
     38 root      20   0       0      0      0 S   0.0   0.0   0:00.25 ksoftirqd/4
     40 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/4:0H-events_highpri
     41 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/5
     42 root      rt   0       0      0      0 S   0.0   0.0   0:00.45 migration/5
     43 root      20   0       0      0      0 S   0.0   0.0   0:00.10 ksoftirqd/5
     45 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/5:0H-events_highpri
     46 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/6
     47 root      rt   0       0      0      0 S   0.0   0.0   0:00.44 migration/6
     48 root      20   0       0      0      0 S   0.0   0.0   0:00.40 ksoftirqd/6

[dougburks]

You said above that the sensor can connect to the necessary ports on the manager, but is there a firewall or other network device between the sensor and the manager that could be interfering with some of the data?

Am I correct in assuming that all of your sensors have unique IP addresses and hostnames?

[jgiuliano2024]

Yes all ports show good. There is a firewall it is set to allow any to any from the sensors to the master. Yes all sensors have unique IP and hostname. when running 2.3 everything was working. The strange thing is the sensor is reporting to the master fine. I am getting alerts and pcap data from it. Only problem is the system metrics in grid cpu 100% is really strange seeing on the system it's only using a few percent. Fleet shows healthy as well. All others look normal for system metrics but this one

[dougburks]

There is a firewall it is set to allow any to any from the sensors to the master.

Is this a simple firewall that just does port blocking or is it a next-gen firewall that could be doing IPS or other kinds of packet manipulation?

Have you checked the logs on the sensor in /opt/so/log/ and specifically telegraf?

Are all devices in your Security Onion sync'd to the same NTP server and do they all currently show the same exact time?

[jgiuliano2024]

It is a next gen firewall but IPS is not active on it. telegraf.log is empty and lastcaptureloss.txt has this in it 1726835982.570755

It does seem time on this sensor is 4 hours off compared to Master and other Sensors
Sensor: Fri Sep 20 12:44:40 PM UTC 2024
Master: Fri Sep 20 04:44:02 PM UTC 2024

All other sensors are showing same time as master.
Investigating time issue.

[jgiuliano2024]

I could not ping ntp from that sensor. I just changed all ntp setting to use our internal ntp server. I will report back once I see if this corrected the issue.

[jgiuliano2024]

That did the trick

Thank you for your help Doug!
I think we can close this issue now and hopefully it help someone else having this problem in the future.

[tsmith-spscc]

@jgiuliano2024 I seem to be having this problem as well, except it's my manager and forward nodes that have the wrong time (which is causing problems to appear on my search node in the grid). I've changed the ntp setting in the SOC to my internal ntp servers, verified on the CLI of each node that they've updated... but my time is still off by 7 hours on the manager and forward nodes. Did you just update ntp via the SOC as well or did you have to do something else to get it to work properly?

[jgiuliano2024]

I updated all nodes in soc you have to open each one in there and enter in your ntp. the I ran a salt '*' state.highstate from cli to be sure everything updated. Once done I had to reboot the node.