Search node CPU at 100% in grid, SO basically non-functional #13644
-
Version2.4.80 Installation MethodSecurity Onion ISO image Descriptionother (please provide detail below) Installation TypeDistributed Locationon-prem with Internet access Hardware SpecsExceeds minimum requirements CPU24 RAM192 Storage for /40 TB Storage for /nsm40 TB Network Traffic Collectionspan port Network Traffic Speeds1Gbps to 10Gbps StatusYes, all services on all nodes are running OK Salt StatusNo, there are no failures LogsYes, there are additional clues in /opt/so/log/ (please provide detail below) DetailDue to issues with my search node server after a graceful shutdown for maintenance, I had to wipe and reload the search node in my Security Onion environment. Originally, I had the OS installed on a separate NVME drive in the server. However, the server BIOS no longer sees that device as a valid boot option, forcing me to re-install SO on the 40 TB RAID virtual drive in the server instead. Both the OS and the /nsm volume are on the same RAID virtual disk now. Since reloading the search node it has been in a faulted state in my SO grid showing 100% CPU usage and "Process Status: Unknown". The OS Uptime metric appears to not be updating as well. However when looking at processes via the CLI, the actual CPU usage is close to 0% across all cores. Running salt-call state.highstate does not return any errors. so-status on both the manager and sensor nodes come back with all services up and running. The securityonion.log file in /opt/so/log/elasticsearch on the manager node is almost 5 GB in size and it appears to be mostly (if not all) WARN messages like the one below: If there is a way to get the sensor node connected and working correctly without having to redo my entire Security Onion deployment I would be grateful. If other information would be helpful for troubleshooting I will do my best to provide it. Thanks. Guidelines
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 19 replies
-
|
Screen shot of the search node grid status: |
Beta Was this translation helpful? Give feedback.
-
|
I was able to re-install SO just now with the OS on the NVME drive and /nsm on the RAID virtual drive. It boots into the OS fine from the NVME drive now. However, the same issue remains. The search node still shows the problems above and is in a fault state. Also, prior to re-installing the search node I removed the previous iteration of it from the grid and rebooted the manager node to clear it from the grid completely. I also did this prior to the other re-install attempts yesterday. |
Beta Was this translation helpful? Give feedback.
-
|
When running "salt * state.highstate", I encouter this error on the search node. It is the only error returned from running the highstate across all nodes: |
Beta Was this translation helpful? Give feedback.
-
|
Well, I re-installed SO one more time, but this time I used a different hostname and IP address for the node. It made no difference. Once added to the grid, it still comes up with the same problem - 100% CPU and Elasticsearch isn't communicating correctly with the manager node, thus SO is essentially non-functional still. |
Beta Was this translation helpful? Give feedback.
-
|
I did notice I had the nodes IP in the fleet section as well as the sensor section. looking at netstat the sensor entry showed traffic the fleet entry did not. I removed the fleet entry and rebooted the node. No change. I also ran so-idstools-restart and rebooted the node one more time. Nogo I was hoping that was the cause it was not. If I come up with something that fixes it I will post it. |
Beta Was this translation helpful? Give feedback.
-
|
I updated from 2.4.80 to 2.4.100 but this did not help at all. |
Beta Was this translation helpful? Give feedback.
-
|
I've done a number of tcpdumps on both the manager and search nodes. What I see is that the search node attempts to connect to the so-elastic-fleet docker container on tcp/8220 but gets a [RST, ACK] response, so the container is essentially non-responsive. If I am reading iptables correctly, it should allow traffic from any non-sobridge source to the docker containers, and specifically the so-elastic-fleet container on tcp/8220... so I am not sure why the traffic is being dropped at the docker container. |
Beta Was this translation helpful? Give feedback.
-
|
This is the output of so-elastic-agent-status on the manager node. It appears that fleet is not responding. |
Beta Was this translation helpful? Give feedback.
-
|
@tsmith-spscc The title of this discussion specifies
Have you run full diagnostics on your hardware? |
Beta Was this translation helpful? Give feedback.
-
Are you able to reach the Elastic Fleet web interface at all? If so, are there any clues there? Have you checked the logs in /opt/so/log for additional clues? I'd start with elasticfleet, kibana, and elasticsearch.
This seems like a strange problem. It may be difficult to troubleshoot asynchronously via Github Discussions in a timely manner. If you need priority assistance, it may be worth your time to consider purchasing Professional Services so that an engineer can work with you in real time via screen share to repair your installation rather than having to start over from scratch. For more information, please see: |
Beta Was this translation helpful? Give feedback.
-
|
Hi Doug, thanks for the response. I have tried looking at the Elastic Fleet interface, but I receive a 404 when trying to access it. I have looked through the elasticsearch log but not kibana or elastic fleet. I'll give those a look as well. I'll contact your professional support services for a quote. Thanks again. |
Beta Was this translation helpful? Give feedback.
-
|
So, as a follow up for anyone who comes across this - Since my last post, I have re-installed Security Onion 2.4.100 from scratch across my entire grid (manager, search and forward nodes) multiple times and the issue keeps coming back. Currently, my search node is still showing 100% CPU in my grid and not providing the status of services running on it. Last night, after rebuilding it, the docker containers were not coming up on the search node. However, this morning they appear to be up and running. So far it looks like SO is back up and functioning at the moment, with the exception of the grid still showing a fault for the search node. I am receiving data from my forward node, which is generating suricata alerts. I am able to access Kibana again. I've looked through the logs on the search node and see no obvious signs of errors. Running a salt highstate across the grid returns no errors. So I'm going to leave it alone for a day or two and see if the fault on the search node in the grid decides to shake out. I'm hesitant to start tuning and adding integrations again until I'm more comfortable that the grid is actually stable. |
Beta Was this translation helpful? Give feedback.
-
|
@tsmith-spscc Based on a similar discussion at #13691, check to make sure that all of your nodes are synced to the same NTP server and have the same exact time on them. |
Beta Was this translation helpful? Give feedback.
-
|
@dougburks Thanks for pointing me to that discussion. This was indeed the issue that I was running into at this final stage of troubeshooting and rebuilding the grid. I had to set the time manually across all my nodes before turning ntp back on though before the time would sync up. And I can't say for sure if fixing the time issue would have resolved the original problems I was running into before I wiped and re-installed all of the nodes in my grid. But, for the time being, everything is green again. One last question - are there any discussions or resources that provide an explainer on the process of restoring the data backed up by the default SO backup process? Is it as simple as untarring the backup files in the relevant directories? |
Beta Was this translation helpful? Give feedback.
-
Please take a look at this discussion and see if it helps: |
Beta Was this translation helpful? Give feedback.
-
|
@jgiuliano2024 If you need help with your deployment, please start a separate discussion and provide detailed information about your deployment. |
Beta Was this translation helpful? Give feedback.
@tsmith-spscc Based on a similar discussion at #13691, check to make sure that all of your nodes are synced to the same NTP server and have the same exact time on them.