Sensor install keeps failing. #15212
Replies: 20 comments 13 replies
-
|
Just for a test, I also installed a frontend on a VM with two interfaces to test and see if there was something else wrong. so-strelka-backend is in a rebooting loop, so-strelka-filestream is missing, so-strelka-frontend is missing, and so-strelka-manager is missing. This is the same as on the Z400 workstation. It is running 2.4.190. Memory is at 36% used, and everything else is down in the single digits. |
Beta Was this translation helpful? Give feedback.
-
|
Would you look in the |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I looked in the sensor folder and the file was empty. The other error log reported an unknown error occurred. I was hoping for something in the logs to give a clue. It reports an unknown error occurred and I followed the installer dialog looking for the logs. The sensor shows up in the manager and I can add it in, but those containers are missing and one is in a restart loop reporting a lookup error. I will be back in the office tomorrow morning to try and grab the error logs from the rebooting container. |
Beta Was this translation helpful? Give feedback.
-
|
Watching processes on the sensor while adopting "soup" is running and installing. |
Beta Was this translation helpful? Give feedback.
-
|
This is the contents of the error.log file. With a directory of the /root. The sosetup.log file is about 45KB long if you want to see that. `[secsensor@onionsensor ~]$ sudo tail -n75 /root/sosetup.log
|
Beta Was this translation helpful? Give feedback.
-
|
This is the sensor status. ❗ Check container logs and /opt/so/log for more details. |
Beta Was this translation helpful? Give feedback.
-
|
I found the so-check command and ran that. It resulted in three errors. `` |
Beta Was this translation helpful? Give feedback.
-
|
The above looks like an error with |
Beta Was this translation helpful? Give feedback.
-
|
Here is the full sosetup.log file. The installer fails with an unknown error, but the ManagerSearch node sees it, and then I can accept the new Forward node into the cluster. |
Beta Was this translation helpful? Give feedback.
-
|
I figured the connection activation was the result of the Forward node trying to talk to the ManagerSearch node and being unable to wait as a pending member. Once I see it show up, I approve it, then I see Salt work on the Forward node. |
Beta Was this translation helpful? Give feedback.
-
|
If salt is working, what happens if you Is this an ISO install or network install on OL9? These partitions do not look like an ISO install: |
Beta Was this translation helpful? Give feedback.
-
|
Noticed an error in your log Opened an issue for this #15233 if you want to test this and see if it resolves the so-setup failure you are seeing you can try this on the sensor run Then you can rerun so-setup and install it as a sensor and let us know if anything new comes up |
Beta Was this translation helpful? Give feedback.
-
|
This install was done via the ISO. I burned the ISO to a USB stick. The only difference in the system from the default was that I used a hardware stripe drive set to hold the NSM partition. Otherwise, I used the default partitions as defined in the ISO setup. I did not define a separate drive for the NSM since the stripe set does not show up as an option in the setup tool. The two Ethernet interfaces were not customized outside of the static IP address assigned to the management. Using the SO network setup system. But I am still seeing the same condition. I did allow SOUP to run on the main node and allowed. I do see that the bridge interface is gone. [root@onionsensor secsensor]# so-status ❗ Check container logs and /opt/so/log for more details. [root@onionsensor secsensor]# This is my routing table.
|
Beta Was this translation helpful? Give feedback.
-
|
It has been a few days, so I figured I would check in to see if there are any other suggestions to try or information I can provide to help resolve the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Can you share the new /root/sosetup.log? What does |
Beta Was this translation helpful? Give feedback.
-
|
I ran into the same problem. Tried to get around the issue by:
All other components (search, receiver, manager etc) all seem to install correctly (as VMs) but I have been unable to have a sensor (or any node that includes a sensor) to install/attach to the Manager |
Beta Was this translation helpful? Give feedback.
-
|
Also tried to install latest on a completely different hypervisor and same result. Going to go back to 2.4.160-20250625 as that's the last ISO that definitely worked for me. I actually had a failure I couldn't solve after a few months of running that last version and now I'm beginning to wonder if it was an upgrade to these later version that caused that failure. |
Beta Was this translation helpful? Give feedback.
-
|
2.4.160-20250625 got stuck on trying to resolve sigs.securityonion.net and then failed with an "unrecoverable error". Throwing in the towel for now. Tried install one more time and still timing out on resolving that host despite the fact resolution for that host is working fine. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I can, These are the interfaces[root@onionsensor ~]# nmcli con show This is the returned result from running so-setup isoInstall had a problem. Please see /root/sosetup.log for details. │ errors.log file[root@onionsensor ~]# cat errors.log This is the sosetup.log file.This is the last so-setup after I ran the so-setup iso command. I also have allowed the sensor to join the ManagerSerach member. |
Beta Was this translation helpful? Give feedback.
-
|
Here's mine. This is a standalone install using securityonion-2.4.160-20250625.iso. I've used an older version on purpose as I think this is telling us something. I am 100% sure I used this ISO previously and didn't have this problem. This is also on a completely different hypervisor from the one where I first ran into this. There is only one thing I can think of that has changed (and might coincide with the observation I made above wrt Proxmox) - the version of Proxmox is different. When all of this was working before, I believe I was on v8 but upgraded to v9 a few weeks after it was released. I believe that's when my grid broke or more specifically, the sensor stopped sending data to the manager/receiver. Again, I have done multiple installations of manager, receiver and search nodes without issue but when installing a sensor node or a standalone node (which of course includes the sensor), this is what happens. Anyhow, here is the log: |
Beta Was this translation helpful? Give feedback.
-
|
Decided something else. Tried changing the capture interface in proxmox by changing from a virtio nic to an E1000. Rebooted, tried to rerun setup and it's doing the "could not resolve host: sigs.securityonion.net" thing again. Tried to resolve on another host to confirm no issues with DNS. Took a few seconds but resolved. I can see the server querying my local DNS (as well as the test system which had no issue resolving that address) and it's getting a response yet it claims to be waiting for a response. This is another variation of the failed install I've seen before and it will now just sit stuck at resolving that address indefinitely. |
Beta Was this translation helpful? Give feedback.
-
|
One install success. This time succeeded with an airgap install, securityonion-2.4.160-20250625.iso and E1000 for the capture interface. Not sure what the pattern is yet but possibly getting closer. Will go back to installing sensor on other hypervisor and try again with E1000 for capture interface but that one is the newest SO version and is also not airgapped. |
Beta Was this translation helpful? Give feedback.
-
|
Install on previous hypervisor (distributed layout) finally worked but after rebooting sensor, it never re-attaches to manager and states "System appears to be starting. No highstate has completed since the system Getting really hard to see any pattern here although using the simulated E1000 in proxmox for the capture interface seems to at least have changed something... |
Beta Was this translation helpful? Give feedback.
-
|
Last one for tonight. Wiped the sensor using the initial boot up from ISO one more time. Then redid the install again and everything finally worked. Still don't know why the previous install didn't work but again, the one thing that has changed here is changing the way the capture interface is presented to SO. As such, my recommendation is to try the same approach. I will create another sensor tomorrow and try everything again but go back to using the virtio NIC as the capture interface and attempt to attach to the same Manager as per above. |
Beta Was this translation helpful? Give feedback.
-
|
@Mr-Dilkington Checkout #15257 looks like a change with proxmox 9 @erlicthemad it looks like its all around the creation of bond0 / unable to manage slave interfaces to get them to join the bond0 interface. This is baremetal install not a VM in a hypervisor like proxmox? Do you have any other NIC available to use as monitor interface? What kind of NIC is it? |
Beta Was this translation helpful? Give feedback.
-
|
I went back down and added an old 3Com network card. 3c509. Now I see this in my interfaces. [secsensor@onionsensor setup]$ sudo nmcli bond0: connected to bond0 docker0: connected (externally) to docker0 sobridge: connected (externally) to sobridge lo: connected (externally) to lo enp1s0: disconnected ens5: disconnected veth12395d7: unmanaged vethdbb44b3: unmanaged DNS configuration: |
Beta Was this translation helpful? Give feedback.
-
|
Following the suggestion from before, I ran the command to delete BOND0. Then ran the so-install. Using both the 3c509 and enp1s0 interface and I wind up in the same situation. As a test, I disabled enp1s0 in the setup, leaving only the 3c509 interface running, and I am still in the same position. |
Beta Was this translation helpful? Give feedback.
-
|
Looking at the strelka-backend, I am seeing the following crash logs. ` Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): When I check from the host there is no problem reaching my onion Manager Search node. |
Beta Was this translation helpful? Give feedback.
-
|
Looking at the containers. When I SH into any running container and need an IP address, all interfaces show up, and I can ping the Manager Search node. The working Strelka nodes are actually Redis servers. |
Beta Was this translation helpful? Give feedback.
-
|
We need to backtrack a bit on the installer failing, it can't add your NIC to the bond interface which is required for a sensor node can you share Have you tried running that journalctl -xe command or reviewing /var/log/messages for errors related to bond0 / NIC interface? |
Beta Was this translation helpful? Give feedback.
-
|
Here is the enp1s0
Here is the ens5 And here is the log messages grepped for bond0 " I see the error in the MTU setting. Both Ethernet interfaces are set to 1500 MTU, but the bond seems to want to be set to 9000 MTU. I'm not sure where it's getting the desire for jumbo frames. I have found a link to change the MTU of the bond0 interfaces. And changed them back down to 1500 to see if that helps. This is what my interfaces look loke now |
Beta Was this translation helpful? Give feedback.
-
|
After making the changes to the bond0 MTU, I see that the so-strelka-backend has started and stabilized. But the other three are still missing. so-strelka-filestream, so-strelka-frontend, and so-strelka-manager. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Version
2.4.190
Installation Method
Security Onion ISO image
Description
installation
Installation Type
Distributed
Location
on-prem with Internet access
Hardware Specs
Exceeds minimum requirements
CPU
12
RAM
48
Storage for /
70G
Storage for /nsm
913G
Network Traffic Collection
span port
Network Traffic Speeds
Less than 1Gbps
Status
No, one or more services are failed (please provide detail below)
Salt Status
No, there are no failures
Logs
No, there are no additional clues
Detail
Hello,
I have been trying to get a Security Onion sensor installed on an HP Z400 workstation with an X5690 CPU. The install ends aborting with an undefined error, and all the logs listed to look at are empty. I started with build 2.4.160 and upgraded to 2.4.170, then I am now at 2.4.190. Each time the install fails with an unknown error. The node shows up, and Salt reports everything is fine, but the Suricata services are reporting that the containers are missing. Even when starting from bare-iron hard drives, where the boot volume is a 460 GB drive partitioned by Onion into 70 GB for the boot and the balance set to /home. The /nsm volume is a raid set at 913GB.
When I look for the logs, there is no content in the error logs, or the logs list an unknown error. Is it possible that the containers are not compatible with the X5690 Xeon processors?
Guidelines
Beta Was this translation helpful? Give feedback.
All reactions