so-status says so-steno and so-suricata missing after upgrade to 2.3.110

[UMHB-InfoSec]

I got SUPER excited to see the new node type in 2.3.110 so decided to go for the upgrade. I was running 2.3.100.

The upgrade went fine, no errors that I was able to see. I waited an hour after the upgrade completed and logged back in. The manager still said all the nodes may need a reboot so I gave the sensors and the search nodes a reboot and then the manager.

The manager came back fine and the search nodes seem to be fine, but the sensor nodes are not happy. I have 5 sensor nodes and they are all experiencing the exact same issue.

As you can see both stenographer and suricata are borked.

I wasn't able to get much info from the logs in /opt/so/logs, here are the latest snippets from the steno and suricata logs:

Steno:

2022-03-10T21:04:54.186280Z T:4d7767 [stenotype.cc:567] Starting, page size is 4096
2022-03-10T21:04:54.186454Z T:4d7767 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2022-03-10T21:04:54.475852Z T:4d7767 [stenotype.cc:619] CHECK(SUCCEEDED(__check_success_error__)) builder.Bind(flag_iface, &v3): No such device
ABORTABORTABORT
/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x42e4a4]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fb54bcd5555]
/usr/bin/stenotype() [0x40412b]
2022/03/10 21:04:54 Stenotype stopped after 1.203248704s: stenotype wait failed: signal: aborted (core dumped)
2022/03/10 21:04:54 Stenotype ran for too little time, crashing to avoid stenotype crash loop

Suricata:

10/3/2022 -- 21:04:55 - <Notice> - This is Suricata version 6.0.4 RELEASE running in SYSTEM mode
10/3/2022 -- 21:04:55 - <Info> - CPUs/cores online: 64
10/3/2022 -- 21:04:55 - <Info> - dropped the caps for main thread
10/3/2022 -- 21:04:55 - <Info> - eve-log output device (regular) initialized: /nsm/eve-%Y-%m-%d-%H:%M.json
10/3/2022 -- 21:04:55 - <Info> - DNP3 log sub-module initialized.
10/3/2022 -- 21:04:55 - <Notice> - JsonDNP3Log logger not enabled: protocol dnp3 is disabled
10/3/2022 -- 21:04:55 - <Info> - DNP3 log sub-module initialized.
10/3/2022 -- 21:04:55 - <Notice> - JsonDNP3Log logger not enabled: protocol dnp3 is disabled
10/3/2022 -- 21:04:55 - <Info> - stats output device (regular) initialized: stats.log
10/3/2022 -- 21:04:55 - <Info> - Running in live mode, activating unix socket
10/3/2022 -- 21:04:57 - <Info> - 1 rule files processed. 24608 rules successfully loaded, 0 rules failed
10/3/2022 -- 21:04:57 - <Info> - Threshold config parsed: 15 rule(s) found
10/3/2022 -- 21:04:57 - <Info> - 24611 signatures processed. 1294 are IP-only rules, 4116 are inspecting packet payload, 19176 inspect application layer, 0 are decoder event only
10/3/2022 -- 21:05:14 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find type for iface "bond0": No such device
10/3/2022 -- 21:05:14 - <Info> - Going to use 31 thread(s)
10/3/2022 -- 21:05:14 - <Info> - Running in live mode, activating unix socket
10/3/2022 -- 21:05:14 - <Info> - Using unix socket file '/var/run/suricata/suricata-command.socket'
10/3/2022 -- 21:05:14 - <Notice> - all 31 packet processing threads, 4 management threads initialized, engine started.
10/3/2022 -- 21:05:14 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find iface bond0: No such device
10/3/2022 -- 21:05:14 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
10/3/2022 -- 21:05:14 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - thread W#01-bond0 failed

Steno is complaining about a missing interface and the suricata log says that it could not find the "bond0" interface, my first thought is "why is my sensor trying to do body repair on my truck", but them I thought better and decided to check my interfaces.

Sure enough, my sniffing interface on the sensor is disabled. All the other interfaces are up, just the interface I use for ingesting traffic from my tap is down.

I was able to bring the interface back up and put it in promiscuous mode via the commands:

ifconfig enp19s0f0 up
ifconfig enp19s0f0 promisc

and I am seeing traffic on the interface now.

# ifconfig enp19s0f0 
enp19s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 00:0e:1e:9c:0f:f0  txqueuelen 1000  (Ethernet)
        RX packets 62733884  bytes 33228999969 (33.2 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 33  memory 0x1c903000000-1c9037fffff  

However suricata and steno still crash when I try to start them. How do I get steno and suricata running again?

I do have some rather important data on this instance and I would like to keep it if at all possible.

Am I borked beyond saving? Am I missing something simple?

[UMHB-InfoSec]

update ... still borked ... but

I had the brainwave of re-running the network setup portion of the install procedure. But when I get to the step to specify the sniffing interface I get an error and the setup bombs:

some creative google-ing led me to try running:

nmcli dev set enp19s0f0 managed yes

and then trying to activate the connection via the network manager TUI utility nmtui

When I attempt to activate I get this error:

but if I run the setup again, and get to the sniffing interface portion, it still fails, but when the docker containers restart they all come up! WOOT!

but ... sad day ... on a reboot the problem comes back.

I'm going to keep pounding away at it, but if anyone has anyone has ideas PLEASE feel free to send them my way!

[UMHB-InfoSec]

Well, I would like to thank you all for coming on this journey of self discovery with me.

I believe I have fixed the issue.

https://docs.securityonion.net/en/latest/installation.html#installation-on-ubuntu-or-centos

Issue this one command

sudo touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf

and reboot, you probably don't have to reboot ... but it is intensely satisfying to see the server come up correctly from a reboot.

That's it, that's all ... hours of my finite time on this globe siphoned off into the abyss ... I can't fault the SO team ... it's right there in the release announcement.

If possible, we recommend that you test the upgrade process on a test deployment before deploying to production.

Perhaps I have learned my lesson ... then again ... perhaps not.

[commgina]

How did you rebooted? I'm having the same problem, suricata and steno missing in all of my sensors... I executed sudo touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf but nothing changed. Im not using DHCP btw, all manual network configuration. This happened after I changed DNS servers on /etc/resolv.conf ..

[UMHB-InfoSec]

Are you still running SO 2.3.x? If you are, please consider upgrading since it is EOL any day now.

I don't know for sure, but the upgrade from 2.3 to 2.4 may be best done as a fresh install.

BUT, to answer your question I just issued the standard reboot command on the CLI
sudo reboot

It sounds like there may be a mistake in your resolv.conf file. Also, IIRC some distros don't like you editing the resolv.conf file directly and you have to edit the interface definition file which then gets processed into a resolv.conf file. But, that is distro and release specific .

This link is an example of what I am talking about: https://askubuntu.com/questions/604760/can-not-edit-resolv-conf

Hope that helps, good luck.