so-suricata and so-stenographer containers are missing

[commgina]

Version
2.3.160

Installation Method
Network Install on Ubuntu 20.04

Installation Type
Distributed

Network Traffic Speeds
Less than 1Gbps

Status
All services on all nodes are running OK on manager but so-suricata and so-stenographer are missing on sensor nodes

Details:

Due to issues with DNS servers, the automatic update capability for rules and updating Security Onion was lost. To address this, initially, we waited for the networking team to resolve the issue. We had to make changes to the DNS configuration files such as /etc/resolv.conf. The DNS problem was resolved; however, after these changes, other issues arose: the so-suricata and so-stenographer containers on the sensor nodes started failing. Sometimes their status was "MISSING," and other times it was "ERROR." Following this, I attempted to modify the files in /etc/netplan and add the IP addresses of the DNS servers that had been added to /etc/resolv.conf. However, this did not resolve the issue. Any ideas how can I solve this?

Stenographer.log

/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x42e4a4]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f1e0e00a555]
/usr/bin/stenotype() [0x40412b]
2024/04/03 12:36:58 Stenotype stopped after 1.038903608s: stenotype wait failed: signal: aborted (core dumped)
2024/04/03 12:36:58 Stenotype ran for too little time, crashing to avoid stenotype crash loop
2024-04-03T12:47:31.434501Z T:9447d7 [stenotype.cc:567] Starting, page size is 4096
2024-04-03T12:47:31.434622Z T:9447d7 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2024-04-03T12:47:31.799149Z T:9447d7 [stenotype.cc:619] CHECK(SUCCEEDED(__check_success_error__)) builder.Bind(flag_iface, &v3): No such device

Suricata.log

Unable to find iface bond0: No such device
3/4/2024 -- 12:37:04 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find iface bond0: No such device
3/4/2024 -- 12:37:04 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
3/4/2024 -- 12:37:04 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - thread W#01-bond0 failed
3/4/2024 -- 12:47:32 - <Notice> - This is Suricata version 6.0.6 RELEASE running in SYSTEM mode
3/4/2024 -- 12:47:32 - <Info> - CPUs/cores online: 20
3/4/2024 -- 12:47:32 - <Info> - dropped the caps for main thread
3/4/2024 -- 12:47:32 - <Info> - eve-log output device (regular) initialized: /nsm/eve-%Y-%m-%d-%H:%M.json
3/4/2024 -- 12:47:32 - <Info> - stats output device (regular) initialized: stats.log
3/4/2024 -- 12:47:32 - <Info> - Running in live mode, activating unix socket
3/4/2024 -- 12:47:34 - <Info> - 1 rule files processed. 36764 rules successfully loaded, 0 rules failed
3/4/2024 -- 12:47:34 - <Info> - Threshold config parsed: 0 rule(s) found
3/4/2024 -- 12:47:35 - <Info> - 36767 signatures processed. 1158 are IP-only rules, 4916 are inspecting packet payload, 30666 inspect application layer, 0 are decoder event only
3/4/2024 -- 12:47:37 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find type for iface "bond0": No such device
3/4/2024 -- 12:47:37 - <Info> - Going to use 8 thread(s)
3/4/2024 -- 12:47:38 - <Info> - Running in live mode, activating unix socket
3/4/2024 -- 12:47:38 - <Info> - Using unix socket file '/var/run/suricata/suricata-command.socket'
3/4/2024 -- 12:47:38 - <Notice> - all 8 packet processing threads, 4 management threads initialized, engine started.

[cm-ops]

What does nmcli con show return? Also, cat /proc/net/bonding/bond0.

[commgina]

NAME                UUID                                  TYPE      DEVICE 
bond0               f7d688d0-d721-4587-8891-c14fb01fb598  bond      --     
bond0-slave-ens192  20973e1c-fae3-4029-8fef-fb296cdc62ac  ethernet  --  

I don't have /bonding/bond0 dir. These are the directories I have in /proc/net:

anycast6   fib_triestat   ip6_mr_cache       ipv6_route  protocols  rt_acct       stat      unix
arp        icmp           ip6_mr_vif         mcfilter    psched     rt_cache      tcp       wireless
connector  icmp6          ip_mr_cache        mcfilter6   ptype      snmp          tcp6      xfrm_stat
dev        if_inet6       ip_mr_vif          netfilter   raw        snmp6         udp
dev_mcast  igmp           ip_tables_matches  netlink     raw6       sockstat      udp6
dev_snmp6  igmp6          ip_tables_names    netstat     route      sockstat6     udplite
fib_trie   ip6_flowlabel  ip_tables_targets  packet      rt6_stats  softnet_stat  udplite6

I don't know if this is useful but I'm having this output in journalctl -u NetworkManager :

ifupdown: interfaces file /etc/network/interfaces doesn't exist
device (lo): carrier: link connected
manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
device (ens160): carrier: link connected
manager: (ens160): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
manager: (ens192): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
manager: (bond0): new Bond device (/org/freedesktop/NetworkManager/Devices/4)
Error: failed to open /run/network/ifstate

[commgina]

Could it be that systemd-networkd and NetworkManager are running at the same time? When I made modifications to the DNS, adding two new entries to the /etc/resolv.conf file, I noticed that systemd-resolved was inactive. I enabled it, thinking it might have been the problem. Perhaps enabling it also activated systemd-networkd, which then started causing issues?

[cm-ops]

What happens if you run nmcli con up bond0? Also, what does this return nmcli con show bond0?

Security Onion 2.3 is EOL on 4/6 (tomorrow) https://blog.securityonion.net/2023/10/6-month-eol-notice-for-security-onion-23.html You should upgrade to the new version 2.4.60 https://blog.securityonion.net/2024/03/security-onion-2460-now-available.html

[commgina]

nmcli con up bond0
Error: Connection activation failed: Activation failed because the device is unmanaged
nmcli con show bond0
connection.id: bond0
connection.uuid: 3e457a9a-0ae5-4e90-8e85-951fda65da45
connection.stable-id: --
connection.type: bond
connection.interface-name: bond0
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.autoconnect-retries: -1 (default)
connection.multi-connect: 0 (default)
connection.auth-retries: -1
connection.timestamp: 1711558411
connection.read-only: no
connection.permissions: --
connection.zone: --
connection.master: --
connection.slave-type: --
connection.autoconnect-slaves: -1 (default)
connection.secondaries: --
connection.gateway-ping-timeout: 0
connection.metered: unknown
connection.lldp: default
connection.mdns: -1 (default)
connection.llmnr: -1 (default)
connection.wait-device-timeout: -1
802-3-ethernet.port: --
802-3-ethernet.speed: 0
802-3-ethernet.duplex: --
802-3-ethernet.auto-negotiate: no
802-3-ethernet.mac-address: --
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist: --
802-3-ethernet.mtu: 1500
802-3-ethernet.s390-subchannels: --
802-3-ethernet.s390-nettype: --
802-3-ethernet.s390-options: --
802-3-ethernet.wake-on-lan: default
802-3-ethernet.wake-on-lan-password: --
ipv4.method: disabled
ipv4.dns: --
ipv4.dns-search: --
ipv4.dns-options: --
ipv4.dns-priority: 0
ipv4.addresses: --
ipv4.gateway: --
ipv4.routes: --
ipv4.route-metric: -1
ipv4.route-table: 0 (unspec)
ipv4.routing-rules: --
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-iaid: --
ipv4.dhcp-timeout: 0 (default)
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.dhcp-fqdn: --
ipv4.dhcp-hostname-flags: 0x0 (none)
ipv4.never-default: no
ipv4.may-fail: yes
ipv4.dad-timeout: -1 (default)
ipv6.method: ignore
ipv6.dns: --
ipv6.dns-search: --
ipv6.dns-options: --
ipv6.dns-priority: 0
ipv6.addresses: --
ipv6.gateway: --
ipv6.routes: --
ipv6.route-metric: -1
ipv6.route-table: 0 (unspec)
ipv6.routing-rules: --
ipv6.ignore-auto-routes: no
ipv6.ignore-auto-dns: no
ipv6.never-default: no
ipv6.may-fail: yes
ipv6.ip6-privacy: -1 (unknown)
ipv6.addr-gen-mode: stable-privacy
ipv6.ra-timeout: 0 (default)
ipv6.dhcp-duid: --
ipv6.dhcp-iaid: --
ipv6.dhcp-timeout: 0 (default)
ipv6.dhcp-send-hostname: yes
ipv6.dhcp-hostname: --
ipv6.dhcp-hostname-flags: 0x0 (none)
ipv6.token: --
bond.options: mode=balance-rr
proxy.method: none
proxy.browser-only: no
proxy.pac-url: --
proxy.pac-script: --

[commgina]

I solved it. The file 10-globally-managed-devices.conf was the problem. I created it when I was trying to solve the DNS issue. Thank you!