Firewall issue and/or OSQuery Agent install problem

[pezhore]

Version

2.4.40

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16

Storage for /

60

Storage for /nsm

250

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Salt errors:

local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 2745438 and was started at 2024, Mar 05 02:57:33.166069 with jid 20240305025733166069

I'm preparing for an infosec event where I will have four teams - each with their own Security Onion instance and their own LAN/DMZ network segments to monitor. I'm running into issues for each of the four teams when I try to get OSquery agents installed on the DMZ systems (but not the LAN). It looks to be firewall related as those DMZ systems can ssh/web browse/etc other LAN systems without any issues. Additionally, there are zero firewall rules at the router between each LAN/DMZ pair.

My question is this - what is the correct way to add more than one IP/CIDR block to the firewall.hostgroups.elastic_agent_endpoint Grid configuration?

Currently, I have the following (where 172.19.0.0/24 is the LAN, where multiple systems successfully installed the OSQuery agent, and 10.29.0.0/24 is the DMZ, where no systems can install OSQuery successfully).

172.19.0.0/24
10.29.0.0/24

OSquery logs seem to indicate connectivity issues - I can confirm that attempting to manually curl the urls below times out.

timestamp=2024-03-05T03:04:40.540375265Z level=info message="Version Information" ElasticAgentVersion=8.7.0 WrapperVersion=2.4.2
timestamp=2024-03-05T03:04:40.540436498Z level=info message="Runtime Data" EnrollmentToken="<TOKEN HERE>" FleetURL/s=https://seconion.ift.int:8220,https://seconion:8220
timestamp=2024-03-05T03:04:40.540447027Z level=info message="Installation Progress" Status="Starting Installation Precheck"
timestamp=2024-03-05T03:04:43.5427778Z level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://seconion.ift.int:8220
timestamp=2024-03-05T03:04:46.545880782Z level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://seconion:8220
timestamp=2024-03-05T03:04:46.545922807Z level=info message="Installation Progress" Status="No Fleet Hosts are accessible - Check connectivity to Fleet Host."
timestamp=2024-03-05T03:04:46.545956094Z level=info message="Installation Progress" Status="Exiting Installer..."

Is there something else I'm missing? Is my Firewall config incorrect?

EDIT: I've double checked and DNS resolves from the DMZ systems, it really does seem to just be the Security Onion firewall, I'm not sure how best to proceed.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[TotieBash]

Your #1720 reference link is for the old SO2.3.x version. The 2.4.x is a little different where the firewall is done at the SOC gui. I believe osquery is part of the elastic_agent so with that said:
https://www.youtube.com/watch?v=cGmQMsFuAvw
https://docs.securityonion.net/en/2.4/elastic-agent.html

[pezhore]

That #1720 link is the default for the discussion, I didn't add it here other than by clicking the "I have read the discussion guidelines".

This is my firewall settings via grid, It sure looks right to me, yet nothing is coming through.

I've now opened up the router firewall between the DMZ/LAN to allow any source, any destination, any port (source or destination) - on both the LAN and DMZ interfaces. Nothing shows up in the router logs that would indicate asymmetrical routing or other network issue.

[pezhore]

Well after pasting, I realize those netmasks are off - attempting to fix and see if that has a positive impact.

Edit: There are no changes after updating the netmasks to the correct value.

[TotieBash]

As per the documentation below and the youtube instruction (minute 8:00), It should be Firewall > hostgroups > elastic_agent_endpoint.

As soon as you do that check if the port are listening:
ss -tlpn | egrep '8220|8443|5055'

Then check iptables
iptables --list -n | egrep '8220|8443|5055'

These key elastic ports are mentioned on the youtube video as well

[pezhore]

I appreciate the quick reply, apologies if I'm coming off as being difficult - at this point I'm fairly certain it's something stupid simple that I'm missing and it's quite frustruating.

Output for ss -tlpn | egrep '8220|8443|5055'

sudo ss -tlpn | egrep '8220|8443|5055'
LISTEN 0      4096         0.0.0.0:8443       0.0.0.0:*    users:(("docker-proxy",pid=12823,fd=4))                                                          
LISTEN 0      4096         0.0.0.0:8220       0.0.0.0:*    users:(("docker-proxy",pid=26403,fd=4))                                                          
LISTEN 0      4096         0.0.0.0:5055       0.0.0.0:*    users:(("docker-proxy",pid=18744,fd=4))

Output for iptables:

iptables --list -n | egrep '8220|8443|5055'
ACCEPT     tcp  --  0.0.0.0/0            172.17.1.21          tcp dpt:8220
ACCEPT     tcp  --  0.0.0.0/0            172.17.1.29          tcp dpt:5055
ACCEPT     tcp  --  0.0.0.0/0            172.17.1.31          tcp dpt:8443
ACCEPT     tcp  --  172.19.0.97          0.0.0.0/0            tcp dpt:8220
ACCEPT     tcp  --  172.19.0.97          0.0.0.0/0            tcp dpt:5055
ACCEPT     tcp  --  172.19.0.97          0.0.0.0/0            tcp dpt:8443
ACCEPT     tcp  --  172.19.0.0/24        0.0.0.0/0            tcp dpt:8220
ACCEPT     tcp  --  172.19.0.0/24        0.0.0.0/0            tcp dpt:5055
ACCEPT     tcp  --  172.19.0.0/24        0.0.0.0/0            tcp dpt:8443
ACCEPT     tcp  --  10.29.0.0/24         0.0.0.0/0            tcp dpt:8220
ACCEPT     tcp  --  10.29.0.0/24         0.0.0.0/0            tcp dpt:5055
ACCEPT     tcp  --  10.29.0.0/24         0.0.0.0/0            tcp dpt:8443
ACCEPT     tcp  --  172.19.0.0/24        0.0.0.0/0            tcp dpt:8220
ACCEPT     tcp  --  172.19.0.0/24        0.0.0.0/0            tcp dpt:5055
ACCEPT     tcp  --  172.19.0.0/24        0.0.0.0/0            tcp dpt:8443
ACCEPT     tcp  --  10.29.0.0/24         0.0.0.0/0            tcp dpt:8220
ACCEPT     tcp  --  10.29.0.0/24         0.0.0.0/0            tcp dpt:5055
ACCEPT     tcp  --  10.29.0.0/24         0.0.0.0/0            tcp dpt:8443

On a side note, I have three interfaces on the Seconion box - the primary management interface is on the LAN (172.19.0.97), and the other two interfaces are for monitoring on LAN and the DMZ.

For some reason, those other interfaces have IP addresses:

$ ip a | grep ens
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 172.19.0.97/24 brd 172.19.0.255 scope global noprefixroute ens192
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.29.0.97/24 brd 10.29.0.255 scope global noprefixroute ens224
4: ens256: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 172.19.0.98/24 brd 172.19.0.255 scope global noprefixroute ens256

If I do some host file shenanigans and add an entry for 10.29.0.97 seconion.ift.int seconion, then the agent installs/registers successfully.

Is there an obvious downside to just using that interface? Should those even have IPs?

[TotieBash]

Minute 10 of the youtube does mentioned to check the log file of the elastic agent installation. On this example, the second line shows it is trying to connect to https://so-manager:8220. The local machine does need name resolution through DNS or, just like what you did, through host file entry. Base on what you are saying it does look like name resolution is your problem.

[pezhore]

So I ended up finding the problem - I'm not sure how those extra interfaces monitoring got IP addresses - but that caused some asymmetric routing issues. DMZ hosts were going through our pfsense box to the LAN seconion, but since the seconion box had an interface with an IP on the DMZ, it wouldn't travers the pfsense on the way back.

We discovered the fix on accident while forcing those nics into monitoring mode with sudo so-monitor-add ens224 - after seconion didn't have an interface with an IP on the DMZ, everything functioned as expected.