Security Onion 2.3.210 WinLogBeat's agent traffic traversing NAT

[Ghost53574]

Good day,

I have been trying to setup Security Onion (SO) WinLogBeat agents, and Wazuh agents, with the PFSense firewall that is all hosted via ESXI. The problem I keep on having is that SO is not responding to the forwarded agent traffic, so the agent times out trying to reach the SO manager.

The network looks like the following:

                       External - [ Enterprise Network 192.168.0.0/24 ] -
[ MS Windows 10 Pro : Workstation 192.168.0.4 ] --- [ MS Windows 2019 Server : 192.168.0.3 ]
                                                ||
                                 [ PFSense : 192.168.0.2 (WAN)  ]
                       Internal -[         : 10.0.0.1    (LAN)  ]
                                        [ SO : 10.0.0.2 ]

External and Internal are vSwitches and port groups assigned to those networks and hosts. The SO configuration is setup in a standalone air-gapped config.

On the PFSense firewall I have tried the following:

1. Traditional PNAT (Pure NAT)
2. Static NAT (1:1 - /w 192.168.0.5 virtual IP)

Each of the NAT rules in PFSense automatically generate firewall allow rules. In addition for testing purposes I allowed all traffic from the the WAN to the LAN network and I have verified from the SO manager interface, via tcpdump, that SO is in fact receiving traffic from the PFSense. This was also checked via capturing traffic on PFSense.

When testing the PNAT configuration on both the Workstation and the Server, in the Enterprise Network, the IP set for the manager IP was 192.168.0.2:9200 for elasticsearch and 192.168.0.2:5044 for logstash. Do note that either configuration was tried separately (elasticsearch versus logstash). During this config the traffic was being forwarded as stated before. What was noticed via PFSense's state view was that all the sessions going from 192.168.0.3 to 10.0.0.2 were labelled SYN/WAIT meaning that the endpoint (SO) never responded.

When testing the SNAT configuration on both the Workstation and the Server, in the Enterprise Network, the IP set for the manager IP was 192.168.0.5:9200 for elasticsearch and 192.168.0.5:5044 for logstash. Do note that either configuration was tried separately (elasticsearch versus logstash). During this config the traffic was being forwarded as stated before. What was noticed via PFSense's state view was that all sessions going from 192.168.0.5 to 10.0.0.2 were labelled as FIN & TIME_WAIT meaning that the endpoint received traffic, and responded by closing the connection.

Also noting for the SNAT config that the traffic sniffed from the manager interface on SO included the HTTP data from WinLogBeats (from the Server) that varied in having it's Host header set to 192.168.0.5. This seemed to be part of the problem on why the traffic was not being sent back. But what is crazy is that not traffic was being sent back at all from Security Onion. SO seemed like it was just choosing to not respond.

Please note that IPtables on SO was updated to allow:

• Analyst traffic from both 10.0.0.0/24 & 192.168.0.0/24
• ElasticSearch REST API traffic from both 10.0.0.0/24 & 192.168.0.0/24
• Logstash traffic from both 10.0.0.0/24 & 192.168.0.0/24
• Wazuh registration and agent traffic from both 10.0.0.0/24 & 192.168.0.0/24

For both NAT configurations while testing the deployment of the Wazuh agent the same behavior occurred. What was even more crazy was that the OSSEC config manager window, after entering in the base64 encoded agent key, actually pulled the agent name and id from the SO server! But subsequently recorded in it's logs that the service couldn't reach the SO manager by timing out.

I'm at a loss for what to try now next to adding another interface to SO and hooking it up to the External vSwitch, which defeats the purpose of NAT'ing SO behind a firewall. I've read a lot of setup blogs for configurations for WinLogBeats, Wazuh and Sysmon. Watched the instructional videos on YouTube that Security Onion Solutions provides, even referenced the Security Onion book we've received and I just can't make it work.

Please also note that we use Velociraptor for the beginning portion of our IR and those agents work just fine with the PNAT config. I also am able to SSH into SO from the External network but can't reach the web interface.

[InfosecGoon]

Is there anything useful in the Logstash logs on the standalone (/opt/so/logs/logstash/logstash.log) indicating a connection error or other misconfiguration?

[Ghost53574]

I had time last week to setup the lab again and I'll share what I found.

The connection in between the Windows 10 Client and the Security Onion installation was fine in the sense that the traffic was being forwarded fine between the PFSense firewall and both endpoints. Security Onion was receiving traffic from the port forwards (from 192.168.0.4) and Security Onion was responding to the traffic. What seems to happen is that after the WinLogBeats agent sends it's configuration (via the logstash configuration of 192.168.20.2:5044) the java based server that sits in the logstash container has an application level exception. This is seen on the network by Security Onion sending a reset packet after the a couple of push packets of data (as Security Onion drops the connection because it gets interrupted).

This is the screenshot from the Sec

urity Onion console.

As stated earlier I had done packet captures on the PFSense firewall and on Security Onion's management interface, they all had the same packet flow ending with a RST packet, after a couple PSH packets and after the three way (or four) TCP handshake. So from the perspective of a network engineer there should be no issues with the connection.

After reading up on some common issues with logstash the recommendations that I saw where setting the pipeline (workers?) to zero, and/or increasing the idle timeout to 120 or greater. I have not tried these methods because I ran out of time testing for this past week (only could complete in one day).

The running configuration on the Windows 10 host has the elastic block commented out and the logstash block uncommented to only include the logstash header and the hosts field which is set to 192.168.20.2:5044. There were no changes to the Security Onions logstash.yml file thus far.

[Ghost53574]

bump?

[Ghost53574]

bump?

[InfosecGoon]

This is a head-scratcher, for sure. What version of SO are you running? Is it the version of Winlogbeat from the SOC Downloads screen? Can you share your winlogbeat.yml file?

[Ghost53574]

I can't because another peer is using the server for something else and wiped it. The version is in the title (2.3.210) for SO, the installed agent is from the SOC Downloads page and the only modification to the winlogbeat.yml file was uncommenting the logstash option and setting the host to 192.168.0.2:5044 and commenting out the elasticsearch option.