Clean install, missing fleet server/agent - Issues with internal firewall perhaps ?

[kawaiipantsu]

Version

2.4.10

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

30

Storage for /

2.1TiB

Storage for /nsm

2.1TiB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Clean ISO Standalone installation on x64-86 intel i5 VM with 6 vcpu (Cores) and 30MiB RAM and 2.1TiB NVMe zfs.
3 NICs, 1 Management NIC, 2 Monitor NICs - 1 receiving soft-tap with internal vlan trunk traffic, 1 receiving raw port mirror from wan outside

The OS installation all goes fine, first login initiates the SO setup.
Management NIC get static IP via DHCP.
Choosing FQDN as my local network uses that, it's still just resolving directly to the local ip.
(Tried also just ip and just hostname - Same result)

Please note, SO is on the DMZ network, but the IP i enter to "allow" Management access is from a different subnet (LAN).
So my thoughts are that perhaps you are not allowing the original IP of the host/network in the firewall but only the one i entered and that's why everything fails when trying to enable the Elastic Agent, as it can't connect to the fleet server ...

Management overview

[DMZ] - SecurityOnion (10.10.10.11) NIC 1
[LAN] - 172.30.30.0/24

Monitor overview (just for info, no a problem here i think)
Also i can see that /nsm/pcap is being populated

[WAN] --- MIRROR/SPAN ------------ [NIC 2]
[DMZ] --- vlan trunk tap ------.-- [NIC 3]
[LAN] --- vlan trunk tap -----´

When done with the ncurses-dialog setup the setup then proceeds to do all the "install/salt/init" thing and this takes quite a while. Everything is going fine. Until the very end when building the Elastic Agents and "enabling the fleet server agent" (im guessing). Where i'm presented with some ERROR messages and within those texts i can see the sentence "Not accessible" and i can track that down to that the Agent install can't reach the Fleet server.

# so-status

                         Security Onion Status
                         Container │ Status  │                 Details
───────────────────────────────────┼─────────┼─────────────────────────
                        so-curator │ running │           Up 24 minutes
                 so-dockerregistry │ running │           Up 50 minutes
                     so-elastalert │ running │           Up 24 minutes
                  so-elastic-fleet │ running │           Up 14 minutes
 so-elastic-fleet-package-registry │ running │ Up 35 minutes (healthy)
                  so-elasticsearch │ running │           Up 40 minutes
                       so-idstools │ running │           Up 42 minutes
                       so-influxdb │ running │ Up 45 minutes (healthy)
                         so-kibana │ running │           Up 32 minutes
                         so-kratos │ running │           Up 47 minutes
                       so-logstash │ running │           Up 36 minutes
                          so-mysql │ running │ Up 42 minutes (healthy)
                          so-nginx │ running │ Up 43 minutes (healthy)
                       so-playbook │ running │       Up About a minute
                          so-redis │ running │           Up 36 minutes
                      so-sensoroni │ running │           Up 43 minutes
                            so-soc │ running │           Up 19 minutes
                       so-soctopus │ running │       Up About a minute
                          so-steno │ running │           Up 30 minutes
                so-strelka-backend │ running │           Up 20 minutes
            so-strelka-coordinator │ running │           Up 20 minutes
             so-strelka-filestream │ running │           Up 19 minutes
               so-strelka-frontend │ running │           Up 20 minutes
             so-strelka-gatekeeper │ running │           Up 20 minutes
                so-strelka-manager │ running │           Up 20 minutes
                       so-suricata │ running │           Up 20 minutes
                       so-telegraf │ running │           Up 43 minutes
                           so-zeek │ running │ Up 28 minutes (healthy)

 ✔ This onion is ready to make your adversaries cry!

### Cleaning up temp files in /nsm/elastic-agent-workspace[ERROR   ] Command '/opt/so/__salt.tmp.nh5imw_d' failed with return cod                                                                                  e: 1
[ERROR   ] stdout:
Installation initiated, view install log for further details.
Not Accessible
Not Accessible
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 218749, 'retcode': 1, 'stdout': '\nInstallation initiated, view install log for further details.\nNot Accessib                                                                                  le\nNot Accessible', 'stderr': ''}
[ERROR   ] Command '/opt/so/__salt.tmp.9guh2e_f' failed with return code: 1
[ERROR   ] stdout:
Installation initiated, view install log for further details.
Not Accessible
Not Accessible
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 219929, 'retcode': 1, 'stdout': '\nInstallation initiated, view install log for further details.\nNot Accessib                                                                                  le\nNot Accessible', 'stderr': ''}
local:
----------
          ID: run_installer
    Function: cmd.script
        Name: salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command 'salt://elasticfleet/files/so_agent-i                                                                                  nstallers/so-elastic-agent_linux_amd64' run"
              Command 'salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64' run
     Started: 19:24:24.933255
    Duration: 37665.979 ms
     Changes:
              ----------
              pid:
                  219929
              retcode:
                  1
              stderr:
              stdout:

                  Installation initiated, view install log for further details.
                  Not Accessible
                  Not Accessible

Summary for local
------------
Succeeded: 0 (changed=1)
Failed:    1
------------
Total states run:     1
Total run time:  37.666 s

# cat errors.log
### Cleaning up temp files in /nsm/elastic-agent-workspace[ERROR   ] Command '/opt/so/__salt.tmp.nh5imw_d' failed with return code: 1
[ERROR   ] stdout:
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 218749, 'retcode': 1, 'stdout': '\nInstallation initiated, view install log for further details.\nNot Accessible\nNot Accessible', 'stderr': ''}
[ERROR   ] Command '/opt/so/__salt.tmp.9guh2e_f' failed with return code: 1
[ERROR   ] stdout:
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 219929, 'retcode': 1, 'stdout': '\nInstallation initiated, view install log for further details.\nNot Accessible\nNot Accessible', 'stderr': ''}
      Result: False

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[kawaiipantsu]

Adding the following into firewall for fleet, agents and so on:

10.10.10.0/24
172.30.30.0/24

Rebooting the SO installation and now there is a agent present called "FleetServer-((hostname))"
So that part seems to be fixed, firewall problem ??

But i'm getting no real data into SO at all.
I can't see anything from Zeek, Suricata and so on.

I'm guessing that i'm still missing the "Grid Node" agent ??
Any advise how to fix this, and is this a installation bug where you forget to add access ?

[kawaiipantsu]

To get back to the main issue, clearly theres a problem with the firewall or something denying access to the host.
Again, the above + reboot fixes the "Fleet Server" part. And now i figured out how to trigger the "Grid node agent installation" manually.

sudo salt-call state.apply elasticfleet.install_agent_grid

This now runs successful and the Grid node agent shows up under Fleet.
And now all the logs are starting to show up (connections etc and so on)

But it still leaves the question, what is the problem doing the installation ?
Completely clean install via ISO choosing STANDALONE no funny business, doing things by the book.

[dougburks]

We've done lots of standalone installations that don't have this issue.

x64-86 intel i5 VM with 6 vcpu (Cores)

What other VMs are running? Is it possible your CPU is oversubscribed?
https://docs.securityonion.net/en/2.4/best-practices.html#installation
https://docs.securityonion.net/en/2.4/hardware.html#virtualization

Management NIC get static IP via DHCP.

Have you tried setting an actual static IP address (with DHCP disabled) to see if that makes any difference?

[kawaiipantsu]

I think it comes down to DHCP overwriting hosts file where you rely on a 127.0.0.1 for the hostname that is accepted in the firewall per default.

[dougburks]

Our official recommendation is to use a static IP address and avoid DHCP:
https://docs.securityonion.net/en/2.4/hardware.html#nic