Elastic Agent install

[tuamortemo]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

other (please provide detail below)

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

16

Storage for /

200gb

Storage for /nsm

800gb

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hello,
Have a problem with connecting Elastic Agent Endpoint to Elastic Fleet Server.
I have distributed type of installation - 1 manager and 1 search in one network, forward and Fleet nodes in another network. The second ones are connected to the manager via VPN. The management interface for them was specified tun0 during installation and, accordingly, the manager communicates with them via their VPN ip.
To connect agents to the fleet server, I need to use its internal ip. In accordance with the instructions (https://docs.securityonion.net/en/2.4/elastic-fleet.html#settings), I added a Custom FQDN so that the agent could connect to it, but there was a certificate problem, because it was issued under vpn ip.
Unfortunately, I don't really understand where to look

Fleet-server:
name: fleet-mt-r
ens192: 10.87.133.50
tun0: 10.99.2.50

And second question is about fleet outputs in settings. There is no entry for the new ip:5055 in the fleet outputs menu

Agent Installing log:

timestamp=2024-04-03T16:47:45.3005136+03:00 level=info message="Version Information" ElasticAgentVersion=8.7.0 WrapperVersion=2.4.2
timestamp=2024-04-03T16:47:45.3554457+03:00 level=info message="Runtime Data" EnrollmentToken="dDFDM2RZNEJzbEMzUUU2S29iRng6Q2JqaWJEVnFRUXVvd0ljTmw3SDAzUQ==" FleetURL/s=https://10.99.1.30:8220,https://manager-so:8220,https://10.87.133.50:8220,https://fleet-mt-r:8220
timestamp=2024-04-03T16:47:45.3559688+03:00 level=info message="Installation Progress" Status="Starting Installation Precheck"
timestamp=2024-04-03T16:47:48.3590166+03:00 level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://10.99.1.30:8220
timestamp=2024-04-03T16:47:50.652759+03:00 level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://manager-so:8220
timestamp=2024-04-03T16:47:50.6778563+03:00 level=info message="Installation Progress" FleetHostConnectivityCheck=Success FleetHostURL=https://10.87.133.50:8220
timestamp=2024-04-03T16:47:50.6783767+03:00 level=info message="Installation Progress" Status="Fleet Host is accessible - Continuing installation"
timestamp=2024-04-03T16:47:50.6783767+03:00 level=info message="Installation Progress" Status="Installation Precheck Complete"
timestamp=2024-04-03T16:47:50.6783767+03:00 level=info message="Installation Progress" Status="Extracting Elastic Agent files"
timestamp=2024-04-03T16:48:02.0240223+03:00 level=info message="Installation Progress" Status="Executing Elastic Agent installer"
timestamp=2024-04-03T16:48:02.0240223+03:00 level=info message="Installation Progress" Status="Executing the following: ./so-elastic-agent_source/elastic-agent/elastic-agent install --url=https://10.87.133.50:8220 --enrollment-token=dDFDM2RZNEJzbEMzUUU2S29iRng6Q2JqaWJEVnFRUXVvd0ljTmw3SDAzUQ== --certificate-authorities=C:\Program Files\Elastic\SO\soca.crt -n"
timestamp=2024-04-03T16:48:24.7950881+03:00 level=error message="Installation Progress" Context="Installing in non-interactive mode.{"log.level":"info","@timestamp":"2024-04-03T16:48:22.327+0300","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Starting enrollment to URL: https://10.87.133.50:8220/\",\"ecs.version\":\"1.6.0\"}\nError: fail to enroll: fail to execute request to fleet-server: x509: certificate is valid for 10.99.2.50, not 10.87.133.50\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html\nError: enroll command failed with exit code: 1\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html\n" ErrorDetails="exit status 1"

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Did you also regenerate the installers after changing to custom FQDN?

[tuamortemo]

Yes. I have reinstall fleet server to change management interface. After adding custom FQDN its not editing Fleet Outputs (10.87.133.50:5055). Agent connect to new fleet server but not sending event

Fleet server hosts

grid-default
10.99.1.30:8220
manager-so:8220
10.87.133.50:8220
fleet-mt-r:8220

Outputs Field

grid-logstash | logstash
10.99.1.30:5055
manager-so:5055
fleet-mt-r:5055

[tuamortemo]

ID: so-elastic-fleet-auto-configure-logstash-outputs
Function: cmd.run
Name: /usr/sbin/so-elastic-fleet-outputs-update
Result: True
Comment: Command "/usr/sbin/so-elastic-fleet-outputs-update" run
Started: 11:55:09.139958
Duration: 8633.594 ms
Changes:
----------
pid:
638367
retcode:
0
stderr:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100  7795  100  7795    0     0  52315      0 --:--:-- --:--:-- --:--:-- 54131
              [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
              [INFO    ] Syncing grains for environment 'base'
              [INFO    ] Loading cache from salt://_grains, for base
              [INFO    ] Caching directory '_grains/' for environment 'base'
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed

                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100   378  100   131  100   247    231    435 --:--:-- --:--:-- --:--:--   666
              100   378  100   131  100   247    231    435 --:--:-- --:--:-- --:--:--   666
          stdout:

              Hashes don't match - update needed.
              Current List: ["10.99.1.30:5055","manager-so:5055","fleet-mt-r:5055"]
              New List: ["10.99.1.30:5055","manager-so:5055","10.87.133.50:5055","10.87.133.22:5055","fleet-mt-r:5055"]
              {
                "statusCode": 400,
                "error": "Bad Request",
                "message": "Output of type \"logstash\" is not usable with policy \"Fleet Server Policy\"."
              }

[tuamortemo]

Fleet Server Policy is in default state

[tuamortemo]

Find the issue. Was my fault when i try to add Outputs by editing it in Kibana. I've create th Fleet Server Policy which SO can't to edit and it cause of Error. After removing policy its look fine! Sorry for the unnecessary discussion