No Fleet Server 2.4

[ColeVan]

Version

2.4.2

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

12

RAM

64

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

I've searched several prior discussions but I could not find the reason for my error. Hopefully, someone can lend a hand.

I have a distributed deployment however, the fleet server did not install properly on the manager, or it did and is not showing up in Elastic fleet. What would be the troubleshooting steps to determine why the fleet sever did not install properly or show up in fleet?

When I check under Agent policies I see under so-grid-nodes_general there are three other Agents. It would appear that the search and two sensor nodes deployed the elastic agent properly. When I run a so-status on the manager everything shows green. When I run a netstat -punta | grep 8220 on search and sensor nodes I see a connection back to my manager IP for fleet.

Very interesting problem.... Not sure where to begin. I ran sudo docker logs so-elastic-fleet > logs and parsed through the file but I still couldn't find the issue... Thoughts?? Thanks in advance!!

Below are some errors I get when running salt-call state.highstate

After a little digging, I see these errors in the master log below.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[ColeVan]

So I found duplicate enrollment keys in two sls files. I commented them out and ran salt-call state.highstate with no errors. Elastic fleet is up and running but still nothing under fleet in kibana.

[ColeVan]

So I rebooted the manager and tried running salt-call state.apply elasticfleet.install_agent_grid to hopefully bring the fleet server up in Kibana and I get the following errors in the error log.

[INFO ] {'pid': 54965, 'retcode': 0, 'stdout': '\nSetting up so-logs-logs policy...\n{"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-lifecycle-so-logs-logs) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-lifecycle-so-logs-logs) within 30s"},"status":503}\n\nSetting up so-logs-elastic_agent.apm_server-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.auditbeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.cloudbeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.endpoint_security-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.alerts-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.api-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.file-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.library-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.network-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.process-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.registry-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-endpoint.events.security-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.filebeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.fleet_server-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.heartbeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.metricbeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.osquerybeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-logs-elastic_agent.packetbeat-logs policy...\n{"acknowledged":true}\n\nSetting up so-suricata-logs policy...\n{"acknowledged":true}\n\nSetting up so-import-logs policy...\n{"acknowledged":true}\n\nSetting up so-kratos-logs policy...\n{"acknowledged":true}\n\nSetting up so-logstash-logs policy...\n{"acknowledged":true}\n\nSetting up so-redis-logs policy...\n{"acknowledged":true}\n\nSetting up so-strelka-logs policy...\n{"acknowledged":true}\n\nSetting up so-syslog-logs policy...\n{"acknowledged":true}\n\nSetting up so-zeek-logs policy...\n{"acknowledged":true}', 'stderr': ''} {"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-lifecycle-so-logs-logs) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-lifecycle-so-logs-logs) within 30s"},"status":503} [ERROR ] Command '/opt/so/__salt.tmp.fo4lnp1q' failed with return code: 1 [ERROR ] stdout: [ERROR ] retcode: 1 [ERROR ] {'pid': 140483, 'retcode': 1, 'stdout': '\nInstallation initiated, view install log for further details.', 'stderr': ''} [ERROR ] Command '/opt/so/__salt.tmp.yefpxx5k' failed with return code: 1 [ERROR ] stdout: [ERROR ] retcode: 1 [ERROR ] {'pid': 141107, 'retcode': 1, 'stdout': '\nInstallation initiated, view install log for further details.', 'stderr': ''} Result: False

The above may be useless since when I run so-elastic-fleet-start it tells me that elastic-fleet is already running.

[ColeVan]

Update. I was finally able to get fleet server and elastic agents from search and sensor nodes to show up in Kibana under fleet tab. After amending the firewall > fleet config everything started showing up. However, the manager appears to be offline still. Still troubleshooting. Any suggestions? I wonder if there are some sort of internal firewall issues going on here.

[defensivedepth]

Do you still get the errors when running a highstate?

That original error you posted looks like maybe so-setup-elastic-fleet was run multiple times?

[ColeVan]

So this is the only error I get when running a highstate. It seems that ElasticSearch can not be reached.

[ColeVan]

Actually, I found more errors.

[defensivedepth]

If you look at the following files on your Manager, do you have duplicate entries for Elastic Fleet configs?

/opt/so/saltstack/local/pillar/minions/$MINIONID.sls

[defensivedepth]

Yes, sounds like something happened with the original Fleet setup and it now needs to be reset. I am actually working on a script right now to reinstall Fleet - unfortunately, it is not as simple as just re-running the setup script for Fleet.

[ColeVan]

we are deploying everything again now. I'll report back if the issue persists.

[harleytomlinsonOMB]

We have this issue too, in our pre-prod SO distributed environment. I've been attempting to configure a fleet node, and while the SO installer on that new fleet node shows fleet state starting and component fleet server healthy, the node never checks in and isn't viewable in grid. (hope I'm using all correct jargon, new to SO)

I've looked at the minions on the manager and don't see fleet. I do see the search node which apparently is fine, along with the manager.

Your last comment was that you're deploying everything again now, so this is a 'nuke & pave' situation or how should we proceed?

Thanks!

[harleytomlinsonOMB]

I found that the fleet node needed to be accepted back in the main SO console > administration > Grid members > pending members. Now the fleet node shows up in minions on the manager /opt/so/.../minions/, elastic agents all show healthy in the fleet dashboard including the new fleet node.

[defensivedepth]

@harleytomlinsonOMB Yep that is correct, good to hear you got it figured out!

[defensivedepth]

@ColeVan How goes it? .30 was released yesterday with that reset script.