UI says: "Add a Fleet server" while the fleet extension is running

[bastien-rcl]

Problem description

I have installed the Fleet extension as covered by the Readme ( adding -f extensions/fleet/fleet-compose.yml to my docker compose command).

I can see the service running with docker compose ps.

Part of my confusion is that I don't grasp how the components should interact together.
My assumption is that when I go to the Kiabana UI -> Management -> Fleet I should be able to enroll an agent in Fleet right away (maybe I'm wrong). But instead, the UI asks me to "Add a Fleet server" and provides the instructions to do so manually.

My understanding is that the Fleet server is already installed through the docker compose service and Kibana should not ask me to install a server.

Is that right ? And then I should start troubleshooting the issue. Or am I mistaken in my understanding and I should really install the Fleet server following the Kibana instructions ? But then what is the Fleet service in the docker compose ?

Thank you !

Extra information

Stack configuration

Docker setup

$ docker version

Client: Docker Engine - Community
 Version:           28.0.1
 API version:       1.48
 Go version:        go1.23.6
 Git commit:        068a01e
 Built:             Wed Feb 26 10:41:12 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.1
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.6
  Git commit:       bbd0a17
  Built:            Wed Feb 26 10:41:12 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker compose version

Docker Compose version v2.33.1

Container logs

$ docker compose logs

(this server is under load and there are a lot of logs. Please specify which log could be useful, if any, and I will search for it)

[antoineco]

Your assumption is correct, the server should be automatically registered and visible in Kibana like in the screenshot attached to the README. I'd suggest to start looking into the logs of the fleet-server service.

[bastien-rcl]

Thanks for your help.

What looks like a possible cause in the Fleet logs is this :

{"log.level":"error","@timestamp":"2025-06-11T10:06:04.284Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

I'm trying to understand why would elastic refuse the connection.
So far I could not see any useful logs in elastic regarding connection issues.

I may add that I have slightly modified the compose file(s) to include Traefik in the mix in order to ease with the TLS certificates. But it should not have anything to do with internal connections between containers in my understanding. It is not fiddling with inter container connections when it comes to Kibana <-> Elastic anyway.

[bastien-rcl]

Now that I come to think of it, [::1], from fleet-server's perspective is itself, not elastic. And that's perfectly normal that it finds nothing on port 9200 on its own container.
So the question is, why does it try to connect to elastic through [::1] ? Where does it get that misleading information ?

[bastien-rcl]

From the fleet container :

root@fleet-server:/usr/share/elastic-agent# cat elastic-agent.yml             
######################################                                        
# Fleet configuration                                                         
######################################                                                                                                                       
outputs:                                                                      
  default:                                                                    
    type: elasticsearch                                                                                                                                      
    hosts: '${ELASTICSEARCH_HOSTS:http://elasticsearch:9200}'                 

$ELASTICSEARCH_HOSTS is empty so I guess it defaults to http://elasticsearch:9200.

And trying to connect to elastic from fleet does seem to work :

root@fleet-server:/usr/share/elastic-agent# curl http://elasticsearch:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","ApiKey"]}},"status":401}

[antoineco]

You can't connect from the Fleet container without credentials, so this error is expected. By default, the Fleet extension attempts to enroll using the elastic user:

docker-elk/extensions/fleet/fleet-compose.yml

Lines 27 to 30 in ac4a01f

	# Enrollment.
	# (a) Auto-enroll using basic authentication
	ELASTICSEARCH_USERNAME: elastic
	ELASTICSEARCH_PASSWORD: ${ELASTIC_PASSWORD:-}

Check the logs of the Fleet container, it should contain entries that can point you into the right direction. For example, it could be that the ELASTIC_PASSWORD in your .env file no longer matches the actual password for that user.

[bastien-rcl]

Thanks @antoineco, maybe my comments above are too verbose, I'll try to summarize them here.

The only error log in fleet is

{"log.level":"error","@timestamp":"2025-06-11T10:06:04.284Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

So basically the network connection can't be established with the elasticsearch container. Which seems normal since the fleet-server container tries to connect to [::1]:9200 which would be the fleet-server container itself, that does not host anything on port 9200. It seems to be mixing up its own container network address for the elasticsearch container address.

In order to prove that, I attach to the fleet-server container and try to curl [::1]:9200 which returns the same network error, as expected (curl: (7) Failed to connect to ::1 port 9200: Connection refused). And then I try another curl from the fleet-server container but instead of [::1]:9200 I curl http://elasticsearch:9200 which works (please ignore the credentials error, this is irrelevant for the test which is just proving that network connectivity works when using the correct address for the elasticsearch container).

So that means that somewhere in the configuration, the fleet-server container uses [::1]:9200 instead of elasticsearch:9200 to reach the elasticsearch container. And obviously that fails. I'm trying to find out where this happens and why. Any hint would be of great help. Thanks.

[antoineco]

My bad, I missed the part about the [::1] address in your previous comments.

Maybe Elastic changed the default URL in v9, I'll need to look at the changes more closely.

Could you please try setting it explicitly, like we do on the tls branch? Simply replace https with http in the snippet below:

docker-elk/extensions/fleet/fleet-compose.yml

Line 33 in e4e9103

ELASTICSEARCH_HOST: https://elasticsearch:9200

[bastien-rcl]

Yes thanks. I need to wait until outside business hours to fiddle with the server. I will report then.

[bastien-rcl]

I did the upgrade to the latest commit, running 9.0.2.
I did add the ELASTICSEARCH_HOST: http://elasticsearch:9200 variable and checked within the running container that it's actually there.

The behavior is different now so I think it did something although I still see some errors trying to connect to [::1}.
Now the fleet-server container keeps restarting with exit code 1.

Here is the log. I try to understand what I could have done wrong.

fleet-server-1  | agent container initialisation - chown paths                                                                                                                                                                                                                                                            
fleet-server-1  | Requesting service_token from Kibana.                                                                                                                                                                                                                                                                   
fleet-server-1  | Created service_token named: token-1749759659060                                                                                                                                                                                                                                                        
fleet-server-1  | Policy selected for enrollment:  fleet-server-policy                                                                                                                                                                                                                                                    
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.049Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).startAgent","file.name":"cmd/enroll_cmd.go","file.line":677},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap p
rocess.","ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.096Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":193},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":48,"agent.vers
ion":"9.0.2","agent.unprivileged":false,"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.302Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"agent is not upgradable, not starting watcher","log":{"source
":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                 
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.303Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":282},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"
}                                                                                                                                                                                                                                                                                                                         
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.303Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":73},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.versio
n":"1.6.0"}                                                                                                                                                                                                                                                                                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":79},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"
inputs":["entity-analytics","unix","logstash/metrics","mongodb/metrics","azure/metrics","rabbitmq/metrics","vsphere/metrics","cloudbeat/cis_aws","pf-elastic-symbolizer","aws-s3","filestream","iis/metrics","meraki/metrics","cloudbeat/asset_inventory_gcp","fleet-server","container","streaming","synthetics/browser",
"http/metrics","memcached/metrics","sql/metrics","zookeeper/metrics","system/metrics","pf-host-agent","linux/metrics","endpoint","audit/file_integrity","azure-blob-storage","cloudfoundry","etw","kafka","syslog","nginx/metrics","stan/metrics","aws-cloudwatch","log","elasticsearch/metrics","uwsgi/metrics","etcd/met
rics","statsd/metrics","packet","azure-eventhub","journald","unifiedlogs","synthetics/http","synthetics/icmp","cloudfoundry/metrics","mssql/metrics","haproxy/metrics","audit/auditd","cometd","http_endpoint","mqtt","oracle/metrics","jolokia/metrics","osquery","cloudbeat","docker","syncgateway/metrics","gcp/metrics
","cloudbeat/cis_azure","cloudbeat/vuln_mgmt_aws","awsfargate/metrics","cloudbeat/cis_k8s","cloudbeat/cis_gcp","cloudbeat/asset_inventory_aws","gcp-pubsub","tcp","beat/metrics","traefik/metrics","mysql/metrics","cel","gcs","lumberjack","salesforce","udp","kibana/metrics","containerd/metrics","httpjson","synthetic
s/tcp","redis/metrics","cloudbeat/asset_inventory_azure","winlog","docker/metrics","apache/metrics","nats/metrics","openai/metrics","panw/metrics","prometheus/metrics","apm","audit/system","benchmark","redis","kubernetes/metrics","postgresql/metrics","windows/metrics","activemq/metrics","kafka/metrics","netflow",
"o365audit","aws/metrics","cloudbeat/cis_eks","pf-elastic-collector"],"ecs.version":"1.6.0"}                                                                                                                                                                                                                              
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilit
ies.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                          
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":85},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.ve
rsion":"1.6.0"}                                                                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":100},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agen
t.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.531Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":164},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"a
ddress":"localhost:6789","ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.537Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":176},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap m
ode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                              
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.537Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":91},"message":"GRPC control socket listening at unix:///usr/share/elasti
c-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"}                                                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":226},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104
857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":127},"message":"Source URI changed from \"https://artifacts.elastic.co/do
wnloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.M
onitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc001257f20), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"
\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.L
imit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg a
pi.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                       
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}                                                                                                                                                                                                                                                                                                                       
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://local
host:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                       
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1450},"message":"Updating running component
 model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":676},"message":"Spawned new component flee
t-server-default: Starting: spawned pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}                                                                                                                                                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default: Starting: spawned pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}                                                                                     
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default-fleet-server: Starting: spawned pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.663Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ppid":48,"service.name":"fleet-server"
,"version":"9.0.2","commit":"5c7037cf","exe":"/usr/share/elastic-agent/data/elastic-agent-5368ce/components/fleet-server","args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-default"],"ecs.version":"1.6.0","service.ty
pe":"fleet-server","pid":68,"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.663Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"
ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                                                                                                                                                                                                   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.666Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":693},"message":"control checkin v2
 protocol has chunking enabled","ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                    
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.666Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":696},"message":"Component state changed fl
eet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}                                                                                                  
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:02.051Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":843},"message":"Fleet Server - waiting for output unit","ecs.version":"1.6.0"}      
fleet-server-1  | {"log.level":"error","@timestamp":"2025-06-12T20:21:08.012Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fl
eet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}                                                                                                                                                                 
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.012Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-se
rver-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"Elasticsearch":{"Headers":null,"Hosts":["localhost:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"http","ProxyDisa
ble":false,"ProxyHeaders":{},"ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":null,"Timeout":90000000000},"Extra":null},"ecs.version":"1.6.0","ecs.version":"1.6.0"}                                                                                                                                 
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.012Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0
","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0"}                                                                                                                                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.012Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","servi
ce.name":"fleet-server","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":536870912,"NumCounters":6400000},"ecs.version":"1.6.0"}                                                      
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-serv
er","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}                                                                                                                                                                                                                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0
","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0"}                                                                                                                                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","servi
ce.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                                                                                                                                                                                                                              
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.
6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                                                                                                                                                                                                                   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.na
me":"fleet-server","service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0"}                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.020Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version
":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.0.2","elasticsearch_version":"9.0.2","@timestamp":"2025-06-12T20:21:08.02Z","ecs.version":"1.6.0"}                                                                                                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","ctx":"policy agent monitor","burst":1,"ecs.version":"1.6.0","service.type":"fleet-server","event_rate":4000,"ecs.version":"1.6.0"}                                  
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0","ecs.version":"1.6.0"}                                       
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0"}                                        
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.022Z","message":"Exposed over insecure HTTP; enablement of TLS is strongly recommended","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                        
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.022Z","message":"Exposed over insecure HTTP; enablement of TLS is strongly recommended","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}     
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.528Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.528Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}         
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.528Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}                                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.035Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.060Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":824},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}                                                   
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:14.060Z","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}                                                                   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.422Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":534},"message":"Starting enrollment to URL: http://0.0.0.0:8220/","ecs.version":"1.6.0"}                                                                    
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"ApiKey fail authentication","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"unauthorized: apikey auth response UL0abJUBZ4ny01R-6_-3: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"UL0abJUBZ4ny01R-6_-3","event.duration":1190712,"ecs.version":"1.6.0","http.request.id":"01JXJWVJ85W7YCZ68618GDJNYB","server.address":"127.0.0.1:8220","ecs.version":"1.6.0"}                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":540},"message":"1st enrollment attempt failed, retrying enrolling to URL: http://0.0.0.0:8220/ with exponential backoff (init 5s, max 10m0s)","ecs.version":"1.6.0"}                                                                       
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"HTTP request error","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"event.duration":1242339,"ecs.version":"1.6.0","http.request.id":"01JXJWVJ85W7YCZ68618GDJNYB","server.address":"127.0.0.1:8220","error.message":"unauthorized: apikey auth response UL0abJUBZ4ny01R-6_-3: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}
","error.type":"*fmt.wrapErrors","service.name":"fleet-server","service.type":"fleet-server","http.response.status_code":401,"ecs.version":"1.6.0"}          
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":559},"message":"Enrollment failed: fail to execute request to fleet-server: st
atus code: 401, fleet-server returned an error: ErrUnauthorized, message: unauthorized","ecs.version":"1.6.0"}                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"perform rollback on enrollment failure","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6
.0","service.type":"fleet-server","http.request.id":"01JXJWVJ85W7YCZ68618GDJNYB","server.address":"127.0.0.1:8220","mod":"enroll","service.name":"fleet-server","error.message":"unauthorized: apikey auth response UL0abJUBZ4ny01R-6_-3: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",
\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"A
piKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\
\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","ecs.version":"1.6.0"}                                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"HTTP Request","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","server.address":"127.
0.0.1:8220","http.request.method":"POST","event.duration":1271904,"service.name":"fleet-server","service.type":"fleet-server","fleet.apikey.id":"UL0abJUBZ4ny01R-6_-3","client.address":"127.0.0.1:46348","http.request.body.bytes":0,"http.version":"1.1","tls.established":false,"http.request.id":"01JXJWVJ85W7YCZ68618
GDJNYB","url.full":"/api/fleet/agents/enroll","http.response.body.bytes":69,"http.response.status_code":401,"ecs.version":"1.6.0"}                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":377},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":389},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-age
nt"},"ecs.version":"1.6.0"}                                                   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-ser
ver","service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0"}  
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.831Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":395},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.832Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"e
lastic-agent"},"ecs.version":"1.6.0"}                                         
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.832Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: us
e of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                      
fleet-server-1  | Error: fail to enroll: fail to execute request to fleet-server: status code: 401, fleet-server returned an error: ErrUnauthorized, message: unauthorized
fleet-server-1  | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
fleet-server-1  | Error: enrollment failed: exit status 1
fleet-server-1  | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
fleet-server-1 exited with code 1

[bastien-rcl]

I tried providing a FLEET_SERVER_SERVICE_TOKEN previously created from the kibana interface.
Before that, the fleet-server container exited with code 1 about every 15 seconds. After setting the token, it still exits with code 1 albeit a bit more gracefully, after reaching its 2 minutes timeout :

Error: fleet-server failed: timed out waiting for Fleet Server to start after 2m0s

The logs look pretty similar to me :

fleet-server-1  | agent container initialisation - chown paths                                                                                                                                                                                                                                                            
fleet-server-1  | Policy selected for enrollment:  fleet-server-policy                                                                                                                                                                                                                                                    fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.206Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).startAgent","file.name":"cmd/enroll_cmd.go","file.line":677},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap p
rocess.","ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                           fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.450Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":193},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":37,"agent.vers
ion":"9.0.2","agent.unprivileged":false,"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                            fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.662Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                 fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.672Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":282},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"
}                                                                                                                                                                                                                                                                                                                         fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.674Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":73},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.versio
n":"1.6.0"}                                                                                                                                                                                                                                                                                                               fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.715Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":79},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"
inputs":["cloudbeat/cis_eks","container","windows/metrics","aws/metrics","haproxy/metrics","kafka/metrics","sql/metrics","traefik/metrics","cel","synthetics/icmp","kibana/metrics","redis/metrics","uwsgi/metrics","cloudbeat/cis_k8s","docker","udp","unix","synthetics/http","docker/metrics","system/metrics","awsfargate/metrics","cloudbeat/cis_gcp","syslog","kubernetes/metrics","mysql/metrics","postgresql/metrics","syncgateway/metrics","nats/metrics","nginx/metrics","cloudbeat/asset_inventory_gcp","pf-elastic-collector","cloudfoundry","journald","linux/metrics","mssql/metrics","oracle/metrics","fleet-server","httpjson","lumb
erjack","etcd/metrics","openai/metrics","stan/metrics","statsd/metrics","cloudbeat/vuln_mgmt_aws","cloudbeat/asset_inventory_aws","pf-host-agent","audit/system","gcp-pubsub","gcs","http_endpoint","cloudfoundry/metrics","cloudbeat/cis_aws","redis","azure/metrics","memcached/metrics","osquery","azure-blob-storage","mqtt","elasticsearch/metrics","winlog","zookeeper/metrics","apm","pf-elastic-symbolizer","aws-s3","cometd","log","http/metrics","gcp/metrics","iis/metrics","jolokia/metrics","beat/metrics","benchmark","salesforce","tcp","filestream","synthetics/browser","activemq/metrics","prometheus/metrics","cloudbeat","cloudb
eat/cis_azure","aws-cloudwatch","azure-eventhub","o365audit","unifiedlogs","synthetics/tcp","panw/metrics","streaming","endpoint","audit/auditd","entity-analytics","mongodb/metrics","rabbitmq/metrics","vsphere/metrics","cloudbeat/asset_inventory_azure","etw","kafka","netflow","logstash/metrics","containerd/metrics","meraki/metrics","packet","audit/file_integrity","apache/metrics"],"ecs.version":"1.6.0"}                                                                                                                                                                                                                              
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.716Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                          
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.716Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":85},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.716Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":100},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                            fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.045Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":164},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"a
ddress":"localhost:6789","ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                           fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.056Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":176},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap m
ode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                              fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.060Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":91},"message":"GRPC control socket listening at unix:///usr/share/elasti
c-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"}                                                                                                            fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.062Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":226},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104
857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.064Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":127},"message":"Source URI changed from \"https://artifacts.elastic.co/do
wnloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.M
onitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc001824030), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"
\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.L
imit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg a
pi.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.065Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}                                                                           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.066Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://local
host:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}           
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.067Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1450},"message":"Updating running component
 model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.218Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":676},"message":"Spawned new component flee
t-server-default: Starting: spawned pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.218Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default-fleet-server: Starting: spawned pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.218Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default: Starting: spawned pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.257Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"pid":54,"service.name":"fleet-server",
"service.type":"fleet-server","version":"9.0.2","commit":"5c7037cf","ppid":37,"exe":"/usr/share/elastic-agent/data/elastic-agent-5368ce/components/fleet-server","args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-defa
ult"],"ecs.version":"1.6.0","ecs.version":"1.6.0"}                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.258Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"
ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                                      
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.264Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":693},"message":"control checkin v2
 protocol has chunking enabled","ecs.version":"1.6.0"}                        
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.264Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":696},"message":"Component state changed fl
eet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:43.213Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":843},"message":"Fleet Server - waiting for output unit","ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"error","@timestamp":"2025-06-12T20:50:53.358Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"error.message":"d
ial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}    
fleet-server-1  | {"log.level":"warn","@timestamp":"2025-06-12T20:50:53.358Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-se
rver-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"Elasticsearch":{"Headers":null,"Hosts":["localhost:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"http","ProxyDisable":false,"ProxyHeaders":{},"
ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":null,"Timeout":90000000000},"Extra":null},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.359Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0
","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0"}                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.359Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server
","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":536870912,"NumCounters":6400000},"ecs.version":"1.6.0","ecs.version":"1.6.0"}
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","state":"STARTING"
,"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"recommended_mb":323
84,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"ecs.version":"1.6.0"}                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","servi
ce.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}  
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.
6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}                                                                      
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","se
rvice.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0","ecs.version":"1.6.0"}

[...]


fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.496Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.na
me":"fleet-server","service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0"}                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.497Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-s
erver-default-fleet-server (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}                                     
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.497Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-s
erver-default (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}                                                              
fleet-server-1  | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6
.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","ecs.version":"1.6.0"}                                                                      
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.498Z","message":"Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","component":{"binary"
:"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}                                               
fleet-server-1  | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name"
:"fleet-server","service.type":"fleet-server","error.message":"failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","ecs.version":"1.6.0"}                                 
fleet-server-1  | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-
server-default-fleet-server (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"source":"elastic-agent"},"component":{"id":"fleet-serv
er-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}                                                                                                                                                            
fleet-server-1  | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-
server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","
state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}                                                                                                                                                                                     
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":377},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}                                                                                                                                                                                                                                                                                                                       
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":389},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-age
nt"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                               
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":793},"message":"Waiting for Elastic Agent to start","ecs.version":"1.6.0"}          
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","sig":"terminated
","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}                                                                                                                                                                                                                                              
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"flee
t-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0","ecs.version":"1.6.0"}                                                                                                                                                                                            
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.413Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":395},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}   
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.413Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"e
lastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                                                                     
fleet-server-1  | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.413Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: us
e of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                                                   
fleet-server-1  | Error: fleet-server failed: timed out waiting for Fleet Server to start after 2m0s                                                                                                                                                                                                                      
fleet-server-1  | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html                                                                                                                                                                                  
fleet-server-1  | Error: enrollment failed: exit status 1                                                                                                                                                                                                                                                                 
fleet-server-1  | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html                                                                                                                                                                                  
fleet-server-1 exited with code 1                                                                                                                                                                                                                                                                                         

[antoineco]

You are right, the Fleet Server is still trying to reach Elasticsearch on localhost: dial tcp [::1]:9200: connect: connection refused.

It will be difficult for me to help without a full view of the system. If it is an option for you, I'd recommend wiping Elasticsearch's data entirely and starting over, to ensure that Kibana initializes everything correctly.

[bastien-rcl]

Yes I have been thinking about wiping it clean. But I really can't afford to lose the data on it.
Do you recommend any way to re-initialize without losing any data ?
I will be looking into the snapshot / restore way. And of course I don't want to restore the settings that are causing this issue.
I guess I can try to run the setup again, it should not be of any harm.

[antoineco]

The setup won't be of any help here, but resetting the Fleet state could. I do not know how to achieve this, but maybe the Elastic forums could point you into the right direction.