Elastic Endpoint failing to communicate with Elastic Agent on host - "BulkQueueConsumer.cpp:186 No valid comms client available"

[DoughnutPeach]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

16

Storage for /

500

Storage for /nsm

500

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Ask

I'm unable to get Elastic Agent working on a couple of hosts (~20/100) in my environment.

Firewall and Security Group rules are consistent across all hosts. The Elastic Endpoint (used by Elastic Defend Integration) is unable to communicate with the Elastic Agent on the same host. This seems to be the root cause of the issue.

Context

Logs from those with failing Elastic Agents shows the following:

/opt/Elastic/Endpoint/state/logs/endpoint-*-.log

{"@timestamp":"2024-03-28T08:20:00.589598228Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":452,"name":"Entry.cpp"}}},"message":"Entry.cpp:452 Failed to apply initial policy from on disk configuration (Initial Policy Application Failed)","process":{"pid":234774,"thread":{"id":234774}}}
{"@timestamp":"2024-03-28T08:20:00.589605835Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":459,"name":"Entry.cpp"}}},"message":"Entry.cpp:459 Starting Endpoint main execution","process":{"pid":234774,"thread":{"id":234774}}}
{"@timestamp":"2024-03-28T08:20:01.545684017Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":127,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:127 Validated agent (233035) is root/admin","process":{"pid":234774,"thread":{"id":234805}}}
{"@timestamp":"2024-03-28T08:20:01.545979188Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":135,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:135 Established stage 1 connection to agent","process":{"pid":234774,"thread":{"id":234805}}}
{"@timestamp":"2024-03-28T08:20:02.547293633Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":925,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:925 Connecting to Agent.","process":{"pid":234774,"thread":{"id":234805}}}
{"@timestamp":"2024-03-28T08:20:05.503000845Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":186,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:186 No valid comms client available","process":{"pid":234774,"thread":{"id":234780}}}
< truncated repeated lines > 
{"@timestamp":"2024-03-28T08:20:50.503972811Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":186,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:186 No valid comms client available","process":{"pid":234774,"thread":{"id":234780}}}
{"@timestamp":"2024-03-28T08:20:55.504073408Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":186,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:186 No valid comms client available","process":{"pid":234774,"thread":{"id":234780}}}
{"@timestamp":"2024-03-28T08:21:00.500432767Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":72,"name":"Logging.cpp"}}},"message":"Logging.cpp:72 Logging directory cleaned up, current size: 527083","process":{"pid":234774,"thread":{"id":234777}}}
{"@timestamp":"2024-03-28T08:21:00.504174923Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":186,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:186 No valid comms client available","process":{"pid":234774,"thread":{"id":234780}}}
{"@timestamp":"2024-03-28T08:21:02.551964404Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":947,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:947 Unable to make GRPC connection in deadline(60s). Fetching connection info again","process":{"pid":234774,"thread":{"id":234805}}}

I noticed that the elastic-endpoint.yaml configuration is unable to pull the correct configuration:

cat elastic-endpoint.yaml 
revision: 0

fleet:
  agent:
    id: 00000000-0000-0000-0000-000000000000
    logging:
      level: info
  host:
    id: 00000000-0000-0000-0000-000000000000

inputs:
  - id: 00000000-0000-0000-0000-000000000000
    name: initial
    revision: 0

Running ./elastic-endpoint diagnostics reveals that there is an communications issue between Agent and Endpoint:

cat analysis.txt 
== Agent<->Endpoint connection analysis ==
Agent and Endpoint are not actively connected

Active connections that are for or may conflict with Agent and Endpoint:
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/elastic-agent (pid 233035) is root/admin and using or recently used connection 'LISTEN (ipv4,tcp) 127.0.0.1:6789 <-> 0.0.0.0:0'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/elastic-agent (pid 233035) is root/admin and using or recently used connection 'LISTEN (ipv4,tcp) 127.0.0.1:6788 <-> 0.0.0.0:0'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/elastic-agent (pid 233035) is root/admin and using or recently used connection 'ESTABLISHED (ipv4,tcp) 127.0.0.1:6789 <-> 127.0.0.1:56662'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/elastic-agent (pid 233035) is root/admin and using or recently used connection 'ESTABLISHED (ipv4,tcp) 127.0.0.1:6789 <-> 127.0.0.1:56666'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/components/osquerybeat (pid 233172) is root/admin and using or recently used connection 'ESTABLISHED (ipv4,tcp) 127.0.0.1:56662 <-> 127.0.0.1:6789'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/components/filebeat (pid 233193) is root/admin and using or recently used connection 'ESTABLISHED (ipv4,tcp) 127.0.0.1:56666 <-> 127.0.0.1:6789'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/components/filebeat (pid 233206) is root/admin and using or recently used connection 'ESTABLISHED (ipv4,tcp) 127.0.0.1:56678 <-> 127.0.0.1:6789'
* An unknown process is using connection 'TIME_WAIT (ipv4,tcp) 127.0.0.1:6788 <-> 127.0.0.1:45239'
* /opt/Elastic/Agent/data/elastic-agent-a92ca3/elastic-agent (pid 233035) is root/admin and using or recently used connection 'ESTABLISHED (ipv4,tcp) 127.0.0.1:6789 <-> 127.0.0.1:56678'

Potential problems:
* There is no active connection between Endpoint and Agent

Not sure where to go from here. I've tried re-installing the Agent/Endpoint, re-enrolling the Agents but nothing seems to work.

Would appreciate any help I can get. Thanks!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Do you have other endpoint agents installed?

[DoughnutPeach]

Negative, only the one.

I ran elastic-endpoint uninstall and elastic-agent uninstall, and cleared /opt/Elastic/*.

ps -ef | grep elastic shows no other agents installed

[defensivedepth]

What OS are in the env? And especially between the failed & working ones?

[DoughnutPeach]

All are running CentOS9 Stream

[defensivedepth]

Is selinux running on any of them? If you do an elastic-agent inspect on both a working and non-working one, do you see any difference between the running config?

[DoughnutPeach]

So I compared between a working and non-working host.

SELinux:

• Both have the same config

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

elastic-agent inspect

• It's the same on both hosts, less the id and access_api_key. This seems to be OK.

Any ideas on how else I can troubleshoot this?

[defensivedepth]

At this point, this seems to be an Elastic Agent issue, not something specific to SO. I would suggest posting over in the Elastic Forums or Elastic Slack.