Upgrade from 2.4.10 to 2.4.20 - Elastic Agents Unhealthy

[chadstout1939]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

128

Storage for /

512 GB

Storage for /nsm

8 TB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have a Managersearch node, with no other nodes right now just to have Elastic agents deployed for testing, with a goal of deploying sensor nodes for monitoring network traffic in the future. I performed the 'sudo soup' upgrade to get upgraded to 2.4.20 last night. After doing so, all agents report as unhealthy.

When I click into the integrations within 'endpoints-initial', and click on the integration for Elastic Defend, it says "Failed to load endpoint policy settings / Cannot read properties of undefined (reading 'value')". I found a similar error within #11148 and applied those fixes. The Elastic Agents remain Unhealthy.

When I create a new Agent Policy with the same integrations, and deploy that to the agents, they show as healthy. So it seems like there is something specifically wrong with the endpoints-initial agent policy. Another person stated that they had certificate issues, and after clearing those up, they changed policies back and forth, and the endpoints-initial policy worked again. But I do not know what they meant by certificate issues. This was reported in #11540.

Please let me know if there is anything I can do to fix this, or if any logs would be helpful.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[defensivedepth]

Please run the following on your ManagerSearch and post the output here:

cat /etc/sohotfix

[chadstout1939]

20231012

[chadstout1939]

I have dug into the agent JSON as well, and found this:

{
"id": "ENDPOINT_INTEGRATION_CONFIG-default",
"type": "",
"status": "FAILED",
"message": "input not supported",
"units": [
{
"id": "ENDPOINT_INTEGRATION_CONFIG-default-ff9af890-573a-11ee-b4c6-db684e030d49",
"type": "input",
"status": "FAILED",
"message": "input not supported"
},
{
"id": "ENDPOINT_INTEGRATION_CONFIG-default",
"type": "output",
"status": "FAILED",
"message": "input not supported"
}
]
},

I also see in the agents' logs that DNS fails to the hostname, but I've set it up to use the IP address for things rather than the host name. In the mean time, I've setup a DNS entry for the Managersearch and tested to make sure it works. 

If there is anything else I can provide, I will be more than happy to do so.

[defensivedepth]

Please follow this procedure:

• Remove the 'endpoints-initial' Agent Policy
• Login to the Manager and run the following
  • sudo rm -f /opt/so/state/eaintegrations.txt
  • sudo so-elastic-fleet-integration-policy-load

That should recreate the endpoints-initial policy.

Also FYI, we have found the root cause of this and fixed it for the next release.

[chadstout1939]

Thank you for your response! Unfortunately, this did not work for me. When I run it, it says:

{"statusCode":500,"error":"Internal Server Error","message":"status_exception\n\tRoot causes:\n\t\tstatus_exception: Could not stop the transforms [endpoint.metadata_current-default-8.8.0] as they timed out [30s]."}

Initial Endpoints Policy - Loading /opt/so/conf/elastic-fleet/integrations/elastic-defend/elastic-defend-endpoints.json
jq: error (at :5): Cannot iterate over null (null)

Integration does not exist - Creating integration
{"statusCode":404,"error":"Not Found","message":"Saved object [ingest-agent-policies/endpoints-initial] not found"}

Initial Endpoints Policy - Loading /opt/so/conf/elastic-fleet/integrations/endpoints-initial/elastic-defend-endpoints.json
jq: error (at :5): Cannot iterate over null (null)

Integration does not exist - Creating integration
{"statusCode":404,"error":"Not Found","message":"Saved object [ingest-agent-policies/endpoints-initial] not found"}

Initial Endpoints Policy - Loading /opt/so/conf/elastic-fleet/integrations/endpoints-initial/osquery.json
jq: error (at :5): Cannot iterate over null (null)

Integration does not exist - Creating integration
{"statusCode":400,"error":"Bad Request","message":"Agent policy endpoints-initial not found"}

Initial Endpoints Policy - Loading /opt/so/conf/elastic-fleet/integrations/endpoints-initial/system-endpoints.json
jq: error (at :5): Cannot iterate over null (null)

Integration does not exist - Creating integration
{"statusCode":404,"error":"Not Found","message":"Saved object [ingest-agent-policies/endpoints-initial] not found"}

Initial Endpoints Policy - Loading /opt/so/conf/elastic-fleet/integrations/endpoints-initial/windows-endpoints.json
jq: error (at :5): Cannot iterate over null (null)

Integration does not exist - Creating integration
{"statusCode":404,"error":"Not Found","message":"Saved object [ingest-agent-policies/endpoints-initial] not found"}

If I run it again, the same series of commands, it comes back blank.

Upgrading endpoint package...

Upgrading f5_bigip package...

Is this something that 2.4.30 might be able to fix, or am I looking to redeploy the Managersearch and elastic agents? Thank you again for your help. Looking forward to 2.4.30!