2.4.20 -> 2.4.30 upgrade seems to have broken Elastic Defend integration

[ross-tsc]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

256

Storage for /nsm

SO AMI does not set up separate LV

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

We upgraded to the latest version released yesterday (2.4.30 from 2.4.20). This seemed to break the Elastic Defend integration on the Elastic Agent policy for EUC in my environment (although the ED integration I added to the grid policy was healthy). I was unable to upgrade the Agent version while it was in error, so I removed Elastic Defend integration from the appropriate policy.

After the agent upgrade succeeded, I went to turn the Elastic Defend integration back on on that policy and received the following error:

Error loading package information - package info for endpoint-8.10.2 does not exist

I also noted that the Elastic Defend integration (on the Integrations page) is missing its logo, and when selected shows that it requires an upgrade. That upgrade fails with the same error above.

I couldn't find anything pertinent in the logs for Elastic Fleet. Is this a known error?

Many thanks
R

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[ross-tsc]

Just built a new distributed grid from scratch with 2.4.30 - the issue does not reoccur. This must have been an upgrade issue of some kind. Leaving this note for posterity in case anyone bumps into this when upgrading.
R

[weslambert]

Yes, this is an issue with upgrades from 2.4.10 or earlier. We are looking into and it will be able to provide more information as it is available.

https://blog.securityonion.net/2023/11/security-onion-2430-now-available.html

[weslambert]

We are working on a solution, but for now, please try the following steps to resolve your issue:

# Remove the old Elastic Defend integration file:

sudo rm -f /opt/so/conf/elastic-fleet/integrations/endpoints-initial/elastic-defend-endpoints.json

# Edit /opt/so/saltstack/default/salt/kibana/defaults.yaml to include the migration settings under the kibana.config section:

kibana:
  config:
    migrations:
      discardCorruptObjects: "8.10.4"

# Restart Kibana:

sudo so-kibana-restart --force 

# Confirm the Fleet API can be reached by ensuring the following command returns output:

sudo so-elastic-fleet-package-list

# Once output is returned, run the following command to re-add the Elastic Defend integration:

sudo /usr/sbin/so-elastic-fleet-integration-policy-elastic-defend

[dougburks]

We've released a hotfix:
https://blog.securityonion.net/2023/11/security-onion-2430-hotfix-20231117-now.html