Upgrade from 2.4.20 to 2.4.40 partailly failed

[UMWP]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

8

RAM

16

Storage for /

240GB

Storage for /nsm

1,8TB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Distributed on-prem deployment with 2 sensors, 1 search and 1 management node.

After soup from 2.4.20 to 2.4.40 so-elasticsearch status is missing on management node
All other nodes are OK on grid status.

sudo salt-call state.highstate on management shows 6 failed.

Logs included.

soup.log
state.highstate.txt

Any help appreciated

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[InfosecGoon]

Are there any useful errors in the Elasticsearch logs on the Manager Node? They're located in /opt/so/log/elasticsearch/securityonion.log.

[petiepooo]

Elasticsearch is not running. In the soup log was:

ID: so-elasticsearch
Function: docker_container.running
Result: False
Comment: One or more requisite failed: ssl./etc/pki/elasticsearch.crt, ca.pki_public_ca_crt, ssl./etc/pki/elasticsearch.key
Started: 09:11:11.382131
Duration: 0.006 ms
Changes:

Well before that, your first false result was regarding the ca.crt, which is noted above in the comment as well. I've seen a similar issue and ended up reinstalling as it was a lab system, but you may get by with recreating all your SSL certs. I think the way to do that is salt-call state.apply ssl.remove -l info then restart everything by either rebooting each node or bouncing the docker service and running so-checkin on each node, starting with the manager.

[UMWP]

First of all thank You very much and I'm so sorry for my terrible mistake.
It's so-elastalert container with status "missing" (and not so-elasticsearch), I'm very sorry about that false info.
Lines from so-elastalert-restart and from /opt/so/log/elastalert/elastalert.log containing errors are included below:

so-elastalert restart.txt

Error.txt

And also Kibana shows "404 page not found"

Thank You very much once again.

[petiepooo]

No. Elastalert is only failing because it cannot reach Elasticsearch. Same with Kibana. To see that, run salt-call state.apply ssl -l info, and you will see the certificate errors that are causing all the dependent containers to fail. My recommendations stand.

[UMWP]

Unfortunately after:
salt-call state.apply ssl.remove -l info

and then after restarting all nodes (management, forwarding, search) it's still the same:
so-elastalert status "missing"

I think I'll have to revert back to 2.4.20 :-(

[petiepooo]

Sorry to hear that. Good luck.