Upgrading 2.4.50 -> 2.4.60 - grid doesn't come up again

[ejgh-oe]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi,

Today I decided to upgrade my SO-installation (distributed installation including manager-, search-, sensor- and several IDH-nodes) from 2.4.50 -> 2.4.60.

soup ran but in the end complained

Soup failed with error 1: Unhandled error
Unhandled error occured, please check /root/soup.log for details.

Checking soup.log I found this:

          ID: wait_for_elasticsearch
    Function: cmd.run
        Name: so-elasticsearch-wait
      Result: False
     Comment: Command "so-elasticsearch-wait" run
     Started: 17:47:30.350685
    Duration: 313249.83 ms
     Changes:   
              ----------
              pid:
                  1688690
              retcode:
                  1
              stderr:
              stdout:
                  Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)
                  Server is not ready
.
.
.
          ID: wait_for_elasticsearch
    Function: cmd.run
        Name: so-elasticsearch-wait
      Result: False
     Comment: Command "so-elasticsearch-wait" run
     Started: 17:47:30.350685
    Duration: 313249.83 ms
     Changes:   
              ----------
              pid:
                  1688690
              retcode:
                  1
              stderr:
              stdout:
                  Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)
                  Server is not ready

First I thought about a timeout and rebooted the manager-node, but alas, it didn't come up again :-(

Grid status shows "Fault" for the manager node:

Likewise "so-status" shows "so-elastalert" as "missing".

Running salt-call state.highstate throws several error messages like the ones I was seeing in the soup.log:

Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (172/300)
Server is not ready

counting all the way up to "(300/300")

The grid status shows Fault for the manager node and I don't get any alerts/events at all in the GUI (which is understandable since elasticsearch seems to have a problem given the logs).

Any ideas on how to get the installation up & running again?

Thanks much in advance for your help

PS: As for the other nodes in the grid, they're at 2.4.60 and show status green (OK) - haven't rebooted any of them yet however:

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[ejgh-oe]

Sorry, forgot the logs mentioned in my previous post...
soup.log

elastalert.log

(NB: Please note that I've removed all normal logs in elastalert.log prior to the upgraded - the full elastalert.log including the normal status messages would be almost 6MB)

[ejgh-oe]

Addition to my post from yesterday:
Overnight the "Fault" state on the manager node went away for some mysterious reason ;-) (elastalert somehow managed to come up without intervention whatsoever) but nevertheless some problems are still there like Kibana, Elastic Fleet, OSQuery manager also don't seem to be working. They show up as running in node status on manager

when when I click on Kibana, ElasticFleet or OSQuery Manager from the main menu instead of the various tools I only get

404 page not found

[petiepooo]

It looks like things started to fall apart where the following log entry appears in soup.log:

Populating the mine with mine_functions for each host.
Minion onion-mgr_manager did not respond. No job will be sent.

You may need to manually update the manager's salt mine with the command sudo salt-call mine.update then run sudo so-checkin to get everything back up to highstate.

[ejgh-oe]

Hi,
That absolutely did the trick - everything's back to normal now :-)
Thanks alot for your help.