Elasticsearch status pending/Kibana 404 after update to 2.4.70

[tsmith-spscc]

Version

2.4.70

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

24

RAM

192

Storage for /

1 TB

Storage for /nsm

40 TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

I came in this morning to find Security Onion no longer able to show alerts. All of my nodes had a status of "Restart", which indicated that they were pending a restart. It looked as though some kind of automatic update had tried to apply itself over the weekend. After rebooting the SO nodes, the Forward node did not show up in the grid. After troubleshooting and looking through logs, I didn't see anything that stood out so I figured I'd try running soup to see if there was an update that didn't complete correctly or something along those lines. Soup upgraded my SO from 2.4.60 to 2.4.70. After the upgrade, the forward node appeared in my grid again with all services up and available. However, the manager node has a persistent status of:

Elasticsearch status: Pending

Also, Kibana will not open. Attempting to do so returns a 404 error.

Running a salt highstate also returns errors. There appears to be some kind of communication error. Salt highstate errors are as follows:

----------
          ID: so-elastic-fleet-auto-configure-logstash-outputs
    Function: cmd.run
        Name: /usr/sbin/so-elastic-fleet-outputs-update
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
              Command "/usr/sbin/so-elastic-fleet-outputs-update" run
     Started: 19:12:54.878092
    Duration: 30038.721 ms
     Changes:
              ----------
              pid:
                  917512
              retcode:
                  1
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed

                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Failed to query for current Logstash Outputs...

----------
          ID: so-elastic-fleet-auto-configure-server-urls
    Function: cmd.run
        Name: /usr/sbin/so-elastic-fleet-urls-update
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-urls-update" run"
              Command "/usr/sbin/so-elastic-fleet-urls-update" run
     Started: 19:13:24.947426
    Duration: 30035.947 ms
     Changes:
              ----------
              pid:
                  918822
              retcode:
                  1
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed

                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Failed to query for current Fleet Server URLs...

----------
          ID: so-elastic-fleet-auto-configure-elasticsearch-urls
    Function: cmd.run
        Name: /usr/sbin/so-elastic-fleet-es-url-update
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
              Command "/usr/sbin/so-elastic-fleet-es-url-update" run
     Started: 19:13:55.013508
    Duration: 30039.241 ms
     Changes:
              ----------
              pid:
                  919241
              retcode:
                  1
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed

                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Failed to query for current Fleet Server Elasticsearch URLs...

----------
          ID: so-elastic-agent-grid-upgrade
    Function: cmd.run
        Name: /usr/sbin/so-elastic-agent-grid-upgrade
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-agent-grid-upgrade" run"
              Command "/usr/sbin/so-elastic-agent-grid-upgrade" run
     Started: 19:14:26.440554
    Duration: 30066.14 ms
     Changes:
              ----------
              pid:
                  920600
              retcode:
                  56
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed

                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (56) Recv failure: Connection reset by peer
                  /usr/sbin/so-elastic-agent-grid-upgrade: line 20: [: : integer expression expected
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed

                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  100     4    0     0  100     4      0   4000 --:--:-- --:--:-- --:--:--  4000
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Initiating upgrades for  Agents to Elastic 8.10.4...

I really don't want to have to re-install Security Onion yet again. Any help is very much appreciated. Thanks.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[tsmith-spscc]

This morning the Elasticsearch status has changed to OK, but Kibana, Elastic Fleet, etc, still give a 404 error. The so-kibana container shows that it's running. Not sure what else I can do here.

[cm-ops]

Would check these logs for any issues:

/opt/so/log/kibana/kibana.log
sudo docker logs so-kibana