After upgrading from 2.4.120 to 2.4.130 on the Tools menu the links for Kibana and Elastic Fleet get a 404 page not found error

[DCraigRich]

Version

2.4.130

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

40

RAM

269.9 GB

Storage for /

314.4 GB

Storage for /nsm

39673. 8GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

It looks like the elastic fleet upgrade failed not connecting to kibana though kibana is running
hightstate.log
soup.log.2.4.130.20250314.013214.log
. I am attaching the soup log from the upgrade and also the log of the errors I see with the salt-call state.highstate. I am not sure how to proceed to try and fix this. Any help is appreciated. Thanks!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[ReezyBoi]

I am getting the same issue after upgrading to .130 from .120. All containers show running as healthy.

The errors seem to evolve around so-elastic-fleet. When I run so-elastic-fleet-restart, there are 6 errors that pop up pertaining to a mix of elastic agent grid updates and elastic integrations. Running so-kibana-restart shows no errors.

[reyesj2]

Can you try running (and share output)

so-status
so-kibana-api-check
so-elastic-fleet-integration-upgrade; echo $?

Can you share any logs for kibana or elasticsearch? They would be located in

/opt/so/log/kibana/kibana.log
/opt/so/log/elasticsearch/securityonion.log

[reyesj2]

Also I think you may have overwritten your soup log with a highstate log. If possible update with the soup log, so I can review.

[DCraigRich]

Can you try running (and share output)

so-status
so-kibana-api-check
so-elastic-fleet-integration-upgrade; echo $?

Can you share any logs for kibana or elasticsearch? They would be located in

/opt/so/log/kibana/kibana.log
/opt/so/log/elasticsearch/securityonion.log

[root@seconion10 secadmin]# so-status

                        Security Onion Status
                         Container │ Status  │               Details
───────────────────────────────────┼─────────┼───────────────────────
                 so-dockerregistry │ running │           Up 26 hours
                     so-elastalert │ running │           Up 26 hours
                  so-elastic-fleet │ running │           Up 26 hours
 so-elastic-fleet-package-registry │ running │ Up 26 hours (healthy)
                  so-elasticsearch │ running │           Up 26 hours
                       so-idstools │ running │           Up 26 hours
                       so-influxdb │ running │ Up 26 hours (healthy)
                         so-kibana │ running │           Up 26 hours
                         so-kratos │ running │           Up 26 hours
                       so-logstash │ running │           Up 26 hours
                          so-nginx │ running │ Up 26 hours (healthy)
                          so-redis │ running │           Up 26 hours
                      so-sensoroni │ running │           Up 26 hours
                            so-soc │ running │           Up 26 hours
                          so-steno │ running │           Up 26 hours
                so-strelka-backend │ running │           Up 26 hours
            so-strelka-coordinator │ running │           Up 26 hours
             so-strelka-filestream │ running │           Up 26 hours
               so-strelka-frontend │ running │           Up 26 hours
             so-strelka-gatekeeper │ running │           Up 26 hours
                so-strelka-manager │ running │           Up 26 hours
                       so-suricata │ running │           Up 26 hours
                       so-telegraf │ running │           Up 26 hours
                           so-zeek │ running │ Up 26 hours (healthy)

 ✔ This onion is ready to make your adversaries cry!

[root@seconion10 secadmin]# so-kibana-api-check
Checking to make sure that Kibana API is up & ready...
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (1/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (2/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (3/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (4/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (5/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (6/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (7/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (8/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (9/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (10/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (11/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (12/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (13/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (14/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (15/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (16/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (17/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (18/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (19/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (20/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (21/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (22/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (23/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (24/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (25/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (26/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (27/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (28/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (29/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (30/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (31/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (32/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (33/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (34/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (35/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (36/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (37/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (38/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (39/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (40/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (41/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (42/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (43/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (44/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (45/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (46/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (47/300)
Server is not ready
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (48/300)
Server is not ready
^C
[root@seconion10 secadmin]# so-elastic-fleet-integration-upgrade; echo $?
Error: Failed to connect to Kibana.

[DCraigRich]

Here is the soup log that got overwritten. I will grab the other stuff also
soup.log.2.4.130.20250314.013214.013214.log

[DCraigRich]

Current Kibana log is empty. Grabbed the last ones there around upgrade. And the securityonion.log
kibana-20250315.log.gz
kibana-20250314.log.gz
kibana-20250313.log.gz
securityonion.log
Thanks!

[reyesj2]

Looking at your elasticsearch/securityonion.log there are errors relating to too many shards. You can try removing a few of your older indices to bring that count down.

so-elasticsearch-query _cat/indices?s=index will show you a list of indices

then you can take one of the indices name you wish to remove (they should be sorted by name. Chose one that is oldest vs trying to delete most recent. Example below where I would delete .ds-logs-suricata.alerts-so-2025.02.10-000001

green  open .ds-logs-suricata.alerts-so-2025.02.10-000001                          sZIZYQAmRDe2Ty-LFpOF7A 1 1      676    0   9.9mb   4.9mb   4.9mb
green  open .ds-logs-suricata.alerts-so-2025.02.11-000002                          _H9Mmvq9SL-LlNZVAa-ZZQ 1 1      780    0   7.9mb   3.9mb   3.9mb
green  open .ds-logs-suricata.alerts-so-2025.03.06-000003                          13tcyyE-TuyYB0lrBMhg9g 1 1     5994    0  47.3mb  23.6mb  23.6mb
green  open .ds-logs-suricata.alerts-so-2025.03.07-000004                          Hu5_2T2eQVSOKEv8LLWUQQ 1 1    13736    0  87.9mb  43.9mb  43.9mb
green  open .ds-logs-suricata.alerts-so-2025.03.08-000005                          WhfWFlinSLmZnUAmlZv0Ug 1 0     4386    0    24mb    24mb    24mb
green  open .ds-logs-suricata.alerts-so-2025.03.09-000006                          dAh4MDjISDi-QpESSkpsMA 1 0     3125    0  17.5mb  17.5mb  17.5mb
green  open .ds-logs-suricata.alerts-so-2025.03.10-000007                          C0YmxwcASUiSwnljm449EA 1 0     2964    0  14.2mb  14.2mb  14.2mb
green  open .ds-logs-suricata.alerts-so-2025.03.11-000008                          ZCU2Qq-xQfaJmYUbkHTzNA 1 0     2968    0  15.9mb  15.9mb  15.9mb
green  open .ds-logs-suricata.alerts-so-2025.03.12-000009                          IbHdLnZMRoye_9gEaRejMw 1 0    11393    0  23.6mb  23.6mb  23.6mb
green  open .ds-logs-suricata.alerts-so-2025.03.13-000010                          EIjJe-uyShaZF1Y1UpuCxQ 1 0    96115    0 205.8mb 205.8mb 205.8mb
green  open .ds-logs-suricata.alerts-so-2025.03.14-000011                          bWF8w2KER8uyhlK7gz40Yg 1 0     3749    0  20.8mb  20.8mb  20.8mb
green  open .ds-logs-suricata.alerts-so-2025.03.15-000012                          UwA7qzHTTFeOf3x-v24Lpg 1 0     3926    0  20.9mb  20.9mb  20.9mb
green  open .ds-logs-suricata.alerts-so-2025.03.16-000013                          xKiIziTRRIOsP-EO5ZVfHA 1 0     2249    0  13.2mb  13.2mb  13.2mb

The delete command would be

so-elasticsearch-query .ds-logs-suricata.alerts-so-2025.02.10-000001 -XDELETE

Looking at your log you need at least 8 shards removed

Once you remove a few try running

so-restart kibana --force
so-kibana-api-check

The expected output from the api check command is

Checking to make sure that Kibana API is up & ready...
Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings' (1/300)
Received expected response; proceeding.

[DCraigRich]

This fixed my issue. Thank you very much!