Kibana 404 Error After Upgrade to 2.4.30

[secopsgeekcode]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

MemTotal: 65443204 kB

Storage for /

1.5TB

Storage for /nsm

1.5TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

After upgrading SO to 2.4.30 I am now getting a 404 Error message with kibana that say: 404 page not found

local:
Data failed to compile

The function ''"state.highstate" 'is running as PID 295783 and was started at 2023, Nov 14 19:13:33.896088 with jid ''20231114191333896088'

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[secopsgeekcode]

Uploading Kibana_sample.log…

[secopsgeekcode]

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-14T20:49:00.641+00:00","message":"Server is stopping; must stop all async operations","error":{"message":"Server is stopping; must stop all async operations","type":"Error","stack_trace":"Error: Server is stopping; must stop all async operations\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:40:18\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":7},"trace":{"id":"89c520873db77cce908715d2eefed4c6"},"transaction":{"id":"7e33ecb4a07e94a8"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-14T20:49:00.642+00:00","message":"Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations","error":{"message":"Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations","type":"Error","stack_trace":"Error: Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations\n at installWithTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:48:11)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":7},"trace":{"id":"89c520873db77cce908715d2eefed4c6"},"transaction":{"id":"7e33ecb4a07e94a8"}}

[cm-ops]

Have you looked in your Elasticsearch log in /opt/so/log/elasticsearch for any issues with the cluster?

[secopsgeekcode]

I will check there as well once I get back home, thanks.

[weslambert]

We are aware of an issue and are currently looking into it. We'll let you know the best course of action as we have more information available.

[secopsgeekcode]

Thank You!

[weslambert]

We are working on a solution, but for now, please try the following steps to resolve your issue:

# Remove the old Elastic Defend integration file:

sudo rm -f /opt/so/conf/elastic-fleet/integrations/endpoints-initial/elastic-defend-endpoints.json

# Edit /opt/so/saltstack/default/salt/kibana/defaults.yaml to include the migration settings under the kibana.config section:

kibana:
  config:
    migrations:
      discardCorruptObjects: "8.10.4"

# Restart Kibana:

sudo so-kibana-restart --force 

# Confirm the Fleet API can be reached by ensuring the following command returns output:

sudo so-elastic-fleet-package-list

# Once output is returned, run the following command to re-add the Elastic Defend integration:

sudo /usr/sbin/so-elastic-fleet-integration-policy-elastic-defend

[secopsgeekcode]

Thanks I will try this out.

[Acewiza]

Saw this before doing the update but pressed on anyway, because I'm a glutton for punishment. Fixed it for me.

[secopsgeekcode]

Uploading Output.txt

This is the results of the commands.

[secopsgeekcode]

That fixed it thank you @weslambert for your help!!!

[secopsgeekcode]

Output.txt

[dougburks]

We've released a hotfix:
https://blog.securityonion.net/2023/11/security-onion-2430-hotfix-20231117-now.html