Kibana Fails to Start after running soup [2.4.70 => 2.4.110]

[ZeroSums]

Version

2.4.110

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

24

RAM

128

Storage for /

24tb

Storage for /nsm

12tb

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Q: Does so-status show all services running?
A:

                 so-dockerregistry │ running    │                    Up 3 hours 
                     so-elastalert │ missing    │
                  so-elastic-fleet │ running    │                    Up 2 hours 
 so-elastic-fleet-package-registry │ running    │          Up 3 hours (healthy) 
                  so-elasticsearch │ running    │                 Up 53 minutes 
                       so-idstools │ running    │                    Up 3 hours 
                       so-influxdb │ running    │          Up 3 hours (healthy) 
                         so-kibana │ running    │                    Up 3 hours 
                         so-kratos │ running    │                    Up 3 hours 
                       so-logstash │ running    │                    Up 3 hours 
                          so-nginx │ running    │          Up 3 hours (healthy) 
                          so-redis │ running    │                    Up 3 hours 
                      so-sensoroni │ running    │                    Up 3 hours 
                            so-soc │ running    │                    Up 3 hours 
                          so-steno │ running    │                    Up 3 hours 
                so-strelka-backend │ restarting │ Restarting (1) 22 seconds ago 
            so-strelka-coordinator │ running    │                    Up 3 hours 
             so-strelka-filestream │ missing    │
               so-strelka-frontend │ missing    │
             so-strelka-gatekeeper │ running    │                    Up 3 hours 
                so-strelka-manager │ missing    │
                       so-suricata │ running    │                    Up 3 hours 
                       so-telegraf │ running    │                    Up 3 hours 
                           so-zeek │ running    │          Up 3 hours (healthy)

Q: Do you get any failures when you run sudo salt-call state.highstate?
A: Yes:

local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 449799 and was started at 2024, Oct 16 15:51:23.491952 with jid 20241016155123491952

Q: Does the SOC Grid page show any failures?
A: I'm unable to access the grid - as I'm getting Error 500 when attempting to access the SOC

Q: Explain your issue. For example: Installation fails when I select this series of options...
A: I updated our SO instance to Latest (2.4.110), and after allowing soup to run patches for the system, I restarted, as per the docs. The moment the system booted, I began seeing errors in /opt/so/log/kibana.log:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:55:39.772+00:00","message":"Error occurred while publishing Fleet metrics: ResponseError: script_exception\n\tCaused by:\n\t\tnull_pointer_exception: cannot access method/field [0] from a null def reference\n\tRoot causes:\n\t\tscript_exception: runtime error","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58654.100816707},"trace":{"id":"d938dbd2946799dc69532e18bbd26cc8"},"transaction":{"id":"983a03830db39084"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:56:12.788+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":10,\"healthy\":10,\"unhealthy\":0,\"offline\":0,\"inactive\":1,\"unenrolled\":1,\"total_all_statuses\":11,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"inactive\":1,\"unenrolled\":0,\"total_all_statuses\":2,\"num_host_urls\":2},\"license_issued_to\":\"securityonion\"}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58687.116251565},"trace":{"id":"bb35e44262b0bdb98c04d143cdda3cab"},"transaction":{"id":"deff95e6e7f726a6"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:56:42.772+00:00","message":"Error occurred while publishing Fleet metrics: ResponseError: script_exception\n\tCaused by:\n\t\tnull_pointer_exception: cannot access method/field [0] from a null def reference\n\tRoot causes:\n\t\tscript_exception: runtime error","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58717.099932091},"trace":{"id":"d938dbd2946799dc69532e18bbd26cc8"},"transaction":{"id":"745f45312247f5d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:13.426+00:00","message":"Unable to determine whether upgrade is being performed","log":{"level":"WARN","logger":"plugins.ml"},"process":{"pid":7,"uptime":58747.754052497},"trace":{"id":"5feeb48ee7c72bb6f87114f2e7ad1873"},"transaction":{"id":"e52432c2c1dcffc7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:13.993+00:00","message":"Beginning fleet setup","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58748.321132668},"trace":{"id":"195c4ae7373343bceaa0fe9d83015754"},"transaction":{"id":"82b267d6c17b1078"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:13.993+00:00","message":"Cleaning old indices","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58748.321505274},"trace":{"id":"195c4ae7373343bceaa0fe9d83015754"},"transaction":{"id":"82b267d6c17b1078"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:14.544+00:00","message":"Failed to fetch latest version of synthetics from registry: Cannot read properties of null (reading 'version')","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58748.871981229},"trace":{"id":"195c4ae7373343bceaa0fe9d83015754"},"transaction":{"id":"82b267d6c17b1078"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:14.691+00:00","message":"Fleet setup completed","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58749.019506744},"trace":{"id":"195c4ae7373343bceaa0fe9d83015754"},"transaction":{"id":"82b267d6c17b1078"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:16.191+00:00","message":"Failed to fetch latest version of security_detection_engine from registry: Cannot read properties of null (reading 'version')","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58750.519398578},"trace":{"id":"d0d301a12b45774f3a20a7c41b7450e6"},"transaction":{"id":"380a8c9ede74dfa7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:21.706+00:00","message":"Failed to fetch latest version of synthetics from registry: Cannot read properties of null (reading 'version')","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58756.034587603},"trace":{"id":"8fe7565ca7b8244b7748aafed57cc913"},"transaction":{"id":"28c6da325635ff81"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:21.711+00:00","message":"Failed to fetch latest version of elastic_agent from registry: Cannot read properties of null (reading 'version')","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58756.03959429},"trace":{"id":"8fe7565ca7b8244b7748aafed57cc913"},"transaction":{"id":"28c6da325635ff81"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-10-16T12:57:21.716+00:00","message":"Failed to fetch latest version of security_detection_engine from registry: Cannot read properties of null (reading 'version')","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":7,"uptime":58756.044638508},"trace":{"id":"8fe7565ca7b8244b7748aafed57cc913"},"transaction":{"id":"28c6da325635ff81"}}

/root/soup.log:

### Preparing soup at Wed Oct 16 02:06:32 PM UTC 2024 ###

Checking if we can access the salt master and that it is ready at: Oct 16 14:06:32
Successfully accessed and salt master ready at: Oct 16 14:06:43
Checking to see if this is a manager.

This is a manager, so we can proceed.

The manager's pillars can be rendered. We can proceed with SOUP.

Checking to see if this is an airgap install.

Found that Security Onion 2.4.110 is currently installed.

Cloning Security Onion github repo into /tmp/sogh/securityonion.
Removing previous upgrade sources.
Cloning the Security Onion Repo.
Verifying we have the latest soup script.
This version of the soup script is up to date. Proceeding.
Let's see if we need to update Security Onion.
Checking to see if there are hotfixes needed
You are already running the latest version of Security Onion.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

Q: Do you get any failures when you run sudo salt-call state.highstate?
A: Yes:

You got this error because a highstate was already running. Please try running it like this:

sudo so-checkin

If a highstate is running, then it should wait for it to complete before trying another highstate.

/root/soup.log:
Preparing soup at Wed Oct 16 02:06:32 PM UTC 2024 ###
Checking if we can access the salt master and that it is ready at: Oct 16 14:06:32
Successfully accessed and salt master ready at: Oct 16 14:06:43
Checking to see if this is a manager.
This is a manager, so we can proceed.
The manager's pillars can be rendered. We can proceed with SOUP.
Checking to see if this is an airgap install.
Found that Security Onion 2.4.110 is currently installed.

Looks like this soup.log is from running soup after the system had been updated to 2.4.110.

When you ran soup to update from 2.4.70 to 2.4.110, were there any errors or did soup complete successfully?