Post 2.4.30 upgrade, no kibana, no fleet

[kvkamin]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

20

RAM

64gb

Storage for /

300G

Storage for /nsm

8TB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

After update to 2.4.30, both kibana and fleet get "404 page not found"

salt-call state.highstate gets these errors:

[root@so-ka salt]# cat minion | grep -v INFO | more
2023-11-15 00:05:21,211 [salt.loaded.int.module.cmdmod:920 ][ERROR ][3251508] Command '/usr/sbin/so-elastic-fleet-outputs-update' failed with return code: 1
2023-11-15 00:05:21,211 [salt.loaded.int.module.cmdmod:922 ][ERROR ][3251508] stdout: Failed to query for current Logstash Outputs...
2023-11-15 00:05:21,211 [salt.loaded.int.module.cmdmod:924 ][ERROR ][3251508] stderr: % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

Error with /usr/sbin/so-elastic-fleet-outputs-update

[root@so-ka ~]# curl -K /opt/so/conf/elasticsearch/curl.config 'http://localhost:5601/api/fleet/outputs/so-manager_logstash'
curl: (56) Recv failure: Connection reset by peer

[root@so-ka ~]# so-status

                      Security Onion Status                           
                     Container │ Status  │                    Details 

───────────────────────────────────┼─────────┼────────────────────────────
so-curator │ running │ Up About an hour
so-dockerregistry │ running │ Up About an hour
so-elastalert │ running │ Up About an hour
so-elastic-fleet │ running │ Up About an hour
so-elastic-fleet-package-registry │ running │ Up About an hour (healthy)
so-elasticsearch │ running │ Up About an hour
so-idstools │ running │ Up About an hour
so-influxdb │ running │ Up About an hour (healthy)
so-kibana │ running │ Up 23 minutes
so-kratos │ running │ Up About an hour
so-logstash │ running │ Up About an hour
so-mysql │ running │ Up About an hour (healthy)
so-nginx │ running │ Up About an hour (healthy)
so-playbook │ running │ Up About an hour
so-redis │ running │ Up About an hour
so-sensoroni │ running │ Up About an hour
so-soc │ running │ Up About an hour
so-soctopus │ running │ Up About an hour
so-telegraf │ running │ Up About an hour

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[Pencat89]

I have the same problem with 404 page not found. Also Osquery has the same issue.

[defensivedepth]

@Pencat89 Please create a new Discussion, making sure to fill out the technical details

[kvkamin]

Here is the soup log from 20-30 update with errors. Gzipped.

~K
soup.log.2.4.30.20231114.195559.gz

[kvkamin]

Running so-kibana-restart yields errors in /opt/so/logs/kibana/kibana.log regarding SSL

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T14:25:12.903+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"6d71765e18683a677048e6abd71a77fd"},"transaction":{"id":"47d755bcf97a24bd"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T14:25:12.946+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"6d71765e18683a677048e6abd71a77fd"},"transaction":{"id":"47d755bcf97a24bd"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T15:15:40.353+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"0038b064075687e43861304c5e094fdf"},"transaction":{"id":"882d31e372699e82"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T15:15:40.398+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"0038b064075687e43861304c5e094fdf"},"transaction":{"id":"882d31e372699e82"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T18:30:24.180+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"c1a0f3cfbe4c7d1d4b8a4ca4df10372a"},"transaction":{"id":"2835afd1094519bc"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T18:30:24.223+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"c1a0f3cfbe4c7d1d4b8a4ca4df10372a"},"transaction":{"id":"2835afd1094519bc"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T19:31:46.375+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"7014894d23481aadab76dda59b8302c0"},"transaction":{"id":"64ee9470b0b89f01"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-15T19:31:46.417+00:00","message":"Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":7},"trace":{"id":"7014894d23481aadab76dda59b8302c0"},"transaction":{"id":"64ee9470b0b89f01"}}

[weslambert]

We are aware of an issue and are currently looking into it. We'll let you know the best course of action as we have more information available.

[weslambert]

We are working on a solution, but for now, please try the following steps to resolve your issue:

# Remove the old Elastic Defend integration file:

sudo rm -f /opt/so/conf/elastic-fleet/integrations/endpoints-initial/elastic-defend-endpoints.json

# Edit /opt/so/saltstack/default/salt/kibana/defaults.yaml to include the migration settings under the kibana.config section:

kibana:
  config:
    migrations:
      discardCorruptObjects: "8.10.4"

# Restart Kibana:

sudo so-kibana-restart --force 

# Confirm the Fleet API can be reached by ensuring the following command returns output:

sudo so-elastic-fleet-package-list

# Once output is returned, run the following command to re-add the Elastic Defend integration:

sudo /usr/sbin/so-elastic-fleet-integration-policy-elastic-defend

[kvkamin]

Works, Thanks! ~K Karl Kamin, CISSP, CCSP President & CEO, Kamin Associates, Inc. IT Consulting and Services 121 Interpark Blvd, Ste 219 | San Antonio, Texas 78216-1845 Main 210.201.7800 | Cell 210.215.1521 | Fax 210.201.7801 Support http://kamin.cc<http://kamin.cc/>

…

________________________________ From: weslambert ***@***.***> Sent: Friday, November 17, 2023 3:53 PM To: Security-Onion-Solutions/securityonion ***@***.***> Cc: Karl Kamin ***@***.***>; Author ***@***.***> Subject: Re: [Security-Onion-Solutions/securityonion] Post 2.4.30 upgrade, no kibana, no fleet (Discussion #11796) We are working on a solution, but for now, please try the following steps to resolve your issue: # Remove the old Elastic Defend integration file: sudo rm -f /opt/so/conf/elastic-fleet/integrations/endpoints-initial/elastic-defend-endpoints.json # Edit /opt/so/saltstack/default/salt/kibana/defaults.yaml to include the migration settings under the kibana.config section: kibana: config: migrations: discardCorruptObjects: "8.10.4" # Restart Kibana: sudo so-kibana-restart --force # Confirm the Fleet API can be reached by ensuring the following command returns output: sudo so-elastic-fleet-package-list # Once output is returned, run the following command to re-add the Elastic Defend integration: sudo /usr/sbin/so-elastic-fleet-integration-policy-elastic-defend — Reply to this email directly, view it on GitHub<#11796 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABZQGZPYDAY7HT5KLMPL75LYE7MGHAVCNFSM6AAAAAA7MWMEC2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMMBTGQ4DE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dougburks]

We've released a hotfix:
https://blog.securityonion.net/2023/11/security-onion-2430-hotfix-20231117-now.html