Kibana 404

[intsec]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16

Storage for /

200G

Storage for /nsm

320G

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

When going to kibana I am getting 404 error. Kibana service is running. The only service that is having issues is elastalert. Below is what I am seeing in my elasticsearch logs.

[2024-04-01T13:44:00,023][ERROR][org.elasticsearch.xpack.security.action.filter.SecurityActionFilter] blocking [cluster:monitor/nodes/stats] operation due to expired license. Cluster health, cluster stats and indices stats
operations are blocked on license expiration. All data operations (read and write) continue to work.
If you have a new license, please update it. Otherwise, please reach out to your support contact.
[2024-04-01T13:44:19,456][ERROR][org.elasticsearch.xpack.security.action.filter.SecurityActionFilter] blocking [cluster:monitor/nodes/stats] operation due to expired license. Cluster health, cluster stats and indices stats
operations are blocked on license expiration. All data operations (read and write) continue to work.
If you have a new license, please update it. Otherwise, please reach out to your support contact.
[2024-04-01T13:44:19,456][WARN ][org.elasticsearch.cluster.InternalClusterInfoService] failed to retrieve node stats
org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [security]

From running salt-call state.highstate I am getting this.

local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 165866 and was started at 2024, Apr 01 13:45:59.201650 with jid 20240401134559201650

From kibana logs I am seeing this.
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-04-01T13:28:09.174+00:00","message":"[.kibana_security_solution] Action failed with 'security_exception\n\tRoot causes:\n\t\tsecurity_exception: current license is non-compliant for [security]'. Retrying attempt 15 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":7},"trace":{"id":"5082275cfc282cd9e23d5e1ad671af3f"},"transaction":{"id":"1c721f099ea59f37"}}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Did you have an Elastic license? What does sudo so-elasticsearch-query _license return?

[intsec]

Status shows expired, type: trial issued_to securityonion From: Chris Morgret ***@***.***> Sent: Tuesday, April 2, 2024 8:38 AM To: Security-Onion-Solutions/securityonion ***@***.***> Cc: Eric Vanderveer ***@***.***>; Author ***@***.***> Subject: Re: [Security-Onion-Solutions/securityonion] Kibana 404 (Discussion #12712) Did you have an Elastic license? What does sudo so-elasticsearch-query _license return? — Reply to this email directly, view it on GitHub<#12712 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATLMUC4XBUREFSNDYKV4P3TY3KRBBAVCNFSM6AAAAABFRU6O2KVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSOBTGUYTS>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[cm-ops]

Do you have the option in Kibana - Stack Management - License Management - Revert to Basic? You can also use the API to do this https://www.elastic.co/guide/en/elasticsearch/reference/current/start-basic.html#start-basic

[intsec]

Well I am getting a 404 when I try to go to Kibana so I tried doing a curl command from the box itself using "curl -X POST "localhost:9200/_license/start_basic?pretty"" and I get this.
curl: (52) Empty reply from server

[cm-ops]

Try sudo so-elasticsearch-query _license/start_basic?acknowledge=true -XPOST.