Alerts, Hunt are giving error 500 and elasticsearch in fault

[csvancetech]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

32

RAM

128 GB

Storage for /

291GB

Storage for /nsm

55T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

After a reboot of Security Onion Manager node, I can no longer access Kibana with my username/password. Elasticsearch is faulted, but will go green after a while. When clicking on Hunt, Dashboards, or Alerts, it gives me an Error 500. so-status says all is fine. I've rebooted several times. When checking Salt Status I get error local:
Data failed to compile:

The function "state.highstate" is running as PID 128822 and was started at 2024, Nov 01 18:59:10.052668 with jid 20241101185910052668

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

That "error" is just telling you a highstate is running.

What is your cluster health? sudo so-elasticsearch-query _cluster/health?pretty If it is red, you have an unassigned shard, if it is yellow, most likely you have a replica shard that is unassigned. If you have red/yellow cluster health, what does this return? sudo so-elasticsearch-query _cluster/allocation/explain?pretty

[csvancetech]

Seems sudo salt-call state.highstate is completing successfully. When running the above command I'm getting green.
{
"cluster_name" : "securityonion",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 462,
"active_shards" : 462,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

I still cannot log into kibana and Alerts, Hunt, Dashboard, and Cases are still giving me error 500 on the SOC Console.

[cm-ops]

Would you look in /opt/so/log/soc/sensoroni-server.log. When you get the 500 error there should be some clue in there as a reason for the 500 error.

[csvancetech]

{"fields":{"query":"{"aggs":{"timeline":{"date_histogram":{"field":"@timestamp","fixed_interval":"10d","min_doc_count":1}}},"query":{"bool":{"filter":[],"must":[{"query_string":{"analyze_wildcard":true,"default_field":"","query":"(NOT so_case.status:closed AND NOT so_case.category:template) AND _index: \":so-case\" AND so_kind:case"}},{"range":{"@timestamp":{"format":"strict_date_optional_time","gte":"2023-11-05T14:15:16-05:00","lte":"2024-11-05T14:15:16-05:00"}}}],"must_not":[],"should":[]}},"size":500}","requestId":"3657592b-e55d-4fdf-8503-3a3503b9c595"},"level":"info","timestamp":"2024-11-05T19:15:20.638243121Z","message":"Searching Elasticsearch"}
{"fields":{"error":"security_exception: action [indices:data/read/search] is unauthorized for user [so_elastic] with effective roles [superuser], because user [so_elastic] is unauthorized to run as [cvance92] -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "security_exception",\n "reason" : "action [indices:data/read/search] is unauthorized for user [so_elastic] with effective roles [superuser], because user [so_elastic] is unauthorized to run as [cvance92]"\n }\n ],\n "type" : "security_exception",\n "reason" : "action [indices:data/read/search] is unauthorized for user [so_elastic] with effective roles [superuser], because user [so_elastic] is unauthorized to run as [cvance92]"\n },\n "status" : 403\n}\n","requestId":"3657592b-e55d-4fdf-8503-3a3503b9c595","requestor":{"id":"897f31e3-c077-437f-bb4c-7bac90e0de72","createTime":"2024-11-05T19:15:20.634477952Z","updateTime":"0001-01-01T00:00:00Z","email":"cvance92","firstName":"","lastName":"","totpStatus":"disabled","oidcStatus":"disabled","webauthnStatus":"enabled","note":"","roles":null,"status":"","searchUsername":"","password":"","passwordChanged":true}},"level":"warn","timestamp":"2024-11-05T19:15:20.639437126Z","message":"Request did not complete successfully"}
{"fields":{"contentLength":115,"elapsedMs":7,"impl":"/build/server/eventhandler.go:71:github.com/security-onion-solutions/securityonion-soc/server.(EventHandler).getEvent","remoteAddr":"172.17.1.1:46892","requestId":"3657592b-e55d-4fdf-8503-3a3503b9c595","requestMethod":"GET","requestPath":"/api/events/","requestQuery":{"eventLimit":["500"],"format":["2006/01/02 3:04:05 PM"],"metricLimit":["100"],"query":["(NOT so_case.status:closed AND NOT so_case.category:template) AND _index:":so-case" AND so_kind:case"],"range":["2023/11/05 02:15:16 PM - 2024/11/05 02:15:16 PM"],"zone":["America/New_York"]},"requestor":{"id":"897f31e3-c077-437f-bb4c-7bac90e0de72","createTime":"2024-11-05T19:15:20.634477952Z","updateTime":"0001-01-01T00:00:00Z","email":"cvance92","firstName":"","lastName":"","totpStatus":"disabled","oidcStatus":"disabled","webauthnStatus":"enabled","note":"","roles":null,"status":"","searchUsername":"","password":"","passwordChanged":true},"sourceIp":"10.3.2.30","statusCode":500},"level":"info","timestamp":"2024-11-05T19:15:20.63952257Z","message":"Handled request"}

[csvancetech]

Seems making a new user fixed my issue. I was able to delete my old user. I can now use Kibana, alerts, hunt, and cases again.

Thanks!