Acknowledge/Escalate alerts throws status 400 after upgrading to 2.4.100

[innovate-support]

Version

2.4.100

Installation Method

Network installation on Debian

Description

other (please provide detail below)

Installation Type

Distributed

Location

cloud

Hardware Specs

Meets minimum requirements

CPU

8

RAM

16

Storage for /

330

Storage for /nsm

330

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Yesterday I ran soup and updated from 2.4.90 to 2.4.100. After that finished my Manager's Elasticsearch Status showed 'Fault'. There were issues with numerous UNASSIGNED shards, and after searching the forums for solutions I was able to get that resolved by adjusting the elasticsearch 'watermark' percentages and waiting for things to auto cleanup. After a reboot things were working OK again accept for Acknowledgements & Escalations, which throws a red error that says 'Request failed with status code 400'. My user has 'superuser' permissions. My coworker who has 'analyst' permissions is getting the same 400 error. I even tried creating a new superuser but that also gets the 400 error.

These errors are logged in SO and I can Hunt for them, but I'm not sure what's gone wrong or how to fix it.
Here's the 400 error log with a bit of redaction.
{"fields":{"contentLength":115,"elapsedMs":9,"impl":"/build/server/eventhandler.go:91:github.com/security-onion-solutions/securityonion-soc/server.(*EventHandler).postAck","remoteAddr":"REDACT:47964","requestId":"REDACT","requestMethod":"POST","requestPath":"/api/events/ack","requestQuery":{},"requestor":{"id":"REDACT","createTime":"2024-09-04T16:16:43.167887832Z","updateTime":"0001-01-01T00:00:00Z","email":"REDACT","firstName":"Test","lastName":"User","totpStatus":"disabled","oidcStatus":"disabled","webauthnStatus":"disabled","note":"","roles":null,"status":"","searchUsername":"","password":"","passwordChanged":false},"sourceIp":"REDACT","statusCode":400},"level":"info","timestamp":"2024-09-04T16:16:43.175046035Z","message":"Handled request"}

This is a ManagerSearch VM with 2 remote Forward nodes, PCAP is turned off, all grid status is green and OK.
Other troubleshooting steps I've taken already:

• rebooted all 3 servers this morning
• ran sudo soup again and all shows good
• ran sudo salt-call state.highstate and all shows good with no failures

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[defensivedepth]

If you go to SOC --> Hunt --> Dropdown query SOC - APP and then make sure that you have Exclude SOC Logs deselected under the Options dropdown, you should see a bunch of logs with log.level - Search for errors or warnings around the timeframe of a 400 error and post those errors here.

[innovate-support]

soc.fields.error content

index_closed_exception: closed -> {
"error" : {
"root_cause" : [
{
"type" : "index_closed_exception",
"reason" : "closed",
"index_uuid" : "ysgm2dzRSsSfAvw63J-wUw",
"index" : ".ds-logs-elastic_agent.endpoint_security-default-2024.09.03-000002"
}
],
"type" : "index_closed_exception",
"reason" : "closed",
"index_uuid" : "ysgm2dzRSsSfAvw63J-wUw",
"index" : ".ds-logs-elastic_agent.endpoint_security-default-2024.09.03-000002"
},
"status" : 400
}

message content

{"fields":{"clientHost":"https://soman:9200/","error":"index_closed_exception: closed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "index_closed_exception",\n "reason" : "closed",\n "index_uuid" : "ysgm2dzRSsSfAvw63J-wUw",\n "index" : ".ds-logs-elastic_agent.endpoint_security-default-2024.09.03-000002"\n }\n ],\n "type" : "index_closed_exception",\n "reason" : "closed",\n "index_uuid" : "ysgm2dzRSsSfAvw63J-wUw",\n "index" : ".ds-logs-elastic_agent.endpoint_security-default-2024.09.03-000002"\n },\n "status" : 400\n}\n"},"level":"error","timestamp":"2024-09-05T16:09:42.839343547Z","message":"Encountered error while updating elasticsearch"}

[innovate-support]

I think I've resolved it. I ran this command to show all the 'closed' indexes. (use your own user/pass)
curl -k -u USERNAME:PASSWORD -X GET "https://localhost:9200/_cat/indices?v"

I then ran this command to 'open' all the 'closed' ones. (specify the closed index name)
curl -k -u USERNAME:PASSWORD -X POST "https://localhost:9200/INDEXNAME/_open?pretty"

After I opened all the closed indexes my Acknowledgements function again.
Thanks for pointing me in the right direction, and for ChatGPT for giving me the commands to run.