Cases gives 500 when creating/viewing cases.

[orchidObsessed]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8-16

RAM

16-32 GB

Storage for /

100

Storage for /nsm

200

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

When attempting to leverage the "Cases" feature in the SOC, I am met with a 500 error banner. This seems to impact the majority of case functionality, but does not prevent me from creating new cases (either from an alert/other source, or from the cases menu directly); only from modifying them afterwards. No other functionality (monitoring, alerting, hunting, etc.) seems to be impacted. Fixes for other 500 errors have not seemed to help here, such as creating new users.

All of my nodes meet or exceed the hardware requirements, and all were installed via the official ISO. There are no errors/warnings in the SOC or any processes. However, some of the logs do reveal what may or may not be noteworthy errors, including NGINX errors (coming from /opt/so/log/nginx/error.log:

2025/02/09 13:19:30 [error] 22#2: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.XXX.XXX, server: soc.mydomain.com, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://192.168.XXX.XXX:9822/api/node", host: "soc.mydomain.com"

As well as errors coming from /opt/so/log/soc/sensoroni-server.log:

{"fields":{"error":"Object not found","requestId":"01431ba0-3c2f-4ab4-806d-3249b7e5fc26","requestor":{"id":"013cd8d8-8459-44f2-bf50-b5683e7a6afa","createTime":"2025-02-09T13:20:59.081840108Z","updateTime":"0001-01-01T00:00:00Z","email":"[email protected]","firstName":"","lastName":"","totpStatus":"disabled","oidcStatus":"disabled","webauthnStatus":"disabled","note":"","roles":null,"status":"","searchUsername":"","password":"","passwordChanged":false}},"level":"warn","timestamp":"2025-02-09T13:20:59.111078711Z","message":"Request did not complete successfully"}

Any help on this would be greatly appreciated!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Can you run so-elasticsearch-query _cat/indices?v=true and provide the output?

Any errors in your elasticsearch log? /opt/so/log/elasticsearch/securityonion.log

[orchidObsessed]

I take it back! Looking at the cluster today, I noticed ElasticSearch randomly bouncing between OK and Pending. Looking at the logs again, I see logs relating to missing a specific shard:

[2025-02-11T16:38:49,793][WARN ][rest.suppressed          ] path: /.ds-logs-*/_eql/search, params: {ignore_unavailable=true, index=.ds-logs-*}, status: 503
org.elasticsearch.action.search.SearchPhaseExecutionException: start
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:416) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.onFailure(CanMatchPreFilterSearchPhase.java:405) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-system.auth-default-2025.02.11-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
        at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:196) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:131) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:410) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
        ... 6 more

And shortly following that message, a log indicating it restarted that shard:

[2025-02-11T16:38:52,364][INFO ][org.elasticsearch.cluster.routing.allocation.AllocationService] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-logs-system.auth-default-2025.02.11-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ds-logs-system.auth-default-2025.02.11-000001][0]]"

This makes me think it is in some way related to users / setup of users. Do you have any guidance here?

[reyesj2]

Looking through this further. I had some custom configuration for elasticsearch indices and detections, and undid all of those

What changes did you make and then later revert?

Are you sure the cluster is going back and forth between OK and PENDING or is just taking a while to initialize? Is this a standalone or a distributed grid where you have more than 1 searchnode? Any chance you had a separate searchnode installed and later removed it without migrating the data over to another searchnode?

That error looks to be from a search request taking place before all the shards are green/available for searching

[orchidObsessed]

I suppose it could be taking a while to initialize, but all nodes report okay and all other features are fully functional (save for when the manager briefly shows Pending on elasticsearch).

The changes I made were to the Elasticsearch global ILM policies (as well as adding tuning rules to some detection rules). Following these changes, I noticed that Elastalert did not always show up as OK under the detections page. I undid both of those changes and reran a highstate, following the steps in issue #13124. As it stands now, Elastalert has not returned to RuleMismatch.

This is a distributed grid, I have two search nodes with identical specs: 8x vCPU, 24GB memory, and a relatively small amount of storage as I trial and refine my setup (100G OS and 256GB NSM). There were no other nodes before this, but I suppose it is possible that I managed to reboot a node at the worst possible time. Is there any way I can rebuild shards/indices? I'm more than happy to delete existing ones if needed, as this is a very "young" environment.

If it is the cluster taking a while to initialize and not data loss, what would I look for there? Once in a while I'll see warnings about writing the cluster state taking too long; would that be indicative of it initializing? This is a sample of one of those logs:

[2025-02-11T16:40:27,843][WARN ][org.elasticsearch.gateway.PersistedClusterStateService] writing cluster state took [24018ms] which is above the warn threshold of [10s]; [skipped writing] global metadata, wrote [0] new mappings, removed [0] mappings and skipped [82] unchanged mappings, wrote metadata for [0] new indices and [1] existing indices, removed metadata for [0] indices and skipped [89] unchanged indices

[reyesj2]

Assuming these are virtual machines? What kind of backing does the storage have? shared network storage?

That error may be indicative of some disk IO issues

[orchidObsessed]

Yup, the search nodes are VMs. Storage is a vSAN, unfortunately running HDDs, so the IO performance may be less than optimal. I was hoping to get this fully set up as a POC before moving it onto a flash array.