Elasticsearch Status: pending. Process status: Fault

[redla1N]

Hello, guys
After update from 2.4.120 to 2.4.160 i have problem, my manager always not working.
My installation have: 3 Forwards, 1 heavy 1 Manager and 1 Search. In manager problem:

Kibana have 404 page not found

Detections, Case,Hunt, dashboards Alerts have 500 status code:

so-status

after reboot (so-elasticsearch-restart --force) i have:

How fix it?

[rh-ops]

Navigate to /opt/so/state and, if present, remove the estemplates and estemplatesinitialload files, then run so-elasticsearch-templates-load, and when that finishes, run so-checkin.

After that run so-elasticsearch-query _health_report?pretty and so-elasticsearch-query _cat/allocation?v and relay the results.

[redla1N]

@rh-ops can u help please?

[rh-ops]

Okay first lets confirm you have all your nodes. Run the command salt \* test.ping on your manager and verify all your nodes are in the output. If all your nodes are present, we can continue.

Go into SOC and navigate to SOC > Administration > Configuration > Elasticsearch > Index Settings > global_overrides > policy > phases > delete > min_age And note what its currently set to. If its at the default of 365, then you'll need to drop it. I'd recommend dropping it based on how long you are required or would like to hold onto data compared to your average rate of data ingestion and storage space. For example, if you have 100TB of storage space, but are ingesting 1TB of data every day, you aren't going to be able to hold more than 100 days of data. Now if you'd like to be on the safe side, just to regain stability, I'd say try dropping it to 90 days. Then clicking the green checkmark to save it. After that go to your cold phase and set that to 60 days, and then go to your warm phase and set that to 30 days. After that, go to the top of the page, selection the "Options" drop down menu and then select "Synchronize Grid". Give that about 5 minutes to run a high-state and then continue with the following instructions.

Go into your SOC, then select Kibana, after logging in, hit the 3-Lined Button at the top left corner of the screen, then scroll down and select Stack Management. That should take you to a new page. Then from the left panel, select "Index Management", then move over to the center-right side of the screen and select "Include hidden indices". Now scroll down to the bottom of the list of indices and change the number of rows to 100 by clicking the "Rows per page" button at the bottom left. From there, look at your indices named .ds-logs-zeek-so, and delete all the ones that are over 90 days old.

Once you've done that, go back into your CLI, run an so-checkin and then when it finishes, run so-elasticsearch-query _cluster/health?pretty, and take note of the number of "unassigned_shards" and "relocating_shards". The amount of unassigned shards should gradually be going down over time as they are relocated. Give it some time to do that and monitor its progress every so often. If all goes well, the status should change from "yellow" to "green", in which case, try running so-elasticsearch-query _cat/allocation?v command again and share the output.

[redla1N]

@rh-ops
salt * test.ping

SOC > Administration > Configuration > Elasticsearch > Index Settings > global_overrides > policy > phases > delete > min_age. My value=10

I reduced all values by 2 times

I cant connect Kibana, 500 error

after write so-elasticsearch-query _cluster/health?pretty

so-elasticsearch-query _cat/allocation?v

[redla1N]

"Go into your SOC, then select Kibana, after logging in, hit the 3-Lined Button at the top left corner of the screen, then scroll down and select Stack Management. That should take you to a new page. Then from the left panel, select "Index Management", then move over to the center-right side of the screen and select "Include hidden indices". Now scroll down to the bottom of the list of indices and change the number of rows to 100 by clicking the "Rows per page" button at the bottom left. From there, look at your indices named .ds-logs-zeek-so, and delete all the ones that are over 90 days old." - how i can delete if kibana not working?

[rh-ops]

Okay so master_not_discovered_exception means at some point your manager got detached from the rest of the cluster. Easiest solution in this case is to just rerun setup, but fair warning, this will wipe your currently stored data.

[redla1N]

@rh-ops im rerun setup, but i have any problems after this.. can u check pls? #14920

I saw Grid, im accepted Grids, but they have problems with connection to manager