Request failed with status code 502

[molfar101]

Hello,

I run a distributed architecture with 4 search nodes, one forward node, and manager node.
I've started getting error 502 on the Hunt, Dashboard, and and Alert pages. However, everything works fine if search node 03 is offline (whewnever I reboot it everything works fine until all modules are back online). I think there is an issue with elasticsearch since I see the following error in the logs:
this node is unhealthy: health check failed on /usr/share/elasticsearch/data

another error:
unexpected failure while sending request [internal:cluster/shard/started] to [{search node name}{Fi_wbbZr....}......

Please point me in the right direction to fix the problem

Thanks

[dougburks]

Have you checked the Elasticsearch logs on the search node?
https://docs.securityonion.net/en/2.3/elasticsearch.html#diagnostic-logging

[molfar101]

I checked logs on healthy nodes and the one that has issues and I see the following error:
[ERROR][io.netty.util.concurrent.DefaultPromise.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?
Also, there is a warning:
[org.elasticsearch.transport.OutboundHandler] send message failed [channel: Netty4TcpChannel{localAddress=/172.17.0.4:9300, remoteAddress=/MasterNodeIP:38348, profile=default}]
Soc logs don't show any errors or warnings.

Please advise. Thank you

[molfar101]

I checked the nodes health and it looks like there is an unassigned shard which causing the error. Any recommendation on how to proceed?

Thank you

[molfar101]

I deleted an index that was causing issue and the problem was solved.
I ran the following command to find an index that was causing issue:
so-elasticsearch-query _cat/indices | grep red
Then I copied the name and deleted it:
so-elasticsearch-query [index name] -XDELETE