If a node is down Alerts does not show until node is back online #13046
-
Version2.4.60 Installation MethodSecurity Onion ISO image Descriptionother (please provide detail below) Installation TypeDistributed Locationon-prem with Internet access Hardware SpecsExceeds minimum requirements CPU16 RAM128 Storage for /1 TB Storage for /nsm16 TB Network Traffic Collectiontap Network Traffic Speeds1Gbps to 10Gbps StatusNo, one or more services are failed (please provide detail below) Salt StatusYes, there are salt failures (please provide detail below) LogsNo, there are no additional clues DetailHello, Seems that on 2.4.X, if any of the nodes in the grid are down for hardware, networking or power issues, no alerts or dashboards show for the remaining nodes. Only when all nodes are able to communicate to/from the manager, will alerts show. Would be nice if there was an option to still show alerts/dashboards from all nodes that are still up until that single node can be brought back online at a later time. This error shows in alerts until all nodes are up and show green in the grid.
Guidelines
|
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 3 replies
-
|
What is your grid made up of? How many searchnodes do you have? Is this a managersearch? What is the output of |
Beta Was this translation helpful? Give feedback.
-
|
I have one manager node and 8 heavy nodes. |
Beta Was this translation helpful? Give feedback.
-
|
I realize now you mentioned all heavy nodes. Is there a particular reason you elected to use heavynodes instead of keeping separate search / forward nodes? Having separate searchnodes would allow you to have replication across the elasticsearch cluster and allowing you to lose a node entirely while keeping data available. Heavynodes are treated as individual elasticsearch clusters that get searched by the manager |
Beta Was this translation helpful? Give feedback.
-
|
It just seemed easier to us for our install need. Would be nice if there was a skip a node option if it can't reach one for network outage or hardware issue so it would still show alerts, dashboard and so on for all the other nodes in the grid. |
Beta Was this translation helpful? Give feedback.
-
|
Making changes like skip_unavailable could give you a false sense of security. If errors are not brought up during your search then you might be under the impression everything is functioning as intended, but you could be missing logs from an entire heavynode. With that important note in mind. You could make the remote clusters (heavynodes) skip_unavailable from Kibana -> stack management -> remote clusters -> edit and toggle the skip_unavailable option I would highly recommend not leaving that enabled as a default. Perhaps if you know a site has a power outage scheduled or networking issues you can enable it temporarily so that you can view data from other heavynodes. However, strongly recommend you go back and disable skip_unavailable. That way you are immediately aware of any issues your heavynodes maybe having. |
Beta Was this translation helpful? Give feedback.
Making changes like skip_unavailable could give you a false sense of security. If errors are not brought up during your search then you might be under the impression everything is functioning as intended, but you could be missing logs from an entire heavynode.
With that important note in mind. You could make the remote clusters (heavynodes) skip_unavailable from Kibana -> stack management -> remote clusters -> edit and toggle the skip_unavailable option
I would highly recommend not leaving that enabled as a default. Perhaps if you know a site has a power outage scheduled or networking issues you can enable it temporarily so that you can view data from other heavynodes. However, strongly r…