-
|
Hello, I got a distributed layout with multiple sensors, a manager and a search node, using many elastic agent integrations. The manager has a disk of 200 GB, and the search node has multiple TBs. Over night the manager /nsm hit 100% usage and was completely full. This shut down Elasticsearch with the disk watermark. My errors were identical to this: https://discuss.elastic.co/t/how-to-solve-we-couldnt-log-you-in-please-try-again-error-in-kibana/332658/4 I see in /nsm/elasticsearch/indicies/ that full indicies are being stored on the manager, which to my understanding I thought the searchnodes should be the only ones storing them. How does 2.4 handle deleting logs based off the directory size? I know in 2.3 it used to be handled by salt/curator/action/delete.yml but now looking at that file in 2.4 it seems entirely broken. The log_size_limit isn't being set properly. Then search nodes also don't have curator installed on them. My manager is a VM and I had a snapshot so I was able to go back to before this error, but that means I don't have any of the SO logs. But I'm assuming this problem will arise again as the /nsm directory on the manager keeps increasing in storage. So is there a way to make sure this directory doesn't reach 100% storage utilization or a way to make the manager not store indicies and just utilize the search nodes? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 16 replies
-
|
Can you run sudo so-elasticsearch-query _cat/shards |
Beta Was this translation helpful? Give feedback.
-
|
Currently don't have the problem since I went back to a different VM snapshot and a lot of the /nsm directory went away, so unsure if this is helpful, but here you go. If it's not helpful I can just see if the problem occurs again. |
Beta Was this translation helpful? Give feedback.
-
|
What is strange is you have shards on your manager node that you shouldn't. Can you do so-elasticsearch-query .ds-logs-zeek-so-2023.08.15-000003/_settings | jq . and let me know the _tier_preference? Also did you have the manager up for a while and have data coming in before the search node was attached? Also can you look at /opt/so/conf/elasticsearch/elasticsearch.conf and confirm that it has data and not data_hot under roles? |
Beta Was this translation helpful? Give feedback.
-
|
_tier_preference is data_hot. The manager node does have the data role, not data_hot. Search has data_hot. Search node was the first node I attached to the manager, but there’s a chance that the search node had some downtime while the manager and sensors were still up and running. |
Beta Was this translation helpful? Give feedback.
-
|
ok you will need to force the zeek shards from the manager node. Not sure why they are not re-allocating on their own. https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html |
Beta Was this translation helpful? Give feedback.
-
|
Search node is automatically relocating shards to the manager. Caught this happening in the act, unsure what's causing this, but the result of Ran A fix to stop the automatic relocating would be great, as it'll eventually fill the manager /nsm disk up and hit the elasticsearch data floor and therefore stop ingesting logs, when the search node still has plenty of space. |
Beta Was this translation helpful? Give feedback.
-
|
Yep, currently it's only has been the zeek indicies being relocated. |
Beta Was this translation helpful? Give feedback.
-
|
Deleted the data stream before your reply edit, so I lost all existing zeek indicies, so don't have an example of a proven "bad index", but my current zeek index outputs this for |
Beta Was this translation helpful? Give feedback.
-
|
Zeek shard got reallocated again. Ran Zeek shard that got allocated to the manager: https://gist.github.com/EddieN17/3e51c01c736545462c48e15c0d5b219c Same zeek index, but different shard that is allocated to searchnode: https://gist.github.com/EddieN17/da1084f1a4edeeef45a54ebc45dadf56 |
Beta Was this translation helpful? Give feedback.
-
|
What are your watermark settings on the searchnode? |
Beta Was this translation helpful? Give feedback.
-
|
Watermark of searchnode:
|
Beta Was this translation helpful? Give feedback.
-
|
Another update: After running |
Beta Was this translation helpful? Give feedback.
-
|
As a current mitigation, I have adjusted the manager's watermark settings to a low of 45%, and a high of 50%, with the flood stage staying at 90%, and this has seemed to stop the search node reallocating the large zeek shards to the manager as moving any of the complete 50 GB shards would make the manager node over the high watermark. |
Beta Was this translation helpful? Give feedback.
-
|
Wanted to give this thread a bump... having the same issue for a fresh SOv2.4.30 distributed install. Like reported above, some Zeek data streams are being stored on the manager node (which led to /nsm reaching 100% and caused manager services to fail). Modifying the elasticsearch watermarks (thanks @EddieN17) appears to be preventing further /nsm space issues. Is there any way to prevent Zeek data streams from being stored on the manager? |
Beta Was this translation helpful? Give feedback.
-
|
The following should keep indices and data streams from creating on your manager. Replace the IP with the IP address of your manager.
|
Beta Was this translation helpful? Give feedback.
-
|
Is this something that is new in the balancing process? Do we need to add this to the salt stack somewhere? |
Beta Was this translation helpful? Give feedback.
-
|
This is due to the manager having the data role for the initial install. Once that setting is made and the indices migrate off you are able to take away the data role from the manager. For some reason even though we are telling elastic to put everything on the data_hot tier it ends up putting things there even though it is not marked as hot. |
Beta Was this translation helpful? Give feedback.
-
|
I had to make this persistent vs transient, ended up losing this config following container restarts/etc. |
Beta Was this translation helpful? Give feedback.
-
|
Just to be clear you're saying that the data role should be removed from the manager node. True? |
Beta Was this translation helpful? Give feedback.
The following should keep indices and data streams from creating on your manager. Replace the IP with the IP address of your manager.
PUT _cluster/settings { "transient" : { "cluster.routing.allocation.exclude._ip" : "10.0.0.1" } }