Disk filling up on manager node > 80% repeatedly

[ejgh-oe]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

10

RAM

32

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,
I'm running a distributed environment where sensor- and search-node run on hardware with plenty of disk space (4TB on sensor-node and 16 TB on search-node). The manager node on the other hand runs on a VM that's got only 500GB of space.

The problem: Disk space on the manager node regularly fills up beyond 80% causing elasticsearch and/or elastalert to shut down. In order to make them start again I've drastically reduced the retention periods for warm, cold and delete phases via the admin gui from their defaults to several days. While this makes elasticsearch/elastalert start again this can't be a long term solution.

So I came up with the following ideas:

• Is it possible to move elasticsearch & co to say the search node altogether? Since the search-node has plenty of disk space (see above) this would give more headroom in the sense of keeping logs for a longer period of time
• If moving elasticsearc & co to the search node is not an option, would it be possible to increase disk size of the manager node without reinstalling the manager node? I thought about giving the VM more diskspace (while being turned off) and then make the manager node (how?) aware of the larger disk space.

Thanks much in advance for any clue

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

Have you seen this section of our documentation?
https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch-node-roles

When building a distributed deployment, the Security Onion manager has to start off with the data node role. If you later join a separate search node, then you may want to migrate the data from the manager to the search node and then remove the data node role from the manager.

[ejgh-oe]

Have you seen this section of our documentation? https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch-node-roles

When building a distributed deployment, the Security Onion manager has to start off with the data node role. If you later join a separate search node, then you may want to migrate the data from the manager to the search node and then remove the data node role from the manager.

Oops, sorry I missed that...
In order not to have any data loss: is changing that setting, i.e. removing the "data" role from the manager node, something that can be done on the fly?

[dougburks]

Yes, you can do that on the fly without data loss if you follow the directions at: https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch-node-roles

Make sure you see the other page that it links to at:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-filtering

[ejgh-oe]

Yes, you can do that on the fly without data loss if you follow the directions at: https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch-node-roles

Make sure you see the other page that it links to at: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-filtering

TL;DR;
Perfect - neatly did the trick - everything working again :-)

Background-info in case somebody else is trying this: My first idea was to remove the data role from the manager node as described above. Unfortunately this resulted in elasticsearch on the mgr node to stop working. So I reverted that setting back, i.e. making the mgr node a data node again -> elasticsearch came back online on the manager node, so everything was back to the state as before.

So I went ahead and did the change as described in https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-filtering, i.e.

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.exclude._ip" : "192.168.5.127"
  }
}

via the Kibana-interface of S.O. (192.168.5.127 being the IP address of the manager node).

Then I had to wait a while (several hours) while the manager- and search-node started to interact heavily (constant high network traffic between the two, basically from manager- to search-node). While this was going on disk-usage of the /nsm partition on the manager node went down gradually and finally stopped changing. At the same time network traffic between manager- and search-node went down too.

Once this happened I removed the data role from the manager node (again) and this time everything worked.

Thanks much for your help - the disk space problem on the manager-node was something I've been dealing with for quite some time...