Manager acting as data node

[kspringer-maf]

Version

2.4.141

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

many

RAM

much

Storage for /

lots

Storage for /nsm

lots

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have a Distributed grid setup with Manager, Search, Receiver, and multiple Forward Sensors. All seems to be running great, except that the Manager /nsm partition is large and full with hundreds of gigs of indices. They rollover and get deleted based on my ILM settings, but I have to keep an eye on it because it's running high and consuming most of the disk. I'm wondering why they are there at all since I have all PCAP disabled, my Manager is not a Search node, and based on my understanding of the distrubuted grid topology my Manager shouldn't be storing so much data on it, all that should live on the Search node.

Skynet had me run a curl command to determine if the Manager role was set as a 'data' node, and it shows that it is. Is this normal?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Would you run sudo so-elasticsearch-query _cat/allocation?v? I want to see what you shard allocation looks like and how much disk you have on your manager and search.

[kspringer-maf]

[InfosecGoon]

It's storing data on your Manager because your Manager is part of the Elasticsearch cluster and, by default, has the "data" role assigned to it -- that's the "d" in "dmrt" in the allocation output above.

If you want to move the data off of there to the Search Node, you can, but it looks like you're mighty close to the limit on the Search Node as well. You'll need to either delete some older stuff or add another Search Node -- or more disk to the current one -- so you've got some place to put it.

More information: https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch-node-roles

[kspringer-maf]

Thanks. I will increase the disk on Search1, and setup a 2nd Search node as well. Once that's complete I will attempt to remove the 'data' role from the Manager. This rolls right into our plan to attach a storage array and offload older logs for long term storage so we have many months of history instead of just a few weeks.

[kspringer-maf]

@InfosecGoon just to confirm before I apply a setting that could mess things up, I just need to remove the 'data' role from my Manager node and the data will move to the Search nodes automatically and all will be well? Or do I have to add 'data_content' to the Manager? If so, what's the difference between 'data' and 'data_content'?

I reviewed https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch-node-roles
and https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/node-settings
but I'm not seeing a clear answer about this.

[kspringer-maf]

I tried removing the 'data' role from the so-manager, and then elasticsearch failed. So I added the 'data' role back and ran sudo so-checkin to fix it.
It seems that simply removing the 'data' role is not all that's needed. Can anyone please advise?

[cm-ops]

You have to move the data off the manager node first. Then remove the data role.

In Kibana > Dev Tools:

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}

Use that transient setting to move the data, once you don't see any shards on your manager with GET /_cat/shards remove the role.