Expanding Sec Onion Grid using Hot - Warm - Cold roles

[ColeVan]

Hopefully someone can point me in the right direction for this one.

I have 5 search nodes that are approximately 50TB each and I'm looking to divide hot, warm, and cold storage roles to potentially increase efficiency when querying the data. What would be the best way to tackle this? Do all the nodes need to have the same storage capacity as the original search nodes 50TB each? I'm wanting to add 4 additional search nodes with the specifications below. Would this work? Should the warm nodes be larger then the hot nodes? Should the colder nodes be larger then everything since it will store data for longer?

Any guidance would be great. Thank you in advance.

Example:
ILM

Hot 30 days

Warm 60 days

Cold 180 days

Overall Retention between search nodes: 80%

Existing - Search Node 1 : 50TB Role: data_hot & data_content

Existing - Search Node 2 : 50TB Role: data_hot & data_content

Existing - Search Node 3 : 50TB Role: data_hot & data_content

Existing - Search Node 4 : 50TB Role: data_hot & data_content

New - Search Node 5 : 50TB Role: data_hot & data_content

New - Search Node 6 : 20TB Role: data_warm

New - Search Node 7 : 20TB Role: data_warm

New - Search Node 8 : 20TB Role: data_cold

[cm-ops]

Typically if the search nodes have different disks, HDD v. SSD, the data_hot and data_content roles are placed on the faster disk. If the disks are all alike and there are a mix of 20TB and 50TB, I would put data_hot and data_content on the smaller of the two.

Keep an eye on the amount of data going on the 20TB nodes and adjust your warm setting to keep them from filling up past watermark settings. If they are all the same size, you would still adjust to move the data to the warm nodes.

For the 50TB nodes, set data_warm and data_cold, unless you are going to separate the data tiers, then assign the appropriate role to the node. To assist in data retention and searching, also look at ILM actions to extend the data https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html For example, shrink and force merge.

[ColeVan]

Thank you for your response. I think we will try and keep the storage size all the same between the nodes. I'll think what I'll do is create two more search nodes with 50T space and makes those the data_warm and data_cold roles and adjust my ILM to reflect the change.

[ColeVan]

I do have one more question. In the event we wanted to resize nodes and redeploy an entirely new grid with proper storage sizing and node roles is there a way to snapshot the indexed data, save it offline, and re-import once the new grid is up?

[cm-ops]

Yes, depending on the size of the data this could take some time to snapshot and then restore. https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html What type of repository would you be using?

[ColeVan]

We have a few NAS's storage locations with plenty of storage. After doing a little bit of research I believe your correct, this would take some time and not be optimal.

One last question. Would it be possible to change an existing search node from hot to warm role? Would there be any major impacts?

[cm-ops]

You can change the node role, Elasticsearch will move data that does not fit the tier_preference currently for ILM. Depending on how much hot tier data there is shards could move for a period of time. From my experience, a 50GB shard with defaults takes approximately 30-45 minutes to relocate. Defaults will move 2 at a time.