Deploying Security Onion with 3 Elasticsearch master-eligible nodes

[rwaight]

Is there a recommended way to deploy Security Onion with 3 master-eligible Elasticsearch nodes? I am testing an SO environment with over 18 nodes and have been running into a unhealthy manager prompt when attempting to add more search nodes to my Security Onion grid (see #1751 (reply in thread) for reference). This has me also thinking about the lack of High Availability (HA) with TRUECLUSTER: true (one of many checks here). I'm hoping to leverage HA in Elasticsearch in the event the SO manager is ever restarted.

The reason I ask is based on this section of the Elasticsearch 'voting-only master-eligible node' guide:

High availability (HA) clusters require at least three master-eligible nodes, at least two of which are not voting-only nodes. Such a cluster will be able to elect a master node even if one of the nodes fails.

Since voting-only nodes never act as the cluster’s elected master, they may require less heap and a less powerful CPU than the true master nodes. However all master-eligible nodes, including voting-only nodes, require reasonably fast persistent storage and a reliable and low-latency network connection to the rest of the cluster, since they are on the critical path for publishing cluster state updates.

I was considering deploying 2 additional VMs as dedicated master-eligible ES nodes and connect them to the cluster with a custom firewall rule (to allow port 9300 between nodes) and custom certificates, but I do not want to go down an unsustainable path of customization to the current SO deployment configurations.

If there is not a recommended way to deploy SO with 3 master-eligible Elasticsearch nodes, would you consider an option to deploy an Other node configuration (from the install menu) for a dedicated master-eligible Elasticsearch node? Ideally, this would include logic to limit the number of master-eligible nodes to 3 (not voting-only nodes), although there is a mitigation plan if there is an odd number.

Elasticsearch docs for reference:

• Designing for resilience
• Bootstraping a cluster
• Setting the initial voting configuration

For context, I saw this question about multiple managers as well as this issue regarding high availability within an SO deployment.

[TOoSmOotH]

If you look at the current logic in the elasticsearch.yml you can change the roles by modifying the minion file for that node and setting a custom role. So you could take an existing search node and assign it the master role in the minion file.

From the elasticsearch.yml:

{%- set NODE_ROLES = salt['pillar.get']('elasticsearch:node_roles', ['data', 'ingest']) %}

So you would just need to add the additional role to it in the minion pillar for an existing search node:

elasticsearch:
  node_roles:
    - data
    - ingest
    - master

[rwaight]

That worked! Thank you very much, @TOoSmOotH!