Added search node, indices 'Unassigned', wants replica's but I set them to '0'

[Jaap79]

Hi,

I also asked SecurityOnion Support on this but maybe someone here has an idea as well:

We added an additional Search Node last Friday. It's been running fine, mostly - data seems to be flowing in like normal. However, I noticed that some of my Kibana Dashboards stopped working because shards are failing.

Turns out that I keep getting indices Started and Unassigned on that new Search Node. We are using the normal clustering (not the True Cluster settings). All other search nodes are fine and everything acts like normal. This last one, however, doesn't.

According to my global.sls pillar file on the Manager (/opt/so/saltstack/local/pillar/global.sls), I should have '0' replicas set:

(...)
elastic:
features: False
elasticsearch:
true_cluster: False
replicas: 0
discovery_nodes: 1
hot_warm_enabled: False
(...)

And further down the file, as an example how indices are set up:
(...)
so-syslog:
index_template:
template:
settings:
index:
number_of_shards: 1
warm: 7
close: 30
delete: 91
so-zeek:
index_template:
template:
settings:
index:
number_of_shards: 1
warm: 7
close: 30
delete: 91
(...)

When I run so-elasticsearch-query _cluster/health on that particular search node:

{"cluster_name":"REDACTED","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":52,"active_shards":52,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":49,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":51.48514851485149}

And when I query the actual shards:
(...)
so-syslog-2023.02.19 0 p STARTED 162491 75.7mb 10.10.x.x REDACTED
so-syslog-2023.02.19 0 r UNASSIGNED
so-zeek-2023.01.29 0 p STARTED 2 67.1kb 10.10.x.x REDACTED
so-zeek-2023.01.29 0 r UNASSIGNED
so-zeek-2023.02.09 0 p STARTED 7 180.6kb 10.10.x.x REDACTED
so-zeek-2023.02.09 0 r UNASSIGNED
(...)

I don't seem to have anything particular set up in the pillar file for the Search Nodes themselves, and they all look alike:

(...)
elasticsearch:
mainip: '10.10.x.x'
mainint: 'ens192'
esheap: '16384m'
esclustername: '{{ grains.host }}'
node_type: 'search'
es_port: 9200
log_size_limit: 3533
node_route_type: 'hot'

logstash_settings:
ls_pipeline_batch_size: 125
ls_input_threads: 1
lsheap: 1000m
ls_pipeline_workers: 16
(...)

Any ideas on how to fix this without closing indices which actually contain data...?

[cm-ops]

This was a template issue and answered with Security Onion Professional Support.