Any guidance would be appreciated. Unable to update replicas settings due to improper permissions.

[Bal33p]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128

Storage for /

500

Storage for /nsm

30tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

more than 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

when applying fix sudo so-elasticsearch-query _cat/shards | grep UN
the following indices on a new Search node install give the following error when trying to apply the fix

{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/settings/update] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana_task_manager_8.10.4_001], this action is granted by the index privileges [manage,all]"}],"type":"security_exception","reason":"action [indices:admin/settings/update] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana_task_manager_8.10.4_001], this action is granted by the index privileges [manage,all]"},"status":403}

.fleet-agents-7 0 r UNASSIGNED
.fleet-servers-7 0 r UNASSIGNED
.transform-notifications-000002 0 r UNASSIGNED
.metrics-endpoint.metadata_united_default 0 r UNASSIGNED
.fleet-policies-leader-7 0 r UNASSIGNED
.kibana_8.10.4_001 0 r UNASSIGNED
.async-search 0 r UNASSIGNED
.transform-internal-007 0 r UNASSIGNED
.kibana_task_manager_8.10.4_001 0 r UNASSIGNED
.geoip_databases 0 r UNASSIGNED

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Bal33p]

I have tried creating a new role and user and granting permissions to all indexes, but nothing has worked.
The list of indexes has grown considerably, giving me the same error.

Could someone please give me the secret sauce to update these indexes since the "FIX" does not work

PUT /_security/role/my_admin_role
{
"cluster": ["manage"],
"indices": [
{
"names": [".fleet-agents-7"],
"privileges": ["indices:admin/settings/update"]
}
]
}

"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "action [indices:admin/settings/update] is unauthorized for user [test] with effective roles [my_admin_role,superuser] on restricted indices [.fleet-agents-7], this action is granted by the index privileges [manage,all]"
}
],
"type": "security_exception",
"reason": "action [indices:admin/settings/update] is unauthorized for user [test] with effective roles [my_admin_role,superuser] on restricted indices [.fleet-agents-7], this action is granted by the index privileges [manage,all]"
},
"status": 403
}

.security-profile-8 0 r UNASSIGNED
.ds-.fleet-fileds-fromhost-data-agent-2024.02.08-000005 0 r UNASSIGNED
.fleet-policies-7 0 r UNASSIGNED
logs-ti_recordedfuture_latest.threat-1 0 r UNASSIGNED
metrics-endpoint.metadata_current_default 0 r UNASSIGNED
.security-7 0 r UNASSIGNED
.kibana-observability-ai-assistant-conversations-000001 0 r UNASSIGNED
.ds-.kibana-event-log-8.10.4-2024.03.05-000003 0 r UNASSIGNED
.ds-.fleet-actions-results-2023.12.29-000001 0 r UNASSIGNED
.internal.alerts-stack.alerts-default-000001 0 r UNASSIGNED
.ds-ilm-history-5-2024.01.28-000002 0 r UNASSIGNED
.geoip_databases 0 r UNASSIGNED
.slo-observability.sli-v2 0 r UNASSIGNED
.fleet-enrollment-api-keys-7 0 r UNASSIGNED
.internal.alerts-security.alerts-default-000001 0 r UNASSIGNED
logs-ti_anomali_latest.threatstream-2 0 r UNASSIGNED
.internal.alerts-observability.apm.alerts-default-000001 0 r UNASSIGNED
.reporting-2024-02-11 0 r UNASSIGNED
.slo-observability.summary-v2 0 r UNASSIGNED
.internal.alerts-observability.metrics.alerts-default-000001 0 r UNASSIGNED
.fleet-servers-7 0 r UNASSIGNED
.ds-.fleet-actions-results-2024.01.28-000002 0 r UNASSIGNED
.apm-agent-configuration 0 r UNASSIGNED
.transform-internal-007 0 r UNASSIGNED
.reporting-2024-01-28 0 r UNASSIGNED
.internal.alerts-observability.logs.alerts-default-000001 0 r UNASSIGNED
.internal.alerts-observability.slo.alerts-default-000001 0 r UNASSIGNED
.fleet-artifacts-7 0 r UNASSIGNED
.kibana_security_session_1 0 r UNASSIGNED
.kibana_8.10.4_001 0 r UNASSIGNED
.ds-.fleet-fileds-fromhost-meta-agent-2024.01.18-000001 0 r UNASSIGNED
.transform-notifications-000002 0 r UNASSIGNED
.ds-.kibana-event-log-8.10.4-2024.01.28-000002 0 r UNASSIGNED
.ds-.fleet-actions-results-2024.02.27-000003 0 r UNASSIGNED
.kibana_alerting_cases_8.10.4_001 0 r UNASSIGNED
logs-ti_otx_latest.dest_pulses_subscribed-1 0 r UNASSIGNED
.slo-observability.summary-v2.temp 0 r UNASSIGNED
.fleet-actions-7 0 r UNASSIGNED
.kibana_analytics_8.10.4_001 0 r UNASSIGNED
.ds-.fleet-fileds-fromhost-meta-agent-2024.02.17-000002 0 r UNASSIGNED
.kibana_task_manager_8.10.4_001 0 r UNASSIGNED
.internal.alerts-observability.uptime.alerts-default-000001 0 r UNASSIGNED
.kibana_ingest_8.10.4_001 0 r UNASSIGNED
.async-search 0 r UNASSIGNED
.apm-custom-link 0 r UNASSIGNED
.ds-ilm-history-5-2024.02.27-000003 0 r UNASSIGNED
.ds-ilm-history-5-2023.12.29-000001 0 r UNASSIGNED
.ds-.kibana-event-log-8.10.4-2023.12.29-000001 0 r UNASSIGNED
.metrics-endpoint.metadata_united_default 0 r UNASSIGNED
.fleet-agents-7 0 r UNASSIGNED
.apm-source-map 0 r UNASSIGNED
.reporting-2024-02-04 0 r UNASSIGNED
.kibana_security_solution_8.10.4_001 0 r UNASSIGNED
.fleet-policies-leader-7 0 r UNASSIGNED

[Bal33p]

In this example, the role has the required permission, as far as I can tell. What am I doing wrong?

[Bal33p]

I just saw the allow_restricted_indices : false
I modified this to true and now this is the error oi am getting
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "Cannot override settings on system indices: [.fleet-agents*] -> [number_of_replicas]"
}
],
"type": "illegal_state_exception",
"reason": "Cannot override settings on system indices: [.fleet-agents*] -> [number_of_replicas]"
},
"status": 500
}

[cm-ops]

elastic/elasticsearch#85799

Some of what you are doing should be possible if your role's allow_restricted_indices flag is set to true. This is a setting that should be used with caution, since it will allow deletion and modification of most system indices. It's documented on the page for Defining Roles.

This setting currently does not allow access to system indices for the GeoIP plugin, fleet indices, or to system datastreams. In these cases, rolling back to a good snapshot may be the only fix.

One of the main problems with system indices right now is that we need better ways to get them out of bad states. We'd like to add automated cleanup, but if that takes too long we may have to roll back our restrictions to allow the sort of direct user maintenance that was possible in previous versions of Elasticsearch.

[Bal33p]

Great find! Thank you so much for the response!
So I guess I am just SOL, since there appears no way to fix it.
Everything was fun until i applied the pending "FIX" not sure what to do at this point

[cm-ops]

What does cluster/allocation/explain show on one of those replicas?

Below as an example.

GET _cluster/allocation/explain
{
  "index": ".fleet-agents-7",
  "shard": 0,
  "primary": false,
}

[Bal33p]

{
"index": ".fleet-agents-7",
"shard": 0,
"primary": false,
"current_state": "unassigned",
"unassigned_info": {
"reason": "REPLICA_ADDED",
"at": "2024-03-08T13:45:14.021Z",
"last_allocation_status": "no_attempt"
},
"can_allocate": "no",
"allocate_explanation": "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions": [
{
"node_id": "XXXUBWZncpw",
"node_name": "XXXconsp02",
"transport_address": "172.24.XXX.XXX:9300",
"node_attributes": {
"transform.config_version": "10.0.0",
"xpack.installed": "true"
},
"node_decision": "no",
"deciders": [
{
"decider": "same_shard",
"decision": "NO",
"explanation": "a copy of this shard is already allocated to this node [[.fleet-agents-7][0], node[XXXVU0IRF67n_UBWZncpw], [P], s[STARTED], a[id=XXXdKcURqq-jwkp5xN8bg], failed_attempts[0]]"
}
]
},
{
"node_id": "XXXXXV0RIij3l5pD4qbMA",
"node_name": "XXXconsp01",
"transport_address": "172.24.XXX.XXX:9300",
"node_attributes": {
"xpack.installed": "true",
"transform.config_version": "10.0.0"
},
"node_decision": "no",
"deciders": [
{
"decider": "disk_threshold",
"decision": "NO",
"explanation": "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=80%], having less than the minimum required [3.1tb] free space, actual free: [2.5tb], actual used: [84.3%]"
}
]
},
{
"node_id": "XXXQLuzD4ZOW2iySQ",
"node_name": "XXXconmp01",
"transport_address": "172.24.XXX.XXX:9300",
"node_attributes": {
"xpack.installed": "true",
"transform.config_version": "10.0.0"
},
"node_decision": "no",
"deciders": [
{
"decider": "filter",
"decision": "NO",
"explanation": """node matches cluster setting [cluster.routing.allocation.exclude] filters [_ip:"172.24.XXX.XXX"]"""
}
]
}
]
}

[Bal33p]

I recently added a new search node, so I don't know if that is what has caused the issue.

[cm-ops]

"node_name": "XXXconsp01" is above the low watermark threshold, that is why ES has not allocated the replica there.

"decider": "disk_threshold",
"decision": "NO",
"explanation": "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=80%], having less than the minimum required [3.1tb] free space, actual free: [2.5tb], actual used: [84.3%]"

[Bal33p]

I am attempting to do an auto reallocations now that i have the new Search Node
I have implemented the following setting. Please advise if this is a mistake and if i should change it.
PUT /_cluster/settings { "persistent": { "cluster.routing.rebalance.enable": "all", "cluster.routing.allocation.allow_rebalance": "indices_all_active" } }

[cm-ops]

indices_all_active is the default, you could use cluster reroute to move shards. What are you node roles set to (manager and search)?