Repair Kibana Indexes

[rjdub2]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

12GB

Storage for /

200GB

Storage for /nsm

5TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

When we were trying to set up snapshots the kibana_analytics_* index became corrupted. It's located on one of the search nodes and I'm guessing it occurred due to some of the service/system restarts and failures we ran into until we got the snapshot settings configured correctly.

I'm trying to find a way to repair/re-initialize Kibana or curious if there's a way to delete just the index. I've tried deleting it to see if Kibana sets it back up on restart, but I can't figure out how to get the role I created with "allow_restricted_indices":true to map to either the user I log in with and can perform curl requests with or the so-elastic user. None of these users are shown in the _security/users api call, but I can see my role and role-mappings.

This is the initial set up of the system and we were configuring snapshots but didn't get to create any yet that I could restore the index from. I'm ok losing all data/indexes since it's the initial setup. What I don't want to have to do is recreate all of the nodes, but that's what I'm thinking I may need to do at this point.

Any thoughts, or suggestions?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Which index? You might have to use the kibana user from secrets.sls to delete it. something along the lines of curl --user 'so_kibana:pw' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/index -XDELETE where the user is so_kibana and pw is the password from secrets.sls.

[petiepooo]

Would the user creds be in /opt/so/saltstack/local/pillar/elasticsearch/auth.sls instead of ../kibana/secrets.sls?

[rjdub2]

You are right, this is where the so_kibana user's password is. I was able to authenticate, but the attempt to delete the index using this but this user isn't granted the ability to delete indexes either.

action [indices:admin/delete] is unauthorized for user [so_kibana] with effective roles [kibana_system] on indices [kibana_analytics_8.10.4_001], this action is granted by the index privileges [delete_index,manage,all]

[cm-ops]

Yes, it is under auth.sls.

[root@so-standalone ~]# curl --user 'so_kibana:$MY_PW' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/.kibana_analytics_8.8.2_001" -XDELETE
{"acknowledged":true}

Worked for me

[rjdub2]

Is it because we are on a newer version? Our nodes are all from SO images and we haven't customized anything. Here is the curl command and full output of the results just in case I'm missing something.

curl --user 'so_kibana:******' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/kibana_analytics_8.10.4_001" -XDELETE

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "action [indices:admin/delete] is unauthorized for user [so_kibana] with effective roles [kibana_system] on indices [kibana_analytics_8.10.4_001], this action is granted by the index privileges [delete_index,manage,all]"
      }
    ],
    "type": "security_exception",
    "reason": "action [indices:admin/delete] is unauthorized for user [so_kibana] with effective roles [kibana_system] on indices [kibana_analytics_8.10.4_001], this action is granted by the index privileges [delete_index,manage,all]"
  },
  "status": 403
}

[cm-ops]

Did you try adding the . --> .kibana_analytics_8.8.2_001 I see you did kibana_analytics_8.8.2_001?

[rjdub2]

Found how to create a role and a new user with the role that will allow you to delete system indexes. This one is specifically for Kibana indexes: https://www.elastic.co/guide/en/kibana/master/resolve-migrations-failures.html#_corrupt_saved_objects

Create the role

curl -u "anyuserwithpermissions" -k -XPUT  https://localhost:9200/_security/role/grant_kibana_system_indices -H "Content-Type: application/json" -d '{"indices": [{"names": [".kibana*"], "privileges": ["all"], "allow_restricted_indices": true}]}'

Create new user with the role

curl -u "anyuserwithpermissions" -k -XPOST  https://localhost:9200/_security/user/kibanaindexuser -H "Content-Type: application/json" -d '{"password" : "********","roles" : [ "superuser", "grant_kibana_system_indices" ]}'

Delete index as new user

curl --user 'kibanaindexuser' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/kibana_analytics_8.10.4_001" -XDELETE