Kibana 404 page not found

[shawnpersona]

Greetings,

After a power outage I have been having issues getting Kibana to load. Currently I have a set up with 17 forward nodes, and a Manager/search node. I am hoping someone has come across this and may be able to help. so-status shows everything is up and running:
Checking container statuses

so-aptcacherng --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-curator ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-dockerregistry ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-elastalert ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-elasticsearch ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-filebeat ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-fleet --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-grafana ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-idstools ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-influxdb ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-kibana -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-kratos -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-logstash ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-mysql --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-nginx --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-playbook ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-redis --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-sensoroni ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-soc ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-soctopus ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-telegraf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-wazuh --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

I have tried a few things suggested for similar errors through the forums with no success.
Such as

• sudo so-elastic-auth
• FORCE_SYNC=true so-user sync
• sudo so-kibana-restart
• sudo so-kibana-space-defaults
• trying to delete .kibana ( curl --user *** -k -XDELETE https://localhost:9200/.kibana) which returns an error

sudo docker logs so-kibana

returns the following results (Tail end)

[2023-02-07T19:05:25.636+00:00][INFO ][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> FATAL. took: 64022ms.
[2023-02-07T19:05:25.637+00:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the CALCULATE_EXCLUDE_FILTERS step after 15 attempts, terminating. The last failure message was: search_phase_execution_exception: [no_shard_available_action_exception] Reason: null
    at migrationStateActionMachine (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/target_node/src/migrations_state_action_machine.js:127:29)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/usr/share/kibana/node_modules/@kbn/core-saved-objects-server-internal/target_node/src/saved_objects_service.js:160:9)
    at Server.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/target_node/src/server.js:328:31)
    at Root.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/target_node/src/root/index.js:63:14)
    at bootstrap (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/target_node/src/bootstrap.js:103:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:180:5)
[2023-02-07T19:05:25.651+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2023-02-07T19:05:25.652+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2023-02-07T19:05:25.652+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
[2023-02-07T19:05:25.656+00:00][ERROR][plugins.ruleRegistry] Error: Server is stopping; must stop all async operations
    at /usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:54:20
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2023-02-07T19:05:25.657+00:00][ERROR][plugins.ruleRegistry] Error: Failure installing common resources shared between all indices. Server is stopping; must stop all async operations
    at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:62:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:76:5)

 FATAL  Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the CALCULATE_EXCLUDE_FILTERS step after 15 attempts, terminating. The last failure message was: search_phase_execution_exception: [no_shard_available_action_exception] Reason: null

[InfosecGoon]

Definitely seems like that .kibana index is the issue - what's the error you get when you try to delete it?

[InfosecGoon]

Try "sudo so-elasticsearch-query .kibana -XDELETE".

It looks like you're trying to delete it but you aren't using Elastic credentials to authenticate so it's being denied -- so-elasticsearch-query is a wrapper script that executes Elasticsearch API calls with the credentials of the built-in so-kibana account.

[shawnpersona]

Looks like I am still getting the same error

sudo so-elasticsearch-query .kibana -XDELETE
[sudo] password for manager:
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

[InfosecGoon]

Okay, let's try it with the so_kibana account instead. My mistake.

Get the so_kibana password from the Elasticsearch auth pillar (salt-call pillar.get elasticsearch:auth), and try running this:

curl --user 'so_kibana:PASSWORD' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/.kibana -XDELETE

Then, if it deletes properly, restart Kibana with so-kibana-restart and try to log in again.

[shawnpersona]

I feel like we are so much closer, I am getting the following error running the above.

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"invalid version format: -XDELETE HTTP/1.1"}],"type":"illegal_argument_exception","reason":"invalid version format: -XDELETE HTTP/1.1"},"status":400}[manager]

[shawnpersona]

Just realized I had the " in the wrong place switched to curl --user 'so_kibana:PASSWORD' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/.kibana" -XDELETE

I now get a different error message

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [.kibana] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [.kibana] matches an alias, specify the corresponding concrete indices instead."},"status":400}

[shawnpersona]

Thank you, I found a reference to .kibana_task_manager_8.3.3_001 in the logs so I used the following.

curl --user so_kibana -s -k -L -H "Content-Type: application/json" "https://localhost:9200/.kibana_task_manager_8.3.3_001" -XDELETE

This issue has now been resolved .

[InfosecGoon]

Excellent, glad to hear it's resolved!