Request failed with status code 500; and Kibana 404 page not found

[emkaxon]

Hello Community,

We have a distributed Security Onion deployment version: 2.3.270

Whenever we try to access "Alerts", "Hunt", "Dashboards", "Cases" on the web interface we're receiving the following Error:

" Request failed with status code 500
See the Help section of the Security Onion documentation for additional troubleshooting guidance. "

Also when trying to open Kibana, it's displaying that Kibana "404 page not found"

Knowing that the status of all the containers is OK and everything is up and running.

We tried restarting and recreating the containers, but we're still facing the same errors

Those are the logs retrieved from /opt/so/log/kibana/kibana.log:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:17:45.215+00:00","message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 124011ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:17:45.269+00:00","message":"[.kibana] Action failed with '[index_not_yellow_timeout] Timeout waiting for the status of the [.kibana_8.7.1_001] index to become 'yellow' Refer to https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 15 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:17:45.270+00:00","message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 124014ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:19:49.223+00:00","message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> FATAL. took: 124008ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:19:49.225+00:00","message":"Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the WAIT_FOR_YELLOW_SOURCE step after 15 attempts, terminating. The last failure message was: [index_not_yellow_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.6.2_001] index to become 'yellow' Refer to https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.","error":{"message":"Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the WAIT_FOR_YELLOW_SOURCE step after 15 attempts, terminating. The last failure message was: [index_not_yellow_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.6.2_001] index to become 'yellow' Refer to https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.","type":"Error","stack_trace":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the WAIT_FOR_YELLOW_SOURCE step after 15 attempts, terminating. The last failure message was: [index_not_yellow_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.6.2_001] index to become 'yellow' Refer to https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.\n at migrationStateActionMachine (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/migrations_state_action_machine.js:98:29)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Promise.all (index 1)\n at SavedObjectsService.start (/usr/share/kibana/node_modules/@kbn/core-saved-objects-server-internal/src/saved_objects_service.js:175:9)\n at Server.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/server.js:342:31)\n at Root.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/root/index.js:64:14)\n at bootstrap (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:130:5)\n at Command. (/usr/share/kibana/src/cli/serve/serve.js:276:5)"},"log":{"level":"FATAL","logger":"root"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:19:49.241+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:19:49.242+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:19:49.242+00:00","message":"Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2023-11-13T17:19:49.281+00:00","message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> FATAL. took: 124012ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":7},"trace":{"id":"444a4d2e0f18bf91ce299af2d328581e"},"transaction":{"id":"0226347d3e27f5ff"}}

Has anyone faced this issue before?

[Mav1814]

Hi,
Check disk spaces. Usually is about space on the disks.

[emkaxon]

Disks were full, we have deleted the oldest indices and now we have enough space. However, after restarting the containers we are still getting this error

[Bal33p]

what was the resolution for this issue