Kibana 404 Error #9850

its-a-me-mario-the-og · 2023-02-24T15:30:30Z

its-a-me-mario-the-og
Feb 24, 2023

Hi, I am having trouble accessing Kibana to look at data from SO. I know the alerts tab is useful but I use Kibana for more in-depth analysis.

SO Version: 2.3.150
Deployment: Standalone
RAM: 64 gbs
Storage: 5 tbs
CPU: i5-8400

When I do the command "sudo docker logs so-kibana" this is the output I get from the command.

[2023-02-22T22:03:44.505+00:00][INFO ][plugins-service] Plugin "telemetryManagementSection" has been disabled since the following direct or transitive dependencies are missing, disabled, or have incompatible types: [telemetry]
[2023-02-22T22:03:44.511+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
[2023-02-22T22:03:44.580+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2023-02-22T22:03:44.620+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-02-22T22:03:44.647+00:00][WARN ][config.deprecation] You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning.
[2023-02-22T22:03:44.647+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-02-22T22:03:44.647+00:00][WARN ][config.deprecation] Setting "security.showInsecureClusterWarning" has been replaced by "xpack.security.showInsecureClusterWarning"
[2023-02-22T22:03:44.815+00:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,licenseManagement,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
[2023-02-22T22:03:44.827+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 4e18661b-73c4-4bd3-93dd-f0171b08684d
[2023-02-22T22:03:44.882+00:00][WARN ][plugins.security.config] Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.
[2023-02-22T22:03:44.899+00:00][WARN ][plugins.security.config] Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.
[2023-02-22T22:03:44.981+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-02-22T22:03:45.001+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-02-22T22:03:45.450+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-02-22T22:03:45.777+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-02-22T22:03:45.777+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-02-22T22:03:45.799+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 10ms.
[2023-02-22T22:03:45.803+00:00][INFO ][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 17ms.
[2023-02-22T22:03:45.805+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 6ms.
[2023-02-22T22:03:45.808+00:00][INFO ][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 5ms.
[2023-02-22T22:03:45.811+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 6ms.
[2023-02-22T22:03:45.815+00:00][INFO ][savedobjects-service] [.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 6ms.
[2023-02-22T22:03:45.817+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 6ms.
[2023-02-22T22:03:45.819+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 5ms.
[2023-02-22T22:03:45.825+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 8ms.
[2023-02-22T22:03:45.829+00:00][INFO ][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 10ms.
[2023-02-22T22:03:45.834+00:00][INFO ][savedobjects-service] [.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 5ms.
[2023-02-22T22:03:45.836+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 2ms.
[2023-02-22T22:03:45.904+00:00][INFO ][savedobjects-service] [.kibana] Starting to process 1079 documents.
[2023-02-22T22:03:45.905+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 68ms.
[2023-02-22T22:03:46.189+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-02-22T22:03:46.336+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 432ms.
[2023-02-22T22:03:46.395+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 570ms.
[2023-02-22T22:03:46.395+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 606ms
[2023-02-22T22:03:46.415+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 79ms.
[2023-02-22T22:03:46.417+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 2ms.
[2023-02-22T22:03:46.419+00:00][INFO ][savedobjects-service] [.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 2ms.
[2023-02-22T22:03:46.425+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 6ms.
[2023-02-22T22:03:46.428+00:00][INFO ][savedobjects-service] [.kibana] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 3ms.
[2023-02-22T22:03:46.430+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 2ms.
[2023-02-22T22:03:46.435+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 5ms.
[2023-02-22T22:03:46.437+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 2ms.
[2023-02-22T22:03:46.466+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 29ms.
[2023-02-22T22:03:48.373+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 1907ms.
[2023-02-22T22:03:48.400+00:00][ERROR][savedobjects-service] [.kibana] Unexpected Elasticsearch ResponseError: statusCode: 500, method: POST, url: /_aliases error: [illegal_state_exception]: alias [.kibana] has is_hidden set to true on indices [.kibana_8.3.3_001] but does not have is_hidden set to true on indices [.kibana_7.13.4_001]; alias must have the same is_hidden setting on all indices,
[2023-02-22T22:03:48.401+00:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 500, method: POST, url: /_aliases error: [illegal_state_exception]: alias [.kibana] has is_hidden set to true on indices [.kibana_8.3.3_001] but does not have is_hidden set to true on indices [.kibana_7.13.4_001]; alias must have the same is_hidden setting on all indices,
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:164:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:214:9)
    at Server.start (/usr/share/kibana/src/core/server/server.js:363:31)
    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2023-02-22T22:03:48.408+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2023-02-22T22:03:48.409+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2023-02-22T22:03:48.410+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped

 FATAL  Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 500, method: POST, url: /_aliases error: [illegal_state_exception]: alias [.kibana] has is_hidden set to true on indices [.kibana_8.3.3_001] but does not have is_hidden set to true on indices [.kibana_7.13.4_001]; alias must have the same is_hidden setting on all indices,

[2023-02-23T21:23:27.963+00:00][INFO ][plugins-service] Plugin "telemetry" is disabled.
[2023-02-23T21:23:27.988+00:00][INFO ][plugins-service] Plugin "telemetryManagementSection" has been disabled since the following direct or transitive dependencies are missing, disabled, or have incompatible types: [telemetry]
[2023-02-23T21:23:27.997+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
[2023-02-23T21:23:28.065+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2023-02-23T21:23:28.095+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-02-23T21:23:28.121+00:00][WARN ][config.deprecation] You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning.
[2023-02-23T21:23:28.121+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-02-23T21:23:28.121+00:00][WARN ][config.deprecation] Setting "security.showInsecureClusterWarning" has been replaced by "xpack.security.showInsecureClusterWarning"
[2023-02-23T21:23:28.285+00:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,licenseManagement,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
[2023-02-23T21:23:28.297+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 4e18661b-73c4-4bd3-93dd-f0171b08684d
[2023-02-23T21:23:28.350+00:00][WARN ][plugins.security.config] Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.
[2023-02-23T21:23:28.367+00:00][WARN ][plugins.security.config] Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.
[2023-02-23T21:23:28.451+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-02-23T21:23:28.468+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-02-23T21:23:28.926+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-02-23T21:23:29.400+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-02-23T21:23:29.401+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-02-23T21:23:29.436+00:00][INFO ][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 22ms.
[2023-02-23T21:23:29.438+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 21ms.
[2023-02-23T21:23:29.440+00:00][INFO ][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 4ms.
[2023-02-23T21:23:29.491+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 53ms.
[2023-02-23T21:23:29.531+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 40ms.
[2023-02-23T21:23:29.539+00:00][INFO ][savedobjects-service] [.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 99ms.
[2023-02-23T21:23:29.545+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 14ms.
[2023-02-23T21:23:29.622+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-02-23T21:23:34.296+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 4757ms.
[2023-02-23T21:23:34.597+00:00][INFO ][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 301ms.
[2023-02-23T21:23:34.604+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 5059ms.
[2023-02-23T21:23:35.789+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 1185ms.
[2023-02-23T21:23:35.790+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 6373ms
[2023-02-23T21:23:41.406+00:00][INFO ][savedobjects-service] [.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 6809ms.
[2023-02-23T21:23:41.409+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 3ms.
[2023-02-23T21:23:42.223+00:00][INFO ][savedobjects-service] [.kibana] Starting to process 1079 documents.
[2023-02-23T21:23:42.223+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 814ms.
[2023-02-23T21:23:42.564+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 341ms.
[2023-02-23T21:23:42.761+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 197ms.
[2023-02-23T21:23:42.765+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 4ms.
[2023-02-23T21:23:47.226+00:00][INFO ][savedobjects-service] [.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 4461ms.
[2023-02-23T21:23:55.385+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 8159ms.
[2023-02-23T21:23:55.389+00:00][INFO ][savedobjects-service] [.kibana] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 4ms.
[2023-02-23T21:23:55.392+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3ms.
[2023-02-23T21:23:55.521+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 129ms.
[2023-02-23T21:23:55.530+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 9ms.
[2023-02-23T21:24:06.470+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 10940ms.
[2023-02-23T21:24:08.681+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 2211ms.
[2023-02-23T21:24:11.951+00:00][ERROR][savedobjects-service] [.kibana] Unexpected Elasticsearch ResponseError: statusCode: 500, method: POST, url: /_aliases error: [illegal_state_exception]: alias [.kibana] has is_hidden set to true on indices [.kibana_8.3.3_001] but does not have is_hidden set to true on indices [.kibana_7.13.4_001]; alias must have the same is_hidden setting on all indices,
[2023-02-23T21:24:11.952+00:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 500, method: POST, url: /_aliases error: [illegal_state_exception]: alias [.kibana] has is_hidden set to true on indices [.kibana_8.3.3_001] but does not have is_hidden set to true on indices [.kibana_7.13.4_001]; alias must have the same is_hidden setting on all indices,
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:164:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:214:9)
    at Server.start (/usr/share/kibana/src/core/server/server.js:363:31)
    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2023-02-23T21:24:11.966+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2023-02-23T21:24:11.967+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2023-02-23T21:24:11.968+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped

 FATAL  Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 500, method: POST, url: /_aliases error: [illegal_state_exception]: alias [.kibana] has is_hidden set to true on indices [.kibana_8.3.3_001] but does not have is_hidden set to true on indices [.kibana_7.13.4_001]; alias must have the same is_hidden setting on all indices,

I thought deleting indices will help but so far nothing has changed. I have went through the documentation and tried the commands they have for Kibana dashboards and everything but nothing as well. I have restarted the service and looked into Elasticsearch because I thought it might be a template problem but nothing there as well. Any help or any point of direction will be greatly appreciated and thank you in advance!

Answered by InfosecGoon

Feb 27, 2023

It sounds like the .kibana index is what's hanging up -- try deleting it with this command:

sudo so-elasticsearch-query .kibana -XDELETE

View full answer

InfosecGoon · 2023-02-27T15:59:38Z

InfosecGoon
Feb 27, 2023

It sounds like the .kibana index is what's hanging up -- try deleting it with this command:

sudo so-elasticsearch-query .kibana -XDELETE

5 replies

its-a-me-mario-the-og Feb 27, 2023
Author

I just tried that command but the output below is what I got instead.

sudo so-elasticsearch-query .kibana -XDELETE
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with roles [superuser] on restricted indices [.kibana,.kibana_7.17.4_001,.kibana_7.13.4_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with roles [superuser] on restricted indices [.kibana,.kibana_7.17.4_001,.kibana_7.13.4_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

I did try it as root as well and the same output came back, is there a way I can make myself authorized to do this?

InfosecGoon Feb 27, 2023

From: #9744

Okay, let's try it with the so_kibana account instead. My mistake.

Get the so_kibana password from the Elasticsearch auth pillar (salt-call pillar.get elasticsearch:auth), and try running this:

curl --user 'so_kibana:PASSWORD' -s -k -L -H "Content-Type: application/json" "https://localhost:9200/.kibana -XDELETE

Then, if it deletes properly, restart Kibana with so-kibana-restart and try to log in again.

its-a-me-mario-the-og Feb 27, 2023
Author

I got the kibana password and put it in but I got this output.

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [.kibana] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [.kibana] matches an alias, specify the corresponding concrete indices instead."},"status":400}

I tried the command a few different ways and put " at the end of /.kibana but so far nothing has worked still

InfosecGoon Feb 28, 2023

Try to find the actual index that is aliased to .kibana:

sudo so-elasticsearch-query _alias | jq | less

Once you find the actual index name, try deleting that instead.

its-a-me-mario-the-og Mar 6, 2023
Author

Great! I found the answer and have access to Kibana through the web interface. Thank you @InfosecGoon for your help!

llucasti · 2023-06-15T10:41:30Z

llucasti
Jun 15, 2023

I have a similar problem but running "sudo so-elasticsearch-query _alias | jq | less" I get multiple enties.

Do I delete all of below? Or just the ones "with task_manager"?

`.kibana_1

.kibana_8.7.1_reindex_temp

".kibana_task_manager_7.17.4_001": {
"aliases": {
".kibana_task_manager": {},
".kibana_task_manager_7.17.4": {}
}
},

"so-kibana-2023.06.14": {
"aliases": {}

".kibana-event-log-7.17.4-000001": {
"aliases": {
".kibana-event-log-7.17.4": {
"is_write_index": true,
"is_hidden": true
}
}
},

".kibana_security_session_1": {
"aliases": {}
},

".kibana_task_manager_8.7.1_reindex_temp": {
"aliases": {}
},

".kibana_7.17.4_001": {
"aliases": {
".kibana": {},
".kibana_7.17.4": {}
}
},

`

1 reply

dougburks Jun 22, 2023
Maintainer

@llucasti This discussion is several months old and has been marked as answered. From #1720:

Start a new discussion instead of replying to somebody else's discussion
Please search to see if you can find similar discussions that may help you. However, in order to avoid confusion, please do NOT reply to somebody else's discussion with your own issue. Instead, please start a new discussion and in that new discussion you can provide a hyperlink to the related discussion.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Kibana 404 Error #9850

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Kibana 404 Error #9850

Uh oh!

its-a-me-mario-the-og Feb 24, 2023

Replies: 2 comments · 6 replies

Uh oh!

InfosecGoon Feb 27, 2023

Uh oh!

its-a-me-mario-the-og Feb 27, 2023 Author

Uh oh!

InfosecGoon Feb 27, 2023

Uh oh!

its-a-me-mario-the-og Feb 27, 2023 Author

Uh oh!

InfosecGoon Feb 28, 2023

Uh oh!

its-a-me-mario-the-og Mar 6, 2023 Author

Uh oh!

llucasti Jun 15, 2023

Uh oh!

dougburks Jun 22, 2023 Maintainer

its-a-me-mario-the-og
Feb 24, 2023

Replies: 2 comments 6 replies

InfosecGoon
Feb 27, 2023

its-a-me-mario-the-og Feb 27, 2023
Author

its-a-me-mario-the-og Feb 27, 2023
Author

its-a-me-mario-the-og Mar 6, 2023
Author

llucasti
Jun 15, 2023

dougburks Jun 22, 2023
Maintainer