Cannot access kibana after 2.3.140 update

[Jacob-Tate]

This is in our test environment but that being said after an update to 2.3.140 it gets a 404 error when attempting to access kibana.

I originally updated using the normal soup method but then saw another post where someone said to try the following:

sudo yum history packages-list salt-master
sudo yum history undo
sudo BRANCH=hotfix/2.3.140 soup

This did not make a difference to the kibana instance.

The following logs are generated:

[REDACTED@REDACTED ~]$ sudo docker logs so-kibana
[2022-07-19T18:40:23.727+00:00][INFO ][plugins-service] Plugin "telemetry" is disabled.
[2022-07-19T18:40:23.811+00:00][INFO ][plugins-service] Plugin "telemetryManagementSection" has been disabled since the following direct or transitive dependencies are missing, disabled, or have incompatible types: [telemetry]
[2022-07-19T18:40:23.837+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
[2022-07-19T18:40:24.041+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2022-07-19T18:40:24.163+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2022-07-19T18:40:24.298+00:00][WARN ][config.deprecation] You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning.
[2022-07-19T18:40:24.300+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2022-07-19T18:40:24.301+00:00][WARN ][config.deprecation] Setting "security.showInsecureClusterWarning" has been replaced by "xpack.security.showInsecureClusterWarning"
[2022-07-19T18:40:25.016+00:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,licenseManagement,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
[2022-07-19T18:40:25.077+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 3bdbd7f5-558a-4343-b871-f83c1b61e724
[2022-07-19T18:40:25.349+00:00][WARN ][plugins.security.config] Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.
[2022-07-19T18:40:25.457+00:00][WARN ][plugins.security.config] Using secure cookies, but SSL is not enabled inside Kibana. SSL must be configured outside of Kibana to function properly.
[2022-07-19T18:40:25.810+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2022-07-19T18:40:25.886+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-07-19T18:40:27.910+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2022-07-19T18:40:28.892+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2022-07-19T18:40:28.893+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2022-07-19T18:40:29.082+00:00][INFO ][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 58ms.
[2022-07-19T18:40:29.106+00:00][INFO ][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 25ms.
[2022-07-19T18:40:29.118+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 83ms.
[2022-07-19T18:40:29.135+00:00][INFO ][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 17ms.
[2022-07-19T18:40:29.160+00:00][INFO ][savedobjects-service] [.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 54ms.
[2022-07-19T18:40:29.166+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 31ms.
[2022-07-19T18:40:29.185+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 1 in 2 seconds.
[2022-07-19T18:40:29.186+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 19ms.
[2022-07-19T18:40:29.224+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 1 in 2 seconds.
[2022-07-19T18:40:29.225+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64ms.
[2022-07-19T18:40:31.203+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 2 in 4 seconds.
[2022-07-19T18:40:31.204+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 2018ms.
[2022-07-19T18:40:31.213+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2022-07-19T18:40:31.232+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 2 in 4 seconds.
[2022-07-19T18:40:31.233+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 2008ms.
[2022-07-19T18:40:35.210+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 3 in 8 seconds.
[2022-07-19T18:40:35.211+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 4007ms.
[2022-07-19T18:40:35.238+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 3 in 8 seconds.
[2022-07-19T18:40:35.238+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 4006ms.
[2022-07-19T18:40:43.217+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 4 in 16 seconds.
[2022-07-19T18:40:43.217+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 8007ms.
[2022-07-19T18:40:43.246+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 4 in 16 seconds.
[2022-07-19T18:40:43.246+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 8008ms.
[2022-07-19T18:40:59.223+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 5 in 32 seconds.
[2022-07-19T18:40:59.224+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 16006ms.
[2022-07-19T18:40:59.251+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 5 in 32 seconds.
[2022-07-19T18:40:59.252+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 16005ms.
[2022-07-19T18:41:31.231+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 6 in 64 seconds.
[2022-07-19T18:41:31.232+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 32008ms.
[2022-07-19T18:41:31.262+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 6 in 64 seconds.
[2022-07-19T18:41:31.262+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 32011ms.
[2022-07-19T18:42:35.242+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 7 in 64 seconds.
[2022-07-19T18:42:35.243+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64011ms.
[2022-07-19T18:42:35.268+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 7 in 64 seconds.
[2022-07-19T18:42:35.269+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64006ms.
[2022-07-19T18:43:39.256+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 8 in 64 seconds.
[2022-07-19T18:43:39.256+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64014ms.
[2022-07-19T18:43:39.273+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 8 in 64 seconds.
[2022-07-19T18:43:39.274+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64005ms.

[weslambert]

Do you have Elastic Auth enabled? If not, try running so-elastic-auth.

If that doesn't help, try the following (as root):

FORCE_SYNC=true so-user sync
so-kibana-restart
so-kibana-space-defaults

For more information about the update process for this version, see:
https://docs.securityonion.net/en/2.3/soup.html#elastic-8

[Jacob-Tate]

This did the trick

FORCE_SYNC=true so-user sync
so-kibana-restart

Thanks!