upgrade so from 2.3.140 to 2.3.150 and kibana 404 page not found

[golfreeze]

Hi team ,

Thank you for your hard working on 2.3.150.

After My team finished upgraded so from 2.3.140 to 2.3.150 and kibana 404 page not found
and so-status
all services look good and could curl

curl -uabc@abc -k https://localhost:9200/_cat/indices | grep 2022.08

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16038 100 16038 0 0 green open so-ossec-2022.08.29 gL_KnoJITkqQuQJp4uUZvg 1 0 1209 0 1.1mb 1.1mb
340green open so-ossec-2022.08.28 QZ8EKnQoQXuiZlTyztOi6Q 1 0 2394 0 1.9mb 1.9mb
k green open so-ossec-2022.08.27 1vbSEdpJQkyaOmWJ_jPcew 1 0 2394 0 2mb 2mb

###see log from "docker logs so-kibana" found as below
====== Start of so-kibana ======
[2022-08-29T11:40:01.834+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64004ms.
[2022-08-29T11:40:01.835+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 15 in 64 seconds.
[2022-08-29T11:40:01.835+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 64004ms.
[2022-08-29T11:41:05.837+00:00][INFO ][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> FATAL. took: 64003ms.
[2022-08-29T11:41:05.838+00:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the SET_SOURCE_WRITE_BLOCK step after 15 attempts, terminating. The last failure message was: security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]
at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:144:29)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 0)
at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:214:9)
at Server.start (/usr/share/kibana/src/core/server/server.js:363:31)
at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2022-08-29T11:41:05.847+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-08-29T11:41:05.847+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2022-08-29T11:41:05.848+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
[2022-08-29T11:41:05.859+00:00][INFO ][savedobjects-service] [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> FATAL. took: 64024ms.

FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the SET_SOURCE_WRITE_BLOCK step after 15 attempts, terminating. The last failure message was: security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [so_kibana] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]
====== End of so-kibana ======

Please kindly help and guide to access kibana again.
Thank you very much.

[WPTechnician]

Just to add to this, not sure if the OP had the same issue but during the upgrade it threw this:

Total states run: 10
Total run time: 2.534 s
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (1/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (2/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (3/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (4/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (5/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (6/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (7/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (8/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (9/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (10/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (11/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (12/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (13/300)
Server is not ready
Waiting for value 'default' at 'http://localhost:5601/api/spaces/space/default' (14/300)
Server is not ready

We are having the same error messages and log entries as the OP after the update. The only difference is the user in their logs is "so-kibana" and ours is "anonymous user"

Thanks!

[dougburks]

When upgrading from Elastic 7 to Elastic 8, did you follow all steps at the link below including enabling Elastic Auth if necessary?
https://docs.securityonion.net/en/2.3/soup.html#elastic-8

[WPTechnician]

So I was also at 2.3.140. I can't seem to pull the new version to see if the fix for that user will work for me. The version being reported as installed is 2.3.150 but running SOUP reports that I am already at the latest version.

[golfreeze]

@WPTechnician Could you try like this "so-elastic-auth true" then "so-kibana-restart" and try to see kibana again .

[WPTechnician]

So I have moved to 2.3.160 and I am still receiving the same error:

Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [anonymous_user] with roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'.

Can I delete that index maybe? Or can I give the anonymous user the required permission to get around that error?

Thanks!

[golfreeze]

@WPTechnician could you try
so-elastic-auth true
then
so-kibana-restart

[WPTechnician]

Ok, so when I ran that at 2.3.150 there was no change. Upgrading to 2.3.160 and running those commands again has resolve the issue for me. Thanks!

[WPTechnician]

Thanks for the suggestion golfreeze. I had thought to try that and unfortunately I am still seeing this in the log:

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-08-30T14:44:55.897+00:00","message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [anonymous_user] with roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 1 in 2 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":7},"span":{"id":"4ae0a087fe76c7cf"},"trace":{"id":"4b05fee74cb61e69dc7b757ae70c00f5"}}