Kibana 404 after updating to 2.3.210

[PlaceYourBets]

Hello guys,

after updating SO to 2.3.210 Kibana won't come up and shows error 404.

Upgrade path was: 2.3.100 -> 2.3.130 -> 2.3.210
I also followed every step mentioned in the docs for soup.
Distributed deployment: 3 nodes (manager, search & forward)

so-status is fine on all nodes

docker logs so-kibana shows:

[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [anonymous_user] with effective roles [superuser] on restricted indices [.kibana_task_manager_7.17.4_001], this action is granted by the index privileges [manage,all]'. Retrying attempt 2 in 4 seconds. [.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> SET_SOURCE_WRITE_BLOCK. took: 2008ms.

FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the SET_SOURCE_WRITE_BLOCK step after 15 attempts, terminating. The last failure message was: security_exception: [security_exception] Reason: action [indices:admin/block/add] is unauthorized for user [anonymous_user] with effective roles [superuser] on restricted indices [.kibana_7.17.4_001], this action is granted by the index privileges [manage,all]

In the past a few people seemed to have the same problem and I already tried (without success):

• rebooting all nodes
• FORCE_SYNC=true so-user sync; so-kibana-restart; so-kibana-space-defaults

Before I did FORCE_SYNC=true so-user sync the unauthorized user was 'so_kibana', now it's 'anonymous_user'.

As far as I understood, it's only a permission issue, but I'm not sure how to solve it. Do you have any ideas?
(https://github.com/elastic/kibana/issues/119342)

[InfosecGoon]

Do you have Elastic Auth enabled? What's the output from this command:

sudo salt-call pillar.get elasticsearch:auth

[PlaceYourBets]

Before the update I checked it and it was 100% enabled. Now it seems to be disabled:

local:
    ----------
    enabled:
        False
    users:
        ----------
        so_beats_user:
            ----------
            pass:
                6qbEJk+OozQ3wbv9RWApSpr;!+2fx3W]B*+cs9,>uk4-qZthlL+L*Mrq*MAJ2#VQ=Y1f;Dm^
            user:
                so_beats
        so_elastic_user:
            ----------
            pass:
                Hd,_!iEXH(N]<?Vrlbx$(&4VH.2f~!d7d18qKB.1bhu4?b694*UBRlW@x&4@:K)d3+MtX3T(
            user:
                so_elastic
        so_kibana_user:
            ----------
            pass:
                RH)hf|SErqzu((3pG;V1jZ8GNAy<w-)8E+vh?&JXju$gIA;FVB~1p!$d,dlpuHe#O@dkZ.)l
            user:
                so_kibana
        so_logstash_user:
            ----------
            pass:
                0;3l_)(ORpaVMUa?2P@<4S_-Y)YgzDd2bgLOU7:_?2=Y@uDofzpQoT3d4n@J[6Qw::Gn<Y&W
            user:
                so_logstash
        so_monitor_user:
            ----------
            pass:
                oqK30ty93i>._V@@+OnBXrQ8L[z3m~!eP-i]v~u#0HAF)xt3ZQ20i^uzjcez(^v+qiPwT_NP
            user:
                so_monitor

I re-enabled it yesterday, but when I came back today it was disabled again... :/

[PlaceYourBets]

Alright, after re-re-enabling so-elastic-auth it is working now. Weird, but I'm happy :)
I saw some salt highstate errors, maybe I ran into some sync issues?! However, case is closed for me.
Thank you!!!