[Fleet] manage_ccr privilege to kibana_system

[juliaElastic]

Relates elastic/kibana#221277

Fleet sync integrations feature might fail with retention leases error. To fix this automatically, Fleet needs manage_ccr privilege to recreate the follower index.

• Have you signed the contributor license agreement? yes
• Have you followed the contributor guidelines? yes
• If submitting code, have you built your formula locally prior to submission with gradle check? yes
• If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed. yes
• If submitting code, have you checked that your submission is for an OS and architecture that we support? yes
• If you are submitting this code for a class then read our policy for that.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[legrego]

Reading through elastic/kibana#221277, it seems like the conditions necessary to trigger this scenario are quite rare.

I am all for improving the UX in this scenario, but granting manage_ccr to the Kibana system user for the sole purpose of autocorrecting a rare condition does not feel like the right tradeoff.

What alternatives have we explored?

[juliaElastic]

Reading through elastic/kibana#221277, it seems like the conditions necessary to trigger this scenario are quite rare.

I am all for improving the UX in this scenario, but granting manage_ccr to the Kibana system user for the sole purpose of autocorrecting a rare condition does not feel like the right tradeoff.

What alternatives have we explored?

We have a knowledge article to advise users to fix this scenario manually: https://support.elastic.dev/knowledge/view/6efaa9e7

I'm not aware of a way to prevent this error altogether.

I understand that it looks like an overkill to grant manage_ccr just to fix a rare error scenario. We can probably park this change for now and revisit later if we get a lot of SDHs. cc @kpollich

[legrego]

@juliaElastic @kpollich Could we instead present a screen to an administrator, asking them to click a button to repair this? With that approach, we can rely on the administrator's privileges, rather than augmenting our system privileges.

[kpollich]

I understand that it looks like an overkill to grant manage_ccr just to fix a rare error scenario. We can probably park this change for now and revisit later if we get a lot of SDHs. cc @kpollich

+1 from me.

The KB article is great, but we should document this publicly in a troubleshooting doc as well. I think this is rare enough that a manual workaround is acceptable.

@juliaElastic @kpollich Could we instead present a screen to an administrator, asking them to click a button to repair this? With that approach, we can rely on the administrator's privileges, rather than augmenting our system privileges.

This is a reasonable suggestion, but I think even putting a screen like this together is probably more effort that this rare edge case error requires. I'd rather start with public docs and revisit this with something more robust if we see support noise.

[juliaElastic]

Okay, thanks for the feedback. I'll close this pr for now.