[Issue]: Breaking Wrong Info on restoring system indices

[stefnestor]

Type of issue

Inaccurate

What documentation page is affected

https://www.elastic.co/docs/manage-data/migrate/migrate-internal-indices

What happened?

You're not allowed to restore system indices via index +8.0 you MUST use feature states. This (a) literally directs you wrong path & (b) uses renames pattern which will prevent those from working in destination cluster.

Additional info

• Example (fleet) how system indices are suppose to be restored: .fleet-actions-results data stream cannot be restored via the fleet feature state elasticsearch#89261
• Some feature states should be restored in conjunction with others. For example kibana and security can be restored by themselves but fleet has kibana dependencies and those should be restored together.

[georgewallace]

@stefnestor Thanks again for this other piece of feedback, we will take a look at this

[eedugon]

@stefnestor , thanks a lot for sharing this.

Quick question before preparing a fix:

Some feature states should be restored in conjunction with others. For example kibana and security can be restored by themselves but fleet has kibana dependencies and those should be restored together.

Is there any doc where that is explained? I don't see any explanation in the doc where feature states are introduced, and neither in the restore feature state doc.

If you share more details about that we can also improve those docs highlighting that fact :)

cc: @shainaraskas

[stefnestor]

Cross-posting from internal : https://www.elastic.co/guide/en/elasticsearch/reference/8.17/snapshots-restore-snapshot.html#restore-feature-state

[cp-elastic]

@stefnestor , thanks a lot for sharing this.

Quick question before preparing a fix:

Some feature states should be restored in conjunction with others. For example kibana and security can be restored by themselves but fleet has kibana dependencies and those should be restored together.

Is there any doc where that is explained? I don't see any explanation in the doc where feature states are introduced, and neither in the restore feature state doc.

If you share more details about that we can also improve those docs highlighting that fact :)

cc: @shainaraskas

I'll second this. I just ran into an issue with a customer for this same thing, Fleet and Kibana feature states have to be restored together, doing "All feature states" didn't work by itself. I would never have known had it not been for this comment.

[eedugon]

@shainaraskas : who could we contact from devs to get extra insights about restoring feature states so we can improve our document per this last comment? #1391 (comment)

[stefnestor]

@shahinakmal will you kindly CC me when you reach out to Dev?

FWIW I was wrong/incomplete you'd also need to consider security:

• From @lucabelluccini 's article (sub-note [REQUEST]: Document we do not test snapshot/restores of Fleet ingest-docs#388)

  A very high level overview:

  • .fleet* for policies, packages, settings, etc...
  • .kibana* for the dashboards and assets installed by the integrations
  • .security* for the API Keys generated by Fleet, deployed to Elastic Agents to allow them to write to Elasticsearch and communicate with Fleet Server
  • The cluster state, as some assets of integrations are Index Templates, Ingest pipelines, etc...

• I can't find my claim source that fleet+kibana may need restored concurrently but it's because at least in ≤v8.8.0 kibana saved objects included ( fleet-fleet-server-host,fleet-message-signing-keys,fleet-preconfiguration-deletion-record,fleet-proxy ). IIUC if you don't restore together you may induce cannot read existing message signing key pair as common support volume (along with other fleet errors)

Also noting from Slack DM, they'd originally tried to use the Kibana UI which had+has flow issues ( elastic/kibana#157662 , elastic/kibana#137368 , elastic/elasticsearch#86606 ) which is why we usually write our docs to the DevTools commands instead.