System data streams issue after upgrading to 9.0 with feature migration API

[tomsonpl]

In 9.0.0-rc1 we identified an issue where system data stream .fleet-actions-results is still problematic after being upgraded from 7.17.28 > 8.18 > 9.0.0-rc1.

Slack thread: https://elastic.slack.com/archives/C8UUBNASY/p1742824896518599

How to reproduce:

1. On 7.17.28 instance run osquery queries or endpoint isolate/release action. That saved actions to .fleet-actions-results
2. Upgrade to 8.18.0
3. At this point there is no issue
4. Resolve all ES critical issues - there was over 40 issues that I had to reindex - however none of them was about fleet-actions-results
5. Upgrade to 9.0.0
6. Issue exists.

This issue pops out in multiple places, eg. fleet, index management etc. But also breaks other functionalities (eg. those using fleet/agents).

Originally posted by @tomsonpl in #122949

Hi @alexey-ivanov-es, unfortunately another issue occurred with the above mentioned data stream. It was found on 9.0.0-rc1.

Is this still in your ownership, or fleet now?

{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: Data stream(s) [.fleet-actions-results] may not be accessed by product [kibana]"
}

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[tomsonpl]

Previously, we encountered a problem where we couldn't upgrade to version 9.0 using the upgrade assistant due to a blocking index .fleet-actions-results. This prevented us from testing the 9.0 behavior.

The current issue might not be limited just to this particular index, but rather generic edge case for system data streams - best to consult @alexey-ivanov-es to understand the general impact. However, in our specific case, using the .fleet-actions-results index (with Osquery and Elastic Defend in Security) caused the following issues after the 9.0 upgrade:

• Crashed index management page
• Issues on the fleet agents page (no agents)
• Not working agent related functionalities (e.g., Elastic Defend response actions, Osquery)

If a different system data stream had issues, other functionalities could be affected as well.
cc: @arpadkiraly

[alexey-ivanov-es]

The problem was:

1. The system indices migration task uses the data stream reindexing task to migrate the fleet-actions-results data stream created by Elasticsearch v7.
2. To track the status of the data stream reindexing task, its state is stored in the cluster state for 24 hours, effectively keeping the task "in progress" during that time.
3. When a new node (master?) joins the cluster, any unfinished persistent tasks are restarted. This doesn't cause issues on v8 nodes, as the task checks the index versions of the data stream's backing indices and sees that no reindexing is needed. However, when a v9 node joins, it sees older index versions and re-triggers reindexing.
4. This results in double migration of the system indices backing the data stream, leaving them with .migrated-migrated- prefix, which breaks permission checks.
5. Double reindexing could also affect non-system data streams. While it would only have performance implications for those, it shouldn't cause any issues beyond that.

Restarting of reindexing task was fixed by: #125587
Permission checks for system data streams was fixed by: #125598

[tomsonpl]

We upgraded the EDR instance with the problematic data stream from 7x > 8x > 9.0 GA today, and can confirm that this has been resolved 👍

Thanks!