[ML] After upgrade from 7.17 to 9.0, anomaly detection jobs annotations can't be updated.

[wwang500]

Step to reproduce:

1. Deploy a 7.17.27 cluster, just choose kibana, es and ml nodes for avoiding other known issues
2. Create and run ml anomaly detections jobs
3. Upgrade it to 8.18, (the upgrade path I did, is 7.17->8.16.3->8.18.0)
4. Fix all critical deprecations issues being reported by Upgrade Assistant
5. Upgrade to 9.0
6. After upgrade, try create and update pre-upgrade annotations

Observed:

Error:

[security_exception
	Root causes:
		security_exception: action [indices:data/write/bulk[s]] is unauthorized for user [found-internal-kibana4-server] with effective roles [found-internal-kibana4-server,kibana_system] on indices [.reindexed-v8-ml-annotations-000001], this action is granted by the index privileges [create_doc,create,delete,index,write,all]]: action [indices:data/write/bulk[s]] is unauthorized for user [found-internal-kibana4-server] with effective roles [found-internal-kibana4-server,kibana_system] on indices [.reindexed-v8-ml-annotations-000001], this action is granted by the index privileges [create_doc,create,delete,index,write,all]

Also, when clearing job notification, I was seeing the similar error:

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[davidkyle]

The problem is that the built in Kibana System user does not have write access to .reindexed-v8-ml-annotations-000001. It has write access to .ml-annotations-000001 but not the reindexed version created by the upgrade assistant.

The annotations are read through a read alias that points to all *ml-annotations-* indices. If the annotation is updated it is written back to the index the document came from.

elastic/elasticsearch#121897 adds read & write permissions for the .reindexed indices created by the UA