Multi-stage cluster addons: Upgrade from faulty cluster stack to one with the same Kubernetes version does not work #212
Description
What steps did you take and what happened:
Create a cluster stack, e.g. with a wrong CEL expression, so that it will never apply the clusteraddons successfully. Then upgrade to another cluster stack that has no issue. In this case, the AfterControlPlaneInitialized hook will stay in the clusteraddon object and the controller will continue to apply based on AfterControlPlaneInitialized, not BeforeClusterUpgrade.
What did you expect to happen:
The controller should realize that the cluster got upgraded and that it has to remove the existing status and also the spec.hook property.
Currrently, this is not possible, as we cannot know the previous cluster stack if it didn't complete successfully. However, we can solve this by writing the cluster stack that should be upgraded to into the annotations. From there we can read it and if it is set and there is a mismatch between the annotation and the current cluster stack we upgrade to (cluster.spec.topology.class), we know that something went wrong previously and that we should remove the status as well as the spec.hook of clusteraddon.