enhance the MWRS rollout gaps

[haoqing0110]

Describe the enhancement
A clear and concise description of what the enhancement request is.

To fix the issue #1201, we introduced Progressing and Degraded(optional) condition in ManifestWork for the rollout case by PR #1207.

Some gaps need to be enhanced:

1. Reduce the requeue time when minSuccessTime is set for rollout. (working on)
2. During rollout, the Degraded condition has a chance to be true; this will cause the cluster rollout status to pass the progressDeadline and rollout to the next cluster. This is unexpected behavior.
3. During rollout, the Progressing condition keeps false with observedGeneration updated, but with old LastTransaction, this will cause the cluster rollout status to pass the minSuccessTime and rollout to the next cluster. This is unexpected behavior.

[youngbupark]

@haoqing0110 @qiujian16 @annelaucg

2. During rollout, the Degraded condition has a chance to be true; this will cause the cluster rollout status to pass the progressDeadline and rollout to the next cluster. This is unexpected behavior.

I wonder if we need to set Applied, Available, Degraded, Progressing conditions to False in order to reset LastTransitionTime. The current implementation doesn't change the Applied and Available during upgrade except for observedGeneration. Shouldn't Applied and Available represent the state of the updated manifest, not the old manifest?

3. During rollout, the Progressing condition keeps false with observedGeneration updated, but with old LastTransaction, this will cause the cluster rollout status to pass the minSuccessTime and rollout to the next cluster. This is unexpected behavior.

Since meta.SetStatusCondition update LastTransitionTime only when Status is changing. If we can reset the status to False when new MW is applied, we can change the time to the latest.

In the new change of MWRS, clusterRolloutStatusFunc doesn't check Applied and Available condition anymore. From my understanding these two conditions shows the final result when all individual resources are applied. my hypothesis was that Progressing condition might be updated before all resources were applied and available. It means Progressing may evaluate the staled resource statuses. To avoid the race condition, we would need to ensure:

1. Progress needs to be reset and should be evaluated after all resource are being available.
2. clusterRolloutStatusFunc ensures the observedGenerations of Applied/Avaialble/Progressing/Degraded are same as MW's Generation. Otherwise we should not evaluate the rolloutstatus.

[qiujian16]

Applied and Available has the lastObeservedGeneration which is to track which generation of the spec the condition reflects.

[qiujian16]

my hypothesis was that Progressing condition might be updated before all resources were applied and available. It means Progressing may evaluate the staled resource statuses. To avoid the race condition, we would need to ensure:

we probably should whenever the lastObservedGeneration of Applied has changed.

[haoqing0110]

2. During rollout, the Degraded condition has a chance to be true; this will cause the cluster rollout status to pass the progressDeadline and rollout to the next cluster. This is unexpected behavior.

I wonder if we need to set Applied, Available, Degraded, Progressing conditions to False in order to reset LastTransitionTime. The current implementation doesn't change the Applied and Available during upgrade except for observedGeneration. Shouldn't Applied and Available represent the state of the updated manifest, not the old manifest?

The second issue is mainly about how to handle the behavior when Degraded has a chance to be true during rollout and the user sets progressDeadline. We might need to distinguish between a degraded status during a rollout and a real degraded state.

3. During rollout, the Progressing condition keeps false with observedGeneration updated, but with old LastTransaction, this will cause the cluster rollout status to pass the minSuccessTime and rollout to the next cluster. This is unexpected behavior.

Yes, need to check both the lastObservedGeneration of Applied and might need to force update the Progressing condition when spec changes as well. The above is a description of the current problem, and root cause will be further analyzed.

[haoqing0110]

PR open-cluster-management-io/sdk-go#163 will fix the issue 1.
PR #1243 will fix the issue 2 & 3, by checking the lastObservedGeneration of Applied, it delays the check on Progressing and Degraded, and avoids the race condition when a rollout starts.