[Umbrella] Performance optimizations planned for v1.15

[zach593]

Features that can be added before the v1.15 release deadline

1. Since we have disabled deepcopy during list operations in #5813 , we should enable the MutationDetector in our e2e tests to prevent any future code from mutating the cache.

   • Enable mutation detection in E2E testing. (@XiShanYongYe-Chang, enable mutation detection in e2e #6541)
   • detector: eliminate the mutation of the informer cache. (@zhzhuang-zju, detector: eliminate the mutation of the informer cache #6544)

2. Introduce the controller-runtime priority queue. This enhancement allows controllers built on controller-runtime to prioritize the latest changes after a restart or leader transition, deferring the processing of inevitable create events. This can significantly reduce downtime during service restarts and failovers. * For this feature, we will likely need detailed test reports to ensure it does not introduce negative side effects.

   • feat: introduce ControllerPriorityQueue feature gate. (@zach593, feat: introduce ControllerPriorityQueue feature gate #6662)

3. Scheduler Performance enhancement

   • Improve the performance of karmada-scheduler. (@LivingCcj, Improve the performance of karmada-scheduler #6597)

Features that may progress before the v1.15 release deadline but are unlikely to be completed

1. Where feasible, migrate async-worker to use controller-runtime controllers. Replace asyncWorker with controller-runtime reconciler #6251

Features unlikely to start before the v1.15 release deadline but are likely to be pursued in the future

1. Add automated e2e benchmark tests in GitHub CI.
2. Introduce controller-runtime warmup. This feature is still under development ✨ Implement warm replica support for controllers kubernetes-sigs/controller-runtime#3192. Combined with the priority queue, it could theoretically reduce downtime to zero during multi-replica leader transitions, which is especially important in large-scale clusters.
3. Continue adding indices to improve list performance wherever applicable.
   • [Umbrella] Use index to optimize list operations [Umbrella] Use index to optimize list operations #6530

---

The above list includes what I am currently aware of. If there are any omissions, please feel free to add or suggest them.

[zach593]

I will create separate issues for each task this week.

[RainbowMango]

Great thanks @zach593 for your hard work~ Take care!

[RainbowMango]

Speaking of the flags in karmada-controller-manager, like:

      --concurrent-cluster-propagation-policy-syncs int                                                                                                                                                                      
                The number of ClusterPropagationPolicy that are allowed to sync concurrently. (default 1)
      --concurrent-cluster-syncs int                                                                                                                                                                                         
                The number of Clusters that are allowed to sync concurrently. (default 5)
      --concurrent-clusterresourcebinding-syncs int                                                                                                                                                                          
                The number of ClusterResourceBindings that are allowed to sync concurrently. (default 5)
      --concurrent-dependent-resource-syncs int                                                                                                                                                                              
                The number of dependent resource that are allowed to sync concurrently. (default 2)
      --concurrent-namespace-syncs int                                                                                                                                                                                       
                The number of Namespaces that are allowed to sync concurrently. (default 1)
      --concurrent-propagation-policy-syncs int                                                                                                                                                                              
                The number of PropagationPolicy that are allowed to sync concurrently. (default 1)
      --concurrent-resource-template-syncs int                                                                                                                                                                               
                The number of resource templates that are allowed to sync concurrently. (default 5)
      --concurrent-resourcebinding-syncs int                                                                                                                                                                                 
                The number of ResourceBindings that are allowed to sync concurrently. (default 5)
      --concurrent-work-syncs int                                                                                                                                                                                            
                The number of Works that are allowed to sync concurrently. (default 5)

1. Shall we revisit the default values? I'm not sure if the default values of these parameters are overly conservative.
2. Shall we enrich the usage to add more instructions, such as when people need to consider enlarging the value?
3. Shall we group all these performance-related flags into a dedicated group like Performance flags?

[godwinpang]

fwiw kubernetes-sigs/controller-runtime#3192 has been merged

[RainbowMango]

Thank you, @godwinpang, for the notice and for the great job you did.

[RainbowMango]

@zach593 I updated the Issue, added some PR links for tracking the progress.

[RainbowMango]

feat: introduce ControllerPriorityQueue feature gate.

FYI. The test report can be found here.
#6662 (comment)

[RainbowMango]

/close
All tasks planned in this release are done. Thank you so much for your excellent job! @zach593 @zhzhuang-zju @XiShanYongYe-Chang @LivingCcj @CharlesQQ

[karmada-bot]

@RainbowMango: Closing this issue.

In response to this:

/close
All tasks planned in this release are done. Thank you so much for your excellent job! @zach593 @zhzhuang-zju @XiShanYongYe-Chang @LivingCcj @CharlesQQ

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.