feat: dynamic ratelimiter for gracefuleviction #6675

whosefriendA · 2025-08-21T11:35:16Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Developed a cluster failure optimized eviction queue and its dynamic rate control and observability based on the design document.
design document： #6613

karmada-bot · 2025-08-21T11:35:22Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign xishanyongye-chang for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gemini-code-assist

Summary of Changes

Hello @whosefriendA, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to Karmada's graceful eviction mechanism by implementing a dynamic rate limiter. The core purpose is to optimize resource eviction during cluster failures, ensuring system stability by adjusting the eviction rate based on the overall health of the clusters. This feature includes the development of a new, failure-optimized eviction queue, along with robust dynamic rate control and comprehensive observability. The changes allow for a more resilient and adaptive system when dealing with unhealthy clusters, preventing aggressive evictions that could exacerbate issues.

Highlights

Dynamic Eviction Rate Limiting: A new DynamicRateLimiter has been implemented, which intelligently adjusts the rate of resource evictions based on the real-time health status of the clusters. This ensures that evictions are performed gracefully, preventing cascading failures during periods of cluster instability.
Specialized Eviction Worker: A dedicated EvictionWorker has been introduced to manage graceful eviction operations. This worker integrates the new dynamic rate limiter and is designed to enhance the robustness of the eviction process.
Configurable Eviction Queue Options: The behavior of the dynamic eviction queue is now highly configurable through new EvictionQueueOptions. These options, exposed via command-line flags, allow administrators to fine-tune parameters such as resource eviction rates, thresholds for unhealthy clusters, and considerations for large-scale deployments.
Enhanced Observability and Metrics: Comprehensive observability has been added for the new dynamic eviction queue. New Prometheus metrics track key indicators such as eviction queue depth, processing latency, counts by resource kind, and overall cluster health (failure rate and faulty cluster count), providing critical insights into system performance.
Integration with Taint Manager: The NoExecuteTaintManager has been updated to seamlessly integrate the new dynamic eviction worker, ensuring that the graceful eviction mechanism is applied consistently across resource and cluster resource bindings.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a dynamic rate limiter for graceful eviction, a significant enhancement for Karmada's stability. The rate limiter adjusts eviction speed based on the overall health of member clusters, preventing cascading failures. The implementation is well-structured, with clear separation of concerns for the rate limiter logic, a new instrumented eviction worker, and configuration options. The addition of comprehensive Prometheus metrics for the new components is also a major plus for observability. I've found one critical issue that needs to be addressed and a minor one to improve usability.

pkg/controllers/cluster/evictionqueue_config/evictionoption.go

codecov-commenter · 2025-08-21T12:08:39Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 65.30612% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.86%. Comparing base (a759915) to head (d4d9ad8).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/controllers/cluster/taint_manager.go	0.00%	36 Missing ⚠️
pkg/metrics/cluster.go	0.00%	26 Missing ⚠️
pkg/controllers/cluster/dynamic_rate_limiter.go	88.05%	6 Missing and 2 partials ⚠️
cmd/controller-manager/app/controllermanager.go	0.00%	7 Missing ⚠️
cmd/controller-manager/app/options/failover.go	0.00%	4 Missing ⚠️
pkg/controllers/cluster/eviction_worker.go	96.19%	3 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6675      +/-   ##
==========================================
+ Coverage   45.74%   45.86%   +0.11%     
==========================================
  Files         689      691       +2     
  Lines       57174    57409     +235     
==========================================
+ Hits        26157    26328     +171     
- Misses      29389    29452      +63     
- Partials     1628     1629       +1

Flag	Coverage Δ
unittests	`45.86% <65.30%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cmd/controller-manager/app/options/failover.go

XiShanYongYe-Chang · 2025-09-22T01:35:29Z

Hi @whosefriendA, all E2E tests have failed. Have you analyzed the root cause of the errors?

pkg/controllers/cluster/dynamic_rate_limiter.go

XiShanYongYe-Chang · 2025-09-22T03:56:00Z

pkg/controllers/cluster/dynamic_rate_limiter.go

+// - Normal rate when system is healthy
+// - Secondary rate when system is unhealthy but large-scale
+// - Zero (halt evictions) when system is unhealthy and small-scale
+func (d *DynamicRateLimiter[T]) getCurrentRate() float32 {


Can we add test code for these new additions?

sure ,I will upload the test code to this pr

XiShanYongYe-Chang · 2025-09-22T04:00:19Z

pkg/controllers/cluster/dynamic_rate_limiter.go

+	"github.com/karmada-io/karmada/pkg/util/fedinformer/genericmanager"
+)
+
+// maxEvictionDelay is the maximum delay for eviction when the rate is 0


This variable refers to the eviction of one resource every 1000 seconds, right?

The maxEvictionDelay := 1000 * time.Second specifies the maximum wait time the queue will impose on an element when the calculated currentRate == 0 (i.e., eviction should be paused). Rather than setting a rate of "one evict every 1000 seconds," it uses a long delay to pause processing, waking up after 1000 seconds to re-evaluate the rate. Normally, the processing interval is time.Duration(1/currentRate * time.Second). For example, if currentRate = 5/s, the queue will process one element every 200ms.

pkg/controllers/cluster/dynamic_rate_limiter.go

pkg/controllers/cluster/eviction_worker.go

whosefriendA · 2025-09-23T09:22:51Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a dynamic rate limiter for graceful eviction of resources, which adjusts its speed based on the overall health of the cluster federation. This is a valuable feature for improving system stability during widespread cluster failures. The implementation is solid, introducing a new EvictionWorker and DynamicRateLimiter along with comprehensive tests and metrics. However, I've identified a critical bug where a necessary InformerManager is not initialized, which would cause a panic at runtime. I've also included several medium-severity suggestions to improve validation, code clarity, and robustness. Addressing these points will make the new feature more reliable and maintainable.

gemini-code-assist · 2025-09-23T09:30:56Z

pkg/controllers/cluster/dynamic_rate_limiter.go

+	clusterGVR := schema.GroupVersionResource{
+		Group:    clusterv1alpha1.GroupName,
+		Version:  "v1alpha1",
+		Resource: "clusters",
+	}


For consistency and better maintainability, consider defining clusterGVR using clusterv1alpha1.SchemeGroupVersion.WithResource("clusters") as is done in the corresponding test file. This avoids hardcoding the group and version strings.

clusterGVR := clusterv1alpha1.SchemeGroupVersion.WithResource("clusters")

gemini-code-assist · 2025-09-23T09:30:56Z

pkg/controllers/cluster/eviction_worker.go

+func (w *evictionWorker) Add(item interface{}) {
+	if item == nil {
+		klog.Warningf("Ignore nil item from queue")
+		return
+	}
+
+	w.queue.Add(item)
+	metrics.RecordEvictionQueueMetrics(w.name, float64(w.queue.Len()))
+
+	// Update resource kind metrics if possible
+	if w.resourceKindFunc != nil {
+		clusterName, resourceKind := w.resourceKindFunc(item)
+		metrics.RecordEvictionKindMetrics(clusterName, resourceKind, true)
+	}
+}
+
+// AddAfter adds an item to the queue after a delay and updates metrics.
+func (w *evictionWorker) AddAfter(item interface{}, duration time.Duration) {
+	if item == nil {
+		klog.Warningf("Ignore nil item from queue")
+		return
+	}
+
+	w.queue.AddAfter(item, duration)
+	metrics.RecordEvictionQueueMetrics(w.name, float64(w.queue.Len()))
+
+	// Update resource kind metrics if possible
+	if w.resourceKindFunc != nil {
+		clusterName, resourceKind := w.resourceKindFunc(item)
+		metrics.RecordEvictionKindMetrics(clusterName, resourceKind, true)
+	}


The methods Add and AddAfter contain duplicated code for recording metrics. To improve maintainability and reduce redundancy, you could extract this logic into a private helper method.

gemini-code-assist · 2025-09-23T09:30:56Z

pkg/controllers/cluster/eviction_worker.go

+
+// processNextWorkItem handles a single item from the queue with metrics tracking.
+// Returns false when the queue is shutting down, true otherwise.
+func (w *evictionWorker) processNextWorkItem(_ context.Context) bool {


The processNextWorkItem function receives a context.Context but it is ignored (named _). The reconcileFunc that is called does not accept a context, which leads to context.TODO() being used deeper in the call stack (e.g., in syncBindingEviction).

This is a missed opportunity for proper cancellation propagation. Consider refactoring ReconcileFunc and the subsequent call chain to accept and use the context, which would allow for more responsive shutdowns and better resource management.

Signed-off-by: whosefriendA <[email protected]>

XiShanYongYe-Chang · 2025-09-29T09:50:08Z

Hi @whosefriendA, can you help squash the commits into one?

karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 21, 2025

karmada-bot requested review from jwcesign, lonelyCZ and zhzhuang-zju August 21, 2025 11:35

karmada-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 21, 2025

gemini-code-assist bot reviewed Aug 21, 2025

View reviewed changes

pkg/controllers/cluster/evictionqueue_config/evictionoption.go Outdated Show resolved Hide resolved

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch 2 times, most recently from d7e5278 to f7f4225 Compare August 21, 2025 11:52

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch 2 times, most recently from 3e7793f to f7928e7 Compare August 24, 2025 16:07

XiShanYongYe-Chang mentioned this pull request Sep 10, 2025

[Umbrella][Summer OSPP 2025] Support Eviction Queue with Dynamic Rate Limiting #6746

Open

7 tasks

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch 4 times, most recently from 3dcbfb8 to 3d141f5 Compare September 19, 2025 06:46

XiShanYongYe-Chang reviewed Sep 20, 2025

View reviewed changes

cmd/controller-manager/app/options/failover.go Show resolved Hide resolved

XiShanYongYe-Chang reviewed Sep 20, 2025

View reviewed changes

cmd/controller-manager/app/options/failover.go Outdated Show resolved Hide resolved

XiShanYongYe-Chang reviewed Sep 20, 2025

View reviewed changes

cmd/controller-manager/app/options/failover.go Outdated Show resolved Hide resolved

XiShanYongYe-Chang reviewed Sep 22, 2025

View reviewed changes

pkg/controllers/cluster/dynamic_rate_limiter.go Show resolved Hide resolved

XiShanYongYe-Chang reviewed Sep 22, 2025

View reviewed changes

pkg/controllers/cluster/eviction_worker.go Outdated Show resolved Hide resolved

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch 2 times, most recently from 2c436ec to 005dc7d Compare September 23, 2025 05:09

gemini-code-assist bot reviewed Sep 23, 2025

View reviewed changes

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch from 005dc7d to 5931aef Compare September 23, 2025 12:24

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch 3 times, most recently from f433fbb to e9c803d Compare September 24, 2025 14:56

whosefriendA added 15 commits September 26, 2025 19:19

feat: dynamic ratelimiter for gracefuleviction

4c681e8

Signed-off-by: whosefriendA <[email protected]>

fix lint

a41b7e0

Signed-off-by: whosefriendA <[email protected]>

fix spell

cc23d45

Signed-off-by: whosefriendA <[email protected]>

fix lint

e133324

Signed-off-by: whosefriendA <[email protected]>

refactor: change evictionoption to failover.go

ac95f1d

Signed-off-by: whosefriendA <[email protected]>

remove unuse define

db9e0fb

Signed-off-by: whosefriendA <[email protected]>

fix lint

0658c78

Signed-off-by: whosefriendA <[email protected]>

fix lint

ed2ebda

Signed-off-by: whosefriendA <[email protected]>

fix some comment

b5a6c2b

Signed-off-by: whosefriendA <[email protected]>

Add test

f60f68e

Signed-off-by: whosefriendA <[email protected]>

fix lint

00a6d3c

Signed-off-by: whosefriendA <[email protected]>

fix lint

0648e81

Signed-off-by: whosefriendA <[email protected]>

fix a code hygiene

105ba4f

Signed-off-by: whosefriendA <[email protected]>

remove evictionworker interface

e29b12c

Signed-off-by: whosefriendA <[email protected]>

fix: change the wrong enqueue rate

5db74da

Signed-off-by: whosefriendA <[email protected]>

whosefriendA force-pushed the feat/graceful_eviction_dynamicratelimit branch from e9c803d to 5db74da Compare September 26, 2025 11:19

karmada-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 26, 2025

whosefriendA added 3 commits September 26, 2025 20:39

fix lint

d94e6f4

Signed-off-by: whosefriendA <[email protected]>

fix lint

957fe71

Signed-off-by: whosefriendA <[email protected]>

fix:add informermanager init

d4d9ad8

Signed-off-by: whosefriendA <[email protected]>

feat: dynamic ratelimiter for gracefuleviction #6675

Are you sure you want to change the base?

feat: dynamic ratelimiter for gracefuleviction #6675

Uh oh!

Conversation

whosefriendA commented Aug 21, 2025

Uh oh!

karmada-bot commented Aug 21, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

codecov-commenter commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XiShanYongYe-Chang commented Sep 22, 2025

Uh oh!

Uh oh!

XiShanYongYe-Chang Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

whosefriendA Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

XiShanYongYe-Chang Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

whosefriendA Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

whosefriendA commented Sep 23, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

XiShanYongYe-Chang commented Sep 29, 2025

Uh oh!

Uh oh!

codecov-commenter commented Aug 21, 2025 •

edited

Loading