Skip to content

Commit 937f44a

Browse files
ingvagabundingvagabund
committed
DevKubeVirtRelieveAndMigrate: introduce dynamic threshold tuning option
Allow users to tune the descheduler LowNodeUtilization's `useDeviationThresholds` parameter with three predifined combination of low and high thresholds.
1 parent 2481164 commit 937f44a

11 files changed

+292
-17
lines changed

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,7 @@ In the future, additional configurations may be introduced through the operator
171171
The profile exposes the following customization:
172172
- `devLowNodeUtilizationThresholds`: Sets experimental thresholds for the LowNodeUtilization strategy.
173173
- `devActualUtilizationProfile`: Enable load-aware descheduling.
174+
- `devDeviationThresholds`: Have the thresholds be based on the average utilization.
174175

175176
### EvictPodsWithPVC
176177
By default, the operator prevents pods with PVCs from being evicted. Enabling this
@@ -198,6 +199,7 @@ the `profileCustomizations` field:
198199
|`devEnableEvictionsInBackground`|`bool`| Enables descheduler's EvictionsInBackground alpha feature. The EvictionsInBackground alpha feature is a subject to change. Currently provided as an experimental feature.|
199200
| `devHighNodeUtilizationThresholds` | `string` | Sets thresholds for the [HighNodeUtilization](https://github.com/kubernetes-sigs/descheduler#highnodeutilization) strategy of the `CompactAndScale` profile in the following ratios: `Minimal` for 10%, `Modest` for 20%, `Moderate` for 30%. Currently provided as an experimental feature.|
200201
|`devActualUtilizationProfile`|`string`| Sets a profile that gets translated into a predefined prometheus query |
202+
| `devDeviationThresholds` | `string` | Have the thresholds be based on the average utilization. Thresholds signify the distance from the average node utilization in the following setting: `Low`: 10%:10%, `Medium`: 20%:20%, `High`: 30%:30% |
201203

202204
## Prometheus query profiles
203205
The operator provides the following profiles:

‎manifests/kube-descheduler-operator.crd.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,15 @@ spec:
110110
LowNodeUtilization plugin can consume the metrics for now.
111111
Currently provided as an experimental feature.
112112
type: string
113+
devDeviationThresholds:
114+
description: devDeviationThresholds enables dynamic thresholds
115+
based on average resource utilization
116+
enum:
117+
- Low
118+
- Medium
119+
- High
120+
- ""
121+
type: string
113122
devEnableEvictionsInBackground:
114123
description: |-
115124
DevEnableEvictionsInBackground enables descheduler's EvictionsInBackground alpha feature.
@@ -185,6 +194,7 @@ spec:
185194
- EvictPodsWithLocalStorage
186195
- EvictPodsWithPVC
187196
- CompactAndScale
197+
- DevKubeVirtRelieveAndMigrate
188198
type: string
189199
type: array
190200
unsupportedConfigOverrides:

‎pkg/apis/descheduler/v1/types_descheduler.go

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,10 @@ type ProfileCustomizations struct {
9090
// LowNodeUtilization plugin can consume the metrics for now.
9191
// Currently provided as an experimental feature.
9292
DevActualUtilizationProfile ActualUtilizationProfile `json:"devActualUtilizationProfile,omitempty"`
93+
94+
// devDeviationThresholds enables dynamic thresholds based on average resource utilization
95+
// +kubebuilder:validation:Enum=Low;Medium;High;""
96+
DevDeviationThresholds *DeviationThresholdsType `json:"devDeviationThresholds,omitempty"`
9397
}
9498

9599
type LowNodeUtilizationThresholdsType string
@@ -121,6 +125,22 @@ var (
121125
CompactModerateThreshold HighNodeUtilizationThresholdsType = "Moderate"
122126
)
123127

128+
type DeviationThresholdsType string
129+
130+
var (
131+
// DeviationThresholdLow sets thresholds to 10%:10% ratio.
132+
// The threshold value is subject to change.
133+
LowDeviationThreshold DeviationThresholdsType = "Low"
134+
135+
// DeviationThresholdMedium sets thresholds to 20%:20% ratio.
136+
// The threshold value is subject to change.
137+
MediumDeviationThreshold DeviationThresholdsType = "Medium"
138+
139+
// DeviationThresholdHigh sets thresholds to 30%:30% ratio.
140+
// The threshold value is subject to change.
141+
HighDeviationThreshold DeviationThresholdsType = "High"
142+
)
143+
124144
// ActualUtilizationProfile sets predefined Prometheus PromQL query
125145
type ActualUtilizationProfile string
126146

@@ -146,7 +166,7 @@ type Namespaces struct {
146166

147167
// DeschedulerProfile allows configuring the enabled strategy profiles for the descheduler
148168
// it allows multiple profiles to be enabled at once, which will have cumulative effects on the cluster.
149-
// +kubebuilder:validation:Enum=AffinityAndTaints;TopologyAndDuplicates;LifecycleAndUtilization;DevPreviewLongLifecycle;LongLifecycle;SoftTopologyAndDuplicates;EvictPodsWithLocalStorage;EvictPodsWithPVC;CompactAndScale
169+
// +kubebuilder:validation:Enum=AffinityAndTaints;TopologyAndDuplicates;LifecycleAndUtilization;DevPreviewLongLifecycle;LongLifecycle;SoftTopologyAndDuplicates;EvictPodsWithLocalStorage;EvictPodsWithPVC;CompactAndScale;DevKubeVirtRelieveAndMigrate
150170
type DeschedulerProfile string
151171

152172
var (

‎pkg/apis/descheduler/v1/zz_generated.deepcopy.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/generated/applyconfiguration/descheduler/v1/profilecustomizations.go

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/operator/target_config_reconciler.go

Lines changed: 39 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -595,23 +595,43 @@ func setExcludedNamespacesForLowNodeUtilizationPlugin(lowNodeUtilizationArgs *no
595595
}
596596
}
597597

598-
func getLowNodeUtilizationThresholds(profileCustomizations *deschedulerv1.ProfileCustomizations) (deschedulerapi.Percentage, deschedulerapi.Percentage, error) {
598+
func getLowNodeUtilizationThresholds(profileCustomizations *deschedulerv1.ProfileCustomizations, ignoreDynamic bool) (deschedulerapi.Percentage, deschedulerapi.Percentage, error) {
599599
lowThreshold := deschedulerapi.Percentage(20)
600600
highThreshold := deschedulerapi.Percentage(50)
601601

602-
if profileCustomizations != nil && profileCustomizations.DevLowNodeUtilizationThresholds != nil {
603-
switch *profileCustomizations.DevLowNodeUtilizationThresholds {
604-
case deschedulerv1.LowThreshold:
605-
lowThreshold = 10
606-
highThreshold = 30
607-
case deschedulerv1.MediumThreshold, "":
608-
lowThreshold = 20
609-
highThreshold = 50
610-
case deschedulerv1.HighThreshold:
611-
lowThreshold = 40
612-
highThreshold = 70
613-
default:
614-
return 0, 0, fmt.Errorf("unknown Descheduler LowNodeUtilization threshold %v, only 'Low', 'Medium' and 'High' are supported", *profileCustomizations.DevLowNodeUtilizationThresholds)
602+
if profileCustomizations != nil {
603+
if !ignoreDynamic && profileCustomizations.DevLowNodeUtilizationThresholds != nil && profileCustomizations.DevDeviationThresholds != nil {
604+
return 0, 0, fmt.Errorf("only one of DevLowNodeUtilizationThresholds and DevDeviationThresholds customizations can be configured simultaneously")
605+
}
606+
if profileCustomizations.DevLowNodeUtilizationThresholds != nil {
607+
switch *profileCustomizations.DevLowNodeUtilizationThresholds {
608+
case deschedulerv1.LowThreshold:
609+
lowThreshold = 10
610+
highThreshold = 30
611+
case deschedulerv1.MediumThreshold, "":
612+
lowThreshold = 20
613+
highThreshold = 50
614+
case deschedulerv1.HighThreshold:
615+
lowThreshold = 40
616+
highThreshold = 70
617+
default:
618+
return 0, 0, fmt.Errorf("unknown Descheduler LowNodeUtilization threshold %v, only 'Low', 'Medium' and 'High' are supported", *profileCustomizations.DevLowNodeUtilizationThresholds)
619+
}
620+
}
621+
if !ignoreDynamic && profileCustomizations.DevDeviationThresholds != nil {
622+
switch *profileCustomizations.DevDeviationThresholds {
623+
case deschedulerv1.LowDeviationThreshold:
624+
lowThreshold = 10
625+
highThreshold = 10
626+
case deschedulerv1.MediumDeviationThreshold:
627+
lowThreshold = 20
628+
highThreshold = 20
629+
case deschedulerv1.HighDeviationThreshold:
630+
lowThreshold = 30
631+
highThreshold = 30
632+
default:
633+
return 0, 0, fmt.Errorf("unknown Descheduler DeviationThresholds threshold %v, only 'Low', 'Medium' and 'High' are supported", *profileCustomizations.DevDeviationThresholds)
634+
}
615635
}
616636
}
617637

@@ -691,7 +711,7 @@ func lifecycleAndUtilizationProfile(profileCustomizations *deschedulerv1.Profile
691711
setExcludedNamespacesForLowNodeUtilizationPlugin(profile.PluginConfigs[2].Args.Object.(*nodeutilization.LowNodeUtilizationArgs), includedNamespaces, excludedNamespaces, protectedNamespaces)
692712
}
693713

694-
lowThreshold, highThreshold, err := getLowNodeUtilizationThresholds(profileCustomizations)
714+
lowThreshold, highThreshold, err := getLowNodeUtilizationThresholds(profileCustomizations, true)
695715
if err != nil {
696716
return nil, err
697717
}
@@ -772,7 +792,7 @@ func relieveAndMigrateProfile(profileCustomizations *deschedulerv1.ProfileCustom
772792
setExcludedNamespacesForLowNodeUtilizationPlugin(profile.PluginConfigs[0].Args.Object.(*nodeutilization.LowNodeUtilizationArgs), includedNamespaces, excludedNamespaces, protectedNamespaces)
773793
}
774794

775-
lowThreshold, highThreshold, err := getLowNodeUtilizationThresholds(profileCustomizations)
795+
lowThreshold, highThreshold, err := getLowNodeUtilizationThresholds(profileCustomizations, false)
776796
if err != nil {
777797
return nil, err
778798
}
@@ -781,6 +801,9 @@ func relieveAndMigrateProfile(profileCustomizations *deschedulerv1.ProfileCustom
781801
args := profile.PluginConfigs[0].Args.Object.(*nodeutilization.LowNodeUtilizationArgs)
782802

783803
if profileCustomizations != nil {
804+
// enable deviation
805+
args.UseDeviationThresholds = profileCustomizations.DevDeviationThresholds != nil && *profileCustomizations.DevDeviationThresholds != ""
806+
784807
if profileCustomizations.DevActualUtilizationProfile != "" {
785808
query, err := utilizationProfileToPrometheusQuery(profileCustomizations.DevActualUtilizationProfile)
786809
if err != nil {

‎pkg/operator/target_config_reconciler_test.go

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,64 @@ func TestManageConfigMap(t *testing.T) {
267267
Data: map[string]string{"policy.yaml": string(bindata.MustAsset("assets/relieveAndMigrateIncludedNamespace.yaml"))},
268268
},
269269
},
270+
{
271+
name: "RelieveAndMigrateDynamicThresholdsLow",
272+
descheduler: &deschedulerv1.KubeDescheduler{
273+
Spec: deschedulerv1.KubeDeschedulerSpec{
274+
Profiles: []deschedulerv1.DeschedulerProfile{"DevKubeVirtRelieveAndMigrate"},
275+
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
276+
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
277+
},
278+
},
279+
},
280+
want: &corev1.ConfigMap{
281+
TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"},
282+
Data: map[string]string{"policy.yaml": string(bindata.MustAsset("assets/relieveAndMigrateDynamicThresholdsLow.yaml"))},
283+
},
284+
},
285+
{
286+
name: "RelieveAndMigrateDynamicThresholdsMedium",
287+
descheduler: &deschedulerv1.KubeDescheduler{
288+
Spec: deschedulerv1.KubeDeschedulerSpec{
289+
Profiles: []deschedulerv1.DeschedulerProfile{"DevKubeVirtRelieveAndMigrate"},
290+
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
291+
DevDeviationThresholds: &deschedulerv1.MediumDeviationThreshold,
292+
},
293+
},
294+
},
295+
want: &corev1.ConfigMap{
296+
TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"},
297+
Data: map[string]string{"policy.yaml": string(bindata.MustAsset("assets/relieveAndMigrateDynamicThresholdsMedium.yaml"))},
298+
},
299+
},
300+
{
301+
name: "RelieveAndMigrateDynamicThresholdsHigh",
302+
descheduler: &deschedulerv1.KubeDescheduler{
303+
Spec: deschedulerv1.KubeDeschedulerSpec{
304+
Profiles: []deschedulerv1.DeschedulerProfile{"DevKubeVirtRelieveAndMigrate"},
305+
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
306+
DevDeviationThresholds: &deschedulerv1.HighDeviationThreshold,
307+
},
308+
},
309+
},
310+
want: &corev1.ConfigMap{
311+
TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"},
312+
Data: map[string]string{"policy.yaml": string(bindata.MustAsset("assets/relieveAndMigrateDynamicThresholdsHigh.yaml"))},
313+
},
314+
},
315+
{
316+
name: "RelieveAndMigrateDynamicAndStaticThresholds",
317+
descheduler: &deschedulerv1.KubeDescheduler{
318+
Spec: deschedulerv1.KubeDeschedulerSpec{
319+
Profiles: []deschedulerv1.DeschedulerProfile{"DevKubeVirtRelieveAndMigrate"},
320+
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
321+
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
322+
DevLowNodeUtilizationThresholds: &deschedulerv1.LowThreshold,
323+
},
324+
},
325+
},
326+
err: fmt.Errorf("only one of DevLowNodeUtilizationThresholds and DevDeviationThresholds customizations can be configured simultaneously"),
327+
},
270328
{
271329
name: "AffinityAndTaintsWithNamespaces",
272330
descheduler: &deschedulerv1.KubeDescheduler{

‎pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsHigh.yaml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
apiVersion: descheduler/v1alpha2
2+
kind: DeschedulerPolicy
3+
profiles:
4+
- name: DevKubeVirtRelieveAndMigrate
5+
pluginConfig:
6+
- args:
7+
evictableNamespaces:
8+
exclude:
9+
- kube-system
10+
- hypershift
11+
- openshift
12+
- openshift-kube-scheduler
13+
targetThresholds:
14+
cpu: 30
15+
memory: 30
16+
pods: 30
17+
thresholds:
18+
cpu: 30
19+
memory: 30
20+
pods: 30
21+
useDeviationThresholds: true
22+
name: LowNodeUtilization
23+
- args:
24+
evictLocalStoragePods: true
25+
name: DefaultEvictor
26+
plugins:
27+
balance:
28+
disabled: null
29+
enabled:
30+
- LowNodeUtilization
31+
deschedule:
32+
disabled: null
33+
enabled: null
34+
filter:
35+
disabled: null
36+
enabled:
37+
- DefaultEvictor
38+
preevictionfilter:
39+
disabled: null
40+
enabled: null
41+
presort:
42+
disabled: null
43+
enabled: null
44+
sort:
45+
disabled: null
46+
enabled: null

‎pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsLow.yaml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
apiVersion: descheduler/v1alpha2
2+
kind: DeschedulerPolicy
3+
profiles:
4+
- name: DevKubeVirtRelieveAndMigrate
5+
pluginConfig:
6+
- args:
7+
evictableNamespaces:
8+
exclude:
9+
- kube-system
10+
- hypershift
11+
- openshift
12+
- openshift-kube-scheduler
13+
targetThresholds:
14+
cpu: 10
15+
memory: 10
16+
pods: 10
17+
thresholds:
18+
cpu: 10
19+
memory: 10
20+
pods: 10
21+
useDeviationThresholds: true
22+
name: LowNodeUtilization
23+
- args:
24+
evictLocalStoragePods: true
25+
name: DefaultEvictor
26+
plugins:
27+
balance:
28+
disabled: null
29+
enabled:
30+
- LowNodeUtilization
31+
deschedule:
32+
disabled: null
33+
enabled: null
34+
filter:
35+
disabled: null
36+
enabled:
37+
- DefaultEvictor
38+
preevictionfilter:
39+
disabled: null
40+
enabled: null
41+
presort:
42+
disabled: null
43+
enabled: null
44+
sort:
45+
disabled: null
46+
enabled: null

0 commit comments

Comments
 (0)