Skip to content

Commit 1936ece

Browse files
openshift-merge-bot[bot]openshift-merge-bot[bot]
authored
Merge pull request #506 from tiraboschi/st_default_enabled
Have the SoftTainter always deployed with KubeVirtRelieveAndMigrate
2 parents 1fc35cd + 8007c71 commit 1936ece

10 files changed

+83
-50
lines changed

‎README.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -174,11 +174,16 @@ That nodeSelector is a top level configuration option affecting all the active d
174174
this profile is not expected to be combined with other profiles
175175
unless all profiles are expected to operate over the same set of nodes.
176176

177+
This profile deploys the soft-tainter component to dynamically set/remove soft taints according to the
178+
same criteria used for load aware descheduling. In case of load-aware descheduling we can potentially have a
179+
relevant asymmetry between the descheduling and successive scheduling decisions.
180+
The soft taints set by the descheduler soft-tainter act as a hint for the scheduler to mitigate
181+
this asymmetry and foster a quicker convergence.
182+
177183
The profile exposes the following customization:
178184
- `devLowNodeUtilizationThresholds`: Sets experimental thresholds for the LowNodeUtilization strategy.
179185
- `devActualUtilizationProfile`: Enable load-aware descheduling.
180186
- `devDeviationThresholds`: Have the thresholds be based on the average utilization.
181-
- `devEnableSoftTainter`: Have the operator deploying the soft-tainter component to dynamically set/remove soft taints according to the same criteria used for load aware descheduling. Scheduling and descheduling decisions are in general fully independent. In case of load-aware descheduling we can potentially have a relevant asymmetry between the descheduling and successive scheduling decisions. The soft taints set by the descheduler soft-tainter act as a hint for the scheduler to mitigate this asymmetry and foster a quicker convergence.
182187

183188
### EvictPodsWithPVC
184189
By default, the operator prevents pods with PVCs from being evicted. Enabling this
@@ -207,7 +212,6 @@ the `profileCustomizations` field:
207212
| `devHighNodeUtilizationThresholds` | `string` | Sets thresholds for the [HighNodeUtilization](https://github.com/kubernetes-sigs/descheduler#highnodeutilization) strategy of the `CompactAndScale` profile in the following ratios: `Minimal` for 10%, `Modest` for 20%, `Moderate` for 30%. Currently provided as an experimental feature.|
208213
|`devActualUtilizationProfile`|`string`| Sets a profile that gets translated into a predefined prometheus query |
209214
| `devDeviationThresholds` | `string` | Have the thresholds be based on the average utilization. Thresholds signify the distance from the average node utilization. An AsymmetricDeviationThreshold will force all nodes below the average to be considered as underutilized to help rebalancing overutilized outliers. Supported settings: `Low`: 10%:10%, `Medium`: 20%:20%, `High`: 30%:30%, `AsymmetricLow`: 0%:10%, `AsymmetricMedium`: 0%:20%, `AsymmetricHigh`: 0%:30% |
210-
| `devEnableSoftTainter` | `bool` | Have the operator deploying the soft-tainter component to dynamic set/remove soft taints as a hint for the scheduler based on the same criteria used for the descheduling |
211215

212216
## Prometheus query profiles
213217
The operator provides the following profiles:

‎manifests/kube-descheduler-operator.crd.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ spec:
133133
DevEnableSoftTainter enables SoftTainter alpha feature.
134134
The EnableSoftTainter alpha feature is a subject to change.
135135
Currently provided as an experimental feature.
136+
Deprecated: DevEnableSoftTainter field is deprecated and ignored.
136137
type: boolean
137138
devHighNodeUtilizationThresholds:
138139
description: |-

‎pkg/apis/descheduler/v1/types_descheduler.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ type ProfileCustomizations struct {
7878
// DevEnableSoftTainter enables SoftTainter alpha feature.
7979
// The EnableSoftTainter alpha feature is a subject to change.
8080
// Currently provided as an experimental feature.
81+
// +kubebuilder:deprecatedversion:warning="devEnableSoftTainter field is deprecated and ignored"
82+
// Deprecated: DevEnableSoftTainter field is deprecated and ignored.
8183
DevEnableSoftTainter bool `json:"devEnableSoftTainter"`
8284

8385
// DevEnableEvictionsInBackground enables descheduler's EvictionsInBackground alpha feature.

‎pkg/operator/target_config_reconciler.go

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1255,12 +1255,6 @@ func (c *TargetConfigReconciler) manageConfigMap(descheduler *deschedulerv1.Kube
12551255
}
12561256
}
12571257

1258-
if descheduler.Spec.ProfileCustomizations != nil && descheduler.Spec.ProfileCustomizations.DevEnableSoftTainter {
1259-
if !slices.Contains(descheduler.Spec.Profiles, deschedulerv1.RelieveAndMigrate) {
1260-
return nil, true, fmt.Errorf("the softtainter can only be used with the %v profile", deschedulerv1.RelieveAndMigrate)
1261-
}
1262-
}
1263-
12641258
// ignore PVC pods by default
12651259
ignorePVCPods := true
12661260
evictLocalStoragePods := false
@@ -1596,9 +1590,10 @@ func (c *TargetConfigReconciler) isKubeVirtDeployed() (bool, error) {
15961590
}
15971591

15981592
func (c *TargetConfigReconciler) isSoftTainterNeeded(descheduler *deschedulerv1.KubeDescheduler) (bool, error) {
1599-
if descheduler.Spec.ProfileCustomizations != nil && descheduler.Spec.ProfileCustomizations.DevEnableSoftTainter {
1593+
if slices.Contains(descheduler.Spec.Profiles, deschedulerv1.RelieveAndMigrate) {
16001594
return true, nil
16011595
}
1596+
16021597
ls, err := labels.Parse(kubeVirtShedulableLabelSelector)
16031598
if err != nil {
16041599
return false, err

‎pkg/operator/target_config_reconciler_test.go

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1170,14 +1170,13 @@ func TestManageSoftTainterDeployment(t *testing.T) {
11701170
checkContainerArgsOnly bool
11711171
}{
11721172
{
1173-
name: "RelieveAndMigrate with the softtainer",
1173+
name: "RelieveAndMigrate",
11741174
descheduler: &deschedulerv1.KubeDescheduler{
11751175
Spec: deschedulerv1.KubeDeschedulerSpec{
11761176
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.RelieveAndMigrate},
11771177
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
11781178
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
11791179
DevActualUtilizationProfile: deschedulerv1.PrometheusCPUCombinedProfile,
1180-
DevEnableSoftTainter: true,
11811180
},
11821181
DeschedulingIntervalSeconds: utilptr.To[int32](10),
11831182
},
@@ -1187,14 +1186,13 @@ func TestManageSoftTainterDeployment(t *testing.T) {
11871186
want: expectedSoftTainterDeployment,
11881187
},
11891188
{
1190-
name: "RelieveAndMigrate without the softtainer and no leftovers on existing nodes",
1189+
name: "LifecycleAndUtilization (without the softtainer) and no leftovers on existing nodes",
11911190
descheduler: &deschedulerv1.KubeDescheduler{
11921191
Spec: deschedulerv1.KubeDeschedulerSpec{
1193-
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.RelieveAndMigrate},
1192+
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.LifecycleAndUtilization},
11941193
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
11951194
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
11961195
DevActualUtilizationProfile: deschedulerv1.PrometheusCPUCombinedProfile,
1197-
DevEnableSoftTainter: false,
11981196
},
11991197
DeschedulingIntervalSeconds: utilptr.To[int32](10),
12001198
},
@@ -1234,14 +1232,13 @@ func TestManageSoftTainterDeployment(t *testing.T) {
12341232
want: nil,
12351233
},
12361234
{
1237-
name: "RelieveAndMigrate without the softtainer but a leftover on existing nodes - 1",
1235+
name: "LifecycleAndUtilization (without the softtainer) but a leftover on existing nodes - 1",
12381236
descheduler: &deschedulerv1.KubeDescheduler{
12391237
Spec: deschedulerv1.KubeDeschedulerSpec{
1240-
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.RelieveAndMigrate},
1238+
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.LifecycleAndUtilization},
12411239
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
12421240
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
12431241
DevActualUtilizationProfile: deschedulerv1.PrometheusCPUCombinedProfile,
1244-
DevEnableSoftTainter: false,
12451242
},
12461243
DeschedulingIntervalSeconds: utilptr.To[int32](10),
12471244
},
@@ -1276,14 +1273,13 @@ func TestManageSoftTainterDeployment(t *testing.T) {
12761273
want: expectedSoftTainterDeployment,
12771274
},
12781275
{
1279-
name: "RelieveAndMigrate without the softtainer but a leftover on existing nodes - 2",
1276+
name: "LifecycleAndUtilization (without the softtainer) but a leftover on existing nodes - 2",
12801277
descheduler: &deschedulerv1.KubeDescheduler{
12811278
Spec: deschedulerv1.KubeDeschedulerSpec{
1282-
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.RelieveAndMigrate},
1279+
Profiles: []deschedulerv1.DeschedulerProfile{deschedulerv1.LifecycleAndUtilization},
12831280
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
12841281
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
12851282
DevActualUtilizationProfile: deschedulerv1.PrometheusCPUCombinedProfile,
1286-
DevEnableSoftTainter: false,
12871283
},
12881284
DeschedulingIntervalSeconds: utilptr.To[int32](10),
12891285
},

‎pkg/softtainter/softtainter.go

Lines changed: 8 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -126,20 +126,6 @@ func (st *softTainter) Reconcile(ctx context.Context, request reconcile.Request)
126126
nodeList[i] = &node
127127
}
128128

129-
enableSoftTainter := false
130-
if des.Spec.ProfileCustomizations != nil && des.Spec.ProfileCustomizations.DevEnableSoftTainter {
131-
enableSoftTainter = true
132-
}
133-
134-
if !enableSoftTainter {
135-
logger.Info("SoftTainter is disabled, cleaning up eventual leftover taints")
136-
err = st.cleanAllSoftTaints(ctx, nodeList)
137-
if err != nil {
138-
return reconcile.Result{}, err
139-
}
140-
return reconcile.Result{RequeueAfter: st.resyncPeriod}, nil
141-
}
142-
143129
var lnargs *nodeutilization.LowNodeUtilizationArgs
144130

145131
for _, p := range policy.Profiles {
@@ -151,11 +137,16 @@ func (st *softTainter) Reconcile(ctx context.Context, request reconcile.Request)
151137
}
152138
}
153139
}
140+
154141
if lnargs == nil {
155-
err := fmt.Errorf("unable to read LowNodeUtilizationArgs")
156-
logger.Error(err, "reconciliation failed")
157-
return reconcile.Result{}, err
142+
logger.Info("SoftTainter is disabled, cleaning up eventual leftover taints")
143+
err = st.cleanAllSoftTaints(ctx, nodeList)
144+
if err != nil {
145+
return reconcile.Result{}, err
146+
}
147+
return reconcile.Result{RequeueAfter: st.resyncPeriod}, nil
158148
}
149+
159150
st.args = &softTainterArgs{
160151
useDeviationThresholds: lnargs.UseDeviationThresholds,
161152
thresholds: lnargs.Thresholds,

‎pkg/softtainter/softtainter_test.go

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,6 @@ func TestReconcile(t *testing.T) {
109109
DeschedulingIntervalSeconds: pointer.Int32(30),
110110
Mode: deschedulerv1.Automatic,
111111
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
112-
DevEnableSoftTainter: true,
113112
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
114113
},
115114
},
@@ -146,7 +145,6 @@ func TestReconcile(t *testing.T) {
146145
DeschedulingIntervalSeconds: pointer.Int32(30),
147146
Mode: deschedulerv1.Automatic,
148147
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
149-
DevEnableSoftTainter: true,
150148
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
151149
},
152150
},
@@ -183,7 +181,6 @@ func TestReconcile(t *testing.T) {
183181
DeschedulingIntervalSeconds: pointer.Int32(30),
184182
Mode: deschedulerv1.Automatic,
185183
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
186-
DevEnableSoftTainter: true,
187184
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
188185
},
189186
},
@@ -226,7 +223,6 @@ func TestReconcile(t *testing.T) {
226223
DeschedulingIntervalSeconds: pointer.Int32(30),
227224
Mode: deschedulerv1.Automatic,
228225
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
229-
DevEnableSoftTainter: true,
230226
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
231227
},
232228
},
@@ -263,7 +259,6 @@ func TestReconcile(t *testing.T) {
263259
DeschedulingIntervalSeconds: pointer.Int32(30),
264260
Mode: deschedulerv1.Automatic,
265261
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
266-
DevEnableSoftTainter: true,
267262
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
268263
},
269264
},
@@ -300,15 +295,14 @@ func TestReconcile(t *testing.T) {
300295
DeschedulingIntervalSeconds: pointer.Int32(30),
301296
Mode: deschedulerv1.Automatic,
302297
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
303-
DevEnableSoftTainter: true,
304298
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
305299
},
306300
},
307301
},
308302
testfilename: "policy.yaml",
309303
},
310304
{
311-
description: "3 nodes, softTainter is disabled",
305+
description: "3 nodes, RelieveAndMigrate is disabled",
312306
nodes: []*corev1.Node{
313307
buildTestNodeWithTaints("node1", true, false, false, true),
314308
buildTestNodeWithTaints("node2", true, false, false, true),
@@ -337,15 +331,14 @@ func TestReconcile(t *testing.T) {
337331
DeschedulingIntervalSeconds: pointer.Int32(30),
338332
Mode: deschedulerv1.Automatic,
339333
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
340-
DevEnableSoftTainter: false,
341334
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
342335
},
343336
},
344337
},
345-
testfilename: "policy.yaml",
338+
testfilename: "policyNoRelieveAndMigrate.yaml",
346339
},
347340
{
348-
description: "3 nodes, softTainter is disabled, leftover taints",
341+
description: "3 nodes, RelieveAndMigrate is disabled, leftover taints",
349342
nodes: []*corev1.Node{
350343
buildTestNodeWithTaints("node1", true, true, false, true),
351344
buildTestNodeWithTaints("node2", true, false, true, true),
@@ -374,12 +367,11 @@ func TestReconcile(t *testing.T) {
374367
DeschedulingIntervalSeconds: pointer.Int32(30),
375368
Mode: deschedulerv1.Automatic,
376369
ProfileCustomizations: &deschedulerv1.ProfileCustomizations{
377-
DevEnableSoftTainter: false,
378370
DevDeviationThresholds: &deschedulerv1.LowDeviationThreshold,
379371
},
380372
},
381373
},
382-
testfilename: "policy.yaml",
374+
testfilename: "policyNoRelieveAndMigrate.yaml",
383375
},
384376
}
385377
for _, tc := range tests {

‎pkg/softtainter/testdata/policyNoRelieveAndMigrate.yaml

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
apiVersion: descheduler/v1alpha2
2+
kind: DeschedulerPolicy
3+
metricsProviders:
4+
- prometheus:
5+
url: https://prometheus-k8s-openshift-monitoring.apps.example.com
6+
source: Prometheus
7+
nodeSelector: kubevirt.io/schedulable=true
8+
profiles:
9+
- name: LifecycleAndUtilization
10+
pluginConfig:
11+
- args:
12+
evictableNamespaces:
13+
exclude:
14+
- kube-system
15+
- hypershift
16+
- openshift
17+
- openshift-kube-descheduler-operator
18+
- openshift-kube-scheduler
19+
metricsUtilization:
20+
prometheus:
21+
query: descheduler:combined_utilization_and_pressure:avg1m
22+
source: Prometheus
23+
targetThresholds:
24+
MetricResource: 10
25+
thresholds:
26+
MetricResource: 10
27+
useDeviationThresholds: true
28+
name: LowNodeUtilization
29+
- args:
30+
evictLocalStoragePods: true
31+
name: DefaultEvictor
32+
plugins:
33+
balance:
34+
disabled: null
35+
enabled:
36+
- LowNodeUtilization
37+
deschedule:
38+
disabled: null
39+
enabled: null
40+
filter:
41+
disabled: null
42+
enabled:
43+
- DefaultEvictor
44+
preevictionfilter:
45+
disabled: null
46+
enabled: null
47+
presort:
48+
disabled: null
49+
enabled: null
50+
sort:
51+
disabled: null
52+
enabled: null

‎test/e2e/bindata/assets/00_kube-descheduler-operator-crd.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ spec:
133133
DevEnableSoftTainter enables SoftTainter alpha feature.
134134
The EnableSoftTainter alpha feature is a subject to change.
135135
Currently provided as an experimental feature.
136+
Deprecated: DevEnableSoftTainter field is deprecated and ignored.
136137
type: boolean
137138
devHighNodeUtilizationThresholds:
138139
description: |-

‎test/e2e/bindata/assets/07_descheduler-operator.cr.devKubeVirtRelieveAndMigrate.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ spec:
1111
- DevKubeVirtRelieveAndMigrate
1212
profileCustomizations:
1313
podLifetime: 10s
14-
devEnableSoftTainter: true
1514
namespaces:
1615
included:
1716
- e2e-testdescheduling

0 commit comments

Comments
 (0)