Skip to content

Commit 6528fbe

Browse files
KunWuLuanKunWuLuan
committed
add kep
Signed-off-by: KunWuLuan <[email protected]>
1 parent 68e40f8 commit 6528fbe

File tree

2 files changed

+182
-0
lines changed

2 files changed

+182
-0
lines changed

‎kep/594-resourcepolicy/README.md

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# Resource Policy
2+
3+
## Table of Contents
4+
5+
<!-- toc -->
6+
- [Summary](#summary)
7+
- [Motivation](#motivation)
8+
- [Use Cases](#use-cases)
9+
- [Goals](#goals)
10+
- [Non-Goals](#non-goals)
11+
- [Proposal](#proposal)
12+
- [API](#api)
13+
- [Implementation Details](#implementation-details)
14+
- [PreFilter](#prefilter)
15+
- [Filter](#filter)
16+
- [Score](#score)
17+
- [PreBind](#prebind)
18+
- [Known limitations](#known-limitations)
19+
- [Test plans](#test-plans)
20+
- [Graduation criteria](#graduation-criteria)
21+
- [Feature enablement and rollback](#feature-enablement-and-rollback)
22+
<!-- /toc -->
23+
24+
## Summary
25+
This proposal introduces a plugin that enables users to set priorities for various resources and define maximum resource consumption limits for workloads across different resources.
26+
27+
## Motivation
28+
A Kubernetes cluster typically consists of heterogeneous machines, with varying SKUs on CPU, memory, GPU, and pricing. To
29+
efficiently utilize the different resources available in the cluster, users can set priorities for machines of different
30+
types and configure resource allocations for different workloads. Additionally, they may choose to delete pods running
31+
on low priority nodes instead of high priority ones.
32+
33+
### Use Cases
34+
35+
1. As a administrator of kubernetes cluster, there are some static but expensive VM instances and some dynamic but cheaper Spot
36+
instances in my cluster. I hope to restrict the resource consumption on each kind of resource for different workloads to limit the cost.
37+
I hope that important workloads in my cluster can be deployed first on static VM instances so that they will not worry about been preempted. And during business peak periods, the Pods that are scaled up are deployed on cheap, spot instances. At the end of the business peak, the Pods on Spot
38+
instances are prioritized to be scaled down.
39+
40+
### Goals
41+
42+
1. Develop a filter plugin to restrict the resource consumption on each kind of resource for different workloads.
43+
2. Develop a score plugin to favor nodes matched by a high priority kind of resource.
44+
3. Automatically setting deletion costs on Pods to control the scaling in sequence of workloads through a controller.
45+
46+
### Non-Goals
47+
48+
1. Scheduler will not delete the pods.
49+
50+
## Proposal
51+
52+
### API
53+
```yaml
54+
apiVersion: scheduling.sigs.x-k8s.io/v1alpha1
55+
kind: ResourcePolicy
56+
metadata:
57+
name: xxx
58+
namespace: xxx
59+
spec:
60+
matchLabelKeys:
61+
- pod-template-hash
62+
podSelector:
63+
key1: value1
64+
strategy: prefer
65+
units:
66+
- name: unit1
67+
max: 10
68+
maxResource:
69+
cpu: 10
70+
nodeSelector:
71+
key1: value1
72+
```
73+
74+
```go
75+
type ResourcePolicy struct {
76+
metav1.TypeMeta `json:",inline"`
77+
metav1.ObjectMeta `json:"metadata,omitempty"`
78+
79+
Spec ResourcePolicySpec `json:"spec"`
80+
Status ResourcePolicyStatus `json:"status,omitempty"`
81+
}
82+
83+
type ResourcePolicySpec struct {
84+
// +optional
85+
// +nullable
86+
// +listType=atomic
87+
Units []Unit `json:"units,omitempty" protobuf:"bytes,1,rep,name=units"`
88+
89+
Selector map[string]string `json:"selector,omitempty" protobuf:"bytes,2,rep,name=selector"`
90+
MatchLabelKeys []string `json:"matchLabelKeys,omitempty" protobuf:"bytes,3,rep,name=matchLabelKeys"`
91+
}
92+
93+
type Unit struct {
94+
Max *int32 `json:"max,omitempty" protobuf:"varint,1,opt,name=max"`
95+
MaxResources v1.ResourceList `json:"maxResources,omitempty" protobuf:"bytes,2,rep,name=maxResources"`
96+
97+
NodeSelector map[string]string `json:"nodeSelector,omitempty" protobuf:"bytes,3,rep,name=nodeSelector"`
98+
99+
PodLabelsToAdd map[string]string `json:"podLabels,omitempty" protobuf:"bytes,4,rep,name=podLabels"`
100+
PodAnnotationsToAdd map[string]string `json:"podAnnotations,omitempty" protobuf:"bytes,5,rep,name=podAnnotations"`
101+
}
102+
103+
type ResourcePolicyStatus struct {
104+
Pods []int64 `json:"pods,omitempty"`
105+
LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`
106+
}
107+
```
108+
109+
Pods will be matched by the ResourcePolicy in same namespace when the `.spec.podSelector`.
110+
ResourcePolicies will never match pods in different namesapces. One pod can not be matched by more than one Resource Policies.
111+
112+
Pods can only be scheduled on units defined in `.spec.units`. Each item in `.spec.units` contains a set of nodes that match the `NodeSelector` which describes a kind of resource in the cluster.
113+
114+
Pods will be scheduled in the order defined by the `.spec.units`.
115+
`.spec.units[].max` define the maximum number of pods that can be scheduled on each unit. If `.spec.units[].max` is not set, pods can always be scheduled on the units except there is no enough resource.
116+
`.spec.units[].maxResource` define the maximum resource that can be scheduled on each unit. If `.spec.units[].maxResource` is not set, pods can always be scheduled on the units except there is no enough resource.
117+
118+
`.spec.matchLabelKeys` indicate how we group the pods matched by `podSelector`, its behavior is like
119+
`.spec.matchLabelKeys` in `PodTopologySpread`.
120+
121+
### Implementation Details
122+
123+
#### PreFilter
124+
PreFilter check if the current pods match only one resource policy. If not, PreFilter will reject the pod.
125+
If yes, PreFilter will get the number of pods on each unit to determine which units are available for the pod
126+
and write this information into cycleState.
127+
128+
#### Filter
129+
Filter check if the node belongs to an available unit. If the node doesn't belong to any unit, we will return unschedulable.
130+
131+
Besides, filter will check if the pods that was scheduled on the unit has already violated the quantity constraint.
132+
If the number of pods has reach the `.spec.unit[].max`, all the nodes in unit will be marked unschedulable.
133+
134+
#### Score
135+
136+
Node score is `100 - (index of the unit)`
137+
138+
#### PreBind
139+
140+
Add annotations and labels to pods to ensure they can be scaled down in the order of the units.
141+
142+
## Known limitations
143+
144+
- Currently deletion costs only take effect on deployment workload.
145+
146+
## Test plans
147+
148+
1. Add detailed unit and integration tests for the plugin and controller.
149+
2. Add basic e2e tests, to ensure all components are working together.
150+
151+
## Graduation criteria
152+
153+
This plugin will not be enabled only when users enable it in scheduler framework and create a resourcepolicy for pods.
154+
So it is safe to be beta.
155+
156+
* Beta
157+
- [ ] Add node E2E tests.
158+
- [ ] Provide beta-level documentation.
159+
160+
## Feature enablement and rollback
161+
162+
Enable resourcepolicy in MultiPointPlugin to enable this plugin, like this:
163+
164+
```yaml
165+
piVersion: kubescheduler.config.k8s.io/v1
166+
kind: KubeSchedulerConfiguration
167+
leaderElection:
168+
leaderElect: false
169+
profiles:
170+
- schedulerName: default-scheduler
171+
plugins:
172+
multiPoint:
173+
enabled:
174+
- name: resourcepolicy
175+
```
176+
177+

‎kep/594-resourcepolicy/kep.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
title: Resourcepolicy
2+
kep-number: 594
3+
authors:
4+
- "@KunWuLuan"
5+
- "@fjding"

0 commit comments

Comments
 (0)