Skip to content

Commit 55ae98a

Browse files
youxuan0714youxuan0714
committed
YUNIKORN-1452
1 parent bf0f2a5 commit 55ae98a

File tree

2 files changed

+209
-0
lines changed

2 files changed

+209
-0
lines changed

‎docs/user_guide/admission_controller.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
---
2+
id: admission_controller
3+
title: Admission Controller
4+
---
5+
6+
<!--
7+
Licensed to the Apache Software Foundation (ASF) under one
8+
or more contributor license agreements. See the NOTICE file
9+
distributed with this work for additional information
10+
regarding copyright ownership. The ASF licenses this file
11+
to you under the Apache License, Version 2.0 (the
12+
"License"); you may not use this file except in compliance
13+
with the License. You may obtain a copy of the License at
14+
15+
http://www.apache.org/licenses/LICENSE-2.0
16+
17+
Unless required by applicable law or agreed to in writing,
18+
software distributed under the License is distributed on an
19+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
20+
KIND, either express or implied. See the License for the
21+
specific language governing permissions and limitations
22+
under the License.
23+
-->
24+
25+
## What is the admission controller
26+
27+
The pod's yaml file can be easily configured through the admission controller. The admission controller automatically mutates the pod spec so that it can be scheduled by the Yunikorn Scheduler. The admission controller is also responsible for validating configuration additions and updates in the configmap.
28+
29+
The admission controller runs in a separate pod, it runs a [mutation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook)
30+
and a [validation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook), where:
31+
32+
1. The `mutation webhook` mutates pod spec by:
33+
- Adding `schedulerName: yunikorn`
34+
- By explicitly specifying the scheduler name, the pod will be scheduled by YuniKorn scheduler.
35+
- Adding `applicationId` label
36+
- When a label `applicationId` exists, reuse the given applicationId.
37+
- When a label `spark-app-selector` exists, reuse the given spark app ID.
38+
- Otherwise, assign a generated application ID for this pod, using convention: `yunikorn-<namespace>-autogen`. This is unique per namespace.
39+
- Adding `queue` label
40+
- When a label `queue` exists, reuse the given queue name. Note, if placement rule is enabled, values set in the label is ignored.
41+
- Otherwise, adds `queue: root.default`
42+
- Adding `disableStateAware` label
43+
- If pod was assigned a generated applicationId by the admission controller, also set `disableStateAware: true`. This causes the generated application to immediately transition from the `Starting` to `Running` state so that it will not block other applications.
44+
2. The `validation webhook` validates the configuration set in the configmap
45+
- This is used to prevent writing malformed configuration into the configmap.
46+
- The validation webhook calls scheduler [validation REST API](https://yunikorn.apache.org/docs/api/scheduler#configuration-validation) to validate configmap updates.
47+
48+
## Admission controller deployment
49+
By default, the admission controller is deployed as part of the YuniKorn Helm chart installation. This can be disabled if necessary (though not recommended) by setting the Helm parameter `embedAdmissionController` to `false`.
50+
51+
On startup, the admission controller performs a series of tasks to ensure that it is properly registered with Kubernetes:
52+
53+
1. Loads a Kubernetes secret called `admission-controller-secrets`. This secret stores a pair of CA certificates which are used to sign the TLS server certificate used by the admission controller.
54+
2. If the secret cannot be found or either CA certificate is within 90 days of expiration, generates new certificate(s). If a certificate is expiring, a new one is generated with an expiration of 12 months in the future. If both certificates are missing or expiring, the second certificate is generated with an expiration of 6 months in the future. This ensures that both certificates do not expire at the same time, and that there is an overlap of trusted certificates.
55+
3. If the CA certificates were created or updated, writes the secrets back to Kubernetes.
56+
4. Generates an ephemeral TLS server certificate signed by the CA certificate with the latest expiration date.
57+
5. Validates, and if necessary, creates or updates the Kubernetes webhook configurations named `yunikorn-admission-controller-validations` and `yunikorn-admission-controller-mutations`. If the CA certificates have changed, the webhooks will also be updated. These webhooks allow the Kubernetes API server to connect to the admission controller service to perform configmap validations and pod mutations.
58+
6. Starts up the admission controller HTTPS server.
59+
60+
Additionally, the admission controller also starts a background task to wait for CA certificates to expire. Once either certificate is expiring within the next 30 days, new CA and server certificates are generated, the webhook configurations are updated, and the HTTPS server is quickly restarted. This ensures that certificates rotate properly without downtime.
61+
62+
In production clusters, it is recommended to deploy the admission controller with two replicas by setting the Helm parameter `admissionController.replicaCount` to `2`. This will ensure that at least one admission controller webhook is reachable by the Kubernetes API server at all times. In this configuration, the CA certificates and webhook configurations are shared between the instances.
63+
64+
## Configure the admission controller
65+
The admission controller can be configured with the `yunikorn-configs` configmap.
66+
67+
Webhook configuration, all entries start with the prefix `admissionController.webHook.`.
68+
69+
| Variable | Default value | Description |
70+
|---------------------------|-----------------------------------------|----------------------------------------------------------------------------|
71+
| `amServiceName` | “yunikorn-addmission-controller-service”| name of the service that the YuniKorn admission controller is registered |
72+
| `schedulerServiceAddress` | “yunikorn-service:9080” | address of the YuniKorn scheduler service |
73+
74+
`amServiceName` is required for the admission controller to register itself properly with Kubernetes, and should normally not be changed.
75+
76+
`schedulerServiceAddress` must be reachable by the admission controller, and is used by the admission controller when validating ConfigMap changes. The admission controller will contact the REST API on the scheduler to validate any proposed ConfigMap changes. This setting should not normally be changed.
77+
78+
:::note NOTE
79+
A change to `amServiceName` and `amServiceName` requires a restart of the YuniKorn admission controller to take effect.
80+
:::
81+
Filtering configuration, all entries start with the prefix `admissionController.filtering.`.
82+
83+
84+
| Variable | Default value | Description |
85+
|----------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------|
86+
| `processNamespaces` | "" | which namespaces will have pods forwarded to YuniKorn for scheduling |
87+
| `bypassNamespaces` | “^kube-system$” | which namespaces will not have pods forwarded to YuniKorn for scheduling |
88+
| `labelNamespaces` | "" | which namespaces will have pods labeled with an applicationId |
89+
| `noLabelNamespaces` | "" | which namespaces will not have pods labeled with an applicationId |
90+
| `generateUniqueAppId`| "false" | Whether to generate unique applicationId for all the apps that do not have an applicationId to start with |
91+
| `defaultQueue` | "root.default" | Default queue name |
92+
If `processNamespaces` is setting is an empty string, pods created in all namespaces will be scheduled by YuniKorn, similarly, if `labelNamespaces` is setting is an empty string, all pods forwarded to YuniKorn will have an `applicationId` label applied.
93+
94+
`bypassNamespaces` and `noLabelNamespaces` act as exception lists for processNamespaces and labelNamespaces respectively.
95+
96+
If `generateUniqueAppId` is set to false, all the apps in a namespace should be bundled under a single applicationId, otherwise, the namespace and pod's uid will be used to generate a unique applicationId for it.
97+
98+
Access control configuration, all entries start with the prefix `admissionController.accessControl.`.
99+
100+
| Variable | Default value | Description |
101+
|--------------------|---------------------------------------|----------------------------------------------------------------------------|
102+
| `bypassAuth` | false | Allow any external user to create pods with user information set |
103+
| `trustControllers` | true | Allow Kubernetes controller users to create pods with user information set |
104+
| `systemUsers` | "^system:serviceaccount:kube-system:" | Regular expression for the allowed controller service account list |
105+
| `externalUsers` | "" | Regular expression for the allowed external user list |
106+
| `externalGroups` | "" | Regular expression for the allowed external group list |
107+
For more information about Access control configuration, please view [User & Group Resolution](user_guide/usergroup_resolution.md).
108+
109+
## Example
110+
The following YAML does not set any information related to yunikorn
111+
```yaml
112+
apiVersion: apps/v1
113+
kind: Deployment
114+
metadata:
115+
labels:
116+
app: nginx
117+
name: nginx
118+
spec:
119+
replicas: 1
120+
selector:
121+
matchLabels:
122+
app: nginx
123+
template:
124+
metadata:
125+
labels:
126+
app: nginx
127+
name: nginx
128+
spec:
129+
container:
130+
- name: nginx
131+
image: "nginx:stable-alpine
132+
resources:
133+
requests:
134+
cpu: "500m"
135+
memory: "1024M"
136+
```
137+
After deploying it to the cluster, use the following command to view the YAML of the deployed pod.
138+
```shell
139+
kubectl get pod <pod-name> -o yaml -n <namespace>
140+
```
141+
It was found that `schedulerName: yunikorn` and labels such as `queue: root.default` and `applicationId: yunikorn-default-autogen` were added.
142+
```yaml
143+
apiVersion: v1
144+
kind: Pod
145+
metadata:
146+
annotations:
147+
yunikorn.apache.org/allow-preemption: "true"
148+
yunikorn.apache.org/user.info: {...}
149+
labels:
150+
app: nginx
151+
applicationId: yunikorn-default-autogen
152+
disablesStateAware: "true"
153+
queuq: root.default
154+
spec:
155+
schedulerName: yunikorn
156+
```
157+
### Configure admission controller via yunikorn-configs
158+
An example configuration of yunikorn-configs.yaml
159+
```yaml
160+
apiVersion: v1
161+
kind: ConfigMap
162+
metadata:
163+
name: yunikorn-configs
164+
data:
165+
admissionController.webHook.amServiceName: "yunikorn-admission-controller-service"
166+
admissionController.webHook.schedulerServiceAddress: "yunikorn-service:9080"
167+
# Schedule only pods in spark-* and mpi-* namespaces with YuniKorn
168+
admissionController.filtering.processNamespaces: "^spark-,^mpi-"
169+
# Don't schedule pods in kube-system or fluentd-* namespaces
170+
admissionController.filtering.bypassNamespaces: "^kube-system$,^fluentd-"
171+
# Add applicationId labels to pods spark-* namespaces
172+
admissionController.filtering.labelNamespaces: "^spark-"
173+
# Don't add applicationId labels to pods noqueue namespaces
174+
admissionController.filtering.noLabelNamespaces: "^noqueue$"
175+
admissionController.filtering.generateUniqueAppId: "false"
176+
# Change default queue to root.mydefault
177+
admissionController.filtering.defaultQueue: "root.mydefault"
178+
admissionController.accessControl.bypassAuth: "false"
179+
admissionController.accessControl.trustControllers: "true"
180+
admissionController.accessControl.systemUsers: "^system:serviceaccount:kube-system:"
181+
admissionController.accessControl.externalUsers: ""
182+
admissionController.accessControl.externalGroups: ""
183+
queues.yaml: |
184+
partitions:
185+
- name: default
186+
placementrules:
187+
- name: tag
188+
value: namespace
189+
create: true
190+
queues:
191+
- name: root
192+
submitacl: '*'
193+
194+
```
195+
Use the following command to create ConfigMap to change settings.
196+
```shell script
197+
kubectl apply -f yunikorn-configs.yaml -n yunikorn
198+
```
199+
Use the following command to check whether the ConfigMap is created successfully.
200+
```shell script
201+
kubectl get ConfigMap -n yunikorn
202+
```
203+
Seeing that yunikorn-configs represents successful creation
204+
```shell script
205+
NAME DATA AGE
206+
yunikorn-configs 14 10s
207+
yunikorn-defaults 0 1d
208+
```

‎sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ module.exports = {
2222
],
2323
'User Guide': [
2424
'user_guide/deployment_modes',
25+
'user_guide/admission_controller',
2526
'user_guide/service_config',
2627
'user_guide/queue_config',
2728
'user_guide/placement_rules',

0 commit comments

Comments
 (0)