Skip to content

Commit a929235

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #8545 from drmorr0/drmorr/fix-kwok-annotation-key
change kwok nodegroup annotation key, and update documentation
2 parents 20f76e9 + cd2db00 commit a929235

File tree

6 files changed

+31
-31
lines changed

6 files changed

+31
-31
lines changed

‎cluster-autoscaler/cloudprovider/kwok/README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -81,13 +81,13 @@ kubectl create configmap kwok-provider-templates --from-file=templates=template-
8181
```
8282
Replace `template-nodes.yaml` with the path to your template nodes file.
8383

84-
If you are using your template nodes in the `kwok-provider-templates` ConfigMap, make sure you have set the correct value for `nodegroups.fromNodeLabelKey`/`nodegroups.fromNodeAnnotation`. Not doing so will make CA not scale up nodes (it won't throw any error either).
84+
If you are using your template nodes in the `kwok-provider-templates` ConfigMap, make sure you have set the correct value for `nodegroups.fromNodeLabelKey`/`nodegroups.fromNodeAnnotationKey`. Not doing so will make CA not scale up nodes (it won't throw any error either).
8585

8686
If you want to use dynamic template nodes,
8787

8888
Set `readNodesFrom` in `kwok-provider-config` ConfigMap to `cluster`. This tells the kwok provider to use live nodes from the cluster as template nodes.
8989

90-
If you are using live nodes from the cluster as template nodes in the `kwok-provider-templates` ConfigMap, make sure you have set the correct value for `nodegroups.fromNodeLabelKey`/`nodegroups.fromNodeAnnotation`. Not doing so will make CA not scale up nodes (it won't throw any error either).
90+
If you are using live nodes from the cluster as template nodes in the `kwok-provider-templates` ConfigMap, make sure you have set the correct value for `nodegroups.fromNodeLabelKey`/`nodegroups.fromNodeAnnotationKey`. Not doing so will make CA not scale up nodes (it won't throw any error either).
9191

9292
### For local development
9393
1. Point your kubeconfig to the cluster where you want to test your changes
@@ -158,11 +158,11 @@ nodegroups:
158158
# nodegroup2: [node2]
159159
fromNodeLabelKey: "node.kubernetes.io/instance-type"
160160

161-
# fromNodeAnnotation's value is used to group nodes together into nodegroups
161+
# fromNodeAnnotationKey's value is used to group nodes together into nodegroups
162162
# (basically same as `fromNodeLabelKey` except based on annotation)
163-
# you can specify either of `fromNodeLabelKey` OR `fromNodeAnnotation`
163+
# you can specify either of `fromNodeLabelKey` OR `fromNodeAnnotationKey`
164164
# (both are not allowed)
165-
fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
165+
fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
166166
# nodes specifies node level config
167167
nodes:
168168
# skipTaint is used to enable/disable adding kwok provider taint on the template nodes
@@ -211,7 +211,7 @@ kwok provider config is a configuration to change the behavior of the kwok provi
211211

212212
### Gotchas
213213
1. The kwok provider by default taints the template nodes with `kwok-provider: true` taint so that production workloads don't get scheduled on these nodes accidentally. You have to tolerate the taint to schedule your workload on the nodes created by the kwok provider. You can turn this off by setting `nodes.skipTaint: true` in the kwok provider config.
214-
2. Make sure the label/annotation for `fromNodeLabelKey`/`fromNodeAnnotation` in the kwok provider config is actually present on the template nodes. If it isn't present on the template nodes, the kwok provider will not be able to create new nodes.
214+
2. Make sure the label/annotation for `fromNodeLabelKey`/`fromNodeAnnotationKey` in the kwok provider config is actually present on the template nodes. If it isn't present on the template nodes, the kwok provider will not be able to create new nodes.
215215
3. Note that the kwok provider makes the following changes to all the template nodes:
216216
(pseudocode)
217217
```

‎cluster-autoscaler/cloudprovider/kwok/kwok_config.go‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -111,20 +111,20 @@ func LoadConfigFile(kubeClient kubeclient.Interface) (*KwokProviderConfig, error
111111
}
112112

113113
if strings.TrimSpace(kwokConfig.Nodegroups.FromNodeLabelKey) == "" &&
114-
strings.TrimSpace(kwokConfig.Nodegroups.FromNodeLabelAnnotation) == "" {
115-
return nil, fmt.Errorf("please specify either 'nodegroups.fromNodeLabelKey' or 'nodegroups.fromNodeAnnotation' in kwok provider config (currently empty or undefined)")
114+
strings.TrimSpace(kwokConfig.Nodegroups.FromNodeAnnotationKey) == "" {
115+
return nil, fmt.Errorf("please specify either 'nodegroups.fromNodeLabelKey' or 'nodegroups.fromNodeAnnotationKey' in kwok provider config (currently empty or undefined)")
116116
}
117117
if strings.TrimSpace(kwokConfig.Nodegroups.FromNodeLabelKey) != "" &&
118-
strings.TrimSpace(kwokConfig.Nodegroups.FromNodeLabelAnnotation) != "" {
119-
return nil, fmt.Errorf("please specify either 'nodegroups.fromNodeLabelKey' or 'nodegroups.fromNodeAnnotation' in kwok provider config (you can't use both)")
118+
strings.TrimSpace(kwokConfig.Nodegroups.FromNodeAnnotationKey) != "" {
119+
return nil, fmt.Errorf("please specify either 'nodegroups.fromNodeLabelKey' or 'nodegroups.fromNodeAnnotationKey' in kwok provider config (you can't use both)")
120120
}
121121

122122
if strings.TrimSpace(kwokConfig.Nodegroups.FromNodeLabelKey) != "" {
123123
kwokConfig.status.groupNodesBy = groupNodesByLabel
124124
kwokConfig.status.key = kwokConfig.Nodegroups.FromNodeLabelKey
125125
} else {
126126
kwokConfig.status.groupNodesBy = groupNodesByAnnotation
127-
kwokConfig.status.key = kwokConfig.Nodegroups.FromNodeLabelAnnotation
127+
kwokConfig.status.key = kwokConfig.Nodegroups.FromNodeAnnotationKey
128128
}
129129

130130
if kwokConfig.Nodes == nil {

‎cluster-autoscaler/cloudprovider/kwok/kwok_config_test.go‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,9 @@ nodegroups:
4949
# nodegroup1: [node1,node3]
5050
# nodegroup2: [node2]
5151
fromNodeLabelKey: "kwok-nodegroup"
52-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
52+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
5353
# (both are not allowed)
54-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
54+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
5555
nodes:
5656
gpuConfig:
5757
# to tell kwok provider what label should be considered as GPU label
@@ -77,9 +77,9 @@ nodegroups:
7777
# nodegroup1: [node1,node3]
7878
# nodegroup2: [node2]
7979
fromNodeLabelKey: "kwok-nodegroup"
80-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
80+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
8181
# (both are not allowed)
82-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
82+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
8383
nodes:
8484
skipTaint: true
8585
gpuConfig:
@@ -104,9 +104,9 @@ nodegroups:
104104
# nodegroup1: [node1,node3]
105105
# nodegroup2: [node2]
106106
fromNodeLabelKey: "kwok-nodegroup"
107-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
107+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
108108
# (both are not allowed)
109-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
109+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
110110
nodes:
111111
gpuConfig:
112112
# to tell kwok provider what label should be considered as GPU label
@@ -131,9 +131,9 @@ nodegroups:
131131
# nodegroup1: [node1,node3]
132132
# nodegroup2: [node2]
133133
fromNodeLabelKey: "kwok-nodegroup"
134-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
134+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
135135
# (both are not allowed)
136-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
136+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
137137
nodes:
138138
skipTaint: true
139139
gpuConfig:
@@ -159,9 +159,9 @@ nodegroups:
159159
# nodegroup1: [node1,node3]
160160
# nodegroup2: [node2]
161161
fromNodeLabelKey: "node.kubernetes.io/instance-type"
162-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
162+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
163163
# (both are not allowed)
164-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
164+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
165165
nodes:
166166
gpuConfig:
167167
# to tell kwok provider what label should be considered as GPU label
@@ -185,9 +185,9 @@ nodegroups:
185185
# nodegroup1: [node1,node3]
186186
# nodegroup2: [node2]
187187
fromNodeLabelKey: "node.kubernetes.io/instance-type"
188-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
188+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
189189
# (both are not allowed)
190-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
190+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
191191
nodes:
192192
gpuConfig:
193193
# to tell kwok provider what label should be considered as GPU label
@@ -213,9 +213,9 @@ nodegroups:
213213
# nodegroup1: [node1,node3]
214214
# nodegroup2: [node2]
215215
fromNodeLabelKey: "node.kubernetes.io/instance-type"
216-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
216+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
217217
# (both are not allowed)
218-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
218+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
219219
nodes:
220220
gpuConfig:
221221
# to tell kwok provider what label should be considered as GPU label

‎cluster-autoscaler/cloudprovider/kwok/kwok_types.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,8 +63,8 @@ type NodeGroup struct {
6363

6464
// NodegroupsConfig defines options for creating nodegroups
6565
type NodegroupsConfig struct {
66-
FromNodeLabelKey string `json:"fromNodeLabelKey" yaml:"fromNodeLabelKey"`
67-
FromNodeLabelAnnotation string `json:"fromNodeLabelAnnotation" yaml:"fromNodeLabelAnnotation"`
66+
FromNodeLabelKey string `json:"fromNodeLabelKey" yaml:"fromNodeLabelKey"`
67+
FromNodeAnnotationKey string `json:"fromNodeAnnotationKey" yaml:"fromNodeAnnotationKey"`
6868
}
6969

7070
// NodeConfig defines config options for the nodes

‎cluster-autoscaler/cloudprovider/kwok/samples/dynamic_nodegroups_config.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ nodegroups:
99
# nodegroup1: [node1,node3]
1010
# nodegroup2: [node2]
1111
fromNodeLabelKey: "node.kubernetes.io/instance-type"
12-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
12+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
1313
# (both are not allowed)
14-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
14+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
1515

1616
nodes:
1717
# kwok provider adds a taint on the template nodes

‎cluster-autoscaler/cloudprovider/kwok/samples/static_nodegroups_config.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ nodegroups:
99
# nodegroup1: [node1,node3]
1010
# nodegroup2: [node2]
1111
fromNodeLabelKey: "node.kubernetes.io/instance-type"
12-
# you can either specify fromNodeLabelKey OR fromNodeAnnotation
12+
# you can either specify fromNodeLabelKey OR fromNodeAnnotationKey
1313
# (both are not allowed)
14-
# fromNodeAnnotation: "eks.amazonaws.com/nodegroup"
14+
# fromNodeAnnotationKey: "eks.amazonaws.com/nodegroup"
1515
nodes:
1616
# kwok provider adds a taint on the template nodes
1717
# so that even if you run the provider in a production cluster

0 commit comments

Comments
 (0)