Skip to content

Commit ebf0c56

Browse files
nikParasyrnikParasyr
committed
Add PriorityQueue feature flag
Add support for controller-runtime PriorityQueue. To do this: - introduce feature flags on the provider - create new feature flag that when set enables the controller-runtime PriorityQueue - document feature flags and how to enable them - document the PriorityQueue feature flag with upstream links
1 parent a9d872e commit ebf0c56

File tree

8 files changed

+194
-2
lines changed

8 files changed

+194
-2
lines changed

‎config/manager/manager.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ spec:
2424
- "--v=2"
2525
- "--diagnostics-address=127.0.0.1:8080"
2626
- "--insecure-diagnostics=true"
27+
- "--feature-gates=PriorityQueue=${EXP_CAPO_PRIORITY_QUEUE:=false}"
2728
image: controller:latest
2829
imagePullPolicy: Always
2930
name: manager

‎docs/book/src/SUMMARY.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
- [Introduction](./introduction.md)
44
- [Getting Started](getting-started.md)
55
- [Configuration](clusteropenstack/configuration.md)
6+
- [Experimental Features](./experimental-features/experimental-features.md)
7+
- [PriorityQueue](./experimental-features/priority-queue.md)
68
- [Topics](./topics/index.md)
79
- [external cloud provider](./topics/external-cloud-provider.md)
810
- [hosted control plane](./topics/hosted-control-plane.md)

‎docs/book/src/experimental-features/experimental-features.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Experimental Features
2+
3+
CAPO now ships with experimental features the users can enable.
4+
5+
Currently CAPO has the following experimental features:
6+
* `PriorityQueue` (env var: `EXP_CAPO_PRIORITY_QUEUE`): [PriorityQueue](./priority-queue.md)
7+
8+
## Enabling Experimental Features for Management Clusters Started with clusterctl
9+
10+
Users can enable/disable features by setting OS environment variables before running `clusterctl init`, e.g.:
11+
12+
```yaml
13+
export EXP_SOME_FEATURE_NAME=true
14+
15+
clusterctl init --infrastructure openstack
16+
```
17+
18+
As an alternative to environment variables, it is also possible to set variables in the clusterctl config file located at `$XDG_CONFIG_HOME/cluster-api/clusterctl.yaml`, e.g.:
19+
```yaml
20+
# Values for environment variable substitution
21+
EXP_SOME_FEATURE_NAME: "true"
22+
```
23+
In case a variable is defined in both the config file and as an OS environment variable, the environment variable takes precedence.
24+
For more information on how to set variables for clusterctl, see [clusterctl Configuration File](https://cluster-api.sigs.k8s.io/clusterctl/configuration)
25+
26+
27+
## Enabling Experimental Features on Existing Management Clusters
28+
29+
To enable/disable features on existing management clusters, users can edit the controller manager
30+
deployments, which will then trigger a restart with the requested features. E.g:
31+
32+
```
33+
kubectl edit -n capo-system deployment.apps/capo-controller-manager
34+
```
35+
```
36+
// Enable/disable available features by modifying Args below.
37+
Args:
38+
--leader-elect
39+
--feature-gates=SomeFeature=true,OtherFeature=false
40+
```
41+
42+
Similarly, to **validate** if a particular feature is enabled, see the arguments by issuing:
43+
44+
```bash
45+
kubectl describe -n capo-system deployment.apps/capo-controller-manager
46+
```
47+
48+
## Enabling Experimental Features for e2e Tests
49+
50+
Features can be enabled by setting them as environmental variables before running e2e tests.
51+
52+
For `ci` this can also be done through updating `./test/e2e/data/e2e_conf.yaml`.
53+
54+
## Enabling Experimental Features on Tilt
55+
56+
On development environments started with `Tilt`, features can be enabled by setting the feature variables in `kustomize_substitutions`, e.g.:
57+
58+
```yaml
59+
kustomize_substitutions:
60+
EXP_CAPO_PRIORITY_QUEUE: 'true'
61+
```
62+
63+
For more details on setting up a development environment with `tilt`, see [Developing with Tilt](../development/development.md#developing-with-tilt)
64+

‎docs/book/src/experimental-features/priority-queue.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Priority Queue
2+
3+
The `PriorityQueue` feature flag enables the usage of the controller-runtime PriorityQueue.
4+
5+
This feature deprioritizes reconciliation of objects that were not edge-triggered (i.e. due to an create/update etc.) and makes the controller more responsive during full resyncs and controller startups.
6+
7+
More information on controller-runtime PriorityQueue:
8+
- [release-notes](https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.20.0)
9+
- [design docs](https://github.com/kubernetes-sigs/controller-runtime/pull/3013)
10+
- [tracking issue](https://github.com/kubernetes-sigs/controller-runtime/issues/2374)
11+
12+
## Enabling Priority Queue through clusterctl
13+
14+
To enable `PriorityQueue` when using clusterctl:
15+
16+
```yaml
17+
export EXP_CAPO_PRIORITY_QUEUE=true
18+
19+
clusterctl init --infrastructure openstack
20+
```
21+
22+
## Enabling Priority Queue directly
23+
24+
To enable `PriorityQueue` directly on the manager:
25+
26+
```
27+
kubectl edit -n capo-system deployment.apps/capo-controller-manager
28+
```
29+
```
30+
Args:
31+
--leader-elect
32+
--feature-gates=PriorityQueue=true
33+
```

‎feature/feature.go‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
/*
2+
Copyright 2022 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// Package feature handles feature gates.
18+
package feature
19+
20+
import (
21+
"k8s.io/apimachinery/pkg/util/runtime"
22+
"k8s.io/component-base/featuregate"
23+
)
24+
25+
const (
26+
// Every capo-specific feature gate should add method here following this template:
27+
//
28+
// // owner: @username
29+
// // alpha: v1.X
30+
// MyFeature featuregate.Feature = "MyFeature".
31+
32+
// PriorityQueue is a feature gate that controls if the controller uses the controller-runtime PriorityQueue
33+
// instead of the default queue implementation.
34+
//
35+
// alpha: v0.14
36+
PriorityQueue featuregate.Feature = "PriorityQueue"
37+
)
38+
39+
func init() {
40+
runtime.Must(MutableGates.Add(defaultCAPOFeatureGates))
41+
}
42+
43+
// defaultCAPOFeatureGates consists of all known capo-specific feature keys.
44+
// To add a new feature, define a key for it above and add it here.
45+
var defaultCAPOFeatureGates = map[featuregate.Feature]featuregate.FeatureSpec{
46+
// Every feature should be initiated here:
47+
PriorityQueue: {Default: false, PreRelease: featuregate.Alpha},
48+
}

‎feature/gates.go‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
/*
2+
Copyright 2022 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package feature
18+
19+
import (
20+
"k8s.io/component-base/featuregate"
21+
)
22+
23+
var (
24+
// MutableGates is a mutable version of DefaultFeatureGate.
25+
// Only top-level commands/options setup and the k8s.io/component-base/featuregate/testing package should make use of this.
26+
// Tests that need to modify featuregate gates for the duration of their test should use:
27+
// defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.<FeatureName>, <value>)()
28+
MutableGates = featuregate.NewFeatureGate()
29+
30+
// Gates is a shared global FeatureGate.
31+
// Top-level commands/options setup that needs to modify this featuregate gate should use DefaultMutableFeatureGate.
32+
Gates featuregate.FeatureGate = MutableGates
33+
)

‎main.go‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,15 @@ import (
3333
logsv1 "k8s.io/component-base/logs/api/v1"
3434
_ "k8s.io/component-base/logs/json/register"
3535
"k8s.io/klog/v2"
36+
"k8s.io/utils/ptr"
3637
clusterv1 "sigs.k8s.io/cluster-api/api/core/v1beta2"
3738
ipamv1 "sigs.k8s.io/cluster-api/api/ipam/v1beta2"
3839
"sigs.k8s.io/cluster-api/util/flags"
3940
ctrl "sigs.k8s.io/controller-runtime"
4041
cache "sigs.k8s.io/controller-runtime/pkg/cache"
4142
client "sigs.k8s.io/controller-runtime/pkg/client"
42-
"sigs.k8s.io/controller-runtime/pkg/client/config"
43+
clientconfig "sigs.k8s.io/controller-runtime/pkg/client/config"
44+
"sigs.k8s.io/controller-runtime/pkg/config"
4345
"sigs.k8s.io/controller-runtime/pkg/controller"
4446
"sigs.k8s.io/controller-runtime/pkg/webhook"
4547

@@ -48,6 +50,7 @@ import (
4850
infrav1alpha1 "sigs.k8s.io/cluster-api-provider-openstack/api/v1alpha1"
4951
infrav1 "sigs.k8s.io/cluster-api-provider-openstack/api/v1beta1"
5052
"sigs.k8s.io/cluster-api-provider-openstack/controllers"
53+
"sigs.k8s.io/cluster-api-provider-openstack/feature"
5154
"sigs.k8s.io/cluster-api-provider-openstack/pkg/metrics"
5255
"sigs.k8s.io/cluster-api-provider-openstack/pkg/record"
5356
"sigs.k8s.io/cluster-api-provider-openstack/pkg/scope"
@@ -167,6 +170,8 @@ func InitFlags(fs *pflag.FlagSet) {
167170
fs.IntVar(&scopeCacheMaxSize, "scope-cache-max-size", 10, "The maximum credentials count the operator should keep in cache. Setting this value to 0 means no cache.")
168171

169172
fs.BoolVar(&showVersion, "version", false, "Show current version and exit.")
173+
174+
feature.MutableGates.AddFlag(fs)
170175
}
171176

172177
// Add RBAC for the authorized diagnostics endpoint.
@@ -199,7 +204,7 @@ func main() {
199204
}()
200205
}
201206

202-
cfg, err := config.GetConfigWithContext(os.Getenv("KUBECONTEXT"))
207+
cfg, err := clientconfig.GetConfigWithContext(os.Getenv("KUBECONTEXT"))
203208
if err != nil {
204209
setupLog.Error(err, "unable to get kubeconfig")
205210
os.Exit(1)
@@ -229,6 +234,8 @@ func main() {
229234
}
230235
}
231236

237+
setupLog.Info(fmt.Sprintf("Feature gates: %+v\n", feature.Gates))
238+
232239
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
233240
Scheme: scheme,
234241
Metrics: *metricsOpts,
@@ -258,6 +265,9 @@ func main() {
258265
),
259266
HealthProbeBindAddress: healthAddr,
260267
LeaderElectionReleaseOnCancel: true,
268+
Controller: config.Controller{
269+
UsePriorityQueue: ptr.To[bool](feature.Gates.Enabled(feature.PriorityQueue)),
270+
},
261271
})
262272
if err != nil {
263273
setupLog.Error(err, "unable to start manager")

‎test/e2e/data/e2e_conf.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,7 @@ variables:
219219
CNI: "../../data/cni/calico.yaml"
220220
CCM: "../../data/ccm/cloud-controller-manager.yaml"
221221
EXP_CLUSTER_RESOURCE_SET: "true"
222+
EXP_CAPO_PRIORITY_QUEUE: "false"
222223
IP_FAMILY: "ipv4"
223224
OPENSTACK_BASTION_IMAGE_NAME: "cirros-0.6.1-x86_64-disk"
224225
OPENSTACK_BASTION_IMAGE_URL: https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/cirros/2022-12-05/cirros-0.6.1-x86_64-disk.img

0 commit comments

Comments
 (0)