Skip to content

Commit 24fee53

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #2823 from nikParasyr/issue-2820
✨ Add PriorityQueue feature flag
2 parents 33d80a2 + 2d8d6f2 commit 24fee53

File tree

8 files changed

+185
-2
lines changed

8 files changed

+185
-2
lines changed

‎config/manager/manager.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ spec:
2424
- "--v=2"
2525
- "--diagnostics-address=127.0.0.1:8080"
2626
- "--insecure-diagnostics=true"
27+
- "--feature-gates=PriorityQueue=${EXP_CAPO_PRIORITY_QUEUE:=false}"
2728
image: controller:latest
2829
imagePullPolicy: Always
2930
name: manager

‎docs/book/src/SUMMARY.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
- [Introduction](./introduction.md)
44
- [Getting Started](getting-started.md)
55
- [Configuration](clusteropenstack/configuration.md)
6+
- [Experimental Features](./experimental-features/experimental-features.md)
7+
- [PriorityQueue](./experimental-features/priority-queue.md)
68
- [Topics](./topics/index.md)
79
- [external cloud provider](./topics/external-cloud-provider.md)
810
- [hosted control plane](./topics/hosted-control-plane.md)

‎docs/book/src/experimental-features/experimental-features.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Experimental Features
2+
3+
CAPO now ships with experimental features the users can enable.
4+
5+
Currently CAPO has the following experimental features:
6+
* `PriorityQueue` (env var: `EXP_CAPO_PRIORITY_QUEUE`): [PriorityQueue](./priority-queue.md)
7+
8+
## Enabling Experimental Features for Management Clusters Started with clusterctl
9+
10+
Users can enable/disable features by setting OS environment variables before running `clusterctl init`, e.g.:
11+
12+
```yaml
13+
export EXP_SOME_FEATURE_NAME=true
14+
15+
clusterctl init --infrastructure openstack
16+
```
17+
18+
As an alternative to environment variables, it is also possible to set variables in the clusterctl config file located at `$XDG_CONFIG_HOME/cluster-api/clusterctl.yaml`, e.g.:
19+
```yaml
20+
# Values for environment variable substitution
21+
EXP_SOME_FEATURE_NAME: "true"
22+
```
23+
In case a variable is defined in both the config file and as an OS environment variable, the environment variable takes precedence.
24+
For more information on how to set variables for clusterctl, see [clusterctl Configuration File](https://cluster-api.sigs.k8s.io/clusterctl/configuration)
25+
26+
27+
## Enabling Experimental Features on Existing Management Clusters
28+
29+
To enable/disable features on existing management clusters, users can edit the controller manager
30+
deployments, which will then trigger a restart with the requested features. E.g:
31+
32+
```
33+
kubectl edit -n capo-system deployment.apps/capo-controller-manager
34+
```
35+
```
36+
// Enable/disable available features by modifying Args below.
37+
spec:
38+
template:
39+
spec:
40+
containers:
41+
- args:
42+
- --leader-elect
43+
- --feature-gates=SomeFeature=true,OtherFeature=false
44+
```
45+
46+
Similarly, to **validate** if a particular feature is enabled, see the arguments by issuing:
47+
48+
```bash
49+
kubectl describe -n capo-system deployment.apps/capo-controller-manager
50+
```
51+
52+
## Enabling Experimental Features for e2e Tests
53+
54+
Features can be enabled by setting them as environmental variables before running e2e tests.
55+
56+
For `ci` this can also be done through updating `./test/e2e/data/e2e_conf.yaml`.
57+
58+
## Enabling Experimental Features on Tilt
59+
60+
On development environments started with `Tilt`, features can be enabled by setting the feature variables in `kustomize_substitutions`, e.g.:
61+
62+
```yaml
63+
kustomize_substitutions:
64+
EXP_CAPO_PRIORITY_QUEUE: 'true'
65+
```
66+
67+
For more details on setting up a development environment with `tilt`, see [Developing with Tilt](../development/development.md#developing-with-tilt)
68+

‎docs/book/src/experimental-features/priority-queue.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Priority Queue
2+
3+
> **Note:** PriorityQueue is available in >= 0.14
4+
5+
The `PriorityQueue` feature flag enables the usage of the controller-runtime PriorityQueue.
6+
7+
This feature deprioritizes reconciliation of objects that were not edge-triggered (i.e. due to an create/update etc.) and makes the controller more responsive during full resyncs and controller startups.
8+
9+
More information on controller-runtime PriorityQueue:
10+
- [release-notes](https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.20.0)
11+
- [design docs](https://github.com/kubernetes-sigs/controller-runtime/pull/3013)
12+
- [tracking issue](https://github.com/kubernetes-sigs/controller-runtime/issues/2374)
13+
14+
## Enabling Priority Queue
15+
16+
You can enable `PriorityQueue` using the following.
17+
18+
- Environment variable: `EXP_CAPO_PRIORITY_QUEUE=true`
19+
- clusterctl.yaml variable: `EXP_CAPO_PRIORITY_QUEUE: true`
20+
- --feature-gates argument: `PriorityQueue=true`

‎feature/feature.go‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
/*
2+
Copyright 2022 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// Package feature handles feature gates.
18+
package feature
19+
20+
import (
21+
"k8s.io/apimachinery/pkg/util/runtime"
22+
"k8s.io/component-base/featuregate"
23+
)
24+
25+
const (
26+
// Every capo-specific feature gate should add method here following this template:
27+
//
28+
// // owner: @username
29+
// // alpha: v1.X
30+
// MyFeature featuregate.Feature = "MyFeature".
31+
32+
// PriorityQueue is a feature gate that controls if the controller uses the controller-runtime PriorityQueue
33+
// instead of the default queue implementation.
34+
//
35+
// alpha: v0.14
36+
PriorityQueue featuregate.Feature = "PriorityQueue"
37+
)
38+
39+
func init() {
40+
runtime.Must(MutableGates.Add(defaultCAPOFeatureGates))
41+
}
42+
43+
// defaultCAPOFeatureGates consists of all known capo-specific feature keys.
44+
// To add a new feature, define a key for it above and add it here.
45+
var defaultCAPOFeatureGates = map[featuregate.Feature]featuregate.FeatureSpec{
46+
// Every feature should be initiated here:
47+
PriorityQueue: {Default: false, PreRelease: featuregate.Alpha},
48+
}

‎feature/gates.go‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
/*
2+
Copyright 2022 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package feature
18+
19+
import (
20+
"k8s.io/component-base/featuregate"
21+
)
22+
23+
var (
24+
// MutableGates is a mutable version of DefaultFeatureGate.
25+
// Only top-level commands/options setup and the k8s.io/component-base/featuregate/testing package should make use of this.
26+
// Tests that need to modify featuregate gates for the duration of their test should use:
27+
// defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.<FeatureName>, <value>)()
28+
MutableGates = featuregate.NewFeatureGate()
29+
30+
// Gates is a shared global FeatureGate.
31+
// Top-level commands/options setup that needs to modify this featuregate gate should use DefaultMutableFeatureGate.
32+
Gates featuregate.FeatureGate = MutableGates
33+
)

‎main.go‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,14 +34,16 @@ import (
3434
logsv1 "k8s.io/component-base/logs/api/v1"
3535
_ "k8s.io/component-base/logs/json/register"
3636
"k8s.io/klog/v2"
37+
"k8s.io/utils/ptr"
3738
clusterv1 "sigs.k8s.io/cluster-api/api/core/v1beta2"
3839
ipamv1 "sigs.k8s.io/cluster-api/api/ipam/v1beta2"
3940
"sigs.k8s.io/cluster-api/controllers/crdmigrator"
4041
"sigs.k8s.io/cluster-api/util/flags"
4142
ctrl "sigs.k8s.io/controller-runtime"
4243
cache "sigs.k8s.io/controller-runtime/pkg/cache"
4344
client "sigs.k8s.io/controller-runtime/pkg/client"
44-
"sigs.k8s.io/controller-runtime/pkg/client/config"
45+
clientconfig "sigs.k8s.io/controller-runtime/pkg/client/config"
46+
"sigs.k8s.io/controller-runtime/pkg/config"
4547
"sigs.k8s.io/controller-runtime/pkg/controller"
4648
"sigs.k8s.io/controller-runtime/pkg/webhook"
4749

@@ -50,6 +52,7 @@ import (
5052
infrav1alpha1 "sigs.k8s.io/cluster-api-provider-openstack/api/v1alpha1"
5153
infrav1 "sigs.k8s.io/cluster-api-provider-openstack/api/v1beta1"
5254
"sigs.k8s.io/cluster-api-provider-openstack/controllers"
55+
"sigs.k8s.io/cluster-api-provider-openstack/feature"
5356
"sigs.k8s.io/cluster-api-provider-openstack/pkg/metrics"
5457
"sigs.k8s.io/cluster-api-provider-openstack/pkg/record"
5558
"sigs.k8s.io/cluster-api-provider-openstack/pkg/scope"
@@ -173,6 +176,8 @@ func InitFlags(fs *pflag.FlagSet) {
173176
fs.StringArrayVar(&skipCRDMigrationPhases, "skip-crd-migration-phases", []string{},
174177
"List of CRD migration phases to skip. Valid values are: StorageVersionMigration, CleanupManagedFields.")
175178
fs.BoolVar(&showVersion, "version", false, "Show current version and exit.")
179+
180+
feature.MutableGates.AddFlag(fs)
176181
}
177182

178183
// Add RBAC for the authorized diagnostics endpoint.
@@ -210,7 +215,7 @@ func main() {
210215
}()
211216
}
212217

213-
cfg, err := config.GetConfigWithContext(os.Getenv("KUBECONTEXT"))
218+
cfg, err := clientconfig.GetConfigWithContext(os.Getenv("KUBECONTEXT"))
214219
if err != nil {
215220
setupLog.Error(err, "unable to get kubeconfig")
216221
os.Exit(1)
@@ -240,6 +245,8 @@ func main() {
240245
}
241246
}
242247

248+
setupLog.Info(fmt.Sprintf("Feature gates: %+v\n", feature.Gates))
249+
243250
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
244251
Scheme: scheme,
245252
Metrics: *metricsOpts,
@@ -269,6 +276,9 @@ func main() {
269276
),
270277
HealthProbeBindAddress: healthAddr,
271278
LeaderElectionReleaseOnCancel: true,
279+
Controller: config.Controller{
280+
UsePriorityQueue: ptr.To[bool](feature.Gates.Enabled(feature.PriorityQueue)),
281+
},
272282
})
273283
if err != nil {
274284
setupLog.Error(err, "unable to start manager")

‎test/e2e/data/e2e_conf.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,7 @@ variables:
219219
CNI: "../../data/cni/calico.yaml"
220220
CCM: "../../data/ccm/cloud-controller-manager.yaml"
221221
EXP_CLUSTER_RESOURCE_SET: "true"
222+
EXP_CAPO_PRIORITY_QUEUE: "false"
222223
IP_FAMILY: "ipv4"
223224
OPENSTACK_BASTION_IMAGE_NAME: "cirros-0.6.1-x86_64-disk"
224225
OPENSTACK_BASTION_IMAGE_URL: https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/cirros/2022-12-05/cirros-0.6.1-x86_64-disk.img

0 commit comments

Comments
 (0)