Skip to content

Commit c8e78e9

Browse files
PhilippMatthesPhilippMatthes
authored
Use hypervisor CRD for filtering (#441)
## Background For virtual machines spawned on the kvm hypervisor, we want to no longer use nova and placement as source of truth. Instead, filters should use the hypervisor crd exposed by the [hypervisor operator](github.com/cobaltcore-dev/openstack-hypervisor-operator) and populated by the [node agent](https://github.com/cobaltcore-dev/kvm-node-agent). This contribution replaces the implementation of all filters that were originally ported from nova accordingly. Afterward, we can disable filters in nova one-by-one, moving the compute placement logic over to cortex. > [!TIP] > You can use the newly added [mirror tool](93fdcc0) to mirror hypervisor resources from our compute cluster over to the local cluster. ## Completion - [x] ~internal/scheduling/decisions/nova/plugins/filters/filter_compute_capabilities.go~ (REMOVED) - [x] internal/scheduling/decisions/nova/plugins/filters/filter_capabilities.go (NEW) - [x] internal/scheduling/decisions/nova/plugins/filters/filter_correct_az.go - [x] internal/scheduling/decisions/nova/plugins/filters/filter_external_customer.go - [x] internal/scheduling/decisions/nova/plugins/filters/filter_has_accelerators.go - [x] internal/scheduling/decisions/nova/plugins/filters/filter_has_enough_capacity.go - [x] internal/scheduling/decisions/nova/plugins/filters/filter_has_requested_traits.go - [x] internal/scheduling/decisions/nova/plugins/filters/filter_host_instructions.go - [x] internal/scheduling/decisions/nova/plugins/filters/filter_maintenance.go (NEW) - [x] internal/scheduling/decisions/nova/plugins/filters/filter_packed_virtqueue.go - [x] ~internal/scheduling/decisions/nova/plugins/filters/filter_project_aggregates.go~ (REMOVED) - [x] internal/scheduling/decisions/nova/plugins/filters/filter_allowed_projects.go (NEW) - [x] ~internal/scheduling/decisions/nova/plugins/filters/filter_disabled.go~ (REMOVED) - [x] internal/scheduling/decisions/nova/plugins/filters/filter_status_conditions.go (NEW) ## Dependencies > [!NOTE] > The scope of this PR is to establish a minimum viable scheduling pipeline with the current state. Extensive refactorings, for example of the filter for requested traits, are out of scope. Hypervisor operator PR: cobaltcore-dev/openstack-hypervisor-operator#217 KVM node agent PR: cobaltcore-dev/kvm-node-agent#40
1 parent 9fc24b1 commit c8e78e9

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+3348
-2258
lines changed

‎api/v1alpha1/step_types.go‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -60,18 +60,6 @@ type StepSpec struct {
6060
// and decisions made by it.
6161
// +kubebuilder:validation:Optional
6262
Description string `json:"description,omitempty"`
63-
64-
// If needed, database credentials for fetching data from the database.
65-
// The secret should contain the following keys:
66-
// - "username": The database username.
67-
// - "password": The database password.
68-
// - "host": The database host.
69-
// - "port": The database port.
70-
// - "database": The database name.
71-
// Note: this field will be removed in the future when db access in scheduler
72-
// steps is no longer needed.
73-
// +kubebuilder:validation:Optional
74-
DatabaseSecretRef *corev1.SecretReference `json:"databaseSecretRef"`
7563
}
7664

7765
const (

‎api/v1alpha1/zz_generated.deepcopy.go‎

Lines changed: 0 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎cmd/main.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ import (
5959
"github.com/cobaltcore-dev/cortex/pkg/monitoring"
6060
"github.com/cobaltcore-dev/cortex/pkg/multicluster"
6161
"github.com/cobaltcore-dev/cortex/pkg/task"
62+
hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1"
6263
"github.com/sapcc/go-bits/httpext"
6364
"github.com/sapcc/go-bits/must"
6465
corev1 "k8s.io/api/core/v1"
@@ -76,6 +77,7 @@ func init() {
7677
utilruntime.Must(v1alpha1.AddToScheme(scheme))
7778
utilruntime.Must(ironcorev1alpha1.AddToScheme(scheme))
7879
utilruntime.Must(corev1.AddToScheme(scheme))
80+
utilruntime.Must(hv1.AddToScheme(scheme))
7981
// +kubebuilder:scaffold:scheme
8082
}
8183

‎config/crd/bases/cortex.cloud_steps.yaml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -55,28 +55,6 @@ spec:
5555
spec:
5656
description: spec defines the desired state of Step
5757
properties:
58-
databaseSecretRef:
59-
description: |-
60-
If needed, database credentials for fetching data from the database.
61-
The secret should contain the following keys:
62-
- "username": The database username.
63-
- "password": The database password.
64-
- "host": The database host.
65-
- "port": The database port.
66-
- "database": The database name.
67-
Note: this field will be removed in the future when db access in scheduler
68-
steps is no longer needed.
69-
properties:
70-
name:
71-
description: name is unique within a namespace to reference a
72-
secret resource.
73-
type: string
74-
namespace:
75-
description: namespace defines the space within which the secret
76-
name must be unique.
77-
type: string
78-
type: object
79-
x-kubernetes-map-type: atomic
8058
description:
8159
description: |-
8260
Additional description of the step which helps understand its purpose

‎config/crd/cortex.cloud_steps.yaml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -55,28 +55,6 @@ spec:
5555
spec:
5656
description: spec defines the desired state of Step
5757
properties:
58-
databaseSecretRef:
59-
description: |-
60-
If needed, database credentials for fetching data from the database.
61-
The secret should contain the following keys:
62-
- "username": The database username.
63-
- "password": The database password.
64-
- "host": The database host.
65-
- "port": The database port.
66-
- "database": The database name.
67-
Note: this field will be removed in the future when db access in scheduler
68-
steps is no longer needed.
69-
properties:
70-
name:
71-
description: name is unique within a namespace to reference a
72-
secret resource.
73-
type: string
74-
namespace:
75-
description: namespace defines the space within which the secret
76-
name must be unique.
77-
type: string
78-
type: object
79-
x-kubernetes-map-type: atomic
8058
description:
8159
description: |-
8260
Additional description of the step which helps understand its purpose

‎dist/chart/templates/crd/cortex.cloud_steps.yaml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -61,28 +61,6 @@ spec:
6161
spec:
6262
description: spec defines the desired state of Step
6363
properties:
64-
databaseSecretRef:
65-
description: |-
66-
If needed, database credentials for fetching data from the database.
67-
The secret should contain the following keys:
68-
- "username": The database username.
69-
- "password": The database password.
70-
- "host": The database host.
71-
- "port": The database port.
72-
- "database": The database name.
73-
Note: this field will be removed in the future when db access in scheduler
74-
steps is no longer needed.
75-
properties:
76-
name:
77-
description: name is unique within a namespace to reference a
78-
secret resource.
79-
type: string
80-
namespace:
81-
description: namespace defines the space within which the secret
82-
name must be unique.
83-
type: string
84-
type: object
85-
x-kubernetes-map-type: atomic
8664
description:
8765
description: |-
8866
Additional description of the step which helps understand its purpose

‎dist/chart/templates/rbac/hypervisor_role.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{{- if .Values.rbac.hypervisor.enable }}
2+
---
3+
apiVersion: rbac.authorization.k8s.io/v1
4+
kind: ClusterRole
5+
metadata:
6+
labels:
7+
{{- include "chart.labels" . | nindent 4 }}
8+
name: {{ .Values.namePrefix }}-manager-role-hypervisor
9+
rules:
10+
- apiGroups:
11+
- kvm.cloud.sap
12+
resources:
13+
- hypervisors
14+
verbs:
15+
- get
16+
- list
17+
- watch
18+
- apiGroups:
19+
- kvm.cloud.sap
20+
resources:
21+
- hypervisors/status
22+
verbs:
23+
- get
24+
{{- end -}}

‎dist/chart/templates/rbac/hypervisor_role_binding.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{{- if .Values.rbac.hypervisor.enable }}
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
kind: ClusterRoleBinding
4+
metadata:
5+
labels:
6+
{{- include "chart.labels" . | nindent 4 }}
7+
name: {{ .Values.namePrefix }}-manager-rolebinding-hypervisor
8+
roleRef:
9+
apiGroup: rbac.authorization.k8s.io
10+
kind: ClusterRole
11+
name: {{ .Values.namePrefix }}-manager-role-hypervisor
12+
subjects:
13+
- kind: ServiceAccount
14+
name: {{ .Values.namePrefix }}-{{ .Values.controllerManager.serviceAccountName }}
15+
namespace: {{ .Release.Namespace }}
16+
{{- end -}}

‎dist/chart/values.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,10 @@ rbac:
5555
enable: false
5656
pods:
5757
enable: false
58+
# Whether hypervisor operator/crd related roles should be deployed.
59+
# See: https://github.com/cobaltcore-dev/openstack-hypervisor-operator
60+
hypervisor:
61+
enable: false
5862

5963
# [CRDs]: To enable the CRDs
6064
crd:

‎go.mod‎

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ module github.com/cobaltcore-dev/cortex
33
go 1.25.0
44

55
require (
6+
github.com/cobaltcore-dev/openstack-hypervisor-operator v0.0.0-20251230105055-37950dd7ff29
67
github.com/go-gorp/gorp v2.2.0+incompatible
78
github.com/gophercloud/gophercloud/v2 v2.9.0
89
github.com/ironcore-dev/ironcore v0.2.4
@@ -20,27 +21,28 @@ require (
2021
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
2122
github.com/Microsoft/go-winio v0.6.2 // indirect
2223
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
23-
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
24+
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
2425
github.com/beorn7/perks v1.0.1 // indirect
2526
github.com/blang/semver/v4 v4.0.0 // indirect
2627
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
27-
github.com/cenkalti/backoff/v5 v5.0.2 // indirect
28+
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
2829
github.com/cespare/xxhash/v2 v2.3.0 // indirect
2930
github.com/containerd/continuity v0.4.5 // indirect
3031
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
3132
github.com/docker/go-connections v0.6.0 // indirect
3233
github.com/docker/go-units v0.5.0 // indirect
33-
github.com/emicklei/go-restful/v3 v3.12.2 // indirect
34+
github.com/emicklei/go-restful/v3 v3.13.0 // indirect
3435
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
3536
github.com/felixge/httpsnoop v1.0.4 // indirect
3637
github.com/fsnotify/fsnotify v1.9.0 // indirect
3738
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
3839
github.com/go-logr/logr v1.4.3 // indirect
3940
github.com/go-logr/stdr v1.2.2 // indirect
4041
github.com/go-logr/zapr v1.3.0 // indirect
41-
github.com/go-openapi/jsonpointer v0.21.1 // indirect
42-
github.com/go-openapi/jsonreference v0.21.0 // indirect
42+
github.com/go-openapi/jsonpointer v0.22.1 // indirect
43+
github.com/go-openapi/jsonreference v0.21.2 // indirect
4344
github.com/go-openapi/swag v0.23.1 // indirect
45+
github.com/go-openapi/swag/jsonname v0.25.1 // indirect
4446
github.com/gogo/protobuf v1.3.2 // indirect
4547
github.com/golang-migrate/migrate/v4 v4.19.1 // indirect
4648
github.com/google/btree v1.1.3 // indirect
@@ -49,7 +51,7 @@ require (
4951
github.com/google/go-cmp v0.7.0 // indirect
5052
github.com/google/uuid v1.6.0 // indirect
5153
github.com/gotestyourself/gotestyourself v2.2.0+incompatible // indirect
52-
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 // indirect
54+
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.1 // indirect
5355
github.com/inconshreveable/mousetrap v1.1.0 // indirect
5456
github.com/josharian/intern v1.0.0 // indirect
5557
github.com/json-iterator/go v1.1.12 // indirect
@@ -73,21 +75,21 @@ require (
7375
github.com/sapcc/go-api-declarations v1.18.0 // indirect
7476
github.com/sirupsen/logrus v1.9.3 // indirect
7577
github.com/spf13/cobra v1.10.1 // indirect
76-
github.com/spf13/pflag v1.0.9 // indirect
77-
github.com/stoewer/go-strcase v1.3.0 // indirect
78+
github.com/spf13/pflag v1.0.10 // indirect
79+
github.com/stoewer/go-strcase v1.3.1 // indirect
7880
github.com/x448/float16 v0.8.4 // indirect
7981
github.com/ziutek/mymysql v1.5.4 // indirect
8082
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
8183
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
8284
go.opentelemetry.io/otel v1.37.0 // indirect
83-
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0 // indirect
84-
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.36.0 // indirect
85+
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.37.0 // indirect
86+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.37.0 // indirect
8587
go.opentelemetry.io/otel/metric v1.37.0 // indirect
8688
go.opentelemetry.io/otel/sdk v1.37.0 // indirect
8789
go.opentelemetry.io/otel/trace v1.37.0 // indirect
8890
go.opentelemetry.io/proto/otlp v1.7.0 // indirect
8991
go.uber.org/multierr v1.11.0 // indirect
90-
go.uber.org/zap v1.27.0 // indirect
92+
go.uber.org/zap v1.27.1 // indirect
9193
go.yaml.in/yaml/v2 v2.4.3 // indirect
9294
go.yaml.in/yaml/v3 v3.0.4 // indirect
9395
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
@@ -101,8 +103,8 @@ require (
101103
golang.org/x/time v0.14.0 // indirect
102104
gomodules.xyz/jsonpatch/v2 v2.5.0 // indirect
103105
google.golang.org/genproto/googleapis/api v0.0.0-20250818200422-3122310a409c // indirect
104-
google.golang.org/genproto/googleapis/rpc v0.0.0-20250818200422-3122310a409c // indirect
105-
google.golang.org/grpc v1.75.0 // indirect
106+
google.golang.org/genproto/googleapis/rpc v0.0.0-20250929231259-57b25ae835d4 // indirect
107+
google.golang.org/grpc v1.75.1 // indirect
106108
google.golang.org/protobuf v1.36.10 // indirect
107109
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
108110
gopkg.in/inf.v0 v0.9.1 // indirect
@@ -117,6 +119,6 @@ require (
117119
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.33.0 // indirect
118120
sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
119121
sigs.k8s.io/randfill v1.0.0 // indirect
120-
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
122+
sigs.k8s.io/structured-merge-diff/v6 v6.3.1 // indirect
121123
sigs.k8s.io/yaml v1.6.0 // indirect
122124
)

0 commit comments

Comments
 (0)