Skip to content

Rabbitmq vhost and user support #1052

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lmiccini wants to merge 1 commit into
base: main
Choose a base branch
from
+450 −52
Open

Rabbitmq vhost and user support #1052

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions api/bases/nova.openstack.org_nova.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,23 @@ spec:
MemcachedInstance is the name of the Memcached CR that the services in the cell will use.
If defined then this takes precedence over Nova.Spec.MemcachedInstance for this cel
type: string
messagingBus:
description: MessagingBus configuration (username, vhost, and
cluster)
properties:
cluster:
description: Name of the cluster
minLength: 1
type: string
user:
description: User - RabbitMQ username
type: string
vhost:
description: Vhost - RabbitMQ vhost name
type: string
required:
- cluster
type: object
metadataServiceTemplate:
description: |-
MetadataServiceTemplate - defines the metadata service dedicated for the
Expand Down Expand Up @@ -1340,6 +1357,22 @@ spec:
description: MemcachedInstance is the name of the Memcached CR that
all nova service will use.
type: string
messagingBus:
description: MessagingBus configuration (username, vhost, and cluster)
properties:
cluster:
description: Name of the cluster
minLength: 1
type: string
user:
description: User - RabbitMQ username
type: string
vhost:
description: Vhost - RabbitMQ vhost name
type: string
required:
- cluster
type: object
metadataContainerImageURL:
description: MetadataContainerImageURL
type: string
Expand Down Expand Up @@ -1648,6 +1681,23 @@ spec:
NodeSelector here acts as a default value and can be overridden by service
specific NodeSelector Settings.
type: object
notificationsBus:
description: NotificationsBus configuration (username, vhost, and
cluster) for notifications
properties:
cluster:
description: Name of the cluster
minLength: 1
type: string
user:
description: User - RabbitMQ username
type: string
vhost:
description: Vhost - RabbitMQ vhost name
type: string
required:
- cluster
type: object
notificationsBusInstance:
description: |-
NotificationsBusInstance is the name of the RabbitMqCluster CR to select
Expand Down
5 changes: 4 additions & 1 deletion api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/emicklei/go-restful/v3 v3.12.2 // indirect
github.com/evanphx/json-patch v5.9.11+incompatible // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
Expand All @@ -44,6 +43,7 @@ require (
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.65.0 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/rabbitmq/cluster-operator/v2 v2.16.0 // indirect
github.com/spf13/pflag v1.0.7 // indirect
github.com/x448/float16 v0.8.4 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
Expand All @@ -69,6 +69,9 @@ require (
sigs.k8s.io/yaml v1.6.0 // indirect
)

// Use lmiccini's infra-operator branch with status.RabbitmqUserRef support
replace github.com/openstack-k8s-operators/infra-operator/apis => github.com/lmiccini/infra-operator/apis v0.0.0-20260116071214-b5e11cdcbab3

// mschuppert: map to latest commit from release-4.16 tag
// must consistent within modules and service operators
replace github.com/openshift/api => github.com/openshift/api v0.0.0-20250711200046-c86d80652a9e //allow-merging
Expand Down
7 changes: 5 additions & 2 deletions api/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
Expand Down Expand Up @@ -64,6 +65,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lmiccini/infra-operator/apis v0.0.0-20260116071214-b5e11cdcbab3 h1:S1WlXTdXw0PA0yZT1acQ1Jwxf/zLGdF2R9DAxbtpVRw=
github.com/lmiccini/infra-operator/apis v0.0.0-20260116071214-b5e11cdcbab3/go.mod h1:ZXwFlspJCdZEUjMbmaf61t5AMB4u2vMyAMMoe/vJroE=
github.com/mailru/easyjson v0.9.0 h1:PrnmzHw7262yW8sTBwxi1PdJA3Iw/EKBa8psRf7d9a4=
github.com/mailru/easyjson v0.9.0/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
Expand All @@ -78,10 +81,10 @@ github.com/onsi/ginkgo/v2 v2.27.3 h1:ICsZJ8JoYafeXFFlFAG75a7CxMsJHwgKwtO+82SE9L8
github.com/onsi/ginkgo/v2 v2.27.3/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
github.com/onsi/gomega v1.38.3 h1:eTX+W6dobAYfFeGC2PV6RwXRu/MyT+cQguijutvkpSM=
github.com/onsi/gomega v1.38.3/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20251223124749-eedb97238c5f h1:xcCGJ/g5vvbWhtEJCbv8UeBneI5yrMawm+CXRsJrJZo=
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20251223124749-eedb97238c5f/go.mod h1:ex8ou6/3ms6ovR+CMXD6XhTlNakm1GhB6UZgagVRNW8=
github.com/openstack-k8s-operators/lib-common/modules/common v0.6.1-0.20251230215914-6ba873b49a35 h1:pF3mJ3nwq6r4qwom+rEWZNquZpcQW/iftHlJ1KPIDsk=
github.com/openstack-k8s-operators/lib-common/modules/common v0.6.1-0.20251230215914-6ba873b49a35/go.mod h1:kycZyoe7OZdW1HUghr2nI3N7wSJtNahXf6b/ypD14f4=
github.com/openstack-k8s-operators/rabbitmq-cluster-operator/v2 v2.6.1-0.20250929174222-a0d328fa4dec h1:saovr368HPAKHN0aRPh8h8n9s9dn3d8Frmfua0UYRlc=
github.com/openstack-k8s-operators/rabbitmq-cluster-operator/v2 v2.6.1-0.20250929174222-a0d328fa4dec/go.mod h1:Nh2NEePLjovUQof2krTAg4JaAoLacqtPTZQXK6izNfg=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
Expand Down
9 changes: 9 additions & 0 deletions api/v1beta1/nova_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ limitations under the License.
package v1beta1

import (
rabbitmqv1 "github.com/openstack-k8s-operators/infra-operator/apis/rabbitmq/v1beta1"
topologyv1 "github.com/openstack-k8s-operators/infra-operator/apis/topology/v1beta1"
condition "github.com/openstack-k8s-operators/lib-common/modules/common/condition"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -49,6 +50,10 @@ type NovaSpecCore struct {
// communicate.
APIMessageBusInstance string `json:"apiMessageBusInstance"`

// +kubebuilder:validation:Optional
// MessagingBus configuration (username, vhost, and cluster)
MessagingBus rabbitmqv1.RabbitMqConfig `json:"messagingBus,omitempty"`

// +kubebuilder:validation:Optional
// +kubebuilder:default={cell0: {cellDatabaseAccount: nova-cell0, hasAPIAccess: true}, cell1: {cellDatabaseAccount: nova-cell1, cellDatabaseInstance: openstack-cell1, cellMessageBusInstance: rabbitmq-cell1, hasAPIAccess: true}}
// Cells is a mapping of cell names to NovaCellTemplate objects defining
Expand Down Expand Up @@ -131,6 +136,10 @@ type NovaSpecCore struct {
// Avoid colocating it with RabbitMqClusterName, APIMessageBusInstance or CellMessageBusInstance used for RPC.
// For particular Nova cells, notifications cannot be disabled, nor configured differently.
NotificationsBusInstance *string `json:"notificationsBusInstance,omitempty"`

// +kubebuilder:validation:Optional
// NotificationsBus configuration (username, vhost, and cluster) for notifications
NotificationsBus *rabbitmqv1.RabbitMqConfig `json:"notificationsBus,omitempty"`
}

// NovaSpec defines the desired state of Nova
Expand Down
53 changes: 52 additions & 1 deletion api/v1beta1/nova_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (
"fmt"

"github.com/google/go-cmp/cmp"
rabbitmqv1 "github.com/openstack-k8s-operators/infra-operator/apis/rabbitmq/v1beta1"
service "github.com/openstack-k8s-operators/lib-common/modules/common/service"
"github.com/robfig/cron/v3"

Expand Down Expand Up @@ -88,6 +89,24 @@ func (spec *NovaSpecCore) Default() {
spec.APITimeout = novaDefaults.APITimeout
}

// Default MessagingBus.Cluster from APIMessageBusInstance if not already set
if spec.MessagingBus.Cluster == "" {
spec.MessagingBus.Cluster = spec.APIMessageBusInstance
}

// Default NotificationsBus if NotificationsBusInstance is specified
if spec.NotificationsBusInstance != nil && *spec.NotificationsBusInstance != "" {
if spec.NotificationsBus == nil {
// Initialize empty NotificationsBus - credentials will be created dynamically
// to ensure separation from MessagingBus (RPC and notifications should never share credentials)
spec.NotificationsBus = &rabbitmqv1.RabbitMqConfig{}
}
// Default cluster name if not already set
if spec.NotificationsBus.Cluster == "" {
spec.NotificationsBus.Cluster = *spec.NotificationsBusInstance
}
}

for cellName, cellTemplate := range spec.CellTemplates {

if cellTemplate.MetadataServiceTemplate.Enabled == nil {
Expand All @@ -106,6 +125,11 @@ func (spec *NovaSpecCore) Default() {
}
}

// Default MessagingBus.Cluster from CellMessageBusInstance if not already set
if cellTemplate.MessagingBus.Cluster == "" {
cellTemplate.MessagingBus.Cluster = cellTemplate.CellMessageBusInstance
}

// "cellTemplate" is a by-value copy, so we need to re-inject the updated version of it into the map
spec.CellTemplates[cellName] = cellTemplate
}
Expand Down Expand Up @@ -315,7 +339,34 @@ func (spec *NovaSpec) ValidateUpdate(old NovaSpec, basePath *field.Path, namespa
// expected to be called by the validation webhook in the higher level meta
// operator
func (spec *NovaSpecCore) ValidateUpdate(old NovaSpecCore, basePath *field.Path, namespace string) field.ErrorList {
errors := spec.ValidateCellTemplates(basePath, namespace)
var errors field.ErrorList

// Reject changes to deprecated messagingBusInstance fields - users should use the new messagingBus fields instead
if spec.APIMessageBusInstance != old.APIMessageBusInstance {
errors = append(errors, field.Forbidden(
basePath.Child("apiMessageBusInstance"),
"apiMessageBusInstance is deprecated and cannot be changed. Please use messagingBus.cluster instead"))
}
Copy link
Contributor

@SeanMooney SeanMooney Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we should allow them to null it out if its set

but we could reject other changes How that is a breaking api change so im not sold on doing that.
it defeats the reason for ahvaing a deprecation period

what woudl be more correct is to not allow setting both at the same time.

i.e you can either use the old field or new filed but not both.

i woudl condier it an error if

cellTemplate.MessagingBus.Cluster and APIMessageBusInstance were ever diffent unless APIMessageBusInstance was nil or ""

we should exicltiy check for an reject that.


if spec.NotificationsBusInstance != nil && old.NotificationsBusInstance != nil &&
*spec.NotificationsBusInstance != *old.NotificationsBusInstance {
errors = append(errors, field.Forbidden(
basePath.Child("notificationsBusInstance"),
"notificationsBusInstance is deprecated and cannot be changed. Please use notificationsBus.cluster instead"))
}

// Check cell template changes
for cellName, cellTemplate := range spec.CellTemplates {
if oldCell, exists := old.CellTemplates[cellName]; exists {
if cellTemplate.CellMessageBusInstance != oldCell.CellMessageBusInstance {
errors = append(errors, field.Forbidden(
basePath.Child("cellTemplates").Key(cellName).Child("cellMessageBusInstance"),
"cellMessageBusInstance is deprecated and cannot be changed. Please use messagingBus.cluster instead"))
}
}
}

errors = append(errors, spec.ValidateCellTemplates(basePath, namespace)...)
Copy link
Contributor

@SeanMooney SeanMooney Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so i shoudl point out that

https://github.com/openstack-k8s-operators/nova-operator/blob/main/api/v1beta1/nova_webhook.go#L128C2-L153

explicit block 2 cells form sharing a rabbit instance since that is not supproted when not using seperat vhost per cell.

we need to enhance ValidateCellTemplates to account for vhost and more imporatnly the new cellTemplate.MessagingBus.Cluster field.

the quetion i have is do we want to enforce that you cant share the same rabbit cluster btween cells via the webhook or jsut document that as unsupproted in our product.

three isnt a reasonable usecase for sharign beyond testing/dev by levrageing vhosts as cells only exist to shard your cloud to supprot scaling beyond the limitats fo a sincel db/rabbit cluster.

my inclindation is to block 2 cells form using the same rabbit cluster in the webhook to ensure that unsupproted configuration are not possible rather then supproting it only for testing.

// Validate top-level TopologyRef
errors = append(errors, topologyv1.ValidateTopologyRef(
spec.TopologyRef, *basePath.Child("topologyRef"), namespace)...)
Expand Down
5 changes: 5 additions & 0 deletions api/v1beta1/novacell_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ limitations under the License.
package v1beta1

import (
rabbitmqv1 "github.com/openstack-k8s-operators/infra-operator/apis/rabbitmq/v1beta1"
topologyv1 "github.com/openstack-k8s-operators/infra-operator/apis/topology/v1beta1"
condition "github.com/openstack-k8s-operators/lib-common/modules/common/condition"
"github.com/openstack-k8s-operators/lib-common/modules/common/tls"
Expand Down Expand Up @@ -51,6 +52,10 @@ type NovaCellTemplate struct {
// communicate in this cell. For cell0 it is unused.
CellMessageBusInstance string `json:"cellMessageBusInstance"`

// +kubebuilder:validation:Optional
// MessagingBus configuration (username, vhost, and cluster)
MessagingBus rabbitmqv1.RabbitMqConfig `json:"messagingBus,omitempty"`

// +kubebuilder:validation:Required
// HasAPIAccess defines if this Cell is configured to have access to the
// API DB and message bus.
Expand Down
8 changes: 8 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

50 changes: 50 additions & 0 deletions config/crd/bases/nova.openstack.org_nova.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,23 @@ spec:
MemcachedInstance is the name of the Memcached CR that the services in the cell will use.
If defined then this takes precedence over Nova.Spec.MemcachedInstance for this cel
type: string
messagingBus:
description: MessagingBus configuration (username, vhost, and
cluster)
properties:
cluster:
description: Name of the cluster
minLength: 1
type: string
user:
description: User - RabbitMQ username
type: string
vhost:
description: Vhost - RabbitMQ vhost name
type: string
required:
- cluster
type: object
metadataServiceTemplate:
description: |-
MetadataServiceTemplate - defines the metadata service dedicated for the
Expand Down Expand Up @@ -1340,6 +1357,22 @@ spec:
description: MemcachedInstance is the name of the Memcached CR that
all nova service will use.
type: string
messagingBus:
description: MessagingBus configuration (username, vhost, and cluster)
properties:
cluster:
description: Name of the cluster
minLength: 1
type: string
user:
description: User - RabbitMQ username
type: string
vhost:
description: Vhost - RabbitMQ vhost name
type: string
required:
- cluster
type: object
metadataContainerImageURL:
description: MetadataContainerImageURL
type: string
Expand Down Expand Up @@ -1648,6 +1681,23 @@ spec:
NodeSelector here acts as a default value and can be overridden by service
specific NodeSelector Settings.
type: object
notificationsBus:
description: NotificationsBus configuration (username, vhost, and
cluster) for notifications
properties:
cluster:
description: Name of the cluster
minLength: 1
type: string
user:
description: User - RabbitMQ username
type: string
vhost:
description: Vhost - RabbitMQ vhost name
type: string
required:
- cluster
type: object
notificationsBusInstance:
description: |-
NotificationsBusInstance is the name of the RabbitMqCluster CR to select
Expand Down
3 changes: 3 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,9 @@ require (

replace github.com/openstack-k8s-operators/nova-operator/api => ./api

// Use lmiccini's infra-operator branch with status.RabbitmqUserRef support
replace github.com/openstack-k8s-operators/infra-operator/apis => github.com/lmiccini/infra-operator/apis v0.0.0-20260116071214-b5e11cdcbab3

// mschuppert: map to latest commit from release-4.18 tag
// must consistent within modules and service operators
replace github.com/openshift/api => github.com/openshift/api v0.0.0-20250711200046-c86d80652a9e //allow-merging
Expand Down
Loading