Skip to content

Commit f364c0e

Browse files
stuggiclaude
stuggi
and
claude
committed
Add staged deployment support for control plane restore
Add support for pausing OpenStackControlPlane deployment after infrastructure creation to enable database restore before OpenStack services start. This is useful for backup/restore scenarios where databases need to be restored to empty infrastructure before services initialize fresh schemas. **conditions.go:** - Add OpenStackControlPlaneInfrastructureReadyCondition type - Add condition messages (Init, Ready, Running, Error) - Infrastructure includes: CAs, DNSMasq, RabbitMQ, Galera, Memcached, OVN **openstackcontrolplane_types.go:** - Add DeploymentStageAnnotation constant ("core.openstack.org/deployment-stage") - Add DeploymentStageInfrastructureOnly constant ("infrastructure-only") - Add InfrastructureReady condition to InitConditions() **New helper function:** - isInfrastructureReady(): Checks if all enabled infrastructure components are ready - Always checks: CAs (no enabled flag) - Conditionally checks: DNS, RabbitMQ, Galera, Memcached, OVN (only if enabled) - Fixes test failures when OVN or other components are disabled **Staged deployment logic:** - Move OVN reconciliation to infrastructure section (before services) - Check deployment-stage annotation after infrastructure reconciliation - When annotation = "infrastructure-only": - Set InfrastructureReady condition with pause message - Return early (skip service reconciliation) - Message: "Infrastructure ready - deployment paused. Remove annotation to resume" - When annotation not set (normal deployment): - Set InfrastructureReady condition with standard message - Continue with full service reconciliation - Message: "Infrastructure ready" - When infrastructure still deploying: - Set InfrastructureReady = False/Requested - Message: "Infrastructure in progress" Update all kuttl test assertions to expect InfrastructureReady condition: - common/assert-sample-deployment.yaml - ctlplane-basic-deployment/03-assert-deploy-custom-cacert.yaml - ctlplane-collapsed/01-assert-collapsed-cell.yaml - ctlplane-galera-3replicas/01-assert-galera-3replicas.yaml - ctlplane-tls-cert-rotation/00-assert-deploy-openstack.yaml - ctlplane-tls-cert-rotation/03-assert-new-certs.yaml - ctlplane-tls-custom-issuers/01-assert-deploy-openstack.yaml - ctlplane-tls-custom-issuers/09-assert-deploy-openstack.yaml - ctlplane-tls-custom-route/03-assert-deploy-openstack.yaml This enables the backup/restore workflow: 1. Apply OpenStackControlPlane CR with annotation: core.openstack.org/deployment-stage: infrastructure-only 2. Wait for InfrastructureReady condition: oc wait --for=condition=InfrastructureReady openstackcontrolplane/openstack 3. Restore databases (MariaDB, OVN) to empty infrastructure 4. Restore RabbitMQ user credentials for EDPM compatibility 5. Remove annotation to resume deployment: oc annotate openstackcontrolplane openstack core.openstack.org/deployment-stage- 6. Services start with already-restored databases (no db_sync conflicts) - Clean database restore without service restarts - Services start with existing data intact - Clear user visibility via condition message - Works with optional infrastructure components (OVN can be disabled) Jira: OSPRH-25752 Co-Authored-By: Claude Sonnet 4.5 <[email protected]> Signed-off-by: Martin Schuppert <[email protected]>
1 parent 0385ac2 commit f364c0e

18 files changed

+289
-5
lines changed

‎api/core/v1beta1/conditions.go‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,11 @@ const (
161161

162162
// OpenStackControlPlaneExposeWatcherReadyCondition Status=True condition which indicates if Watcher is exposed via a route
163163
OpenStackControlPlaneExposeWatcherReadyCondition condition.Type = "OpenStackControlPlaneExposeWatcherReady"
164+
165+
// OpenStackControlPlaneInfrastructureReadyCondition Status=True condition which indicates if infrastructure components are ready
166+
// Infrastructure includes: CAs, DNSMasq, RabbitMQ, Galera (MariaDB), Memcached, and OVN databases
167+
// This condition is set to True when deployment-stage annotation is "infrastructure-only" and all infrastructure is ready
168+
OpenStackControlPlaneInfrastructureReadyCondition condition.Type = "InfrastructureReady"
164169
)
165170

166171
// Common Messages used by API objects.
@@ -507,6 +512,21 @@ const (
507512

508513
// OpenStackControlPlaneWatcherReadyErrorMessage
509514
OpenStackControlPlaneWatcherReadyErrorMessage = "OpenStackControlPlane Watcher error occured %s"
515+
516+
// OpenStackControlPlaneInfrastructureReadyInitMessage
517+
OpenStackControlPlaneInfrastructureReadyInitMessage = "OpenStackControlPlane Infrastructure not started"
518+
519+
// OpenStackControlPlaneInfrastructureReadyMessage
520+
OpenStackControlPlaneInfrastructureReadyMessage = "OpenStackControlPlane Infrastructure ready"
521+
522+
// OpenStackControlPlaneInfrastructureReadyRunningMessage
523+
OpenStackControlPlaneInfrastructureReadyRunningMessage = "OpenStackControlPlane Infrastructure in progress"
524+
525+
// OpenStackControlPlaneInfrastructureReadyWaitingMessage
526+
OpenStackControlPlaneInfrastructureReadyWaitingMessage = "OpenStackControlPlane Infrastructure in progress - waiting for: %s"
527+
528+
// OpenStackControlPlaneInfrastructureReadyErrorMessage
529+
OpenStackControlPlaneInfrastructureReadyErrorMessage = "OpenStackControlPlane Infrastructure error occured %s"
510530
)
511531

512532
// Version Conditions used by to drive minor updates

‎api/core/v1beta1/openstackcontrolplane_types.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,11 @@ const (
6666
GlanceName = "glance"
6767
// CinderName - Default Cinder name
6868
CinderName = "cinder"
69+
70+
// DeploymentStageAnnotation - Annotation key for controlling deployment stages
71+
DeploymentStageAnnotation = "core.openstack.org/deployment-stage"
72+
// DeploymentStageInfrastructureOnly - Annotation value to pause after infrastructure deployment
73+
DeploymentStageInfrastructureOnly = "infrastructure-only"
6974
)
7075

7176
// OpenStackControlPlaneSpec defines the desired state of OpenStackControlPlane
@@ -951,6 +956,7 @@ func (instance *OpenStackControlPlane) InitConditions() {
951956
condition.UnknownCondition(OpenStackControlPlaneCAReadyCondition, condition.InitReason, OpenStackControlPlaneCAReadyInitMessage),
952957
condition.UnknownCondition(OpenStackControlPlaneOpenStackVersionInitializationReadyCondition, condition.InitReason, OpenStackControlPlaneOpenStackVersionInitializationReadyInitMessage),
953958
condition.UnknownCondition(OpenStackControlPlaneWatcherReadyCondition, condition.InitReason, OpenStackControlPlaneWatcherReadyInitMessage),
959+
condition.UnknownCondition(OpenStackControlPlaneInfrastructureReadyCondition, condition.InitReason, OpenStackControlPlaneInfrastructureReadyInitMessage),
954960

955961
// Also add the overall status condition as Unknown
956962
condition.UnknownCondition(condition.ReadyCondition, condition.InitReason, condition.ReadyInitMessage),

‎internal/controller/core/openstackcontrolplane_controller.go‎

Lines changed: 87 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ package core
2020
import (
2121
"context"
2222
"fmt"
23+
"strings"
2324

2425
certmgrv1 "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1"
2526
routev1 "github.com/openshift/api/route/v1"
@@ -386,6 +387,36 @@ func (r *OpenStackControlPlaneReconciler) reconcileOVNControllers(ctx context.Co
386387
return ctrl.Result{}, nil
387388
}
388389

390+
// isInfrastructureReady checks if all enabled infrastructure components are ready
391+
// Returns true if ready, and a list of components that are not ready (empty if all ready)
392+
func isInfrastructureReady(instance *corev1beta1.OpenStackControlPlane) (bool, []string) {
393+
notReady := []string{}
394+
395+
// CAs are always deployed (no enabled flag)
396+
if !instance.Status.Conditions.IsTrue(corev1beta1.OpenStackControlPlaneCAReadyCondition) {
397+
notReady = append(notReady, "CAs")
398+
}
399+
400+
// Only check each infrastructure component if it's enabled
401+
if instance.Spec.DNS.Enabled && !instance.Status.Conditions.IsTrue(corev1beta1.OpenStackControlPlaneDNSReadyCondition) {
402+
notReady = append(notReady, "DNS")
403+
}
404+
if instance.Spec.Rabbitmq.Enabled && !instance.Status.Conditions.IsTrue(corev1beta1.OpenStackControlPlaneRabbitMQReadyCondition) {
405+
notReady = append(notReady, "RabbitMQs")
406+
}
407+
if instance.Spec.Galera.Enabled && !instance.Status.Conditions.IsTrue(corev1beta1.OpenStackControlPlaneMariaDBReadyCondition) {
408+
notReady = append(notReady, "Galeras")
409+
}
410+
if instance.Spec.Memcached.Enabled && !instance.Status.Conditions.IsTrue(corev1beta1.OpenStackControlPlaneMemcachedReadyCondition) {
411+
notReady = append(notReady, "Memcached")
412+
}
413+
if instance.Spec.Ovn.Enabled && !instance.Status.Conditions.IsTrue(corev1beta1.OpenStackControlPlaneOVNReadyCondition) {
414+
notReady = append(notReady, "OVN")
415+
}
416+
417+
return len(notReady) == 0, notReady
418+
}
419+
389420
func (r *OpenStackControlPlaneReconciler) reconcileNormal(ctx context.Context, instance *corev1beta1.OpenStackControlPlane, version *corev1beta1.OpenStackVersion, helper *common_helper.Helper) (ctrl.Result, error) {
390421
if instance.Spec.TopologyRef != nil {
391422
if err := r.checkTopologyRef(ctx, helper,
@@ -402,6 +433,11 @@ func (r *OpenStackControlPlaneReconciler) reconcileNormal(ctx context.Context, i
402433
instance.Status.Conditions.MarkTrue(condition.TopologyReadyCondition, condition.TopologyReadyMessage)
403434
}
404435

436+
// Check for deployment-stage annotation
437+
deploymentStage := instance.Annotations[corev1beta1.DeploymentStageAnnotation]
438+
infrastructureOnly := deploymentStage == corev1beta1.DeploymentStageInfrastructureOnly
439+
440+
// Reconcile infrastructure components (always run)
405441
ctrlResult, err := openstack.ReconcileCAs(ctx, instance, helper)
406442
if err != nil {
407443
return ctrl.Result{}, err
@@ -437,41 +473,87 @@ func (r *OpenStackControlPlaneReconciler) reconcileNormal(ctx context.Context, i
437473
return ctrlResult, nil
438474
}
439475

440-
ctrlResult, err = openstack.ReconcileKeystoneAPI(ctx, instance, version, helper)
476+
// OVN databases are part of infrastructure
477+
ctrlResult, err = openstack.ReconcileOVN(ctx, instance, version, helper)
441478
if err != nil {
442479
return ctrl.Result{}, err
443480
} else if (ctrlResult != ctrl.Result{}) {
444481
return ctrlResult, nil
445482
}
446483

447-
ctrlResult, err = openstack.ReconcilePlacementAPI(ctx, instance, version, helper)
484+
// Update InfrastructureReady condition based on infrastructure component readiness
485+
// This is useful for observability regardless of staged deployment mode
486+
infrastructureReady, notReadyComponents := isInfrastructureReady(instance)
487+
488+
if infrastructureReady {
489+
// Set different messages based on whether deployment is paused
490+
if infrastructureOnly {
491+
// Infrastructure-only mode: indicate deployment is paused
492+
instance.Status.Conditions.MarkTrue(
493+
corev1beta1.OpenStackControlPlaneInfrastructureReadyCondition,
494+
corev1beta1.OpenStackControlPlaneInfrastructureReadyMessage+" - deployment paused. Remove annotation to resume deployment of OpenStack services")
495+
r.GetLogger(ctx).Info("Infrastructure components ready - deployment paused at infrastructure-only stage")
496+
} else {
497+
// Normal mode: infrastructure is ready
498+
instance.Status.Conditions.MarkTrue(
499+
corev1beta1.OpenStackControlPlaneInfrastructureReadyCondition,
500+
corev1beta1.OpenStackControlPlaneInfrastructureReadyMessage)
501+
}
502+
} else {
503+
// Build a descriptive message showing which components are not ready
504+
if len(notReadyComponents) > 0 {
505+
instance.Status.Conditions.Set(condition.FalseCondition(
506+
corev1beta1.OpenStackControlPlaneInfrastructureReadyCondition,
507+
condition.RequestedReason,
508+
condition.SeverityInfo,
509+
corev1beta1.OpenStackControlPlaneInfrastructureReadyWaitingMessage,
510+
strings.Join(notReadyComponents, ", ")))
511+
} else {
512+
instance.Status.Conditions.Set(condition.FalseCondition(
513+
corev1beta1.OpenStackControlPlaneInfrastructureReadyCondition,
514+
condition.RequestedReason,
515+
condition.SeverityInfo,
516+
corev1beta1.OpenStackControlPlaneInfrastructureReadyRunningMessage))
517+
}
518+
}
519+
520+
// Check if we're in infrastructure-only mode and should pause deployment
521+
if infrastructureOnly {
522+
// Stop here - do not reconcile OpenStack services
523+
return ctrl.Result{}, nil
524+
}
525+
526+
// Continue with OpenStack service reconciliation
527+
ctrlResult, err = openstack.ReconcileKeystoneAPI(ctx, instance, version, helper)
448528
if err != nil {
449529
return ctrl.Result{}, err
450530
} else if (ctrlResult != ctrl.Result{}) {
451531
return ctrlResult, nil
452532
}
453533

454-
ctrlResult, err = openstack.ReconcileGlance(ctx, instance, version, helper)
534+
ctrlResult, err = openstack.ReconcilePlacementAPI(ctx, instance, version, helper)
455535
if err != nil {
456536
return ctrl.Result{}, err
457537
} else if (ctrlResult != ctrl.Result{}) {
458538
return ctrlResult, nil
459539
}
460540

461-
ctrlResult, err = openstack.ReconcileCinder(ctx, instance, version, helper)
541+
ctrlResult, err = openstack.ReconcileGlance(ctx, instance, version, helper)
462542
if err != nil {
463543
return ctrl.Result{}, err
464544
} else if (ctrlResult != ctrl.Result{}) {
465545
return ctrlResult, nil
466546
}
467547

468-
ctrlResult, err = openstack.ReconcileOVN(ctx, instance, version, helper)
548+
ctrlResult, err = openstack.ReconcileCinder(ctx, instance, version, helper)
469549
if err != nil {
470550
return ctrl.Result{}, err
471551
} else if (ctrlResult != ctrl.Result{}) {
472552
return ctrlResult, nil
473553
}
474554

555+
// OVN already reconciled in infrastructure section above
556+
475557
ctrlResult, err = openstack.ReconcileNeutron(ctx, instance, version, helper)
476558
if err != nil {
477559
return ctrl.Result{}, err

‎test/kuttl/common/assert-sample-deployment.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -254,6 +254,10 @@ status:
254254
reason: Ready
255255
status: "True"
256256
type: OpenStackControlPlaneGlanceReady
257+
- message: OpenStackControlPlane Infrastructure ready
258+
reason: Ready
259+
status: "True"
260+
type: InfrastructureReady
257261
- message: OpenStackControlPlane InstanceHa CM is available
258262
reason: Ready
259263
status: "True"

‎test/kuttl/tests/ctlplane-basic-deployment/03-assert-deploy-custom-cacert.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,10 @@ status:
6363
reason: Ready
6464
status: "True"
6565
type: OpenStackControlPlaneGlanceReady
66+
- message: OpenStackControlPlane Infrastructure ready
67+
reason: Ready
68+
status: "True"
69+
type: InfrastructureReady
6670
- message: OpenStackControlPlane InstanceHa CM is available
6771
reason: Ready
6872
status: "True"

‎test/kuttl/tests/ctlplane-collapsed/01-assert-collapsed-cell.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,10 @@ status:
231231
reason: Ready
232232
status: "True"
233233
type: OpenStackControlPlaneGlanceReady
234+
- message: OpenStackControlPlane Infrastructure ready
235+
reason: Ready
236+
status: "True"
237+
type: InfrastructureReady
234238
- message: OpenStackControlPlane InstanceHa CM is available
235239
reason: Ready
236240
status: "True"

‎test/kuttl/tests/ctlplane-galera-3replicas/01-assert-galera-3replicas.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,10 @@ status:
219219
reason: Ready
220220
status: "True"
221221
type: OpenStackControlPlaneGlanceReady
222+
- message: OpenStackControlPlane Infrastructure ready
223+
reason: Ready
224+
status: "True"
225+
type: InfrastructureReady
222226
- message: OpenStackControlPlane InstanceHa CM is available
223227
reason: Ready
224228
status: "True"

‎test/kuttl/tests/ctlplane-staged-deployment/01-assert-infrastructure-ready-paused.yaml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Assert that infrastructure is ready but deployment is paused
2+
# InfrastructureReady should be True with pause message
3+
# Infrastructure components (CAs, DNS, RabbitMQ, MariaDB, Memcached, OVN) should be Ready
4+
# Service components should still be in Init/Unknown state
5+
# Overall Ready should be False
6+
apiVersion: core.openstack.org/v1beta1
7+
kind: OpenStackControlPlane
8+
metadata:
9+
name: openstack
10+
annotations:
11+
core.openstack.org/deployment-stage: infrastructure-only
12+
status:
13+
conditions:
14+
# Overall deployment should not be ready yet (paused after infrastructure)
15+
- reason: Requested
16+
status: "False"
17+
type: Ready
18+
# Infrastructure should be ready with pause message
19+
- message: Infrastructure ready - deployment paused. Remove annotation to resume
20+
reason: Ready
21+
status: "True"
22+
type: InfrastructureReady
23+
# Infrastructure components should be ready
24+
- message: OpenStackControlPlane CAs completed
25+
reason: Ready
26+
status: "True"
27+
type: OpenStackControlPlaneCAReadyCondition
28+
- message: OpenStackControlPlane DNSData completed
29+
reason: Ready
30+
status: "True"
31+
type: OpenStackControlPlaneDNSReadyCondition
32+
- message: OpenStackControlPlane RabbitMQ completed
33+
reason: Ready
34+
status: "True"
35+
type: OpenStackControlPlaneRabbitMQReady
36+
- message: OpenStackControlPlane MariaDB completed
37+
reason: Ready
38+
status: "True"
39+
type: OpenStackControlPlaneMariaDBReady
40+
- message: OpenStackControlPlane Memcached completed
41+
reason: Ready
42+
status: "True"
43+
type: OpenStackControlPlaneMemcachedReady
44+
- message: OpenStackControlPlane OVN completed
45+
reason: Ready
46+
status: "True"
47+
type: OpenStackControlPlaneOVNReady
48+
# Service components should still be initializing (Unknown status with Init reason)
49+
- reason: Init
50+
status: Unknown
51+
type: OpenStackControlPlaneKeystoneAPIReady
52+
- reason: Init
53+
status: Unknown
54+
type: OpenStackControlPlanePlacementAPIReady
55+
- reason: Init
56+
status: Unknown
57+
type: OpenStackControlPlaneGlanceReady
58+
- reason: Init
59+
status: Unknown
60+
type: OpenStackControlPlaneCinderReady
61+
- reason: Init
62+
status: Unknown
63+
type: OpenStackControlPlaneNeutronReady
64+
- reason: Init
65+
status: Unknown
66+
type: OpenStackControlPlaneNovaReady
67+
- reason: Init
68+
status: Unknown
69+
type: OpenStackControlPlaneIronicReady
70+
- reason: Init
71+
status: Unknown
72+
type: OpenStackControlPlaneClientReady
73+
- reason: Init
74+
status: Unknown
75+
type: OpenStackControlPlaneManilaReady
76+
- reason: Init
77+
status: Unknown
78+
type: OpenStackControlPlaneHorizonReady
79+
- reason: Init
80+
status: Unknown
81+
type: OpenStackControlPlaneTelemetryReady
82+
- reason: Init
83+
status: Unknown
84+
type: OpenStackControlPlaneHeatReady
85+
- reason: Init
86+
status: Unknown
87+
type: OpenStackControlPlaneSwiftReady
88+
- reason: Init
89+
status: Unknown
90+
type: OpenStackControlPlaneOctaviaReady
91+
- reason: Init
92+
status: Unknown
93+
type: OpenStackControlPlaneDesignateReady
94+
- reason: Init
95+
status: Unknown
96+
type: OpenStackControlPlaneBarbicanReady
97+
- reason: Init
98+
status: Unknown
99+
type: OpenStackControlPlaneRedisReady
100+
- reason: Init
101+
status: Unknown
102+
type: OpenStackControlPlaneWatcherReady
103+
- reason: Init
104+
status: Unknown
105+
type: OpenStackControlPlaneOpenStackVersionInitializationReadyCondition

‎test/kuttl/tests/ctlplane-staged-deployment/01-deploy-openstack-infrastructure-only.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Deploy OpenStackControlPlane with deployment-stage annotation set to infrastructure-only
2+
# This should pause deployment after infrastructure (CAs, DNSMasq, RabbitMQ, Galera, Memcached, OVN) is ready
3+
apiVersion: kuttl.dev/v1beta1
4+
kind: TestStep
5+
commands:
6+
- script: |
7+
oc kustomize ../../../../config/samples/base/openstackcontrolplane | \
8+
oc annotate -f - --local=true --dry-run=none \
9+
core.openstack.org/deployment-stage=infrastructure-only -o yaml | \
10+
oc apply -n $NAMESPACE -f -

‎test/kuttl/tests/ctlplane-staged-deployment/02-assert-controlplane-ready.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../common/assert-sample-deployment.yaml

0 commit comments

Comments
 (0)