Skip to content

🐛 Fix OpenStackServer reconciliation stuck when cluster is unpaused #2833

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
k8s-ci-robot merged 1 commit into from
Jan 13, 2026
Merged

🐛 Fix OpenStackServer reconciliation stuck when cluster is unpaused #2833

k8s-ci-robot merged 1 commit into from
Jan 13, 2026
+259 −13

Conversation

@bnallapeta
Copy link
Contributor

@bnallapeta bnallapeta commented Nov 13, 2025

What this PR does / why we need it:
When a cluster is paused (e.g., during a pivot operation), OpenStackServer resources stop reconciling. However, when the cluster is unpaused, they don't resume because the controller doesn't watch for cluster pause/unpause events.

This PR adds a watch on Cluster resources so OpenStackServers are re-queued when their parent cluster transitions from paused to unpaused state.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2824

TODOs:

  • squashed commits
  • if necessary:
    • includes documentation
    • adds unit tests

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 13, 2025
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 13, 2025
@netlify
Copy link

netlify bot commented Nov 13, 2025 •
edited
Loading

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name Link
🔨 Latest commit 676ceaa
🔍 Latest deploy log https://app.netlify.com/projects/kubernetes-sigs-cluster-api-openstack/deploys/6965cf013d5a170008d57368
😎 Deploy Preview https://deploy-preview-2833--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@EmilienM
Copy link
Contributor

EmilienM commented Nov 19, 2025

/approve

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 19, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: EmilienM

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2025
@bnallapeta
Copy link
Contributor Author

bnallapeta commented Nov 20, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Nov 20, 2025
lentzi90
lentzi90 reviewed Nov 21, 2025
View reviewed changes
controllers/openstackserver_controller.go Outdated
Comment on lines 775 to 779
// Don't handle deleted clusters
if !c.DeletionTimestamp.IsZero() {
log.V(4).Info("Cluster has a deletion timestamp, skipping mapping.")
return nil
}
Copy link
Contributor

@lentzi90 lentzi90 Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need to clean up if it is deleting? I am thinking that there could be a situation where the user paused the cluster, then deleted (forgetting to unpause first), then unpause.

Copy link
Contributor Author

@bnallapeta bnallapeta Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, to address this edge case, here's what I did:
I've moved the pause check after the deletion timestamp check (similar to OpenStackCluster controller). Now the reconciliation flow is:

  1. Check if server is being deleted -> proceed with deletion (regardless of pause state)
  2. If not deleted, check if cluster is paused -> skip reconciliation
  3. Otherwise, proceed normally

This ensures deletion always proceeds even if the cluster was paused when deletion started, then later unpaused.

Copy link
Contributor Author

@bnallapeta bnallapeta Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lentzi90 ptal ^^

@lentzi90
Copy link
Contributor

lentzi90 commented Dec 11, 2025

I am a bit worried that this will break clusterctl move. We will probably need to handle all these cases:

  1. Cluster paused and has deletion timestamp: do nothing <- This is now broken I think (we delete even when paused)
  2. Cluster unpaused and has deletion timestamp: reconcileDelete <- This was missing in first version I think
  3. Cluster paused and no deletion timestamp: do nothing ✔️
  4. Cluster unpaused and no deletion timestamp: reconcileNormal ✔️

The full test is testing clusterctl move, so let's check
/test pull-cluster-api-provider-openstack-e2e-full-test

@bnallapeta
Copy link
Contributor Author

bnallapeta commented Jan 13, 2026

/test pull-cluster-api-provider-openstack-e2e-full-test

lentzi90
lentzi90 reviewed Jan 13, 2026
View reviewed changes
Copy link
Contributor

@lentzi90 lentzi90 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 13, 2026
@bnallapeta
Copy link
Contributor Author

bnallapeta commented Jan 13, 2026

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 13, 2026
@k8s-ci-robot k8s-ci-robot merged commit 92268d3 into kubernetes-sigs:main Jan 13, 2026
14 checks passed
@github-project-automation github-project-automation bot moved this from Inbox to Done in CAPO Roadmap Jan 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lentzi90 lentzi90 lentzi90 left review comments

@EmilienM EmilienM Awaiting requested review from EmilienM

Assignees

@lentzi90 lentzi90

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

CAPO Roadmap
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

OpenstackServer doesn't retry when Cluster was paued

4 participants

@bnallapeta @EmilienM @k8s-ci-robot @lentzi90