Skip to content

Comments

K8SPXC-1152: restore stucks on operator restart#1610

Merged
hors merged 18 commits intomainfrom
dev/K8SPXC-1152
Feb 28, 2025
Merged

K8SPXC-1152: restore stucks on operator restart#1610
hors merged 18 commits intomainfrom
dev/K8SPXC-1152

Conversation

@pooknull
Copy link
Contributor

@pooknull pooknull commented Jan 30, 2024

K8SPXC-1152 Powered by Pull Request Badge

https://perconadev.atlassian.net/browse/K8SPXC-1152

CHANGE DESCRIPTION

Problem:
If operator pod is restarted during a restore, it can't continue to the restore process.

Cause:
The current design of the restore process is not designed to continue on operator restart.

Solution:
We should refactor the restore code so that the operator can catch up with the current state of the restore and continue.

CHECKLIST

Jira

  • Is the Jira ticket created and referenced properly?
  • Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
  • Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

  • Is an E2E test/test case added for the new feature/change?
  • Are unit tests added where appropriate?
  • Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

  • Are all needed new/changed options added to default YAML files?
  • Are the manifests (crd/bundle) regenerated if needed?
  • Did we add proper logging messages for operator actions?
  • Did we ensure compatibility with the previous version or cluster upgrade process?
  • Does the change support oldest and newest supported PXC version?
  • Does the change support oldest and newest supported Kubernetes version?

@pull-request-size pull-request-size bot added the size/XXL 1000+ lines label Jan 30, 2024
@pooknull pooknull marked this pull request as ready for review February 5, 2024 13:56
inelpandzic
inelpandzic previously approved these changes Feb 6, 2024
Comment on lines 92 to 96
rr := reconcile.Result{
RequeueAfter: time.Second * 5,
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

honestly I'm not happy to depend on RequeueAfter but I guess there's no other way

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a way to not depend on RequeueAfter, but it will take more time to implement. I would like to do it in a separate PR.

inelpandzic
inelpandzic previously approved these changes Feb 13, 2024
https://perconadev.atlassian.net/browse/K8SPXC-1152

fix unit test

fix for pvc restores

refactor

fix tests

fix `security-context` test

add unit-test

improvements

add TODO comment
@pooknull pooknull marked this pull request as draft January 9, 2025 13:02
@pooknull pooknull marked this pull request as ready for review January 20, 2025 09:21
@pooknull pooknull requested a review from gkech as a code owner January 20, 2025 09:21
@egegunes egegunes added this to the v1.17.0 milestone Jan 21, 2025
var oldHAProxySize int32
if cluster.Spec.HAProxy != nil {
oldHAProxySize = cluster.Spec.HAProxy.Size
switch statusState {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should move code for each case to a separate function? this would make code more readable

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


func setStatus(ctx context.Context, cl client.Client, cr *api.PerconaXtraDBClusterRestore, state api.BcpRestoreStates, comments string) error {
cr.Status.State = state
switch state {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this swtich here really needed since are handling only one case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pooknull pooknull requested review from egegunes and gkech February 7, 2025 13:04
gkech
gkech previously approved these changes Feb 7, 2025
@JNKPercona
Copy link
Collaborator

Test name Status
affinity-8-0 passed
auto-tuning-8-0 passed
cross-site-8-0 passed
custom-users-8-0 passed
demand-backup-cloud-8-0 passed
demand-backup-encrypted-with-tls-8-0 passed
demand-backup-8-0 passed
demand-backup-parallel-8-0 passed
haproxy-5-7 passed
haproxy-8-0 passed
init-deploy-5-7 passed
init-deploy-8-0 passed
limits-8-0 passed
monitoring-2-0-8-0 passed
one-pod-5-7 passed
one-pod-8-0 passed
pitr-8-0 passed
pitr-gap-errors-8-0 passed
proxy-protocol-8-0 passed
proxysql-sidecar-res-limits-8-0 passed
pvc-resize-5-7 passed
pvc-resize-8-0 passed
recreate-8-0 passed
restore-to-encrypted-cluster-8-0 passed
scaling-proxysql-8-0 passed
scaling-8-0 passed
scheduled-backup-5-7 passed
scheduled-backup-8-0 passed
security-context-8-0 passed
smart-update1-8-0 passed
smart-update2-8-0 passed
storage-8-0 passed
tls-issue-cert-manager-ref-8-0 passed
tls-issue-cert-manager-8-0 passed
tls-issue-self-8-0 passed
upgrade-consistency-8-0 passed
upgrade-haproxy-5-7 passed
upgrade-haproxy-8-0 passed
upgrade-proxysql-5-7 passed
upgrade-proxysql-8-0 passed
users-5-7 passed
users-8-0 passed
validation-hook-8-0 passed
We run 43 out of 43

commit: 4eddfda
image: perconalab/percona-xtradb-cluster-operator:PR-1610-4eddfda2

@hors hors merged commit e82f671 into main Feb 28, 2025
17 checks passed
@hors hors deleted the dev/K8SPXC-1152 branch February 28, 2025 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XXL 1000+ lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants