K8SPSMDB-1211: handle FULL CLUSTER CRASH error during the restore
#1926
Conversation
pull-request-size
bot
added
size/XL
pkg/controller/common/common.go
Outdated
| @@ -0,0 +1,70 @@ | |||
| package common | |||
There was a problem hiding this comment.
I think packages named common, utils, etc., tend to be vague, as they imply shared logic without a clearly defined domain or separation of concerns.
In this file, the main struct is CommonReconciler, but it's not clear what exactly is being reconciled. The struct also mixes responsibilities: as it's constructing and returning heterogeneous components like backup.PBM, mongo.Client, a scheme, and a k8s client.
To improve clarity and maintainability, I'd suggest:
-
Keeping the scheme and the Kubernetes client in
ReconcilePerconaServerMongoDB, and having related function with receivers of typeReconcilePerconaServerMongoDB. -
Splitting out PBM-related logic into a dedicated PBM factory/service.
-
Doing the same for the MongoClientProvider.
pull-request-size
bot
added
size/XXL
pull-request-size
bot
added
size/XL
pooknull
requested review from
jvpasinatto,
eleo007,
valmiranogueira,
hors,
egegunes and
nmarukovich
as code owners
| finished, err := r.finishPhysicalRestore(ctx, cluster) | ||
| if err != nil { | ||
| log.Error(err, "Failed to recover the cluster after the restore") | ||
| // status.State = psmdbv1.RestoreStateReady |
There was a problem hiding this comment.
please remove commented code
| cfg, err := cli.ReadConfig(ctx) | ||
| if err != nil { | ||
| return errors.Wrap(err, "read replset config") | ||
| } | ||
|
|
||
| if err := cli.WriteConfig(ctx, cfg, true); err != nil { | ||
| return errors.Wrap(err, "reconfigure replset") | ||
| } |
There was a problem hiding this comment.
i don't get what we are doing here, we are reconfiguring replset but we are not changing anything in config?
There was a problem hiding this comment.
We are doing the same thing as in the (*ReconcilePerconaServerMongoDB) handleReplicaSetNoPrimary:
percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go
Lines 763 to 770 in 3db09f5
commit: c15daba |
https://perconadev.atlassian.net/browse/K8SPSMDB-1211
DESCRIPTION
Problem:
During the physical restore, the operator detects a
FULL CLUSTER CRASHand attempts to resolve the issue. The operator log contains theFULL CLUSTER CRASHlog message, which should not be logged because this error occurs 100% of the time during the physical restore.Solution:
The solution is to perform the same action the
(*ReconcilePerconaServerMongoDB) handleReplicaSetNoPrimarymethod does after the physical restore. Once PBM has finished the restore, the operator should recreate the statefulsets and add thepercona.com/restore-in-progressannotation to them and handle theFULL CLUSTER CRASHstate. Afterwards, thepercona.com/restore-in-progressannotation should be removed from the statefulsets.CHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
compare/*-oc.yml)?Config/Logging/Testability