Skip to content

K8SPSMDB-1451: Wait until primary elected after rs initialization #2149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hors merged 8 commits into from
Jan 5, 2026
+30 −1
Merged

K8SPSMDB-1451: Wait until primary elected after rs initialization #2149

Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1bf9e34
K8SPSMDB-1451: Wait until primary elected after rs initialization
egegunes Dec 16, 2025
c684204
address review comments
egegunes Dec 18, 2025
15f463a
Merge branch 'main' into K8SPSMDB-1451
egegunes Dec 18, 2025
8b57e54
Apply suggestions from code review
egegunes Dec 18, 2025
65db8e0
Apply suggestions from code review
egegunes Dec 18, 2025
6234b53
Merge branch 'main' into K8SPSMDB-1451
egegunes Dec 22, 2025
1bb3527
Merge branch 'main' into K8SPSMDB-1451
egegunes Jan 5, 2026
9e380b0
Merge branch 'main' into K8SPSMDB-1451
hors Jan 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion pkg/controller/perconaservermongodb/mgo.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ import (
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/util/retry"
"k8s.io/utils/ptr"
logf "sigs.k8s.io/controller-runtime/pkg/log"

Expand Down Expand Up @@ -724,8 +726,35 @@ func (r *ReconcilePerconaServerMongoDB) handleReplsetInit(ctx context.Context, c
return nil, nil, fmt.Errorf("exec rs.initiate: %v / %s / %s", err, outb.String(), errb.String())
}

backoff := wait.Backoff{
Steps: 5,
Duration: 50 * time.Millisecond,
Factor: 5.0,
Jitter: 0.1,
}
err = retry.OnError(backoff, func(err error) bool { return true }, func() error {
Copy link
Copy Markdown
Contributor

@gkech gkech Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to also retry permanent errors, like auth, for example?

Copy link
Copy Markdown
Contributor Author

@egegunes egegunes Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think yes. at this point there shouldn't be any auth errors any way. and in the worst case scenario we'll retry for 6 seconds.

var stderr, stdout bytes.Buffer

hello := []string{"sh", "-c",
mongoCmd + " --quiet --eval 'db.hello().isWritablePrimary'"}
err := r.clientcmd.Exec(ctx, &pod, "mongod", hello, nil, &stdout, &stderr, false)
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable shadowing detected. The 'err' variable is declared with ':=' inside the retry function, which shadows the 'err' variable from the outer scope (line 736). This should use '=' instead of ':=' since 'err' is already declared in the outer scope. Variable shadowing can lead to confusion and potential bugs where the wrong error variable is used.

Suggested change
err := r.clientcmd.Exec(ctx, &pod, "mongod", hello, nil, &stdout, &stderr, false)
err = r.clientcmd.Exec(ctx, &pod, "mongod", hello, nil, &stdout, &stderr, false)

Copilot uses AI. Check for mistakes.
if err != nil {
return errors.Wrapf(err, "run hello stdout: %s, stderr: %s", stdout.String(), stderr.String())
}

out := strings.TrimSpace(stdout.String())
if out != "true" {
return errors.Errorf("%s is not the writable primary", pod.Name)
}

log.Info(pod.Name+" is the writable primary", "replset", replsetName)

return nil
})
Comment on lines +716 to +740
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new backoff retry logic for waiting until a primary is elected lacks test coverage. Consider adding unit tests to verify the retry behavior, including successful primary election, timeout scenarios, and error handling. Other functions in mgo_test.go demonstrate the testing pattern used in this package.

Copilot uses AI. Check for mistakes.
if err != nil {
return nil, nil, errors.Wrap(err, "wait for replset initialization")
}
log.Info("replset initialized", "replset", replsetName, "pod", pod.Name)
time.Sleep(time.Second * 5)

log.Info("creating user admin", "replset", replsetName, "pod", pod.Name, "user", api.RoleUserAdmin)
userAdmin, err := getInternalCredentials(ctx, r.client, cr, api.RoleUserAdmin)
Expand Down
Loading