Skip to content

fix: tidb pd scale in multiple instances #2269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cjc7373 merged 2 commits into from
Nov 25, 2025
Merged

fix: tidb pd scale in multiple instances #2269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2365aa5
fix: br 7.5 won't restore system table by default
cjc7373 Nov 25, 2025
f0821d2
fix: member leave fails
cjc7373 Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion addons/tidb/dataprotection/restore.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
setStorageVar

# shellcheck disable=SC2086
/br restore full --pd "$PD_ADDRESS" --storage "s3://$BUCKET$DP_BACKUP_BASE_PATH?access-key=$ACCESS_KEY_ID&secret-access-key=$SECRET_ACCESS_KEY" --s3.endpoint "$ENDPOINT" $BR_EXTRA_ARGS
/br restore full --pd "$PD_ADDRESS" --storage "s3://$BUCKET$DP_BACKUP_BASE_PATH?access-key=$ACCESS_KEY_ID&secret-access-key=$SECRET_ACCESS_KEY" --s3.endpoint "$ENDPOINT" $BR_EXTRA_ARGS --with-sys-table
9 changes: 8 additions & 1 deletion addons/tidb/scripts/pd_memberleave.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@ echo "$KB_LEAVE_MEMBER_POD_NAME"
res=$(/pd-ctl member delete name "$KB_LEAVE_MEMBER_POD_NAME")
echo "$res"
not_found_pattern="Failed to delete member.*not found"
if [[ $res != "Success!" && ! $res =~ $not_found_pattern ]]; then
# this redirect_not_leader_pattern is a workaround when scaling in multiple pd instances,
# pd cluster may fail to create a new leader until the deleted members' pod is stopped.
# See: https://github.com/apecloud/apecloud/issues/13893#issuecomment-3574029284
# In the future, when scale in is implemented in instanceset controller, we can have a better
# solution that does memberLeave one by one.
redirect_not_leader_pattern="redirect to not leader"
if [[ $res != "Success!" && ! $res =~ $not_found_pattern && ! $res =~ $redirect_not_leader_pattern ]]; then
exit 1
fi
echo "member leave success"
1 change: 1 addition & 0 deletions addons/tidb/templates/cmpd-tidb.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ spec:
description: tidb's SQL layer
serviceKind: MySQL
updateStrategy: BestEffortParallel
podManagementPolicy: Parallel
vars:
- name: CLUSTER_NAMESPACE
valueFrom:
Expand Down
1 change: 1 addition & 0 deletions addons/tidb/templates/cmpd-tikv.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ spec:
provider: kubeblocks
description: a distributed transactional key-value database
updateStrategy: BestEffortParallel
podManagementPolicy: Parallel
vars:
- name: CLUSTER_NAMESPACE
valueFrom:
Expand Down