You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: best-practices/pd-scheduling-best-practices.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -296,7 +296,9 @@ If a TiKV node fails, PD defaults to setting the corresponding node to the **dow
296
296
297
297
Practically, if a node failure is considered unrecoverable, you can immediately take it offline. This makes PD replenish replicas soon in another node and reduces the risk of data loss. In contrast, if a node is considered recoverable, but the recovery cannot be donein 30 minutes, you can temporarily adjust `max-store-down-time` to a larger value to avoid unnecessary replenishment of the replicas and resources waste after the timeout.
298
298
299
-
In TiDB v5.2.0, TiKV introduces the mechanism of slow TiKV node detection. By sampling the requests in TiKV, this mechanism works out a score ranging from 1 to 100. A TiKV node with a score higher than or equal to 80 is marked as slow. You can add [`evict-slow-store-scheduler`](/pd-control.md#scheduler-show--add--remove--pause--resume--config--describe) to detect and schedule slow nodes. If only one TiKV is detected as slow, and the slow score reaches the limit (80 by default), the Leader in this node will be evicted (similar to the effect of `evict-leader-scheduler`).
299
+
Starting from TiDB v5.2.0, TiKV introduces a mechanism to detect slow-disk nodes. By sampling the requests in TiKV, this mechanism works out a score ranging from 1 to 100. A TiKV node with a score higher than or equal to 80 is marked as slow. You can add [`evict-slow-store-scheduler`](/pd-control.md#scheduler-show--add--remove--pause--resume--config--describe) to schedule slow nodes. If only one TiKV node is detected as slow, and its slow score reaches the limit (80 by default), the Leaders on that node will be evicted (similar to the effect of `evict-leader-scheduler`).
300
+
301
+
Starting from v8.5.5, TiKV introduces a mechanism to detect slow-network nodes. Similar to slow-disk node detection, this mechanism identifies slow nodes by probing network latency between TiKV nodes and calculating a score. You can enable this mechanism using [`enable-network-slow-store`](/pd-control.md#scheduler-config-evict-slow-store-scheduler).
Starting from v8.5.5, if your TiDB cluster and BR are running in an Azure Virtual Machine (VM) or Azure Kubernetes Service (AKS) environment and Azure managed identities have been assigned to the nodes, you can use Azure managed identities for authentication.
208
+
209
+
Before using this method, ensure that you have granted the permissions (such as `Storage Blob Data Contributor`) to the corresponding managed identity to access the target storage account in the [Azure Portal](https://azure.microsoft.com/).
210
+
211
+
- **System-assigned managed identity**:
212
+
213
+
When using a system-assigned managed identity, there is no need to configure any Azure-related environment variables. You can run the BR backup command directly.
> Ensure that the `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, and `AZURE_CLIENT_SECRET` environment variables are **not**setin the runtime environment. Otherwise, the Azure SDK might prioritize other authentication methods, preventing the managed identity from taking effect.
223
+
224
+
- **User-assigned managed identity**:
225
+
226
+
When using a user-assigned managed identity, you need to configure the `AZURE_CLIENT_ID` environment variable in the runtime environment of TiKV and BR, set its value to the client ID of the managed identity, and then run the BR backup command. The detailed steps are as follows:
227
+
228
+
1. Configure the client ID for TiKV when starting with TiUP:
229
+
230
+
The following steps use the TiKV port `24000` and the systemd service name `tikv-24000` as an example:
231
+
232
+
1. Open the systemd service editor by running the following command:
233
+
234
+
```shell
235
+
systemctl edit tikv-24000
236
+
```
237
+
238
+
2. Set the `AZURE_CLIENT_ID` environment variable to your managed identity client ID:
239
+
240
+
```ini
241
+
[Service]
242
+
Environment="AZURE_CLIENT_ID=<your-client-id>"
243
+
```
244
+
245
+
3. Reload the systemd configuration and restart TiKV:
246
+
247
+
```shell
248
+
systemctl daemon-reload
249
+
systemctl restart tikv-24000
250
+
```
251
+
252
+
2. Configure the `AZURE_CLIENT_ID` environment variable for BR:
253
+
254
+
```shell
255
+
export AZURE_CLIENT_ID="<your-client-id>"
256
+
```
257
+
258
+
3. Back up data to Azure Blob Storage using the following BR command:
Full Restore <--------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
147
+
Split&Scatter Region <--------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
Restore Meta Files <--------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
Copy file name to clipboardExpand all lines: br/br-checkpoint-restore.md
+78-4Lines changed: 78 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ If your TiDB cluster is large and cannot afford to restore again after a failure
15
15
16
16
## Implementation principles
17
17
18
-
The implementation of checkpoint restore is divided into two parts: snapshot restore and log restore. For more information, see [Implementation details](#implementation-details).
18
+
The implementation of checkpoint restore is divided into two parts: snapshot restore and log restore. For more information, see [Implementation details: store checkpoint data in the downstream cluster](#implementation-details-store-checkpoint-data-in-the-downstream-cluster) and [Implementation details: store checkpoint data in the external storage](#implementation-details-store-checkpoint-data-in-the-external-storage).
19
19
20
20
### Snapshot restore
21
21
@@ -65,7 +65,11 @@ After a restore failure, avoid writing, deleting, or creating tables in the clus
65
65
66
66
Cross-major-version checkpoint recovery is not recommended. For clusters where `br` recovery fails using the Long-Term Support (LTS) versions prior to v8.5.0, recovery cannot be continued with v8.5.0 or later LTS versions, and vice versa.
67
67
68
-
## Implementation details
68
+
## Implementation details: store checkpoint data in the downstream cluster
69
+
70
+
> **Note:**
71
+
>
72
+
> Starting from v8.5.5, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter.
69
73
70
74
Checkpoint restore operations are divided into two parts: snapshot restore and PITR restore.
71
75
@@ -81,8 +85,78 @@ If the restore fails and you try to restore backup data with different checkpoin
81
85
82
86
[PITR (Point-in-time recovery)](/br/br-pitr-guide.md) consists of snapshot restore and log restore phases.
83
87
84
-
During the initial restore, `br` first enters the snapshot restore phase. This phase follows the same process as the preceding [snapshot restore](#snapshot-restore-1): BR records the checkpoint data, the upstream cluster ID, and BackupTS of the backup data (that is, the start time point `start-ts` of log restore) in the `__TiDB_BR_Temporary_Snapshot_Restore_Checkpoint` database. If restore fails during this phase, you cannot adjust the `start-ts` of log restore when resuming checkpoint restore.
88
+
During the initial restore, `br` first enters the snapshot restore phase. BR records the checkpoint data, the upstream cluster ID, BackupTS of the backup data (that is, the start time point `start-ts` of log restore) and the restored time point `restored-ts` of log restore in the `__TiDB_BR_Temporary_Snapshot_Restore_Checkpoint` database. If restore fails during this phase, you cannot adjust the `start-ts` and `restored-ts` of log restore when resuming checkpoint restore.
85
89
86
90
When entering the log restore phase during the initial restore, `br` creates a `__TiDB_BR_Temporary_Log_Restore_Checkpoint` database in the target cluster. This database records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually delete the `__TiDB_BR_Temporary_Log_Restore_Checkpoint` database and retry with a different backup.
87
91
88
-
Before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data.
92
+
Note that before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. **Deleting data from `mysql.tidb_pitr_id_map` arbitrarily might lead to inconsistent PITR restore data.**
93
+
94
+
> **Note:**
95
+
>
96
+
> To ensure compatibility with clusters of earlier versions, starting from v8.5.5, if the system table `mysql.tidb_pitr_id_map` does not exist in the restore cluster, the `pitr_id_map` data will be written to the log backup directory. The file name is `pitr_id_maps/pitr_id_map.cluster_id:{downstream-cluster-ID}.restored_ts:{restored-ts}`.
97
+
98
+
## Implementation details: store checkpoint data in the external storage
99
+
100
+
> **Note:**
101
+
>
102
+
> Starting from v8.5.5, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. For example:
103
+
>
104
+
> ```shell
105
+
> ./br restore full -s "s3://backup-bucket/backup-prefix" --checkpoint-storage "s3://temp-bucket/checkpoints"
106
+
>```
107
+
108
+
In the external storage, the directory structure of the checkpoint data is as follows:
109
+
110
+
- Root path `restore-{downstream-cluster-ID}` uses the downstream cluster ID `{downstream-cluster-ID}` to distinguish between different restore clusters.
111
+
- Path `restore-{downstream-cluster-ID}/log` stores log file checkpoint data during the log restore phase.
112
+
- Path `restore-{downstream-cluster-ID}/sst` stores checkpoint data of the SST files that are not backed up by log backup during the log restore phase.
113
+
- Path `restore-{downstream-cluster-ID}/snapshot` stores checkpoint data during the snapshot restore phase.
114
+
115
+
```
116
+
.
117
+
`-- restore-{downstream-cluster-ID}
118
+
|-- log
119
+
||-- checkpoint.meta
120
+
||-- data
121
+
|||-- {uuid}.cpt
122
+
|||-- {uuid}.cpt
123
+
||`-- {uuid}.cpt
124
+
||-- ingest_index.meta
125
+
|`-- progress.meta
126
+
|-- snapshot
127
+
||-- checkpoint.meta
128
+
||-- checksum
129
+
|||-- {uuid}.cpt
130
+
|||-- {uuid}.cpt
131
+
||`-- {uuid}.cpt
132
+
|`-- data
133
+
||-- {uuid}.cpt
134
+
||-- {uuid}.cpt
135
+
|`-- {uuid}.cpt
136
+
`-- sst
137
+
`-- checkpoint.meta
138
+
```
139
+
140
+
Checkpoint restore operations are divided into two parts: snapshot restore and PITR restore.
141
+
142
+
### Snapshot restore
143
+
144
+
During the initial restore, `br` creates a `restore-{downstream-cluster-ID}/snapshot` path in the specified external storage. In this path, `br` records checkpoint data, the upstream cluster ID, and the BackupTS of the backup data.
145
+
146
+
If the restore fails, you can retry it using the same command. `br` will automatically read the checkpoint information from the specified external storage path and resume from the last restore point.
147
+
148
+
If the restore fails and you try to restore backup data with different checkpoint information to the same cluster, `br` reports an error. It indicates that the current upstream cluster ID or BackupTS is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup.
149
+
150
+
### PITR restore
151
+
152
+
[PITR (Point-in-time recovery)](/br/br-pitr-guide.md) consists of snapshot restore and log restore phases.
153
+
154
+
During the initial restore, `br` first enters the snapshot restore phase. BR records the checkpoint data, the upstream cluster ID, BackupTS of the backup data (that is, the start time point `start-ts` of log restore) and the restored time point `restored-ts` of log restore in the `restore-{downstream-cluster-ID}/snapshot` path. If restore fails during this phase, you cannot adjust the `start-ts` and `restored-ts` of log restore when resuming checkpoint restore.
155
+
156
+
When entering the log restore phase during the initial restore, `br` creates a `restore-{downstream-cluster-ID}/log` path in the specified external storage. This path records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup.
157
+
158
+
Note that before entering the log restore phase during the initial restore, `br` constructs a mapping of the database and table IDs in the upstream and downstream clusters at the `restored-ts` time point. This mapping is persisted in the checkpoint storage with the file name `pitr_id_maps/pitr_id_map.cluster_id:{downstream-cluster-ID}.restored_ts:{restored-ts}` to prevent duplicate allocation of database and table IDs. **Deleting files from the directory `pitr_id_maps` arbitrarily might lead to inconsistent PITR restore data.**
159
+
160
+
> **Note:**
161
+
>
162
+
> To ensure compatibility with clusters of earlier versions, starting from v8.5.5, if the system table `mysql.tidb_pitr_id_map` does not exist in the restore cluster and the `--checkpoint-storage` parameter is not specified, the `pitr_id_map` data will be written to the log backup directory. The file name is `pitr_id_maps/pitr_id_map.cluster_id:{downstream-cluster-ID}.restored_ts:{restored-ts}`.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?v=4&size=40){kind=avatar}
0 commit comments