You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The consequence of this is that all servers may be different in which databases they are hosting.
26
26
Losing a server in a cluster may cause some databases to lose a member while others are unaffected.
@@ -39,23 +39,28 @@ image::disaster.svg[width="400", title="Example of a cluster disaster", role=pop
39
39
40
40
|Database A
41
41
|All allocations are lost.
42
-
|The database needs to be recreated from a backup.
42
+
|The database needs to be recreated from a backup since there are no available allocations left in the cluster.
43
43
44
44
|Database B
45
45
|The primary allocation is lost, and the secondary allocation is available.
46
-
|The database needs to be recreated, but can be based on available allocations in the cluster.
46
+
|The database needs to be recreated since it has lost a majority of primary allocations and is therefore write-unvailable.
47
+
However, the recreation can be based on the secondary allocation still present on a healthy server, so a backup is not required.
48
+
The recreated database will be as up-to-date as the secondary allocation was at the time of the disaster.
47
49
48
50
|Database C
49
51
|Two primary allocations and a secondary one are lost.
50
-
|The database needs to be recreated, but can be based on available allocations in the cluster.
52
+
|The database needs to be recreated since it has lost a majority of primary allocations and is therefore write-unavailable.
53
+
However, the recreation can be based on the primary and secondary allocations still present on healthy servers, so a backup is not required.
54
+
The recreated database will reflect the state of the most up-to-date surviving primary or secondary allocation.
51
55
52
56
|Database D
53
57
|One primary allocation and two secondary allocations are lost.
54
-
|The database will move when a server is deallocated.
58
+
|The database remains write-available, allowing it to automatically move allocations from lost servers to available ones when the lost servers are deallocated.
59
+
Therefore, the database does not need to be recreated even though some allocations have been lost.
55
60
56
61
|Database E
57
62
|Stays unaffected.
58
-
|No action is required.
63
+
|None of the database's allocations were affected by the disaster, so no action is required.
59
64
|===
60
65
61
66
Although databases C and D share the same topology, their primaries and secondaries are allocated differently, requiring distinct recovery strategies in this disaster example.
@@ -227,7 +232,7 @@ image::servers-cordoned.svg[width="400", title="Cordon unavailable servers", rol
227
232
. For each `Cordoned` server, make sure a new *unconstrained* server has been added to the cluster to take its place.
228
233
See xref:clustering/servers.adoc#cluster-add-server[Add a server to the cluster] for more information.
229
234
+
230
-
If servers were added in the <<make-the-system-database-write-available, Make the `system` database write-available>> step of this guide (like it is done in the current disaster recovery example), additional servers might not be needed here.
235
+
If servers were added in the <<make-the-system-database-write-available, Make the `system` database write-available>> step of this guide (like it has been done in the current disaster recovery example), additional servers might not be needed here.
231
236
It is important that the new servers are unconstrained, or deallocating servers might be blocked even though enough servers were added.
232
237
+
233
238
[NOTE]
@@ -269,13 +274,15 @@ If any database has `currentStatus` = `quarantined` on an available server, recr
269
274
If you recreate databases using xref:database-administration/standard-databases/recreate-database.adoc#undefined-servers[undefined servers] or xref:database-administration/standard-databases/recreate-database.adoc#undefined-servers-backup[undefined servers with fallback backup], the store might not be recreated as up-to-date as possible in certain edge cases where the `system` database has been restored.
image::servers-cordoned-databases-moved.svg[width="400", title="All write-unavailable databases were recreated", role=popup]
273
278
274
279
. For each `Cordoned` server, run `DEALLOCATE DATABASES FROM SERVER cordoned-server-id` on one of the available servers.
275
280
This will move all database allocations from this server to an available server in the cluster.
276
281
+
277
282
image::servers-deallocated.svg[width="400", title="Deallocate databases from unavailable servers", role=popup]
278
283
+
284
+
Note that the database D was still write-available, which means the allocations can be moved from lost servers to available ones when the lost servers are deallocated.
285
+
+
279
286
[NOTE]
280
287
=====
281
288
This operation might fail if enough unconstrained servers were not added to the cluster to replace lost servers.
@@ -284,8 +291,11 @@ Another reason is that some available servers are also `Cordoned`.
284
291
285
292
. For each deallocating or deallocated server, run `DROP SERVER deallocated-server-id`.
After dropping the deallocated servers, you still have to ensure that all moved and recreated databases are write-available.
298
+
For this purpose, follow the steps <<write-available-databases-steps, below>>.
289
299
290
300
[[make-databases-write-available]]
291
301
=== Make databases write-available
@@ -323,6 +333,7 @@ Instead, check that the primary is allocated on an available server and that it
323
333
A stricter verification can be done to verify that all databases are in their desired states on all servers.
324
334
For the stricter check, run `SHOW DATABASES` and verify that `requestedStatus` = `currentStatus` for all database allocations on all servers.
325
335
336
+
[[write-available-databases-steps]]
326
337
==== Path to correct state
327
338
328
339
Use the following steps to make all databases in the cluster write-available again.
@@ -350,7 +361,7 @@ Recreating a database will not complete if one of the following messages is disp
350
361
** `No store found on any of the seeders ServerId1, ServerId2...`
351
362
. For each database which will not complete recreation, recreate them from backup using xref:database-administration/standard-databases/recreate-database.adoc#uri-seed[Backup as seed].
0 commit comments