You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: modules/ROOT/pages/clustering/multi-region-deployment/disaster-recovery.adoc
+20-14Lines changed: 20 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -152,9 +152,6 @@ Use the following steps to regain write availability for the `system` database i
152
152
They create a new `system` database from the most up-to-date copy of the `system` database that can be found in the cluster.
153
153
It is important to get a `system` database that is as up-to-date as possible, so it corresponds to the view before the disaster closely.
154
154
155
-
.Guide
156
-
[%collapsible]
157
-
====
158
155
159
156
[NOTE]
160
157
=====
@@ -180,10 +177,12 @@ Be aware that not replacing servers can cause cluster overload when databases ar
180
177
=====
181
178
+
182
179
. On each server, run `bin/neo4j-admin database load system --from-path=[path-to-dump] --overwrite-destination=true` to load the current `system` database dump.
180
+
+
181
+
image::system-db-restored.svg[width="400", title="The `system` database is restored and unconstrained servers are added", role=popup]
182
+
+
183
183
. On each server, ensure that the discovery settings are correct.
184
184
See xref:clustering/setup/discovery.adoc[Cluster server discovery] for more information.
185
185
. Start the Neo4j process on all servers.
186
-
====
187
186
188
187
189
188
[[make-servers-available]]
@@ -217,16 +216,17 @@ This is done in two different steps:
217
216
* Any allocations that cannot move by themselves require the database to be recreated so that they are forced to move.
218
217
* Any allocations that can move will be instructed to do so by deallocating the server.
219
218
220
-
.Guide
221
-
[%collapsible]
222
-
====
219
+
223
220
. For each `Unavailable` server, run `CALL dbms.cluster.cordonServer("unavailable-server-id")` on one of the available servers.
224
221
This prevents new database allocations from being moved to this server.
. For each `Cordoned` server, make sure a new *unconstrained* server has been added to the cluster to take its place.
226
226
See xref:clustering/servers.adoc#cluster-add-server[Add a server to the cluster] for more information.
227
227
+
228
-
If servers were added in the <<make-the-system-database-write-available, Make the `system` database write-available>> step of this guide, additional servers might not be needed here.
229
-
It is important that the new servers are unconstrained, or deallocating servers might be blocked even though enough servers were added.
228
+
If servers were added in the <<make-the-system-database-write-available, Make the `system` database write-available>> step of this guide (like it is done in the current disaster recovery example), additional servers might not be needed here.
229
+
It is important that the new servers are unconstrained, or deallocating servers might be blocked even though enough servers were added.
230
230
+
231
231
[NOTE]
232
232
=====
@@ -266,10 +266,14 @@ If any database has `currentStatus` = `quarantined` on an available server, recr
266
266
=====
267
267
If you recreate databases using xref:database-administration/standard-databases/recreate-database.adoc#undefined-servers[undefined servers] or xref:database-administration/standard-databases/recreate-database.adoc#undefined-servers-backup[undefined servers with fallback backup], the store might not be recreated as up-to-date as possible in certain edge cases where the `system` database has been restored.
. For each `Cordoned` server, run `DEALLOCATE DATABASES FROM SERVER cordoned-server-id` on one of the available servers.
271
273
This will move all database allocations from this server to an available server in the cluster.
272
274
+
275
+
image::servers-deallocated.svg[width="400", title="Deallocate databases from unavailable servers", role=popup]
276
+
+
273
277
[NOTE]
274
278
=====
275
279
This operation might fail if enough unconstrained servers were not added to the cluster to replace lost servers.
@@ -278,7 +282,7 @@ Another reason is that some available servers are also `Cordoned`.
278
282
279
283
. For each deallocating or deallocated server, run `DROP SERVER deallocated-server-id`.
280
284
This removes the server from the cluster's view.
281
-
====
285
+
282
286
283
287
284
288
[[make-databases-write-available]]
@@ -318,13 +322,12 @@ A stricter verification can be done to verify that all databases are in their de
318
322
For the stricter check, run `SHOW DATABASES` and verify that `requestedStatus` = `currentStatus` for all database allocations on all servers.
319
323
320
324
==== Path to correct state
325
+
321
326
Use the following steps to make all databases in the cluster write-available again.
322
327
They include recreating any databases that are not write-available and identifying any recreations that will not complete.
323
328
Recreations might fail for different reasons, but one example is that the checksums do not match for the same transaction on different servers.
324
329
325
-
.Guide
326
-
[%collapsible]
327
-
====
330
+
328
331
. Identify all write-unavailable databases by running `CALL dbms.cluster.statusCheck([])` as described in the <<#example-verification, Example verification>> part of this disaster recovery step.
329
332
Filter out all databases desired to be stopped, so that they are not recreated unnecessarily.
330
333
. Recreate every database that is not write-available and has not been recreated previously.
@@ -345,4 +348,7 @@ Recreating a database will not complete if one of the following messages is disp
345
348
** `No store found on any of the seeders ServerId1, ServerId2...`
346
349
. For each database which will not complete recreation, recreate them from backup using xref:database-administration/standard-databases/recreate-database.adoc#uri-seed[Backup as seed].
0 commit comments