-
Notifications
You must be signed in to change notification settings - Fork 83
Add a page on how to remove a server from the cluster on AWS #2217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
99 changes: 99 additions & 0 deletions
99
modules/ROOT/pages/cloud-deployments/neo4j-cluster-cloud.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| :description: Tha page describes how to manage the Neo4j cluster on AWS. | ||
| :page-role: enterprise-edition | ||
|
|
||
| [[neo4j-cluster-cloud-deployments]] | ||
| = Neo4j cluster in self-managed cloud deployments | ||
|
|
||
| Before diving into the topic, it is important to understand basics about Neo4j's clustering. | ||
|
|
||
| Neo4j cluster consists of a homogenous pool of servers that collectively run a number of databases. | ||
| The servers can operate in two different database-hosting modes: _primary_ and _secondary_. | ||
| A server can simultaneously act as a primary host for one or more databases and as a secondary host for other databases. | ||
|
|
||
| For more details on operational and application aspects of Neo4j's clustering, refer to the xref::clustering/index.adoc[Clustering in Neo4j]. | ||
|
|
||
| For information on how to manage databases and servers in a cluster, see respectively xref::clustering/databases.adoc[] and xref::clustering/servers.adoc[]. | ||
|
|
||
|
|
||
| == Neo4j cluster on AWS | ||
|
|
||
| Neo4j does not provide Amazon Machine Images (AMIs) with a pre-installed version of the product. | ||
| The Neo4j AWS Marketplace listings (and listings on GitHub) use CloudFormation templates that deploy and configure Neo4j dynamically with a shell script. | ||
|
|
||
|
|
||
| // === Neo4j cluster and auto-scaling groups on AWS | ||
|
|
||
|
|
||
| === Remove a server from the Neo4j cluster | ||
|
|
||
| Rolling updates on Amazon Machine Images (AMIs) often involve their rotating. | ||
NataliaIvakina marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
NataliaIvakina marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| However, simply removing Neo4j servers from the target Network Load Balancer (NLB) one by one does not prevent requests from being routed to them. | ||
| This occurs because the NLB and Neo4j server-side routing operate independently and do not share awareness of a server availability. | ||
|
|
||
| To correctly remove a server from the cluster and reintroduce it after the update, follow the steps outlined below: | ||
|
|
||
| . Remove the server from the AWS NLB. | ||
| This prevents external clients from sending requests to the server. | ||
|
|
||
| . Since Neo4j's cluster routing (server-side routing) does not use the NLB, you need to ensure that queries are not routed to the server. | ||
| To do this, you have to cleanly shut down the server. | ||
|
|
||
| .. Run the following query to check servers are hosting all their assigned databases. | ||
| The query should return no results: | ||
| + | ||
| [source, cypher, role=noplay] | ||
| ---- | ||
| SHOW SERVERS YIELD name, hosting, requestedHosting, serverId WHERE requestedHosting <> hosting | ||
| ---- | ||
|
|
||
| .. Use the following query to check all databases are in their expected state. | ||
| The query should return no results: | ||
| + | ||
| [source, cypher, role=noplay] | ||
| ---- | ||
| SHOW DATABASES YIELD name, address, currentStatus, requestedStatus, statusMessage WHERE currentStatus <> requestedStatus RETURN name, address, currentStatus, requestedStatus, statusMessage | ||
| ---- | ||
|
|
||
| .. To stop the Neo4j service, run the following command: | ||
| + | ||
| [source, shell, role=copy] | ||
| ---- | ||
| sudo systemctl stop neo4j | ||
| ---- | ||
| + | ||
| To configure the timeout period for waiting on active transactions to either complete or be terminated before the shutdown, modify the setting xref::configuration/configuration-settings.adoc#config_db.shutdown_transaction_end_timeout[`db.shutdown_transaction_end_timeout`] in the _neo4j.conf_ file. | ||
| `db.shutdown_transaction_end_timeout` defaults to 10 seconds. | ||
| + | ||
| The environment variable `NEO4J_SHUTDOWN_TIMEOUT` determines how long the system will wait for Neo4j to stop before forcefully terminating the process. | ||
| You can change this using `systemctl edit neo4j.service`. | ||
| By default, `NEO4J_SHUTDOWN_TIMEOUT` is set to 120 seconds. | ||
| If the shutdown process exceeds this limit, it is considered failed. | ||
| You may need to increase the value if the system serves long-running transactions. | ||
|
|
||
| .. Verify that the shutdown process has finished successfully by checking the _neo4j.log_ for relevant log messages confirming the shutdown. | ||
|
|
||
|
|
||
| . When everything is updated or fixed, start the servers one by one again. | ||
| .. Run `systemctl start neo4j`. | ||
| .. Once the server has been restarted, confirm it is running successfully. | ||
| + | ||
| Run the following command and check the server has state `Enabled` and health `Available`. | ||
| + | ||
| [source, cypher, role=noplay] | ||
| ---- | ||
| SHOW SERVERS WHERE name = [server-id]; | ||
| ---- | ||
|
|
||
| .. Confirm that the server has started all the databases that it should. | ||
| + | ||
| This command shows any databases that are not in their expected state: | ||
| + | ||
| [source, cypher, role=noplay] | ||
| ---- | ||
| SHOW DATABASES YIELD name, address, currentStatus, requestedStatus, serverID WHERE currentStatus <> requestedStatus AND serverID = [server-id] RETURN name, address, currentStatus, requestedStatus | ||
| ---- | ||
|
|
||
| . Reattach the server to the NLB. | ||
| Once the server is stable and caught up, add it back to the AWS NLB target group. | ||
|
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.