|
1 | 1 |
|
2 | | -# AWS Blockchain Node Runners |
3 | | - |
4 | | -[AWS Blockchain Node Runners (Node Runners)](https://aws-samples.github.io/aws-blockchain-node-runners/docs/intro) is an open-source initiative aimed at simplifying the deployment of self-managed blockchain nodes on AWS using vetted deployment blueprints and infrastructure configurations. Node Runners is designed to solve for the common challenges customers face in architecting and deploying blockchain nodes on AWS, such as identifying the optimal configurations for each specific blockchain protocol’s node client. |
5 | | - |
6 | | -## Architecture Overview |
7 | | - |
8 | | -This blueprint has two configuration options for running Ethereum nodes and supports several Ethereum client combinations. You can set up a single JSON RPC node or multiple nodes in highly-available setup using the instructions below. |
9 | | - |
10 | | -### Single RPC node setup |
11 | | - |
12 | | - |
13 | | -This setup is for small scale PoC or development environments. It deploys a single EC2 instance with both consensus and execution clients. The RPC port is exposed only to internal IP range of the VPC, while P2P ports allow external access to keep the clients synced. |
14 | | - |
15 | | -### Highly available setup |
16 | | - |
17 | | - |
18 | | -1. An ongoing data synchronization process is configured with nodes in the Ethereum network with a sync node and RPC nodes. |
19 | | -2. The sync node is used to create a copy of node's state data in Amazon S3 bucket. |
20 | | -3. When new RPC nodes are provisioned, they copy state data from Amazon S3 bucket to speed up the initial sync process. |
21 | | -4. Applications and smart contract development tools access highly available RPC nodes behind the Application Load Balancer. |
22 | | - |
23 | | -## Well-Architected |
24 | | - |
25 | | -<details> |
26 | | -<summary>Review the for pros and cons of this solution.</summary> |
27 | | - |
28 | | -### Well-Architected Checklist |
29 | | - |
30 | | -This is the Well-Architected checklist for Ethereum nodes implementation of the AWS Blockchain Node Runner app. This checklist takes into account questions from the [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected/) which are relevant to this workload. Please feel free to add more checks from the framework if required for your workload. |
31 | | - |
32 | | -| Pillar | Control | Question/Check | Remarks | |
33 | | -|:------------------------|:----------------------------------|:---------------------------------------------------------------------------------|:-----------------| |
34 | | -| Security | Network protection | Are there unnecessary open ports in security groups? | Please note that Erigon snap sync port remains open for non-erigon clients, i.e. Port 42069 (TCP/UDP). | |
35 | | -| | | Traffic inspection | AWS WAF could be implemented for traffic inspection. Additional charges will apply. | |
36 | | -| | Compute protection | Reduce attack surface | This solution uses Amazon Linux 2 AMI. You may choose to run hardening scripts on it. | |
37 | | -| | | Enable people to perform actions at a distance | This solution uses AWS Systems Manager for terminal session, not ssh ports. | |
38 | | -| | Data protection at rest | Use encrypted Amazon Elastic Block Store (Amazon EBS) volumes | This solution uses encrypted Amazon EBS volumes. | |
39 | | -| | | Use encrypted Amazon Simple Storage Service (Amazon S3) buckets | This solution uses Amazon S3 managed keys (SSE-S3) encryption. | |
40 | | -| | Data protection in transit | Use TLS | The AWS Application Load balancer currently uses HTTP listener. Create HTTPS listener with self signed certificate if TLS is desired. | |
41 | | -| | Authorization and access control | Use instance profile with Amazon Elastic Compute Cloud (Amazon EC2) instances | This solution uses AWS Identity and Access Management (AWS IAM) role instead of IAM user. | |
42 | | -| | | Following principle of least privilege access | In sync node, root user is not used (using special user "ethereum" instead"). | |
43 | | -| | Application security | Security focused development practices | cdk-nag is being used with appropriate suppressions. | |
44 | | -| Cost optimization | Service selection | Use cost effective resources | AWS Graviton-based Amazon EC2 instances are being used, which are cost effective compared to Intel/AMD instances. | |
45 | | -| | Cost awareness | Estimate costs | One sync node with m7g.2xlarge for geth-lighthouse configuration (2048GB ssd) will cost around US$430 per month in the US East (N. Virginia) region. Additional charges will apply if you choose to deploy RPC nodes with load balancer. | |
46 | | -| Reliability | Resiliency implementation | Withstand component failures | This solution uses AWS Application Load Balancer with RPC nodes for high availability. If sync node fails, Amazon S3 backup can be used to reinstate the nodes. | |
47 | | -| | Data backup | How is data backed up? | Data is backed up to Amazon S3 using [s5cmd](https://github.com/peak/s5cmd) tool. | |
48 | | -| | Resource monitoring | How are workload resources monitored? | Resources are being monitored using Amazon CloudWatch dashboards. Amazon CloudWatch custom metrics are being pushed via CloudWatch Agent. | |
49 | | -| Performance efficiency | Compute selection | How is compute solution selected? | Compute solution is selected based on best price-performance, i.e. AWS Graviton-based Amazon EC2 instances. | |
50 | | -| | Storage selection | How is storage solution selected? | Storage solution is selected based on best price-performance, i.e. gp3 Amazon EBS volumes with optimal IOPS and throughput. | |
51 | | -| | Architecture selection | How is the best performance architecture selected? | s5cmd tool has been chosen for Amazon S3 uploads/downloads because it gives better price-performance compared to Amazon EBS snapshots (including Fast Snapshot Restore, which can be expensive). | |
52 | | -| Operational excellence | Workload health | How is health of workload determined? | Health of workload is determined via AWS Application Load Balancer Target Group Health Checks, on port 8545. | |
53 | | -| Sustainability | Hardware & services | Select most efficient hardware for your workload | This solution uses AWS Graviton-based Amazon EC2 instances which offer the best performance per watt of energy use in Amazon EC2. | |
54 | | - |
55 | | -</details> |
56 | | - |
57 | | -## Solution Walkthrough |
58 | | - |
59 | | -### Open AWS CloudShell |
60 | | - |
61 | | -To begin, ensure you login to your AWS account with permissions to create and modify resources in IAM, EC2, EBS, VPC, S3, KMS, and Secrets Manager. |
62 | | - |
63 | | -From the AWS Management Console, open the [AWS CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html), a web-based shell environment. If unfamiliar, review the [2-minute YouTube video](https://youtu.be/fz4rbjRaiQM) for an overview and check out [CloudShell with VPC environment](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html) that we'll use to test nodes API from internal IP address space. |
64 | | - |
65 | | -Once ready, you can run the commands to deploy and test blueprints in the CloudShell. |
66 | | - |
67 | | -### Clone this repository and install dependencies |
68 | | - |
69 | | -```bash |
70 | | -git clone https://github.com/aws-samples/aws-blockchain-node-runners.git |
71 | | -cd aws-blockchain-node-runners |
72 | | -npm install |
73 | | -``` |
74 | | - |
75 | | -> **NOTE:** *In this tutorial we will set all major configuration through environment variables, but you also can modify parameters in `config/config.ts`.* |
76 | | -
|
77 | | -### Prepare to deploy nodes |
78 | | - |
79 | | -1. Make sure you are in the root directory of the cloned repository |
80 | | - |
81 | | -2. If you have deleted or don't have the default VPC, create default VPC |
82 | | - |
83 | | -```bash |
84 | | -aws ec2 create-default-vpc |
85 | | -``` |
86 | | - |
87 | | -> **NOTE:** *You may see the following error if the default VPC already exists: `An error occurred (DefaultVpcAlreadyExists) when calling the CreateDefaultVpc operation: A Default VPC already exists for this account in this region.`. That means you can just continue with the following steps.* |
88 | | -
|
89 | | -> **NOTE:** *The default VPC must have at least two public subnets in different Availability Zones, and public subnet must set `Auto-assign public IPv4 address` to `YES`* |
90 | | -
|
91 | | -3. Configure your Node Runners Ethereum blueprint deployment |
| 2 | +# Configure your Node Runners Ethereum - Geth Lighthouse |
92 | 3 |
|
93 | 4 | To specify the Ethereum client combination you wish to deploy, create your own copy of `.env` file and edit it using your preferred text editor. The contents of your file for a Geth / Lighthouse node deployment is as follows, which uses a sample config from the repository: |
94 | 5 | ```bash |
|
98 | 9 | cp ./sample-configs/.env-geth-lighthouse .env |
99 | 10 | nano .env |
100 | 11 | ``` |
101 | | -> **NOTE:** *You can find more examples inside the `sample-configs` directory, which illustrate other Ethereum client combinations.* |
102 | | -
|
103 | | - |
104 | | -4. Deploy common components such as IAM role, and Amazon S3 bucket to store data snapshots |
105 | | - |
106 | | -```bash |
107 | | -pwd |
108 | | -# Make sure you are in aws-blockchain-node-runners/lib/ethereum |
109 | | -npx cdk deploy eth-common |
110 | | -``` |
111 | | - |
112 | | -### Option 1: Single RPC Node |
113 | | - |
114 | | -1. Deploy Single RPC Node |
115 | | - |
116 | | -```bash |
117 | | -pwd |
118 | | -# Make sure you are in aws-blockchain-node-runners/lib/ethereum |
119 | | -npx cdk deploy eth-single-node --json --outputs-file single-node-deploy.json |
120 | | -``` |
121 | | -> **NOTE:** *The default VPC must have at least two public subnets in different Availability Zones, and public subnet must set `Auto-assign public IPv4 address` to `YES`* |
122 | | -
|
123 | | -2. After starting the node you need to wait for the inital syncronization process to finish. It may take from half a day to about 6-10 days depending on the client combination and the state of the network. You can use Amazon CloudWatch to track the progress. There is a script that publishes CloudWatch metrics every 5 minutes, where you can watch `sync distance` for consensus client and `blocks behind` for execution client. When the node is fully synced those two metrics shold show 0. To see them: |
124 | | - |
125 | | - - Navigate to [CloudWatch service](https://console.aws.amazon.com/cloudwatch/) (make sure you are in the region you have specified for `AWS_REGION`) |
126 | | - - Open `Dashboards` and select `eth-sync-node-<your-eth-client-combination>` from the list of dashboards. |
127 | | - |
128 | | -4. Once the initial synchronization is done, you should be able to access the RPC API of that node from within the same VPC. The RPC port is not exposed to the Internet. Turn the following query against the private IP of the single RPC node you deployed: |
129 | | - |
130 | | -```bash |
131 | | -INSTANCE_ID=$(cat single-node-deploy.json | jq -r '..|.node-instance-id? | select(. != null)') |
132 | | -NODE_INTERNAL_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --query 'Reservations[*].Instances[*].PrivateIpAddress' --output text) |
133 | | -echo "NODE_INTERNAL_IP=$NODE_INTERNAL_IP" |
134 | | -``` |
135 | | - |
136 | | -Copy output from the last `echo` command with `NODE_INTERNAL_IP=<internal_IP>` and open [CloudShell tab with VPC environment](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html) to access internal IP address space. Paste `NODE_INTERNAL_IP=<internal_IP>` into the new CloudShell tab. Then query the API: |
137 | | - |
138 | | -``` bash |
139 | | -# IMPORTANT: Run from CloudShell VPC environment tab |
140 | | -# We query token balance of Beacon deposit contract: https://etherscan.io/address/0x00000000219ab540356cbb839cbe05303d7705fa |
141 | | -curl http://$NODE_INTERNAL_IP:8545 -X POST -H "Content-Type: application/json" \ |
142 | | ---data '{"method":"eth_getBalance","params":["0x00000000219ab540356cBB839Cbe05303d7705Fa", "latest"],"id":1,"jsonrpc":"2.0"}' |
143 | | -``` |
144 | | - |
145 | | -The result should be like this (the actual balance might change): |
146 | | - |
147 | | -```javascript |
148 | | -{"jsonrpc":"2.0","id":1,"result":"0xe791d050f91d9949d344d"} |
149 | | -``` |
150 | | - |
151 | | -### Option 2: Highly Available RPC Nodes |
152 | | - |
153 | | -1. Deploy Sync Node |
154 | | - |
155 | | -```bash |
156 | | -pwd |
157 | | -# Make sure you are in aws-blockchain-node-runners/lib/ethereum |
158 | | -npx cdk deploy eth-sync-node --json --outputs-file sync-node-deploy.json |
159 | | -``` |
160 | | -**NOTE:** *The default VPC must have at least two public subnets in different Availability Zones, and public subnet must set `Auto-assign public IPv4 address` to `YES`* |
161 | | - |
162 | | -2. After starting the node you need to wait for the inital syncronization process to finish. It may take from half a day to about 6-10 days depending on the client combination and the state of the network. You can use Amazon CloudWatch to track the progress. There is a script that publishes CloudWatch metrics every 5 minutes, where you can watch `sync distance` for consensus client and `blocks behind` for execution client. When the node is fully synced those two metrics shold show 0. To see them: |
163 | | - |
164 | | - - Navigate to [CloudWatch service](https://console.aws.amazon.com/cloudwatch/) (make sure you are in the region you have specified for `AWS_REGION`) |
165 | | - - Open `Dashboards` and select `eth-sync-node-<your-eth-client-combination>` from the list of dashboards. |
166 | | - |
167 | | -Once synchronization process is over, the script will automatically stop both clients and copy all the contents of the `/data` directory to your snapshot S3 bucket. That may take from 30 minutes to about 2 hours. During the process on the dashboard you will see lower CPU and RAM utilization but high data disc throughput and outbound network traffic. The script will automatically start the clients after the process is done. |
168 | | - |
169 | | -> **Note:** *The snapshot backup process will automatically run every day at midnight of the time zone were the sync node runs. To change the schedule, modify `crontab` of the root user on the node's EC2 instance.* |
170 | | -
|
171 | | -3. Configure and deploy 2 RPC Nodes |
172 | | - |
173 | | -```bash |
174 | | -pwd |
175 | | -# Make sure you are in aws-blockchain-node-runners/lib/ethereum |
176 | | -npx cdk deploy eth-rpc-nodes --json --outputs-file rpc-node-deploy.json |
177 | | -``` |
178 | | - |
179 | | -4. Give the new RPC nodes about 30 minutes (up to 2 hours for Erigon(???)) to initialize and then run the following query against the load balancer behind the RPC node created |
180 | | - |
181 | | -```bash |
182 | | -export ETH_RPC_ABL_URL=$(cat rpc-node-deploy.json | jq -r '..|.alburl? | select(. != null)') |
183 | | -echo ETH_RPC_ABL_URL=$ETH_RPC_ABL_URL |
184 | | -``` |
185 | | - |
186 | | -```bash |
187 | | -# IMPORTANT: Run from CloudShell VPC environment tab |
188 | | -# We query token balance of Beacon deposit contract: https://etherscan.io/address/0x00000000219ab540356cbb839cbe05303d7705fa |
189 | | -curl http://$ETH_RPC_ABL_URL:8545 -X POST -H "Content-Type: application/json" \ |
190 | | - --data '{"method":"eth_getBalance","params":["0x00000000219ab540356cBB839Cbe05303d7705Fa", "latest"],"id":1,"jsonrpc":"2.0"}' |
191 | | -``` |
192 | | - |
193 | | -The result should be like this (the actual balance might change): |
194 | | - |
195 | | -```javascript |
196 | | -{"jsonrpc":"2.0","id":1,"result":"0xe791d050f91d9949d344d"} |
197 | | -``` |
198 | | - |
199 | | - If the nodes are still starting and catching up with the chain, you will see the following repsonse: |
200 | | - |
201 | | -```HTML |
202 | | - <html> |
203 | | - <head><title>503 Service Temporarily Unavailable</title></head> |
204 | | - <body> |
205 | | - <center><h1>503 Service Temporarily Unavailable</h1></center> |
206 | | - </body> |
207 | | -``` |
208 | | - |
209 | | -> **NOTE:** By default and for security reasons the load balancer is available only from within the default VPC in the region where it is deployed. It is not available from the Internet and is not open for external connections. Before opening it up please make sure you protect your RPC APIs. |
210 | | -
|
211 | | -### Clearing up and undeploying everything |
212 | | - |
213 | | -Destroy RPC Nodes, Sync Nodes and Comon components |
214 | | - |
215 | | -```bash |
216 | | -# Setting the AWS account id and region in case local .env file is lost |
217 | | - export AWS_ACCOUNT_ID=<your_target_AWS_account_id> |
218 | | - export AWS_REGION=<your_target_AWS_region> |
219 | | - |
220 | | -pwd |
221 | | -# Make sure you are in aws-blockchain-node-runners/lib/ethereum |
222 | | - |
223 | | -# Destroy Single RPC Node |
224 | | -cdk destroy eth-single-node |
225 | | - |
226 | | -# Destroy RPC Nodes |
227 | | -cdk destroy eth-rpc-nodes |
228 | | - |
229 | | -# Destroy Sync Node |
230 | | -cdk destroy eth-sync-node |
231 | | - |
232 | | -# You need to manually delete an s3 bucket with a name similar to 'eth-snapshots-$accountid-eth-nodes-common' on the console,firstly empty the bucket,secondly delete the bucket,and then execute |
233 | | -# Delete all common components like IAM role and Security Group |
234 | | -cdk destroy eth-common |
235 | | -``` |
236 | | - |
| 12 | +> **NOTE:** *You can find more examples inside the `sample-configs` directory, which illustrate other Ethereum client combinations.* |
0 commit comments