Base. Added Instance Store volume option for storage

Author: vlasonfa

lib/base/README.md

@@ -122,7 +122,7 @@ npx cdk deploy base-common
    > cdk bootstrap aws://ACCOUNT-NUMBER/REGION
    > ```
 
-### From your Cloud9: Deploy Single Node
+### Option 1: Deploy Single Node
 
 1. For L1 node you you can set your own URLs in `BASE_L1_EXECUTION_ENDPOINT` and `BASE_L1_CONSENSUS_ENDPOINT` properties of `.env` file. It can be one of [the providers recommended by Base](https://docs.base.org/tools/node-providers) or you can run your own Ethereum node [with Node Runner Ethereum blueprint](https://aws-samples.github.io/aws-blockchain-node-runners/docs/Blueprints/Ethereum) (tested with geth-lighthouse combination). For example:
 
@@ -163,12 +163,48 @@ aws ssm start-session --target $INSTANCE_ID --region $AWS_REGION
 curl -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' http://localhost:8545
 ```
 
+### Option 2: Highly Available RPC Nodes
+
+1. For L1 node you you can set your own URLs in `BASE_L1_EXECUTION_ENDPOINT` and `BASE_L1_CONSENSUS_ENDPOINT` properties of `.env` file. It can be one of [the providers recommended by Base](https://docs.base.org/tools/node-providers) or you can run your own Ethereum node [with Node Runner Ethereum blueprint](https://aws-samples.github.io/aws-blockchain-node-runners/docs/Blueprints/Ethereum) (tested with geth-lighthouse combination). For example:
+
+```bash
+#For Sepolia:
+BASE_L1_EXECUTION_ENDPOINT="https://ethereum-sepolia-rpc.publicnode.com"
+BASE_L1_CONSENSUS_ENDPOINT="https://ethereum-sepolia-beacon-api.publicnode.com"
+```
+
+2. Deploy Base RPC Node and wait for it to sync. For Mainnet it might a day when using snapshots or about a week if syncing from block 0. You can use snapshots provided by the Base team by setting `BASE_RESTORE_FROM_SNAPSHOT="true"` in `.env` file.
+
+   ```bash
+      pwd
+      # Make sure you are in aws-blockchain-node-runners/lib/base
+      npx cdk deploy base-ha-nodes --json --outputs-file ha-nodes-deploy.json
+   ```
+
+2. Give the new RPC **full** nodes about 2-3 hours (24 hours for **archive** node) to initialize and then run the following query against the load balancer behind the RPC node created
+
+   ```bash
+      export RPC_ALB_URL=$(cat ha-nodes-deploy.json | jq -r '..|.alburl? | select(. != null)')
+      echo $RPC_ALB_URL
+   ```
+
+   Periodically check [Geth Syncing Status](https://geth.ethereum.org/docs/fundamentals/logs#syncing). Run the following query from within the same VPC and against the private IP of the load balancer fronting your nodes:
+
+   ```bash
+   curl http://$RPC_ALB_URL:8545 -X POST -H "Content-Type: application/json" \
+   --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
+   ```
+
+**NOTE:** By default and for security reasons the load balancer is available only from within the default VPC in the region where it is deployed. It is not available from the Internet and is not open for external connections. Before opening it up please make sure you protect your RPC APIs.
+
 ### Monitoring
-Every 5 minutes a script on the Base node publishes to CloudWatch service the metrics for current block for L1/L2 clients as well as blocks behind metric for L1 and minutes buehind for L2. When the node is fully synced the blocks behind metric should get to 4 and minutes behind should get down to 0. To see the metrics:
+Every 5 minutes a script on the Base node publishes to CloudWatch service the metrics for current block for L1/L2 clients as well as blocks behind metric for L1 and minutes behind for L2. When the node is fully synced the blocks behind metric should get to 4 and minutes behind should get down to 0. To see the metrics for **single node only**:
 
 - Navigate to CloudWatch service (make sure you are in the region you have specified for AWS_REGION)
 - Open Dashboards and select `base-single-node-<network>-<your_ec2_instance_id>` from the list of dashboards.
 
+Metrics for **ha nodes** configuration is not yet implemented (contributions are welcome!)
+
 ## From your Cloud9: Clear up and undeploy everything
 
 1. Undeploy all Nodes and Common stacks
@@ -184,6 +220,9 @@ pwd
 # Undeploy Single Node
 npx cdk destroy base-single-node
 
+# Undeploy HA Nodes
+npx cdk destroy base-ha-nodes
+
 # Delete all common components like IAM role and Security Group
 npx cdk destroy base-common
 ```

lib/base/app.ts

@@ -5,6 +5,7 @@ import * as cdk from 'aws-cdk-lib';
 import * as config from "./lib/config/baseConfig";
 import {BaseCommonStack} from "./lib/common-stack";
 import {BaseSingleNodeStack} from "./lib/single-node-stack";
+import {BaseHANodesStack} from "./lib/ha-nodes-stack";
 
 const app = new cdk.App();
 cdk.Tags.of(app).add("Project", "AWSBase");
@@ -28,3 +29,22 @@ new BaseSingleNodeStack(app, "base-single-node", {
   snapshotUrl: config.baseNodeConfig.snapshotUrl,
   dataVolume: config.baseNodeConfig.dataVolume,
 });
+
+new BaseHANodesStack(app, "base-ha-nodes", {
+  stackName: `base-ha-nodes-${config.baseNodeConfig.baseNodeConfiguration}-${config.baseNodeConfig.baseNetworkId}`,
+  env: { account: config.baseConfig.accountId, region: config.baseConfig.region },
+
+  instanceType: config.baseNodeConfig.instanceType,
+  instanceCpuType: config.baseNodeConfig.instanceCpuType,
+  baseNetworkId: config.baseNodeConfig.baseNetworkId,
+  baseNodeConfiguration: config.baseNodeConfig.baseNodeConfiguration,
+  restoreFromSnapshot: config.baseNodeConfig.restoreFromSnapshot,
+  l1ExecutionEndpoint: config.baseNodeConfig.l1ExecutionEndpoint,
+  l1ConsensusEndpoint: config.baseNodeConfig.l1ConsensusEndpoint,
+  snapshotUrl: config.baseNodeConfig.snapshotUrl,
+  dataVolume: config.baseNodeConfig.dataVolume,
+
+  albHealthCheckGracePeriodMin: config.haNodeConfig.albHealthCheckGracePeriodMin,
+  heartBeatDelayMin: config.haNodeConfig.heartBeatDelayMin,
+  numberOfNodes: config.haNodeConfig.numberOfNodes
+});

lib/base/lib/assets/setup-instance-store-volumes.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+source /etc/environment
+
+if [[ "$DATA_VOLUME_TYPE" == "instance-store" ]]; then
+  echo "Data volume type is instance store"
+  export DATA_VOLUME_ID=/dev/$(lsblk -lnb | awk 'max < $4 {max = $4; vol = $1} END {print vol}')
+fi
+
+if [ -n "$DATA_VOLUME_ID" ]; then
+  if [ $(df --output=target | grep -c "/data") -lt 1 ]; then
+    echo "Checking fstab for Data volume"
+
+    mkfs.ext4 $DATA_VOLUME_ID
+    echo "Data volume formatted. Mounting..."
+    # Waiting wihtouht using sleep as it sometimes just hangs....
+    coproc read -t 10 && wait "$!" || true
+    DATA_VOLUME_UUID=$(lsblk -fn -o UUID  $DATA_VOLUME_ID)
+    DATA_VOLUME_FSTAB_CONF="UUID=$DATA_VOLUME_UUID /data ext4 defaults 0 2"
+    echo "DATA_VOLUME_ID="$DATA_VOLUME_ID
+    echo "DATA_VOLUME_UUID="$DATA_VOLUME_UUID
+    echo "DATA_VOLUME_FSTAB_CONF="$DATA_VOLUME_FSTAB_CONF
+
+    # Check if data disc is already in fstab and replace the line if it is with the new disc UUID
+    if [ $(grep -c "data" /etc/fstab) -gt 0 ]; then
+      SED_REPLACEMENT_STRING="$(grep -n "/data" /etc/fstab | cut -d: -f1)s#.*#$DATA_VOLUME_FSTAB_CONF#"
+      cp /etc/fstab /etc/fstab.bak
+      sed -i "$SED_REPLACEMENT_STRING" /etc/fstab
+    else
+      echo $DATA_VOLUME_FSTAB_CONF | sudo tee -a /etc/fstab
+    fi
+
+    sudo mount -a
+
+    chown bcuser:bcuser -R /data
+  else
+    echo "Data volume is mounted, nothing changed"
+  fi
+fi

lib/base/lib/assets/user-data/node.sh

@@ -6,10 +6,12 @@ LIFECYCLE_HOOK_NAME=${_LIFECYCLE_HOOK_NAME_}
 AUTOSCALING_GROUP_NAME=${_AUTOSCALING_GROUP_NAME_}
 RESOURCE_ID=${_NODE_CF_LOGICAL_ID_}
 ASSETS_S3_PATH=${_ASSETS_S3_PATH_}
+DATA_VOLUME_TYPE=${_DATA_VOLUME_TYPE_}
 {
   echo "LIFECYCLE_HOOK_NAME=$LIFECYCLE_HOOK_NAME"
   echo "AUTOSCALING_GROUP_NAME=$AUTOSCALING_GROUP_NAME"
   echo "ASSETS_S3_PATH=$ASSETS_S3_PATH"
+  echo "DATA_VOLUME_TYPE=$DATA_VOLUME_TYPE"
 } >> /etc/environment
 
 arch=$(uname -m)
@@ -34,6 +36,7 @@ yum -y install amazon-cloudwatch-agent collectd jq yq gcc ncurses-devel aws-cfn-
 wget $YQ_URI -O /usr/bin/yq && chmod +x /usr/bin/yq
 
 # install aria2 a p2p downloader
+cd /tmp
 
 if [ "$arch" == "x86_64" ]; then
   wget https://github.com/q3aql/aria2-static-builds/releases/download/v1.36.0/aria2-1.36.0-linux-gnu-64bit-build1.tar.bz2
@@ -235,37 +238,48 @@ ExecStart=/home/bcuser/bin/node.sh
 WantedBy=multi-user.target
 EOF'
 
-echo "Signaling completion to CloudFormation to continue with volume mount"
-/opt/aws/bin/cfn-signal --stack $STACK_NAME --resource $RESOURCE_ID --region $REGION
+if [[ "$LIFECYCLE_HOOK_NAME" == "none" ]]; then
+  echo "We run single node setup. Signaling completion to CloudFormation to continue with volume mount"
+  /opt/aws/bin/cfn-signal --stack $STACK_NAME --resource $RESOURCE_ID --region $REGION
+fi
 
 echo "Preparing data volume"
 
-echo "Wait for one minute for the volume to be available"
-sleep 60
-
-if $(lsblk | grep -q nvme1n1); then
-  echo "nvme1n1 is found. Configuring attached storage"
+echo "Wait for one minute for the volume to become available"
+sleep 60s
 
-  if [ "$FORMAT_DISK" == "false" ]; then
-    echo "Not creating a new filesystem in the disk. Existing data might be present!!"
-  else
-    mkfs -t ext4 /dev/nvme1n1
-  fi
+if [[ "$DATA_VOLUME_TYPE" == "instance-store" ]]; then
+  echo "Data volume type is instance store"
 
-  sleep 10
-  # Define the line to add to fstab
-  uuid=$(lsblk -n -o UUID /dev/nvme1n1)
-  line="UUID=$uuid /data ext4 defaults 0 2"
+  cd /opt
+  chmod +x /opt/setup-instance-store-volumes.sh
 
-  # Write the line to fstab
-  echo $line | sudo tee -a /etc/fstab
+  (crontab -l; echo "@reboot /opt/setup-instance-store-volumes.sh >/tmp/setup-instance-store-volumes.log 2>&1") | crontab -
+  crontab -l
 
-  mount -a
+  DATA_VOLUME_ID=/dev/$(lsblk -lnb | awk 'max < $4 {max = $4; vol = $1} END {print vol}')
 
 else
-  echo "nvme1n1 is not found. Not doing anything"
+  echo "Data volume type is EBS"
+
+  DATA_VOLUME_ID=/dev/$(lsblk -lnb | awk -v VOLUME_SIZE_BYTES="$DATA_VOLUME_SIZE" '{if ($4== VOLUME_SIZE_BYTES) {print $1}}')
 fi
 
+mkfs -t ext4 $DATA_VOLUME_ID
+echo "waiting for volume to get UUID"
+  OUTPUT=0;
+  while [ "$OUTPUT" = 0 ]; do 
+    DATA_VOLUME_UUID=$(lsblk -fn -o UUID $DATA_VOLUME_ID)
+    OUTPUT=$(echo $DATA_VOLUME_UUID | grep -c - $2)
+    echo $OUTPUT
+  done
+DATA_VOLUME_FSTAB_CONF="UUID=$DATA_VOLUME_UUID /data ext4 defaults 0 2"
+echo "DATA_VOLUME_ID="$DATA_VOLUME_ID
+echo "DATA_VOLUME_UUID="$DATA_VOLUME_UUID
+echo "DATA_VOLUME_FSTAB_CONF="$DATA_VOLUME_FSTAB_CONF
+echo $DATA_VOLUME_FSTAB_CONF | tee -a /etc/fstab
+mount -a
+
 lsblk -d
 
 chown -R bcuser:bcuser /data

lib/base/lib/config/baseConfig.interface.ts

@@ -23,3 +23,9 @@ export interface BaseBaseNodeConfig extends configTypes.BaseNodeConfig {
     l1ConsensusEndpoint: string;
     snapshotUrl: string;
 }
+
+export interface BaseHAConfig {
+    albHealthCheckGracePeriodMin: number;
+    heartBeatDelayMin: number;
+    numberOfNodes: number;
+}

lib/base/lib/config/baseConfig.ts

@@ -39,3 +39,9 @@ export const baseNodeConfig: configTypes.BaseBaseNodeConfig = {
         throughput: process.env.BASE_DATA_VOL_THROUGHPUT ? parseInt(process.env.BASE_DATA_VOL_THROUGHPUT): 700,
     },
 };
+
+export const haNodeConfig: configTypes.BaseHAConfig = {
+    albHealthCheckGracePeriodMin: process.env.BASE_HA_ALB_HEALTHCHECK_GRACE_PERIOD_MIN ? parseInt(process.env.BASE_HA_ALB_HEALTHCHECK_GRACE_PERIOD_MIN) : 10,
+    heartBeatDelayMin: process.env.BASE_HA_NODES_HEARTBEAT_DELAY_MIN ? parseInt(process.env.BASE_HA_NODES_HEARTBEAT_DELAY_MIN) : 40,
+    numberOfNodes: process.env.BASE_HA_NUMBER_OF_NODES ? parseInt(process.env.BASE_HA_NUMBER_OF_NODES) : 2
+};

lib/base/lib/assets/node-cw-dashboard.ts renamed to lib/base/lib/constructs/node-cw-dashboard.ts

@@ -0,0 +1,141 @@
+import * as cdk from "aws-cdk-lib";
+import * as cdkConstructs from "constructs";
+import * as ec2 from "aws-cdk-lib/aws-ec2";
+import * as iam from "aws-cdk-lib/aws-iam";
+import { AmazonLinuxGeneration, AmazonLinuxImage } from "aws-cdk-lib/aws-ec2";
+import * as s3Assets from "aws-cdk-lib/aws-s3-assets";
+import * as configTypes from "./config/baseConfig.interface";
+import { BaseNodeSecurityGroupConstruct } from "./constructs/base-node-security-group";
+import * as fs from "fs";
+import * as path from "path";
+import * as constants from "../../constructs/constants";
+import { HANodesConstruct } from "../../constructs/ha-rpc-nodes-with-alb";
+import * as nag from "cdk-nag";
+
+export interface BaseHANodesStackProps extends cdk.StackProps {
+    instanceType: ec2.InstanceType;
+    instanceCpuType: ec2.AmazonLinuxCpuType;
+    baseNetworkId: configTypes.BaseNetworkId;
+    baseNodeConfiguration: configTypes.BaseNodeConfiguration;
+    restoreFromSnapshot: boolean;
+    l1ExecutionEndpoint: string,
+    l1ConsensusEndpoint: string,
+    snapshotUrl: string,
+    dataVolume: configTypes.BaseDataVolumeConfig;
+    albHealthCheckGracePeriodMin: number;
+    heartBeatDelayMin: number;
+    numberOfNodes: number;
+}
+
+export class BaseHANodesStack extends cdk.Stack {
+    constructor(scope: cdkConstructs.Construct, id: string, props: BaseHANodesStackProps) {
+        super(scope, id, props);
+
+        const REGION = cdk.Stack.of(this).region;
+        const STACK_NAME = cdk.Stack.of(this).stackName;
+        const lifecycleHookName = STACK_NAME;
+        const autoScalingGroupName = STACK_NAME;
+
+        const {
+            instanceType,
+            instanceCpuType,
+            baseNetworkId,
+            baseNodeConfiguration,
+            restoreFromSnapshot,
+            l1ExecutionEndpoint,
+            l1ConsensusEndpoint,
+            dataVolume,
+            albHealthCheckGracePeriodMin,
+            heartBeatDelayMin,
+            numberOfNodes
+        } = props;
+
+        // using default vpc
+        const vpc = ec2.Vpc.fromLookup(this, "vpc", { isDefault: true });
+
+        // setting up the security group for the node from BSC-specific construct
+        const instanceSG = new BaseNodeSecurityGroupConstruct(this, "security-group", { vpc: vpc });
+
+        // getting the IAM Role ARM from the common stack
+        const importedInstanceRoleArn = cdk.Fn.importValue("BaseNodeInstanceRoleArn");
+
+        const instanceRole = iam.Role.fromRoleArn(this, "iam-role", importedInstanceRoleArn);
+
+        // making our scripts and configs from the local "assets" directory available for instance to download
+        const asset = new s3Assets.Asset(this, "assets", {
+            path: path.join(__dirname, "assets")
+        });
+
+        asset.bucket.grantRead(instanceRole);
+
+        // parsing user data script and injecting necessary variables
+        const nodeScript = fs.readFileSync(path.join(__dirname, "assets", "user-data", "node.sh")).toString();
+        const dataVolumeSizeBytes = dataVolume.sizeGiB * constants.GibibytesToBytesConversionCoefficient;
+
+        const modifiedInitNodeScript = cdk.Fn.sub(nodeScript, {
+            _REGION_: REGION,
+            _ASSETS_S3_PATH_: `s3://${asset.s3BucketName}/${asset.s3ObjectKey}`,
+            _STACK_NAME_: STACK_NAME,
+            _NODE_CF_LOGICAL_ID_: constants.NoneValue,
+            _DATA_VOLUME_TYPE_: dataVolume.type,
+            _DATA_VOLUME_SIZE_: dataVolumeSizeBytes.toString(),
+            _NETWORK_ID_: baseNetworkId,
+            _NODE_CONFIG_: baseNodeConfiguration,
+            _LIFECYCLE_HOOK_NAME_: lifecycleHookName,
+            _AUTOSCALING_GROUP_NAME_: autoScalingGroupName,
+            _RESTORE_FROM_SNAPSHOT_: restoreFromSnapshot.toString(),
+            _FORMAT_DISK_: "true",
+            _L1_EXECUTION_ENDPOINT_: l1ExecutionEndpoint,
+            _L1_CONSENSUS_ENDPOINT_: l1ConsensusEndpoint,
+            _SNAPSHOT_URL_: props.snapshotUrl,
+        });
+
+        const rpcNodes = new HANodesConstruct(this, "rpc-nodes", {
+            instanceType,
+            dataVolumes: [dataVolume],
+            machineImage: new ec2.AmazonLinuxImage({
+                generation: AmazonLinuxGeneration.AMAZON_LINUX_2,
+                kernel:ec2.AmazonLinuxKernel.KERNEL5_X,
+                cpuType: instanceCpuType
+            }),
+            role: instanceRole,
+            vpc,
+            securityGroup: instanceSG.securityGroup,
+            userData: modifiedInitNodeScript,
+            numberOfNodes,
+            rpcPortForALB: 8545,
+            albHealthCheckGracePeriodMin,
+            heartBeatDelayMin,
+            lifecycleHookName: lifecycleHookName,
+            autoScalingGroupName: autoScalingGroupName
+        });
+
+
+
+        new cdk.CfnOutput(this, "alb-url", { value: rpcNodes.loadBalancerDnsName });
+
+        // Adding suppressions to the stack
+        nag.NagSuppressions.addResourceSuppressions(
+            this,
+            [
+                {
+                    id: "AwsSolutions-AS3",
+                    reason: "No notifications needed"
+                },
+                {
+                    id: "AwsSolutions-S1",
+                    reason: "No access log needed for ALB logs bucket"
+                },
+                {
+                    id: "AwsSolutions-EC28",
+                    reason: "Using basic monitoring to save costs"
+                },
+                {
+                    id: "AwsSolutions-IAM5",
+                    reason: "Need read access to the S3 bucket with assets"
+                }
+            ],
+            true
+        );
+    }
+}