Solana. Added ARM support, new Agave client support, removed unused consensus option

Author: vlasonfa

lib/solana/README.md

@@ -4,7 +4,7 @@
 |:--------------------:|
 | [@frbrkoala](https://github.com/frbrkoala) |
 
-Solana nodes on AWS can be deployed in 3 different configurations: Consensus, base RPC and extended RPC with secondary indexes. In addition, you can choose to deploy those configurations as a single node or a highly available (HA) nodes setup. See below the details on single node and HA deployment setups.
+Solana nodes on AWS can be deployed in 2 different configurations: base RPC and extended RPC with secondary indexes. In addition, you can choose to deploy those configurations as a single node or a highly available (HA) nodes setup and use x86 or ARM-powered EC2 instances (powered by Graviton 4). See below the details on single node and HA deployment setups.
 
 ## Overview of Deployment Architectures for Single and HA setups
 
@@ -84,9 +84,8 @@ This is the Well-Architected checklist for Solana nodes implementation of the AW
 
 | Usage pattern  | Ideal configuration  | Primary option on AWS  | Data Transfer Estimates | Config reference |
 |---|---|---|---|---|
-| 1/ Consensus node | 48 vCPU, 384 GiB RAM, Accounts volume: 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: 2TB, 9K IOPS, 700 MB/s throughput   | r7a.12xlarge, Accounts volume: 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: 2TB, 9K IOPS, 700 MB/s throughput | Proportional to the amount at stake. Between 200TB to 400TB/month  | [.env-sample-consensus](./sample-configs/.env-sample-consensus) |
-| 2/ Base RPC node (no secondary indexes) | 48 vCPU, 384 GiB RAM, Accounts volume: EBS gp3, 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: EBS gp3, 2TB, 9K IOPS, 700 MB/s throughput   | r7a.12xlarge, Accounts volume: EBS gp3, 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: EBS gp3, 2TB, 9K IOPS, 700 MB/s throughput | 13-15TB/month (no staking) | [.env-sample-baserpc](./sample-configs/.env-sample-baserpc) |
-| 3/ Extended RPC node (with all secondary indexes) | 96 vCPU, 768 GiB RAM, Accounts volume: 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: 2TB, 9K IOPS, 700 MB/s throughput  | r7a.24xlarge, Accounts volume: EBS io2, 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: EBS gp3, 2TB, 9K IOPS, 700 MB/s throughput | 13-15TB/month (no staking) | [.env-sample-extendedrpc](./sample-configs/.env-sample-extendedrpc) |
+| 1/ Base RPC node (no secondary indexes) | 48 vCPU, 384 GiB RAM, Accounts volume: EBS gp3, 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: EBS gp3, 2TB, 9K IOPS, 700 MB/s throughput   | r7a.12xlarge, Accounts volume: EBS gp3, 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: EBS gp3, 2TB, 9K IOPS, 700 MB/s throughput | 13-15TB/month (no staking) | [.env-sample-baserpc-x86](./sample-configs/.env-sample-baserpc-x86) |
+| 2/ Extended RPC node (with all secondary indexes) | 96 vCPU, 768 GiB RAM, Accounts volume: 500GiB, 7K IOPS, 700 MB/s throughput, Data volume: 2TB, 9K IOPS, 700 MB/s throughput  | I8g.18xlarge, Accounts volume: Instance Store, Data volume: Instance Store | 20-38TB/month (no staking) | [.env-sample-extendedrpc-arm](./sample-configs/.env-sample-extendedrpc-arm) |
 </details>
 
 ## Setup Instructions
@@ -126,10 +125,10 @@ Create your own copy of `.env` file and edit it to update with your AWS Account
 # Make sure you are in aws-blockchain-node-runners/lib/solana
 cd lib/solana
 pwd
-cp ./sample-configs/.env-sample-baserpc .env
+cp ./sample-configs/.env-sample-baserpc-arm .env
 nano .env
 ```
-> **NOTE:** *You can find more examples inside `sample-configs` directory.*
+> **NOTE:** *You can find more examples inside `sample-configs` directory: ARM-powered and x86-powered setups, base and extended RPC configurations.*
 
 
 4. Deploy common components such as IAM role, and Amazon S3 bucket to store data snapshots

lib/solana/lib/assets/setup-instance-store-volumes.sh

@@ -51,6 +51,11 @@ if [ -n "$ACCOUNTS_VOLUME_ID" ]; then
     echo "Checking fstab for Accounts volume"
 
     sudo mkfs.xfs -f $ACCOUNTS_VOLUME_ID
+    if [ "$?" == 1 ]; then
+      echo "Volume $ACCOUNTS_VOLUME_ID is busy, trying /dev/nvme3n1"
+      export ACCOUNTS_VOLUME_ID=/dev/nvme3n1
+      sudo mkfs.xfs -f $ACCOUNTS_VOLUME_ID
+    fi
     sleep 10
     ACCOUNTS_VOLUME_UUID=$(lsblk -fn -o UUID $ACCOUNTS_VOLUME_ID)
     ACCOUNTS_VOLUME_FSTAB_CONF="UUID=$ACCOUNTS_VOLUME_UUID /data/solana/accounts xfs defaults 0 2"

lib/solana/lib/assets/solana/node-consensus-template.sh

@@ -29,4 +29,5 @@ __ENTRY_POINTS__ \
 --limit-ledger-size \
 --accounts /data/solana/accounts \
 --incremental-snapshot-interval-slots 0 \
+--no-port-check \
 --log -

lib/solana/lib/assets/solana/node-heavy-rpc-template.sh

@@ -41,4 +41,5 @@ __ENTRY_POINTS__ \
 --account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6 \
 --account-index-exclude-key TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA \
 --incremental-snapshot-interval-slots 0 \
+--no-port-check \
 --log -

lib/solana/lib/assets/solana/node-light-rpc-template.sh

@@ -36,4 +36,5 @@ __ENTRY_POINTS__ \
 --limit-ledger-size \
 --accounts /data/solana/accounts \
 --incremental-snapshot-interval-slots 0 \
+--no-port-check \
 --log -

lib/solana/lib/assets/user-data/node.sh

@@ -23,26 +23,35 @@ set +e
 source /etc/environment
 
 apt-get -yqq update
-apt-get -yqq install awscli jq unzip python3-pip chrony
+apt-get -yqq install jq unzip python3-pip chrony
 apt install unzip
 
 cd /opt
 
-echo "Downloading assets zip file"
-aws s3 cp $ASSETS_S3_PATH ./assets.zip --region $AWS_REGION
-unzip -q assets.zip
-
 echo "Install and configure CloudWatch agent"
+arch=$(uname -m)
 if [ "$arch" == "x86_64" ]; then
   CW_AGENT_BINARY_URI=https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
+  AWS_CLI_BINARY_URI=https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip
 else
   CW_AGENT_BINARY_URI=https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb
+  AWS_CLI_BINARY_URI=https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip
 fi
 
+
+echo "Intalling AWS CLI"
+curl "$AWS_CLI_BINARY_URI" -o "awscliv2.zip"
+unzip awscliv2.zip
+sudo ./aws/install
+
+echo "Downloading assets zip file"
+aws s3 cp $ASSETS_S3_PATH ./assets.zip --region $AWS_REGION
+unzip -q assets.zip
+
+echo 'Install & configure CloudWatch Agent'
 wget -q $CW_AGENT_BINARY_URI
 dpkg -i -E amazon-cloudwatch-agent.deb
 
-echo 'Configuring CloudWatch Agent'
 mkdir -p /opt/aws/amazon-cloudwatch-agent/etc/
 cp /opt/cw-agent.json /opt/aws/amazon-cloudwatch-agent/etc/custom-amazon-cloudwatch-agent.json
 
@@ -110,13 +119,13 @@ sysctl -p /etc/sysctl.d/20-solana-mmaps.conf
 sysctl -p /etc/sysctl.d/20-solana-udp-buffers.conf
 sysctl -p /etc/sysctl.d/20-solana-additionals.conf
 
-systemctl daemon-reload
-
 bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
 # Increase process file descriptor count limit
 * - nofile 1000000
 EOF"
 
+systemctl daemon-reload
+
 echo 'Preparing fs for Solana installation'
 mkdir /data
 mkdir /data/solana
@@ -131,7 +140,7 @@ usermod -aG solana solana
 if [[ "$STACK_ID" != "none" ]]; then
   echo "Install CloudFormation helper scripts"
   mkdir -p /opt/aws/
-  pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
+  pip3 install --break-system-packages https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
   ln -s /usr/local/init/ubuntu/cfn-hup /etc/init.d/cfn-hup
 
   echo "Configuring CloudFormation helper scripts"
@@ -140,7 +149,7 @@ if [[ "$STACK_ID" != "none" ]]; then
   sed -i "s;__AWS_STACK_ID__;\"$STACK_ID\";g" /etc/cfn/cfn-hup.conf
   sed -i "s;__AWS_REGION__;\"$AWS_REGION\";g" /etc/cfn/cfn-hup.conf
 
-  mkdir -p /etc/cfn/hooks.d/
+  mkdir -p /etc/cfn/hooks.d/system  
   mv /opt/cfn-hup/cfn-auto-reloader.conf /etc/cfn/hooks.d/cfn-auto-reloader.conf
   sed -i "s;__AWS_STACK_NAME__;\"$STACK_NAME\";g" /etc/cfn/hooks.d/cfn-auto-reloader.conf
   sed -i "s;__AWS_REGION__;\"$AWS_REGION\";g" /etc/cfn/hooks.d/cfn-auto-reloader.conf

lib/solana/lib/constructs/solana-node-security-group.ts

@@ -20,16 +20,25 @@ export interface SolanaNodeSecurityGroupConstructProps {
       const sg = new ec2.SecurityGroup(this, `rpc-node-security-group`, {
         vpc,
         description: "Security Group for Blockchain nodes",
-        allowAllOutbound: true,
+        allowAllOutbound: false,
       });
 
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.HTTP, "allow HTTP");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.HTTPS, "allow HTTPS");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(123), "allow NTP");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(8899), "allow Solana RPC");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(8001), "allow TCP Solana P2P");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(8801), "allow TCP Solana P2P");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.udpRange(8000,8014), "allow UDP Solana P2P");
+      sg.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.udpRange(8800,8814), "allow UDP Solana P2P");
+
       // Public ports
-      sg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcpRange(8800, 8814), "P2P protocols (gossip, turbine, repair, etc)");
-      sg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.udpRange(8800, 8814), "P2P protocols (gossip, turbine, repair, etc)");
+      sg.addIngressRule(ec2.Peer.ipv4(vpc.vpcCidrBlock), ec2.Port.tcpRange(8800, 8814), "allow internal TCP P2P protocols (gossip, turbine, repair, etc)");
+      sg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.udpRange(8800, 8814), "allow all UDP P2P protocols (gossip, turbine, repair, etc)");
 
       // Private ports restricted only to the VPC IP range
-      sg.addIngressRule(ec2.Peer.ipv4(vpc.vpcCidrBlock), ec2.Port.tcp(8899), "RPC port HTTP (user access needs to be restricted. Allowed access only from internal IPs)");
-      sg.addIngressRule(ec2.Peer.ipv4(vpc.vpcCidrBlock), ec2.Port.tcp(8900), "RPC port WebSocket (user access needs to be restricted. Allowed access only from internal IPs)");
+      sg.addIngressRule(ec2.Peer.ipv4(vpc.vpcCidrBlock), ec2.Port.tcp(8899), "allow internal RPC port HTTP (user access needs to be restricted. Allowed access only from internal IPs)");
+      sg.addIngressRule(ec2.Peer.ipv4(vpc.vpcCidrBlock), ec2.Port.tcp(8900), "allow internal RPC port WebSocket (user access needs to be restricted. Allowed access only from internal IPs)");
 
       this.securityGroup = sg
 

lib/solana/lib/ha-nodes-stack.ts

@@ -71,10 +71,10 @@ export class SolanaHANodesStack extends cdk.Stack {
         asset.bucket.grantRead(instanceRole);
 
         // Use Ubuntu 20.04 LTS image for amd64. Find more: https://discourse.ubuntu.com/t/finding-ubuntu-images-with-the-aws-ssm-parameter-store/15507
-        let ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id"
+        let ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id"
         // Setting up the node using generic Single Node constract
         if (instanceCpuType === ec2.AmazonLinuxCpuType.ARM_64) {
-            ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/20.04/stable/current/arm64/hvm/ebs-gp2/ami-id"
+            ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/24.04/stable/current/arm64/hvm/ebs-gp3/ami-id"
         }
 
         // Parsing user data script and injecting necessary variables

lib/solana/lib/single-node-stack.ts

@@ -75,10 +75,10 @@ export class SolanaSingleNodeStack extends cdk.Stack {
         asset.bucket.grantRead(instanceRole);
 
         // Use Ubuntu 20.04 LTS image for amd64. Find more: https://discourse.ubuntu.com/t/finding-ubuntu-images-with-the-aws-ssm-parameter-store/15507
-        let ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id"
+        let ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id"
         // Setting up the node using generic Single Node constract
         if (instanceCpuType === ec2.AmazonLinuxCpuType.ARM_64) {
-            ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/20.04/stable/current/arm64/hvm/ebs-gp2/ami-id"
+            ubuntu204stableImageSsmName = "/aws/service/canonical/ubuntu/server/24.04/stable/current/arm64/hvm/ebs-gp3/ami-id"
         }
 
         const node = new SingleNodeConstruct(this, "sync-node", {

lib/solana/sample-configs/.env-sample-consensus renamed to lib/solana/sample-configs/.env-sample-baserpc-arm

@@ -8,11 +8,11 @@ AWS_REGION="us-east-2"
 
 ## Common configuration parameters ##
 SOLANA_CLUSTER="mainnet-beta"                 # All options:  "mainnet-beta", "testnet", "devnet"
-SOLANA_NODE_CONFIGURATION="consensus"                   # All options:   "consensus", "baserpc", "extendedrpc"
-SOLANA_VERSION="2.0.18"                       # Latest required version of Agave above 2.x. Check for latest Mainnet version https://github.com/anza-xyz/agave/releases
+SOLANA_NODE_CONFIGURATION="baserpc"                   # All options:   "consensus", "baserpc", "extendedrpc"
+SOLANA_VERSION="2.0.19"                       # Latest required version of Agave above 2.x. Check for latest Mainnet version https://github.com/anza-xyz/agave/releases
 
-SOLANA_INSTANCE_TYPE="r7a.12xlarge"
-SOLANA_CPU_TYPE="x86_64"                      # All options: "x86_64", "ARM_64". IMPORTANT: Make sure the CPU type matches the instance type used
+SOLANA_INSTANCE_TYPE="r8g.12xlarge"
+SOLANA_CPU_TYPE="ARM_64"                      # All options: "x86_64", "ARM_64". IMPORTANT: Make sure the CPU type matches the instance type used
 # Data volume configuration
 SOLANA_DATA_VOL_TYPE="gp3"                    # Other options: "io1" | "io2" | "gp3" | "instance-store" . IMPORTANT: Use "instance-store" option only with instance types that support that feature, like popular for node im4gn, d3, i3en, and i4i instance families
 SOLANA_DATA_VOL_SIZE="2000"                   # Current required data size to keep both smapshot archive and unarchived version of it