Update/add systemd (#1)

* update readme
* make code more readable
* make variable names more descriptive
* add doc comments to functions
* make calc_* functions consistent
* modularize code
* use shared common utilty functions
* install to /usr/local and symlink to /usr/bin
* isolate upstart installation
* isolate systemd installation
* use a config file
* use trap for stopping log
* move service installs
* move utils location
* create global install script
* suppress error messages when detecting init system
* echo detected init system
* simplify config value retrieval
* fix logthis
* start service last on installation
* add more statistics to logging
* add logging of number of devices
* make logging messages consistent

Author: wleepang

README.md

@@ -1,25 +1,26 @@
 # Amazon Elastic Block Store Autoscale
 
-This is an example of a small daemon process that monitors a BTRFS filesystem mountpoint and automatically expands it when free space falls below a configured threshold. New [Amazon EBS](https://aws.amazon.com/ebs/) volumes are added to the instance as necessary and the underlying [BTRFS filesystem](http://btrfs.wiki.kernel.org) expands while still mounted. As new devices are added, the BTRFS metadata blocks are rebalanced to mitigate the risk that space for metadata will not run out.
+This is an example of a daemon process that monitors a BTRFS filesystem mountpoint and automatically expands it when free space falls below a configured threshold. New [Amazon EBS](https://aws.amazon.com/ebs/) volumes are added to the instance as necessary and the underlying [BTRFS filesystem](http://btrfs.wiki.kernel.org) expands while still mounted. As new devices are added, the BTRFS metadata blocks are rebalanced to mitigate the risk that space for metadata will not run out.
 
 ## Assumptions:
 
-1. That this code is running on a AWS EC2 instance
-2. The instance has a IAM Instance Profile with appropriate permissions to create and attache new EBS volumes. Ssee the [IAM Instance Profile](#iam_instance_profile) section below for more details
-3. That prerequisites are installed on the instance.
+1. Code is running on an AWS EC2 instance
+2. The instance is using a Linux based OS with either **upstart** or **systemd** system initialization
+3. The instance has a IAM Instance Profile with appropriate permissions to create and attach new EBS volumes. See the [IAM Instance Profile](#iam_instance_profile) section below for more details
+4. That prerequisites are installed on the instance.
 
 Provided in this repo are:
 
 1. A python [script](bin/create-ebs-volume.py) that creates and attaches new EBS volumes to the current instance
 2. The daemon [script](bin/ebs-autoscale) that monitors disk space and expands the BTRFS filesystem by leveraging the above script to add EBS volumes, expand the filesystem, and rebalance the metadata blocks
-2. A template for an [upstart configuration file](templates/ebs-autoscale.conf.template)
-2. A [logrotate configuration file](templates/ebs-autoscale.logrotate) which should not be needed but may as well be in place for long-running instances.
-5. A [initialization script](bin/init-ebs-autoscale.sh) to configure and install all of the above
-6. A [cloud-init](templates/cloud-init-userdata.yaml) file for user-data that installs required packages and runs the initialization script. By default this creates a mount point of `/scratch` on a encrypted 20GB EBS volume. To change the mount point, edit the file.
+3. Service definitions for [upstart](service/upstart/ebs-autoscale.conf) and [systemd](service/systemd/ebs-autoscale.service)
+4. Configuration files for the [service](config/ebs-autoscale.json) and [logrotate](config/ebs-autoscale.logrotate)
+5. An [installation script](install.sh) to configure and install all of the above
+6. An example [cloud-init](templates/cloud-init-userdata.yaml) script that can be used as EC2 instance user-data for automated installation
 
 ## Installation
 
-The easiest way to set up an instance is to provide a launch call with the userdata [cloud-init script](templates/cloud-init-userdata.yaml). Here is an example of launching the [Amazon ECS-Optimized  AMI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html) in us-east-1 using this file:
+The easiest way to set up an instance is to provide a launch call with the userdata [cloud-init script](templates/cloud-init-userdata.yaml). Here is an example of launching the [Amazon ECS-Optimized AMI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html) in us-east-1 using this file:
 
 ```bash
 aws ec2 run-instances --image-id ami-5253c32d \
@@ -31,6 +32,7 @@ aws ec2 run-instances --image-id ami-5253c32d \
   --iam-instance-profile Name=MyInstanceProfileWithProperPermissions
 ```
 
+that installs required packages and runs the initialization script. By default this creates a mount point of `/scratch` on a encrypted 20GB EBS volume. To change the mount point, edit the file.
 
 ## A note on IAM Instance Profile
 

bin/create-ebs-volume.py

@@ -36,7 +36,6 @@
 import boto3
 from botocore.exceptions import ClientError
 
-## TODO: CLI arguments
 parameters = argparse.ArgumentParser(description="Create a new EBS Volume and attach it to the current instance")
 parameters.add_argument("-s","--size", type=int, required=True)
 parameters.add_argument("-t","--type", type=str, default="gp2")
@@ -71,7 +70,7 @@ def get_metadata(key):
     return urllib.urlopen(("/").join(['http://169.254.169.254/latest/meta-data', key])).read()
 
 
-# create a EBS volume
+# create an EBS volume
 def create_and_attach_volume(size=10, vol_type="gp2", encrypted=True, max_attached_volumes=16, max_created_volumes=256):
     instance_id  = get_metadata("instance-id")
     availability_zone = get_metadata("placement/availability-zone")
@@ -143,5 +142,12 @@ def create_and_attach_volume(size=10, vol_type="gp2", encrypted=True, max_attach
 
 if __name__ == '__main__':
     args = parameters.parse_args()
-    print(create_and_attach_volume(args.size), end='')
+    print(
+        create_and_attach_volume(
+            size=args.size,
+            vol_type=args.type,
+            encrypted=args.encrypted
+        ), 
+        end=''
+    )
     sys.stdout.flush()

bin/ebs-autoscale

@@ -28,49 +28,57 @@
 #  IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 #  POSSIBILITY OF SUCH DAMAGE.
 
-if [ "$#" -ne "1" ]; then
-  echo "USAGE: $0 <MOUNT POINT>"
-  exit 1
-fi
+. /usr/local/amazon-ebs-autoscale/shared/utils.sh
 
-logthis () {
-  echo "[`date`] $1"
-}
+initialize
+
+MAX_LOGICAL_VOLUME_SIZE=$(get_config_value .limits.max_logical_volume_size)
+MAX_EBS_VOLUME_COUNT=$(get_config_value .limits.max_ebs_volume_count)
 
-MP=$1
+MOUNTPOINT=$(get_config_value .mountpoint)
 BASEDIR=$(dirname $0)
-AZ=$(curl -s  http://169.254.169.254/latest/meta-data/placement/availability-zone/)
 
-logthis "EBS Autoscaling mountpoint: ${MP}"
+starting
+trap "stopping; exit" INT TERM KILL
 
-while [ -z "${AZ}" ]; do
-  logthis "Metadata service did not return AZ. Trying again."
-  sleep 1
-  AZ=$(curl -s  http://169.254.169.254/latest/meta-data/placement/availability-zone/)
-done
-RG=$(echo ${AZ} | sed -e 's/[a-z]$//')
-logthis "Region = $RG."
-IN=$(curl -s  http://169.254.169.254/latest/meta-data/instance-id)
-DRIVE_LETTERS=({a..z})
+logthis "EBS Autoscaling mountpoint: ${MOUNTPOINT}"
+logthis "Region = $AWS_REGION"
+logthis "Availability Zone = $AWS_AZ"
 
 # make sure that this device is mounted.
-until [ -d "${MP}" ]; do
+until [ -d "${MOUNTPOINT}" ]; do
   sleep 1
 done
+
+get_num_devices() {
+  echo $(ls /dev/sd* | grep -v -E '[0-9]$' | wc -l)
+}
+
 calc_threshold() {
-  local num_devices=$(ls /dev/sd* | grep -v -E '[0-9]$' | wc -l)
+  # calculates percent utilization threshold for adding additional ebs volumes
+  # as more ebs volumes are added, the threshold level increases
+
+  local num_devices=$(get_num_devices)
   local threshold=50
-  if [ "$num_devices" -gt "4" ] && [ "$num_devices" -le "6" ]; then
+
+  if [ "$num_devices" -ge "4" ] && [ "$num_devices" -le "6" ]; then
     threshold=80
-  elif [ "$num_devices" -gt "6" ] && [ "$num_devices" -le "10"   ]; then
+  elif [ "$num_devices" -gt "6" ] && [ "$num_devices" -le "10" ]; then
+    threshold=90
+  elif [ "$num_devices" -gt "10" ]; then
     threshold=90
   else
     threshold=50
   fi
+
   echo ${threshold}
 }
+
 calc_new_size() {
-  local num_devices=$1
+  # calculates the size to use for new ebs volumes to expand space
+  # new volume sizes increase as the number of attached volumes increase
+  
+  local num_devices=$(get_num_devices)
   local new_size=150
 
   if [ "$num_devices" -ge "4" ] && [ "$num_devices" -le "6" ]; then
@@ -82,49 +90,68 @@ calc_new_size() {
   else
     new_size=150
   fi
+
   echo ${new_size}
 }
 
 add_space () {
-  local num_devices=$(ls /dev/sd* | grep -v -E '[0-9]$' | wc -l)
-  if [ "${num_devices}" -ge "16" ]; then
+  local num_devices=$(get_num_devices)
+  if [ "${num_devices}" -ge "$MAX_EBS_VOLUME_COUNT" ]; then
     logthis "No more volumes can be safely added."
     return 0
   fi
-  local curr_size=$(df  -BG ${MP} | grep ${MP} | awk '{print $2} ' | cut -d'G' -f1)
-  if [ "${curr_size}" -lt "16384" ]; then
-    local vol_size=$(calc_new_size ${num_devices})
-    logthis "Extending LV ${MP} by  ${vol_size}GB"
 
-    DV=$(python ${BASEDIR}/create-ebs-volume.py -s ${vol_size})
+  local curr_size=$(df -BG ${MOUNTPOINT} | grep ${MOUNTPOINT} | awk '{print $2} ' | cut -d'G' -f1)
+  if [ "${curr_size}" -lt "$MAX_LOGICAL_VOLUME_SIZE" ]; then
+    local vol_size=$(calc_new_size)
+    logthis "Extending logical volume ${MOUNTPOINT} by ${vol_size}GB"
+
+    DEVICE=$(python ${BASEDIR}/create-ebs-volume.py -s ${vol_size})
 
     exit_status=$?
     if [ $exit_status -eq 0 ]; then
-      logthis "adding volume to filesystem"
-      btrfs device add ${DV} ${MP}
-      btrfs balance start -m ${MP}
-      logthis "Finished extending device."
+      logthis "Adding device ${DEVICE} to logical volume ${MOUNTPOINT}"
+      btrfs device add ${DEVICE} ${MOUNTPOINT}
+      btrfs balance start -m ${MOUNTPOINT}
+      logthis "Finished extending logical volume"
 
     else
-      logthis "Error creating or attaching volume"
+      logthis "Error creating or attaching EBS volume"
     fi
 
   fi
 }
 
-COUNT=300
+# number of event loops between utilization status log lines
+# helps to limit the log file size
+# utilization detection is not affected by this
+LOG_INTERVAL=$(get_config_value .logging.log_interval)
+
+# initialized value for log lines
+# report on first run
+LOG_COUNT=$LOG_INTERVAL
+
+# time in seconds between event loops
+# keep this low so that rapid increases in utilization are detected
+DETECTION_INTERVAL=$(get_config_value .detection_interval)
+
 THRESHOLD=$(calc_threshold)
 while true; do
-  F=$(df -BG  ${MP} | grep -v Filesystem | awk '{print $5}' | cut -d"%" -f1 -)
-  if  [ $F -ge "${THRESHOLD}" ]; then
-    logthis "LOW DISK ($F): Adding more."
+  NUM_DEVICES=$(get_num_devices)
+  STATS=$(df -BG  ${MOUNTPOINT} | grep -v Filesystem)
+  TOTAL_SIZE=$(echo ${STATS} | awk '{print $2}')
+  USED=$(echo ${STATS} | awk '{print $3}')
+  AVAILABLE=$(echo ${STATS} | awk '{print $4}')
+  PCT_UTILIZATION=$(echo ${STATS} | awk '{print $5}' | cut -d"%" -f1 -)
+  if  [ $PCT_UTILIZATION -ge "${THRESHOLD}" ]; then
+    logthis "LOW DISK (${PCT_UTILIZATION}%): Adding more."
     add_space
   fi
-  if [ "${COUNT}" -ge "300" ]; then
-    logthis "Threshold -> ${THRESHOLD} :: Used% -> ${F}%"
-    COUNT=0
+  if [ "${LOG_COUNT}" -ge "${LOG_INTERVAL}" ]; then
+    logthis "Devices ${NUM_DEVICES} : Size ${TOTAL_SIZE} : Used ${USED} : Aailable ${AVAILABLE} : Used% ${PCT_UTILIZATION}% : Threshold ${THRESHOLD}%"
+    LOG_COUNT=0
   fi
   THRESHOLD=$(calc_threshold)
-  COUNT=$(expr $COUNT + 1 )
-  sleep 1
+  LOG_COUNT=$(expr $LOG_COUNT + 1 )
+  sleep $DETECTION_INTERVAL
 done

config/ebs-autoscale.json

@@ -0,0 +1,12 @@
+{
+    "mountpoint": "/scratch",
+    "detection_interval": 1,
+    "limits": {
+        "max_logical_volume_size": 16384,
+        "max_ebs_volume_count": 16
+    },
+    "logging": {
+        "log_file": "/var/log/ebs-autoscale.log",
+        "log_interval": 300
+    }
+}

templates/ebs-autoscale.logrotate config/ebs-autoscale.logrotatetemplates/ebs-autoscale.logrotate renamed to config/ebs-autoscale.logrotate

@@ -0,0 +1,107 @@
+#!/bin/sh
+# Copyright 2018 Amazon.com, Inc. or its affiliates.
+#
+#  Redistribution and use in source and binary forms, with or without
+#  modification, are permitted provided that the following conditions are met:
+#
+#  1. Redistributions of source code must retain the above copyright notice,
+#  this list of conditions and the following disclaimer.
+#
+#  2. Redistributions in binary form must reproduce the above copyright
+#  notice, this list of conditions and the following disclaimer in the
+#  documentation and/or other materials provided with the distribution.
+#
+#  3. Neither the name of the copyright holder nor the names of its
+#  contributors may be used to endorse or promote products derived from
+#  this software without specific prior written permission.
+#
+#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
+#  BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+#  THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+#  INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+#  (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+#  SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+#  HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+#  STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
+#  IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+#  POSSIBILITY OF SUCH DAMAGE.
+
+set -e
+
+function printUsage() {
+  echo "USAGE: $0 <MOUNT POINT> [<DEVICE>]"
+}
+
+if [ "$#" -lt "1" ]; then
+  printUsage
+  exit 1
+fi
+
+MOUNTPOINT=$1
+DEVICE=$2
+BASEDIR=$(dirname $0)
+
+. ${BASEDIR}/shared/utils.sh
+
+initialize
+
+# Install executables
+# make executables available on standard PATH
+mkdir -p /usr/local/amazon-ebs-autoscale/{bin,shared}
+cp ${BASEDIR}/bin/{create-ebs-volume.py,ebs-autoscale} /usr/local/amazon-ebs-autoscale/bin
+chmod +x /usr/local/amazon-ebs-autoscale/bin/*
+ln -sf /usr/local/amazon-ebs-autoscale/bin/* /usr/local/bin/
+ln -sf /usr/local/amazon-ebs-autoscale/bin/* /usr/bin/
+
+
+# copy shared assets
+cp ${BASEDIR}/shared/utils.sh /usr/local/amazon-ebs-autoscale/shared
+
+
+## Install configs
+# install the logrotate config
+cp ${BASEDIR}/config/ebs-autoscale.logrotate /etc/logrotate.d/ebs-autoscale
+
+# install default config
+sed -e "s#/scratch#${MOUNTPOINT}#" ${BASEDIR}/config/ebs-autoscale.json > /etc/ebs-autoscale.json
+
+
+## Create filesystem
+if [ -e $MOUNTPOINT ] && ! [ -d $MOUNTPOINT ]; then
+  echo "ERROR: $MOUNTPOINT exists but is not a directory."
+  exit 1
+elif ! [ -e $MOUNTPOINT ]; then
+  mkdir -p $MOUNTPOINT
+fi
+
+# If a device is not given, or if the device is not valid
+# create a new 20GB volume
+if [ -z "${DEVICE}" ] || [ ! -b "${DEVICE}" ]; then
+  DEVICE=$(create-ebs-volume.py --size 20)
+fi
+
+# create and mount the BTRFS filesystem
+mkfs.btrfs -f -d single $DEVICE
+mount $DEVICE $MOUNTPOINT
+
+# add entry to fstab
+# allows non-root users to mount/unmount the filesystem
+echo -e "${DEVICE}\t${MOUNTPOINT}\tbtrfs\tdefaults\t0\t0" |  tee -a /etc/fstab
+
+
+## Install service
+INIT_SYSTEM=$(detect_init_system 2>/dev/null)
+case $INIT_SYSTEM in
+  upstart|systemd)
+    echo "$INIT_SYSTEM detected"
+    cd ${BASEDIR}/service/$INIT_SYSTEM
+    . ./install.sh
+    ;;
+
+  *)
+    echo "Could not install EBS Autoscale - unsupported init system"
+    exit 1
+esac
+cd ${BASEDIR}

install.sh

@@ -0,0 +1,10 @@
+[Unit]
+Description=Amazon EBS Autoscale
+After=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/ebs-autoscale
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

service/systemd/ebs-autoscale.service

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# install systemd service
+cp ebs-autoscale.service /usr/lib/systemd/system/ebs-autoscale.service
+
+# enable the service and start
+systemctl daemon-reload
+systemctl enable ebs-autoscale.service
+systemctl start ebs-autoscale.service