Merge 1.4.0 release into master

Author: bwbarrett

CHANGELOG.rst

@@ -2,6 +2,17 @@
 CHANGELOG
 =========
 
+1.4.0
+=====
+* change:``cfncluster``: `cfncluster stop` will terminate compute
+  instances, but not stop the master node.
+* feature:``cfncluster``: CfnCluster no longer maintains a whitelist
+  of supported instance types, so new platforms are supported on day
+  of launch (including C5).
+* bugfix:``cfncluster-cookbook``: Support for NVMe instance store
+* updates:``ami``: Update all base AMIs to latest patch levels
+* bugfix:``cfncluster-node``: Fixed long scaling times with SLURM
+
 1.3.2
 =====
 * feature:``cfncluster``: Add support for r2.xlarge/t2.2xlarge,

CONTRIBUTORS.txt

@@ -1,12 +1,17 @@
 List of contributors:
 
  - Dougal Ballantyne - dougalb at amazon dot com
+ - Brian Barrett - bbarrett at amazon dot com
  - Adam Boeglin - boeglina at amazon dot com
+ - Jordan Cherry - cherryj at amazon dot com
+ - Mohan Gandhi - mohgan at amazon dot com
+ - Chris Gorski - cgorski at amazon dot com
+ - Raghu Raja - craghun at amazon dot com
  - Sean Smith - sensmit at amazon dot com
- - Brian Barrett - bbarrett at amazon dot com
 
+ - Josh Clausman - jclausman at semanticmachines dot com
  - Kenneth Daily - kmdaily at gmail dot com
+ - Karl Gutwin - karl dot gutwin at biogen dot com
  - John Lilley - johnbot at ltech dot edu
  - Nelson R Monserrate - MonserrateNelson at JohnDeere dot com
- - Karl Gutwin - karl dot gutwin at biogen dot com
- - Josh Clausman - jclausman at semanticmachines dot com
+

README.rst

@@ -5,21 +5,21 @@ CfnCluster
    :target: https://travis-ci.org/awslabs/cfncluster
    :alt: Build Status
 
-CfnCluster ("cloud formation cluster") is a framework that deploys and maintains high performance computing clusters on Amazon Web Services (AWS). Developed by AWS, CfnCluster facilitates both quick start proof of concepts (POCs) and production deployments. CfnCluster supports many different types of clustered applications and can easily be extended to support different frameworks. The CLI is stateless, everything is done using CloudFormation or resources within AWS.
-
-Known Issues
-============
-
-* CfnCluster 1.3.2 supports the I3 and F1 instance families.  However,
-  it will not automatically setup a single, RAIDed volume across all
-  available instance store volumes (like it will for other instance
-  families which support instance storage).  This will be fixed in a
-  future release.
+CfnCluster ("cloud formation cluster") is a framework that deploys and
+maintains high performance computing clusters on Amazon Web Services
+(AWS). Developed by AWS, CfnCluster facilitates both quick start proof
+of concepts (POCs) and production deployments. CfnCluster supports
+many different types of clustered applications and can easily be
+extended to support different frameworks. The CLI is stateless,
+everything is done using CloudFormation or resources within AWS.
 
 Documentation
 =============
 
-Documentation is part of the project and is published to - https://cfncluster.readthedocs.io/. Of most interest to new users is the Getting Started Guide - https://cfncluster.readthedocs.io/en/latest/getting_started.html.
+Documentation is part of the project and is published to -
+https://cfncluster.readthedocs.io/. Of most interest to new users is
+the Getting Started Guide -
+https://cfncluster.readthedocs.io/en/latest/getting_started.html.
 
 Issues
 ======

amis.txt

@@ -1,75 +1,75 @@
 # centos6
-ap-northeast-1: ami-10686677
-ap-northeast-2: ami-bb77abd5
-ap-south-1: ami-0a710e65
-ap-southeast-1: ami-9b1596f8
-ap-southeast-2: ami-c7b9afa4
-ca-central-1: ami-a6f847c2
-eu-central-1: ami-56d67239
-eu-west-1: ami-abd5c9cd
-eu-west-2: ami-cabcabae
-sa-east-1: ami-b491f9d8
-us-east-1: ami-a293c9b4
-us-east-2: ami-2cd3f549
-us-west-1: ami-65c2e005
-us-west-2: ami-6bb1b812
+ap-northeast-1: ami-aadd65cc
+ap-northeast-2: ami-b204a3dc
+ap-south-1: ami-6fb5fb00
+ap-southeast-1: ami-4dde8d2e
+ap-southeast-2: ami-9eda2ffc
+ca-central-1: ami-6855ee0c
+eu-central-1: ami-4540cc2a
+eu-west-1: ami-2e07b457
+eu-west-2: ami-2a4d534e
+sa-east-1: ami-3c682c50
+us-east-1: ami-2d4ed357
+us-east-2: ami-816d44e4
+us-west-1: ami-e6d4ef86
+us-west-2: ami-376db34f
 # centos7
-ap-northeast-1: ami-0e6b6569
-ap-northeast-2: ami-9377abfd
-ap-south-1: ami-546d123b
-ap-southeast-1: ami-da1596b9
-ap-southeast-2: ami-afa6b0cc
-ca-central-1: ami-62f94606
-eu-central-1: ami-9cd571f3
-eu-west-1: ami-6dd8c40b
-eu-west-2: ami-9cb9aef8
-sa-east-1: ami-5092fa3c
-us-east-1: ami-d29cc6c4
-us-east-2: ami-c3d2f4a6
-us-west-1: ami-17fcde77
-us-west-2: ami-22b2bb5b
+ap-northeast-1: ami-73d96115
+ap-northeast-2: ami-9107a0ff
+ap-south-1: ami-33b7f95c
+ap-southeast-1: ami-d8dd8ebb
+ap-southeast-2: ami-79c6331b
+ca-central-1: ami-3656ed52
+eu-central-1: ami-d945c9b6
+eu-west-1: ami-aa00b3d3
+eu-west-2: ami-984d53fc
+sa-east-1: ami-23682c4f
+us-east-1: ami-c24ed3b8
+us-east-2: ami-bb6c45de
+us-west-1: ami-4dd6ed2d
+us-west-2: ami-b370aecb
 # alinux
-ap-northeast-1: ami-586b653f
-ap-northeast-2: ami-ea71ad84
-ap-south-1: ami-1f720d70
-ap-southeast-1: ami-1f1b987c
-ap-southeast-2: ami-56b9af35
-ca-central-1: ami-65f94601
-eu-central-1: ami-7fd17510
-eu-west-1: ami-6ddac60b
-eu-west-2: ami-f1bbac95
-sa-east-1: ami-a990f8c5
-us-east-1: ami-2492c832
-us-east-2: ami-a6d3f5c3
-us-west-1: ami-39c2e059
-us-west-2: ami-f4b0b98d
+ap-northeast-1: ami-31dd6557
+ap-northeast-2: ami-dc07a0b2
+ap-south-1: ami-32b7f95d
+ap-southeast-1: ami-7edf8c1d
+ap-southeast-2: ami-c3c431a1
+ca-central-1: ami-8051eae4
+eu-central-1: ami-5a4bc735
+eu-west-1: ami-ac00b3d5
+eu-west-2: ami-044c5260
+sa-east-1: ami-aa692dc6
+us-east-1: ami-c652cfbc
+us-east-2: ami-ed6f4688
+us-west-1: ami-10d7ec70
+us-west-2: ami-4f6db337
 # ubuntu1404
-ap-northeast-1: ami-466c6221
-ap-northeast-2: ami-2877ab46
-ap-south-1: ami-11700f7e
-ap-southeast-1: ami-be1497dd
-ap-southeast-2: ami-37a4b254
-ca-central-1: ami-eefc438a
-eu-central-1: ami-bcd773d3
-eu-west-1: ami-83d4c8e5
-eu-west-2: ami-5cb9ae38
-sa-east-1: ami-6d91f901
-us-east-1: ami-a192c8b7
-us-east-2: ami-e2d3f587
-us-west-1: ami-b5c2e0d5
-us-west-2: ami-f5b0b98c
+ap-northeast-1: ami-12da6274
+ap-northeast-2: ami-4c05a222
+ap-south-1: ami-16b7f979
+ap-southeast-1: ami-cbde8da8
+ap-southeast-2: ami-d3c530b1
+ca-central-1: ami-0851ea6c
+eu-central-1: ami-bd4ac6d2
+eu-west-1: ami-2e06b557
+eu-west-2: ami-cf4d53ab
+sa-east-1: ami-f36a2e9f
+us-east-1: ami-3d49d447
+us-east-2: ami-8f6d44ea
+us-west-1: ami-bcd4efdc
+us-west-2: ami-726fb10a
 # ubuntu1604
-ap-northeast-1: ami-ce6c62a9
-ap-northeast-2: ami-d072aebe
-ap-south-1: ami-446f102b
-ap-southeast-1: ami-951596f6
-ap-southeast-2: ami-cca6b0af
-ca-central-1: ami-a7fa45c3
-eu-central-1: ami-40d6722f
-eu-west-1: ami-e6d5c980
-eu-west-2: ami-5eb9ae3a
-sa-east-1: ami-df90f8b3
-us-east-1: ami-8c9fc59a
-us-east-2: ami-54d5f331
-us-west-1: ami-0dfddf6d
-us-west-2: ami-c4b1b8bd
+ap-northeast-1: ami-b2da62d4
+ap-northeast-2: ami-0a05a264
+ap-south-1: ami-11b4fa7e
+ap-southeast-1: ami-f8df8c9b
+ap-southeast-2: ami-15c73277
+ca-central-1: ami-3156ed55
+eu-central-1: ami-dd4bc7b2
+eu-west-1: ami-9802b1e1
+eu-west-2: ami-c14f51a5
+sa-east-1: ami-34692d58
+us-east-1: ami-6353ce19
+us-east-2: ami-2d6e4748
+us-west-1: ami-68d5ee08
+us-west-2: ami-5e6cb226

cli/cfncluster/cfncluster.py

@@ -24,15 +24,15 @@
 
 from . import cfnconfig
 
-logger = logging.getLogger('cfncluster.cfncluster')
-
 def version(args):
     config = cfnconfig.CfnClusterConfig(args)
     print(config.version)
 
 def create(args):
-    print('Starting: %s' % (args.cluster_name))
+    logger = logging.getLogger('cfncluster.cfncluster.create')
+    logger.info('Beginning cluster creation for cluster: %s' % (args.cluster_name))
 
+    logger.debug('Building cluster config based on args')
     # Build the config based on args
     config = cfnconfig.CfnClusterConfig(args)
 
@@ -44,6 +44,7 @@ def create(args):
     except ValueError:
         pass
 
+
     # Get the MasterSubnetId and use it to determine AvailabilityZone
     try:
         i = [p[0] for p in config.parameters].index('MasterSubnetId')
@@ -52,22 +53,36 @@ def create(args):
             vpcconn = boto.vpc.connect_to_region(config.region,aws_access_key_id=config.aws_access_key_id,
                                                  aws_secret_access_key=config.aws_secret_access_key)
             availability_zone = str(vpcconn.get_all_subnets(subnet_ids=master_subnet_id)[0].availability_zone)
-        except boto.exception.BotoServerError as e:
-            print(e.message)
+        except Exception as e:
+            logger.critical(e.message)
             sys.exit(1)
         config.parameters.append(('AvailabilityZone', availability_zone))
     except ValueError:
         pass
 
+
+
     capabilities = ["CAPABILITY_IAM"]
-    cfnconn = boto.cloudformation.connect_to_region(config.region,aws_access_key_id=config.aws_access_key_id,
-                                                    aws_secret_access_key=config.aws_secret_access_key)
     try:
-        logger.debug((config.template_url, config.parameters))
-        stack = cfnconn.create_stack(('cfncluster-' + args.cluster_name),template_url=config.template_url,
+        cfnconn = boto.cloudformation.connect_to_region(config.region,aws_access_key_id=config.aws_access_key_id,
+                                                        aws_secret_access_key=config.aws_secret_access_key)
+    except Exception as e:
+        logger.critical(e.message)
+        sys.exit(1)
+
+
+
+
+    try:
+        stack_name = 'cfncluster-' + args.cluster_name
+        logger.info("Creating stack named: " + stack_name)
+        stack = cfnconn.create_stack(stack_name,template_url=config.template_url,
                                      parameters=config.parameters, capabilities=capabilities,
                                      disable_rollback=args.norollback, tags=config.tags)
+        logger.info("foo")
         status = cfnconn.describe_stacks(stack)[0].stack_status
+
+
         if not args.nowait:
             while status == 'CREATE_IN_PROGRESS':
                 status = cfnconn.describe_stacks(stack)[0].stack_status
@@ -82,12 +97,13 @@ def create(args):
         else:
             status = cfnconn.describe_stacks(stack)[0].stack_status
             print('Status: %s' % status)
-    except boto.exception.BotoServerError as e:
-        print(e.message)
-        sys.exit(1)
     except KeyboardInterrupt:
         print('\nExiting...')
         sys.exit(0)
+    except Exception as e:
+        logger.critical(e.message)
+        sys.exit(1)
+
 
 def update(args):
     print('Updating: %s' % (args.cluster_name))
@@ -152,27 +168,11 @@ def update(args):
         sys.exit(0)
 
 def start(args):
-    # 1) Start master server
-    # 2) Set resource limits on compute fleet to min/max/desired = 0/max/0
-    print('Starting: %s' % args.cluster_name)
+    # Set resource limits on compute fleet to min/max/desired = 0/max/0
+    print('Starting compute fleet : %s' % args.cluster_name)
     stack_name = ('cfncluster-' + args.cluster_name)
     config = cfnconfig.CfnClusterConfig(args)
-    master_server_id = get_master_server_id(stack_name, config)
-    ec2conn = boto.ec2.connect_to_region(config.region,aws_access_key_id=config.aws_access_key_id,
-                                                    aws_secret_access_key=config.aws_secret_access_key)
-    try:
-        response = ec2conn.start_instances(master_server_id)
-    except boto.exception.BotoServerError as e:
-        if e.message.endswith("does not exist"):
-            print(e.message)
-            sys.stdout.flush()
-            sys.exit(0)
-        else:
-            raise e
-    except KeyboardInterrupt:
-        print('\nExiting...')
-        sys.exit(0)
-
+    
     # Set asg limits
     max_queue_size = [param[1] for param in config.parameters if param[0] == 'MaxQueueSize']
     max_queue_size = max_queue_size[0] if len(max_queue_size) > 0 else 10
@@ -184,40 +184,16 @@ def start(args):
     asg = get_asg(stack_name=stack_name, config=config)
     set_asg_limits(asg=asg, min=min_queue_size, max=max_queue_size, desired=desired_queue_size)
 
-    # Poll for status
-    poll_master_server_state(stack_name, config)
-
 def stop(args):
-    # 1) Set resource limits on compute fleet to min/max/desired = 0/0/0
-    # 2) Stop master server
-    print('Stopping: %s' % args.cluster_name)
+    # Set resource limits on compute fleet to min/max/desired = 0/0/0
+    print('Stopping compute fleet : %s' % args.cluster_name)
     stack_name = ('cfncluster-' + args.cluster_name)
     config = cfnconfig.CfnClusterConfig(args)
 
     # Set Resource limits
     asg = get_asg(stack_name=stack_name, config=config)
     set_asg_limits(asg=asg, min=0, max=0, desired=0)
 
-    # Stop master Server
-    master_server_id = get_master_server_id(stack_name, config)
-    ec2conn = boto.ec2.connect_to_region(config.region,aws_access_key_id=config.aws_access_key_id,
-                                                    aws_secret_access_key=config.aws_secret_access_key)
-    try:
-        response = ec2conn.stop_instances(master_server_id)
-    except boto.exception.BotoServerError as e:
-        if e.message.endswith("does not exist"):
-            print(e.message)
-            sys.stdout.flush()
-            sys.exit(0)
-        else:
-            raise e
-    except KeyboardInterrupt:
-        print('\nExiting...')
-        sys.exit(0)
-    
-    # Poll for status
-    poll_master_server_state(stack_name, config)
-
 def list(args):
     config = cfnconfig.CfnClusterConfig(args)
     cfnconn = boto.cloudformation.connect_to_region(config.region,aws_access_key_id=config.aws_access_key_id,

cli/cfncluster/cfnconfig.py

@@ -178,7 +178,7 @@ def __init__(self, args):
                                   compute_subnet_cidr=('ComputeSubnetCidr',None),
                                   compute_subnet_id=('ComputeSubnetId', 'VPCSubnet'), use_public_ips=('UsePublicIps',
                                                                                                       None),
-                                  ssh_from=('AccessFrom', None), access_from=('AccessFrom', None), 
+                                  ssh_from=('AccessFrom', None), access_from=('AccessFrom', None),
                                   additional_sg=('AdditionalSG','VPCSecurityGroup'),
                                   vpc_security_group_id=('VPCSecurityGroupId','VPCSecurityGroup')
                                   )
@@ -321,8 +321,8 @@ def __init__(self, args):
         # Handle extra parameters supplied on command-line
         try:
             if self.args.extra_parameters is not None:
-                self.parameters = dict(self.parameters)
-                self.__temp_dict = dict(list(self.parameters.items()) + list(self.args.extra_parameters.items()))
+                self.__temp_dict = dict(self.parameters)
+                self.__temp_dict.append(dict(self.args.extra_parameters))
                 self.__dictlist = []
                 for key, value in self.__temp_dict.items():
                     temp = [str(key),str(value)]