Updated docs and a few bits for v0.0.18

Author: dougalb

.gitignore

@@ -7,3 +7,4 @@ dist/
 build/
 *.egg-info/
 .idea/
+.DS_Store

CHANGELOG.rst

@@ -12,6 +12,7 @@ CHANGELOG
 * feature:``cfncluster``: Support for additional VPC security group
 * updates:``ami``: Pulled latest NVIDIA driver 340.65
 * feature:``cli``: Added support for version command
+* updates:``cli``: Removed unimplemented stop command from CLI
 
 0.0.17
 ======

README.md

@@ -0,0 +1,9 @@
+cfncluster
+==========
+
+cfncluster is a framework that deploys and maintains HPC clusters on AWS. It is reasonably agnostic to what the cluster is for and can easily be extended to support different frameworks. The CLI is stateless, everything is done using CloudFormation or resources within AWS.
+
+Documentation
+=============
+
+Documentation is part of the project and is published to - http://cfncluster.readthedocs.org/. Of most interest to new users is the Getting Started Guide - http://cfncluster.readthedocs.org/en/latest/getting_started.html.

README.rst

@@ -1,7 +1,7 @@
 cfncluster is an example framework for launching clusters.
 
     usage: cfncluster [-h] [--config CONFIG_FILE] [--region REGION] [--nowait]
-                      {create,update,stop,delete,status,list,instances,sshmaster}
+                      {create,update,delete,status,list,instances,version}
                       ...
 
     cfncluster is the a tool to launch and manage cluster.
@@ -10,11 +10,11 @@ cfncluster is an example framework for launching clusters.
       {create,update,stop,delete,status,list,instances,sshmaster}
         create              creates a cluster
         update              update a running cluster
-        stop                stop a cluster
         delete              delete a cluster
         status              pull the current status of the cluster
         list                display a list of stacks associated with cfncluster
         instances           display a list of all instances in a cluster
+        version             display the version of cfncluster cli
 
     optional arguments:
       -h, --help            show this help message and exit

cli/README

@@ -20,9 +20,6 @@
 def create(args):
     cfncluster.create(args)
 
-def stop(args):
-    pass
-
 def status(args):
     cfncluster.status(args)
 
@@ -106,11 +103,6 @@ def main():
                          help='reset the current ASG desired capacity to initial config values')
     pupdate.set_defaults(func=update)
 
-    pstop = subparsers.add_parser('stop', help='stop a cluster')
-    pstop.add_argument("cluster_name", type=str, default=None,
-                        help='stop a cfncluster with the provided name.')
-    pstop.set_defaults(func=stop)
-
     pdelete = subparsers.add_parser('delete', help='delete a cluster')
     pdelete.add_argument("cluster_name", type=str, default=None,
                         help='delete a cfncluster with the provided name.')

cli/cfncluster/cli.py

@@ -20,8 +20,8 @@ def read(fname):
     return open(os.path.join(os.path.dirname(__file__), fname)).read()
 
 console_scripts = ['cfncluster = cfncluster.cli:main']
-version = "0.0.18"
-requires = ['boto>=2.34'] 
+version = "0.0.99"
+requires = ['boto>=2.35.1'] 
 
 if sys.version_info[:2] == (2, 6):
     # For python2.6 we have to require argparse since it

cli/setup.py

@@ -3,54 +3,85 @@
 AWS Services used in cfncluster
 ===============================
 
-The following Amazon Web Services(AWS) services are used in cfncluster. 
+The following Amazon Web Services (AWS) services are used in cfncluster. 
 
-* CloudFormation
-* IAM
-* SNS
-* SQS
-* EC2
-* AutoScaling
-* EBS
-* Cloudwatch
-* S3
-* DynamoDB
+* AWS CloudFormation
+* AWS Identity and Access Management (IAM)
+* Amazon SNS
+* Amazon SQS
+* Amazon EC2
+* Auto Scaling
+* Amazon EBS
+* Amazon Cloud Watch
+* Amazon S3
+* Amazon DynamoDB
 
-CloudFormation
---------------
+AWS CloudFormation
+------------------
 
-CloudFormation is the core service used by cfncluster. Each cluster is representated as a stack. All resources required by the cluster are defined within the cfncluster CloudFormation template. cfncluster cli commands typically map to CloudFormation stack commands, such as create, update and delete. Instances launched within a cluster make HTTPS calls to the CloudFormation Endpoint for the region the cluster is launched in.
+AWS CloudFormation is the core service used by cfncluster. Each cluster is representated as a stack. All resources required by the cluster are defined within the cfncluster CloudFormation template. cfncluster cli commands typically map to CloudFormation stack commands, such as create, update and delete. Instances launched within a cluster make HTTPS calls to the CloudFormation Endpoint for the region the cluster is launched in.
 
-IAM
----
+For more details about AWS CloudFormation, see http://aws.amazon.com/cloudformation/
 
-Idenity and Access Management is used within cfncluster to provide an EC2 IAM Role for the instances. This role is a least privilged role specifically created for each cluster. cfncluster instances are given access only to the specific API calls that are required to deploy and manage the cluster. 
+AWS Identity and Access Management (IAM)
+----------------------------------------
 
-SNS
----
+IAM is used within cfncluster to provide an Amazon EC2 IAM Role for the instances. This role is a least privilged role specifically created for each cluster. cfncluster instances are given access only to the specific API calls that are required to deploy and manage the cluster. 
 
-Simple Notification Service is used to receive notifications from Autoscaling. These events are called life cycle events, and are generated when an instance lauches or terminates in an Autoscaling Grpoup. Within cfncluster, the SNS topic for the Autoscaling Group is subnscibred to an SQS queue.
+For more details about AWS Identity and Access Management, see http://aws.amazon.com/iam/
 
-SQS
----
+Amazon SNS
+----------
+
+Amazon Simple Notification Service is used to receive notifications from Auto Scaling. These events are called life cycle events, and are generated when an instance lauches or terminates in an Autoscaling Grpoup. Within cfncluster, the Amazon SNS topic for the Autoscaling Group is subnscibred to an Amazon SQS queue.
+
+For more details about Amazon SNS, see http://aws.amazon.com/sns/
+
+Amazon SQS
+----------
 
-Simple Queuing Service is used to hold notifications(messages) from Autoscaling, sent through SNS. This decouples the 
+Amazon Simple Queuing Service is used to hold notifications(messages) from Auto Scaling, sent through Amazon SNS and notifications from the ComputeFleet instanes. This decouples the sending of notifications from the receiving and allows the Master to handle them through polling. The MasterServer runs Amazon SQSwatcher and polls the queue. AutoScaling and the ComputeFleet instanes post messages to the queue.
 
-EC2
----
+For more details about Amazon SQS, see http://aws.amazon.com/sqs/
 
-Autscaling
+Amazon EC2
 ----------
 
-EBS
----
+Amazon EC2 provides the compute for cfncluster. The MasterServer and ComputeFleet are EC2 instances. Any instance type that support HVM can be selected. The MasterServer and ComputeFleet can be different instance types and the ComputeFleet can also be laucnhed as Spot instances. Ephmeral storage found on the instances is mounted as a RAID0 volume.
+
+For more details about Amazon EC2, see http://aws.amazon.com/ec2/
+
+Auto Scaling
+------------
+
+Auto Scaling is used to manage the ComputeFleet instances. These are managed as an AutoScaling Group and can either be elastic driven by workload or static driven by the config. 
+
+For more details about Auto Scaling, see http://aws.amazon.com/autoscaling/
 
-Cloudwatch
+Amazon EBS
 ----------
 
-S3
---
+Amazon EBS provides the storage for the shared volume. Any EBS settings can be passed through the config. EBS volumes can either be initialized empty or from an exisiting EBS snapshot.
+
+For more details about Amazon EBS, see http://aws.amazon.com/ebs/
+
+Amazon Cloud Watch
+------------------
+
+Amazon Cloud Watch provides metric collection and alarms for cfncluster. The MasterServer publishs pending tasks(jobs) for each cluster. Two alarms are defined that based on parameters defined in the config will automatically increase the size of the ComputeFleet Auto Scaling group.
+
+For more details, see http://aws.amazon.com/cloudwatch/
+
+Amazon S3
+---------
+
+Amazon S3 is used to store the cfncluster templates. Each region has a bucket with all templates. Within cfncluster, access to S3 can be controlled to allow CLI/SDK tools to use S3.
+
+For more details, see http://aws.amazon.com/s3/
+
+Amazon DynamoDB
+---------------
 
-DynamoDB
---------
+Amazon DynamoDB is used to store minimal state of the cluster. The MasterServer tracks provisioned instances in a DynamoDB table.
 
+For more details, see http://aws.amazon.com/dynamodb/

docs/source/aws_services.rst

@@ -27,9 +27,11 @@ Additional Resources
 
 * `cfncluster Source Repository`_
 * `cfncluster Issue Tracker`_
+* `cfncluster Webcast - HPC Scalability in the Cloud`_
 
 .. _cfncluster Issue Tracker: https://github.com/awslabs/cfncluster/issues
 .. _cfncluster Source Repository: https://github.com/awslabs/cfncluster
+.. _cfncluster Webcast - HPC Scalability in the Cloud: https://www.youtube.com/watch?v=iHtzy6_WytE
 
 Indices and tables
 ==================

docs/source/index.rst

@@ -1,4 +1,18 @@
 .. _s3_resources:
 
 working with S3
-===============
+===============
+
+Accessing S3 within cfncluster can be controlled through two parameters in the cfncluster config.
+
+::
+
+  # Specify S3 resource which cfncluster nodes will be granted read-only access
+  # (defaults to NONE for the default template)
+  s3_read_resource = NONE
+  # Specify S3 resource which cfncluster nodes will be granted read-write access
+  # (defaults to NONE for the default template)
+  s3_read_write_resource = NONE
+
+Both parameters except either ``*`` or a valid S3 ARN. For details of how to specify S3 ARNs, please see http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3
+