Merge branch 'master' into develop

Author: Pang, Lee

docs/core-env/create-custom-compute-resources.md

@@ -1,19 +1,16 @@
 # Creating Custom Compute Resources
 
 Genomics is a data-heavy workload and requires some modification to the defaults
-used for batch job processing. In particular, we need to be able to scale the
-storage used by the instances Tasks/Jobs run on to meet unpredictable runtime 
-demands.
+used for batch job processing. In particular, instances running the Tasks/Jobs need scalable storage to meet unpredictable runtime demands.
 
 By default, AWS Batch relies upon the [Amazon ECS-Optimized AMI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html)
-as the image used to for instances it launches to run jobs.  What this image 
-provides is sufficient in most cases.  Specialized needs, such as the large 
+to launch container instances for running jobs.  This is sufficient in most cases, but specialized needs, such as the large 
 storage requirements noted above, require customization of the base AMI.
 
 This section provides two methods for customizing the base ECS-Optimized AMI 
-that adds an expandable working directory that the Jobs will use to write data.
-That directory will be monitored by a process that inspects the free space 
-available and adds more EBS volumes and expands the filesystem on the fly, like so:
+that adds an expandable working directory for jobs to write data.
+A process will monitor the directory and add more EBS volumes on the fly to expand the free space 
+based on the capacity threshold, like so:
 
 ![Autoscaling EBS storage](images/ebs-autoscale.png)
 
@@ -92,4 +89,4 @@ Once your AMI is created, you will need to jot down its unique AMI Id.  You will
 need this when creating compute resources in AWS Batch.
 
 !!! note
-    This is considered advanced use.  All documentation and CloudFormation templates hereon assumes use of EC2 Launch Templates.
+    This is considered advanced use.  All documentation and CloudFormation templates hereon assumes use of EC2 Launch Templates.

docs/core-env/create-iam-roles.md

@@ -1,19 +1,19 @@
-# Creating IAM Roles
+# Permissions
 
-Below are IAM roles that your job execution environment in AWS Batch will use:
+## Create IAM Roles
 
-* Batch Service Role:
+IAM roles that your job execution environment in AWS Batch will use include:
+
+* **Batch Service Role (required)**:
 
-    (required)
     Role used by AWS Batch to call other AWS services on its behalf.
     AWS Batch makes calls to other AWS services on your behalf to manage the resources that you use with the service. Before you can use the service, you must have an IAM policy and role that provides the necessary permissions to AWS Batch.
     [(Learn More)](https://docs.aws.amazon.com/batch/latest/userguide/service_IAM_role.html)
 
-* Batch Instance Profile:
+* **Batch Instance Profile (required)**:
 
-    (required)
     Role that defines service permissions for EC2 instances launched by AWS Batch.
-    For example, this is used to specify policies that allow access to specific S3 buckets and modify storage on the intance (shown below).
+    For example, this is used to specify policies that allow access to specific S3 buckets and modify storage on the instance (shown below).
     [(Learn More)](https://docs.aws.amazon.com/batch/latest/userguide/instance_IAM_role.html)
 
 ```yaml
@@ -46,16 +46,14 @@ Below are IAM roles that your job execution environment in AWS Batch will use:
             Resource: "*"
 ```
 
-* Batch SpotFleet Role:
+* **Batch SpotFleet Role (depends)**:
     
-    (depends)
-    This is role is needed if you intend to launch spot instances from AWS Batch.
+    This role is needed if you intend to launch spot instances from AWS Batch.
     If you create a managed compute environment that uses Amazon EC2 Spot Fleet Instances, you must create a role that grants the Spot Fleet permission to bid on, launch, tag, and terminate instances on your behalf.
     [(Learn More)](https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html)
 
-* Batch Job Role:
+* **Batch Job Role (optional)**:
 
-    (optional)
     Role used to provide service permissions to individual jobs.
     Jobs can run without an IAM role. In that case, they inherit the
     permissions of the instance they run on.

docs/core-env/create-s3-bucket.md

@@ -1,4 +1,4 @@
-# Creating an S3 Bucket
+# Data Storage
 
 You will need a robust location to store your input and output data.  As mentioned
 previously, genomics data files are fairly large.  In addition to input sample
@@ -12,6 +12,8 @@ The following are key criteria for storing data for genomics workflows
 * durable
 * capable of handling large files
 
+## Create an S3 Bucket
+
 Amazon S3 buckets meet all of the above conditions.
 
 You can use an existing bucket for your workflows, or you can create a new one using the CloudFormation template below.

docs/core-env/setup-aws-batch.md

@@ -14,19 +14,19 @@ A [job definition](http://docs.aws.amazon.com/batch/latest/userguide/job_definit
 
 Jobs are submitted to [job queues](http://docs.aws.amazon.com/batch/latest/userguide/job_queues.html) where they reside until they can be scheduled to run on Amazon EC2 instances within a compute environment. An AWS account can have multiple job queues, each with varying priority. This gives you the ability to closely align the consumption of compute resources with your organizational requirements.
 
-[Compute environments](http://docs.aws.amazon.com/batch/latest/userguide/compute_environments.html) provision and manage your EC2 instances and other compute resources that are used to run your AWS Batch jobs. Job queues are mapped to one more compute environments and a given environment can also be mapped to one or more job queues. This many-to-many relationship is defined by the compute environment order and job queue priority properties.
+[Compute environments](http://docs.aws.amazon.com/batch/latest/userguide/compute_environments.html) provision and manage your EC2 instances and other compute resources that are used to run your AWS Batch jobs. Job queues are mapped to one or more compute environments and a given environment can also be mapped to one or more job queues. This many-to-many relationship is defined by the compute environment order and job queue priority properties.
 
 The following diagram shows a general overview of how the AWS Batch resources interact.
 
 ![AWS Batch environment](https://d2908q01vomqb2.cloudfront.net/1b6453892473a467d07372d45eb05abc2031647a/2018/04/23/AWSBatchresoucreinteract-diagram.png)
 
 For more information, watch the [How AWS Batch Works](https://www.youtube.com/watch?v=T4aAWrGHmxQ) video.
 
-### Requirements for AWS Batch Jobs
+## AWS Batch Jobs Requirements
 
-AWS Batch does not make assumptions on the structure and requirements that Jobs take with respect to inputs and outputs. Batch Jobs may take data streams, files, or only parameters as input, and produce the same variaty for output, inclusive of files, metadata changes, updates to databases, etc. Batch assumes that each application handles their own input/output requirements.
+AWS Batch does not make assumptions on the structure and requirements that Jobs take with respect to inputs and outputs. Batch Jobs may take data streams, files, or only parameters as input, and produce the same variety for output, inclusive of files, metadata changes, updates to databases, etc. Batch assumes that each application handles their own input/output requirements.
 
-A common pattern for bioinformatics tooling is that files such as genomic sequence data are both inputs and outputs to/from a process. Many bioinformatics tools have also been developed to run in traditional Linux-based compute clusters with shared filesystems, and are not necessarily optimized for cloud computing.
+A common pattern for bioinformatics tooling is that files such as genomic sequence data are both inputs and outputs to/from a process. Many bioinformatics tools have also been developed to run in traditional Linux-based compute clusters with shared filesystems and are not necessarily optimized for cloud computing.
 
 The set of common requirements for genomics on AWS Batch are:
 
@@ -36,19 +36,19 @@ The set of common requirements for genomics on AWS Batch are:
 
 * Multitenancy:
 
-    Multiple container jobs may run concurrently on the same instance. In these situations, it’s essential that your job writes to a unique subdirectory.
+    Multiple container jobs may run concurrently on the same instance. In these situations, it is essential that your job writes to a unique subdirectory.
 
 * Data cleanup:
 
     As your jobs complete and write the output back to S3, it is a good idea to delete the scratch data generated by that job on your instance. This allows you to optimize for cost by reusing EC2 instances if there are jobs remaining in the queue, rather than terminating the EC2 instances.
 
-## What you will need
+## AWS Batch Environment
 
 A complete AWS Batch environment consists of the following:
 
 1. A Compute Environment that utilizes [EC2 Spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) for cost-effective computing
 2. A Compute Environment that utilizes EC2 on-demand (e.g. [public pricing](https://aws.amazon.com/ec2/pricing/on-demand/)) instances for high-priority work that can't risk job interruptions or delays due to insufficient Spot capacity.
-3. A default Job Queue that utilizes the Spot compute environment first, but falls back to the on-demand compute environment if there is spare capacity already there.
+3. A default Job Queue that utilizes the Spot compute environment first, but falls back to the on-demand compute environment if there is spare capacity available.
 4. A high-priority Job Queue that leverages the on-demand and Spot CE's (in that order) and has higher priority than the default queue.
 
 The CloudFormation template below will create all of the above.

docs/orchestration/cromwell/cromwell-overview.md

@@ -6,7 +6,7 @@
 system for scientific workflows developed by the [Broad Institute](https://broadinstitute.org/)
 and supports job execution using [AWS Batch](https://aws.amazon.com/batch/).
 
-## TL;DR
+## Full Stack Deployment (TL;DR)
 
 If you need a Cromwell server backed by AWS **now** and will worry about the
 details later, use the CloudFormation template below.
@@ -16,7 +16,8 @@ details later, use the CloudFormation template below.
 {{ cfn_stack_row("Cromwell All-in-One", "Cromwell", "cromwell/cromwell-aio.template.yaml", "Create all resources needed to run Cromwell on AWS: an S3 Bucket, AWS Batch Environment, and Cromwell Server Instance") }}
 
 When the above stack is complete, navigate to the `HostName` that is generated
-in the outputs to access Cromwell via its SwaggerUI.
+in the outputs to access Cromwell via its SwaggerUI, which provides a simple web interface for submitting workflows.
+
 
 ![cromwell on aws](images/cromwell-all-in-one.png)
 
@@ -36,7 +37,7 @@ these setup.
 !!! note
     For a Cromwell server that will run multiple workflows, or workflows with many
     steps (e.g. ones with large scatter steps), it is recommended to setup a
-    database to use to store workflow metadata.
+    database to store workflow metadata.
 
 ## Custom Compute Resource with Cromwell Additions
 
@@ -49,7 +50,7 @@ Once complete, you will have a resource ID to give to AWS Batch to setup compute
 
 ## Cromwell Server
 
-To ensure the highest level of security, and robustness for long running workflows,
+To ensure the highest level of security and robustness for long running workflows,
 it is recommended that you use an EC2 instance as your Cromwell server for submitting
 workflows to AWS Batch.
 
@@ -222,7 +223,7 @@ backend {
 
 The above file uses the [default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) for authorization.
 
-Replace the following with values appropriate for your accoutn and workload:
+Replace the following with values appropriate for your account and workload:
 
 * `<your region>` : the AWS region your S3 bucket and AWS Batch environment are
   deployed into - e.g. `us-east-1`
@@ -235,7 +236,7 @@ Replace the following with values appropriate for your accoutn and workload:
 
 !!! note
     The CloudFormation template above automatically starts Cromwell on launch.
-    Use the instructions below are if you are provisioning your own EC2 instance.
+    Use the instructions below if you are provisioning your own EC2 instance.
 
 Log into your server using SSH.  If you setup a port tunnel, you can interact
 with Cromwell's REST API from your local machine:
@@ -266,13 +267,13 @@ your local machine by navigating to:
 
 ## Running a workflow
 
-To submit a workflow to your Cromwell server, you can use:
+To submit a workflow to your Cromwell server, you can use any of the following:
 
 * Cromwell's SwaggerUI in a web-browser
 * a REST client like [Insomnia](https://insomnia.rest/) or [Postman](https://www.getpostman.com/)
-* or, the command line with `curl`
+* the command line with `curl`
 
 After submitting a workflow, you can monitor the progress of tasks via the
 AWS Batch console.
 
-The next section provides some examples of running Crommwell on AWS.
+The next section provides some examples of running Crommwell on AWS.

docs/orchestration/step-functions/step-functions-overview.md

@@ -4,7 +4,7 @@
 
 [AWS Step Functions](https://aws.amazon.com/step-functions/) is a service that allows you to orchestrate other AWS services, such as Lambda, Batch, SNS, and Glue, making it easy to coordinate the components of distributed applications as a series of steps in a visual workflow.
 
-In the context of genomics workflows, the combination of Step Functions with Batch and Lambda constitutes a robust and scalable task orchestration solution.
+In the context of genomics workflows, the combination of AWS Step Functions with Batch and Lambda constitutes a robust, scalable, and serverless task orchestration solution.
 
 ## TL;DR
 
@@ -20,15 +20,15 @@ read through the rest of this page.
 
 To get started using AWS Step Functions for genomics workflows you'll need the following setup in your AWS account:
 
-1. The core set of resources (S3 Bucket, IAM Roles, AWS Batch) described in the [Getting Started](../../../core-env/introduction/) section.
+* The core set of resources (S3 Bucket, IAM Roles, AWS Batch) described in the [Getting Started](../../../core-env/introduction/) section.
 
-## Step Functions Execution Role
+## AWS Step Functions Execution Role
 
-A Step Functions Execution role is an IAM role that allows Step Functions to execute other AWS services via the state machine.
+An AWS Step Functions Execution role is an IAM role that allows Step Functions to execute other AWS services via the state machine.
 
-This can be created automatically during the "first-run" experience in the Step Functions console when you create your first state machine.  The policy attached to the role will depend on the specifc tasks you incorporate into your state machine.
+This can be created automatically during the "first-run" experience in the AWS Step Functions console when you create your first state machine.  The policy attached to the role will depend on the specifc tasks you incorporate into your state machine.
 
-For state machines that use AWS Batch for job execution and send events to CloudWatch, the should have an Execution role with the following inline policy:
+State machines that use AWS Batch for job execution and send events to CloudWatch should have an Execution role with the following inline policy:
 
 ```json
 {
@@ -60,7 +60,7 @@ For state machines that use AWS Batch for job execution and send events to Cloud
 
 ## Step Functions State Machine
 
-Workflows in AWS Step Functions are built using [Amazon States Language](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html) (ASL), a declarative, JSON-based, structured language used to define your state machine, a collection of states, that can do work (Task states), determine which states to transition to next (Choice states), stop an execution with an error (Fail states), and so on.
+Workflows in AWS Step Functions are built using [Amazon States Language](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html) (ASL), a declarative, JSON-based, structured language used to define your state machine, a collection of states that can do work (Task states), determine which states to transition to next (Choice states), stop an execution with an error (Fail states), and so on.
 
 ### Building workflows with AWS Step Functions
 
@@ -178,7 +178,7 @@ An example Job Definition for the `bwa-mem` sequence aligner is shown below:
 
 ### State Machine Batch Job Tasks
 
-Conveniently for genomics workflows, Step Functions has built-in integration with AWS Batch (and [several other services](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-connectors.html)), and provides snippets of code to make developing your state-machine
+Conveniently for genomics workflows, AWS Step Functions has built-in integration with AWS Batch (and [several other services](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-connectors.html)), and provides snippets of code to make developing your state-machine
 Batch tasks easier.
 
 ![Manage a Batch Job Snippet](images/sfn-batch-job-snippet.png)