Merge pull request #23 from wleepang/sfn-example

Add sfn example

Author: wleepang

docs/orchestration/step-functions/images/cfn-stack-outputs-statemachineinput.png

@@ -8,11 +8,17 @@ In the context of genomics workflows, the combination of AWS Step Functions with
 
 ## Full Stack Deployment (TL;DR)
 
-If you need something up and running in a hurry, a fully automated setup process
-is provided at this GitHub repository:
+If you need something up and running in a hurry, the follwoing CloudFormation template will create everything you need to run an example genomics workflow using `bwa-mem`, `samtools`, and `bcftools`.
+
+| Name | Description | Source | Launch Stack |
+| -- | -- | :--: | :--: |
+{{ cfn_stack_row("AWS Step Functions All-in-One Example", "AWSGenomicsWorkflow", "step-functions/sfn-aio.template.yaml", "Create all resources needed to run a genomics workflow with Step Functions: an S3 Bucket, AWS Batch Environment, State Machine, Batch Job Definitions, and container images") }}
+
+Another example that uses a scripted setup process is provided at this GitHub repository:
 
 [AWS Batch Genomics](https://github.com/aws-samples/aws-batch-genomics)
 
+
 If you are interested in creating your own solution with AWS Step Functions and AWS Batch,
 read through the rest of this page.
 
@@ -219,3 +225,39 @@ Inputs to a state machine that uses the above `BwaMemTask` would look like this:
 ```
 
 When the Task state completes Step Functions will add information to a new `status` key under `bwa-mem` in the JSON object.  The complete object will be passed on to the next state in the workflow.
+
+## Example state machine
+
+All of the above is created by the following CloudFormation template.
+
+| Name | Description | Source | Launch Stack |
+| -- | -- | :--: | :--: |
+{{ cfn_stack_row("AWS Step Functions Example", "SfnExample", "step-functions/sfn-example.template.yaml", "Create a Step Functions State Machine, Batch Job Definitions, and container images to run an example genomics workflow") }}
+
+!!! note
+    The stack above needs to create several IAM Roles.  You must have administrative privileges in your AWS Account for this to succeed.
+
+### Running the workflow
+
+When the stack above completes, go to the outputs tab and copy the JSON string provided in `StateMachineInput`.
+
+![cloud formation output tab](./images/cfn-stack-outputs-tab.png)
+![example state-machine input](./images/cfn-stack-outputs-statemachineinput.png)
+
+Next head to the AWS Step Functions console and select the state-machine that was created.
+
+![select state-machine](./images/sfn-console-statemachine.png)
+
+Click the "Start Execution" button.
+
+![start execution](./images/sfn-console-start-execution.png)
+
+In the dialog that appears, paste the input JSON into the "Input" field, and click the "Start Execution" button.  (A unique execution ID will be automatically generated).
+
+![start execution dialog](./images/sfn-console-start-execution-dialog.png)
+
+You will then be taken to the execution tracking page where you can monitor the progress of your workflow.
+
+![execution tracking](./images/sfn-console-execution-inprogress.png)
+
+The workflow takes approximately 5-6hrs to complete on `r4.2xlarge` SPOT instances.

docs/orchestration/step-functions/images/cfn-stack-outputs-tab.png

@@ -60,6 +60,7 @@ Below are the stand-alone CloudFormation templates for each of the sub-stacks. T
 
 | Name | Description | Source | Launch Stack |
 | -- | -- | :--: | :--: |
+{{ cfn_stack_row("AWS Step Functions Example", "SfnExample", "step-functions/sfn-example.template.yaml", "Create a Step Functions State Machine, Batch Job Definitions, and container images to run an example genomics workflow") }}
 {{ cfn_stack_row("Cromwell Server", "CromwellServer", "cromwell/cromwell-server.template.yaml", "Create an EC2 instance and an IAM instance profile to run Cromwell") }}
 {{ cfn_stack_row("Nextflow Resources", "NextflowResources", "nextflow/nextflow-resources.template.yaml", "Create Nextflow specific resources needed to run on AWS: an S3 Bucket for nextflow config and workflows, AWS Batch Job Definition for a Nextflow head node, and an IAM role for the nextflow head node job") }}
 

docs/orchestration/step-functions/images/sfn-console-execution-inprogress.png

@@ -0,0 +1,2 @@
+push.sh
+job-definition.json

docs/orchestration/step-functions/images/sfn-console-start-execution-dialog.png

@@ -0,0 +1,48 @@
+FROM ubuntu:18.04 AS build
+
+ARG VERSION=1.9
+
+# Metadata
+LABEL container.base.image="ubuntu:18.04"
+LABEL software.name="BCFtools"
+LABEL software.version=${VERSION}
+LABEL software.description="Utilities for variant calling and manipulating files in the Variant Call Format (VCF) and its binary counterpart BCF"
+LABEL software.website="http://www.htslib.org"
+LABEL software.documentation="http://www.htslib.org/doc/bcftools.html"
+LABEL software.license="MIT/Expat"
+LABEL tags="Genomics"
+
+# System and library dependencies
+RUN apt-get -y update && \
+    apt-get -y install \
+      autoconf \
+      automake \
+      make \
+      gcc \
+      perl \
+      zlib1g-dev \
+      libbz2-dev \
+      liblzma-dev \
+      libcurl4-gnutls-dev \
+      libssl-dev \
+      libncurses5-dev \
+      wget && \
+    apt-get clean
+
+# Application installation
+RUN wget -O /bcftools-${VERSION}.tar.bz2 \
+  https://github.com/samtools/bcftools/releases/download/${VERSION}/bcftools-${VERSION}.tar.bz2 && \
+  tar xvjf /bcftools-${VERSION}.tar.bz2 && rm /bcftools-${VERSION}.tar.bz2
+
+WORKDIR /bcftools-${VERSION}
+RUN ./configure && make
+
+FROM ubuntu:18.04 AS final
+COPY --from=build /bcftools-*/bcftools /usr/local/bin
+
+RUN apt-get -y update && \
+    apt-get -y install \
+      libcurl3-gnutls && \
+    apt-get clean
+
+ENTRYPOINT ["bcftools"]