updated documentation (#143)

Author: ErinWeisbart

documentation/DCP-documentation/_toc.yml

@@ -1,8 +1,12 @@
 # Table of contents
 
 format: jb-book
-root: overview
 parts:
+- caption: FAQ
+  chapters:
+  - file: overview
+  - file: overview_2
+  - file: costs
 - caption: Running DCP
   chapters:
   - file: step_0_prep
@@ -15,7 +19,8 @@ parts:
   - file: step_4_monitor
 - caption:
   chapters:
+  - file: dashboard
   - file: advanced_configuration
   - file: AWS_hygiene_scripts
   - file: troubleshooting_runs
-  - file: versions
+  - file: versions

documentation/DCP-documentation/costs.md

@@ -0,0 +1,36 @@
+# What does Distributed-CellProfiler cost?
+
+Distributed-CellProfiler is run by a series of three commands, only one of which incurs costs at typical scale of usage:
+
+[`setup`](step_1_configuration.md) creates a queue in SQS and a cluster, service, and task definition in ECS. 
+ECS is entirely free. 
+SQS queues are free to create and use up to 1 million requests/month.
+
+[`submitJobs`](step_2_submit_jobs.md) places messages in the SQS queue which is free (under 1 million requests/month).
+
+[`startCluster`](step_3_start_cluster.md) is the only command that incurs costs with initiation of your spot fleet request, creating machine alarms, and optionally creating a run dashboard. 
+
+The spot fleet is the major cost of running Distributed-CellProfiler, exact pricing of which depends on the number of machines, type of machines, and duration of use. 
+Your bid is configured in the [config file](step_1_configuration.md).
+
+Spot fleet costs can be minimized/stopped in multiple ways:
+1) We encourage the use of [`monitor`](step_4_monitor.md) during your job to help minimize the spot fleet cost as it automatically scales down your spot fleet request as your job queue empties and cancels your spot fleet request when you have no more jobs in the queue.
+Note that you can also perform a more aggressive downscaling of your fleet by monitor by engaging Cheapest mode (see [`more information here`](step_4_monitor.md)).
+2) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to perform the same cleanup (without the automatic scaling).
+3) If you want to abort and clean up a run, you can purge your SQS queue in the [AWS SQS console](https://console.aws.amazon.com/sqs/) (by selecting your queue and pressing Actions => Purge) and then initiate [`monitor`](step_4_monitor.md) to perform the same cleanup.
+4) You can stop the spot fleet request directly in the [AWS EC2 console](https://console.aws.amazon.com/ec2/) by going to Instances => Spot Requests, selecting your spot request, and pressing Actions => Cancel Spot Request.
+
+After the spot fleet has started, a Cloudwatch instance alarm is automatically placed on each instance in the fleet.
+Cloudwatch instance alarms [are currently $0.10/alarm/month](https://aws.amazon.com/cloudwatch/pricing/).
+Cloudwatch instance alarm costs can be minimized/stopped in multiple ways:
+1) If you run monitor during your job, it will automatically delete Cloudwatch alarms for any instance that is no longer in use once an hour while running and at the end of a run.
+2) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to delete Cloudwatch alarms for any instance that is no longer in use.
+3) In [AWS Cloudwatch console](https://console.aws.amazon.com/cloudwatch/) you can select unused alarms by going to Alarms => All alarms. Change Any State to Insufficient Data, select all alarms, and then Actions => Delete.
+4) We provide a [hygiene script](hygiene.md) that will clean up old alarms for you.
+
+Cloudwatch Dashboards [are currently free](https://aws.amazon.com/cloudwatch/pricing/) for 3 Dashboards with up to 50 metrics per month and are $3 per dashboard per month after that. 
+Cloudwatch Dashboard costs can be minimized/prevented in multiple ways:
+1) You can choose not to have Distributed-CellProfiler create a Dashboard by setting `CREATE_DASHBOARD = 'False'` in your [config file](step_1_configuration.md).
+2) We encourage the use of [`monitor`](step_4_monitor.md) during your job as if you have set `CLEAN_DASHBOARD = 'True'` in your [config file](step_1_configuration.md) it will automatically delete your Dashboard when your job is done.
+3) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to perform the same cleanup (without the automatic scaling).
+4) You can manually delete Dashboards in the [Cloudwatch Console]((https://console.aws.amazon.com/cloudwatch/)) by going to Dashboards, selecting your Dashboard, and selecting Delete.

documentation/DCP-documentation/dashboard.md

@@ -0,0 +1,99 @@
+# AWS Cloudwatch Dashboard
+![Cloudwatch Dashboard Overview](images/dashboard_overview.png)
+
+AWS Cloudwatch Dashboards are “customizable home pages in the CloudWatch console that you can use to monitor your resources in a single view and create customized views of the metrics and alarms for your AWS resources.” 
+A Dashboard is full of widgets, each of which you create and customize to report on a separate AWS metric. 
+Distributed-CellProfiler has the option to auto-create a Cloudwatch Dashboard for each run and the option to clean it up when you are done.
+These options are set in [your config file](step_1_configuration.md).
+
+The Dashboard setup that DS auto-populates is helfpul for monitoring a run as it is occurring or for a post-mortem to better understand a previous run. 
+Some things you can see include: whether your machines are sized appropriately for your jobs, how stable your spot fleet is, whether your jobs are failing and if so if they’re failing in a consistent manner. 
+All told, this can help you understand and optimize your resource usage, thus saving you time and money
+
+## FulfilledCapacity:
+![Fulfilled Capacity widget](images/fulfilledcapacity.png)
+
+This widget shows the number of machines in your spot fleet that are fulfilled, i.e. how many machines you actually have at any given point. 
+After a short spin-up time after initiating a run, you hope to see a straight line at the number of machines requested in your fleet and then a steady decrease at the end of a run as monitor scales your fleet down to match the remaining jobs. 
+
+Some number of small dips are all but inevitable as machines crash and are replaced or AWS takes some of your capacity and gives it to a higher bidder. 
+However, every time there is a dip, it means that a machine that was running a job is no longer running it and any progress on that job is lost. 
+The job will hang out as “Not Visible” in your SQS queue until it reaches the amount of time set by SQS_MESSAGE_VISIBILITY in [your config file](step_1_configuration.md). 
+For quick jobs, this doesn’t have much of an impact, but for jobs that take many hours, this can be frustrating and potentially expensive.
+
+If you’re seeing lots of dips or very large dips, you may be able to prevent this in future runs by 1) requesting a different machine type 2) bidding a larger amount for your machines 3) changing regions. 
+You can also check if blips coincide with AWS outages, in which case there’s nothing you can do, it’s just bad luck (that’s what happened with the large dip in the example above).
+
+## NumberOfMessagesReceived/Deleted
+
+![NumberofMessagesReceived/Deleted](images/messages_deleted_received.png)
+
+This widget shows you in bulk whether your jobs are completing or erroring. 
+NumberOfMessagesDeleted shows messages deleted from the queue after the job has successfully completed. 
+NumberOfMessagesReceived shows both messages that are deleted from the queue as well as messages that are put back in the queue because they errored. 
+You hope to see that the two lines track on top of each other because that means no messages are erroring. 
+If there are often gaps between the lines then it means a fraction of your jobs are erroring and you’ll need to figure out why (see MemoryUtilization and Show Errors or look directly in your Cloudwatch Logs for insights).
+
+## MemoryUtilization
+
+![Memory Utilization](images/memoryutilization.png)
+
+Insufficient memory is the error that we most often encounter (as we try to use the smallest machines possible for economy’s sake) so we like to look at memory usage. 
+Note that this is showing memory utilization in bulk for your cluster, not for individual machines. 
+Because different machines reach memory intensive steps at different points in time, and because we’re looking at an average across 5 minute windows, the max percentage you see is likely to be much less than 100%, even if you are using all the memory in your machines at some points.
+
+# MessagesVisible/NotVisible
+
+![MessagesVisible/NotVisible](images/messages_change_slope.png)
+
+Visible messages are messages waiting in your queue. 
+Hidden messages (aka MessagesNotVisible) have been started and will remain hidden until either they are completed and therefore removed from the queue or they reach the time set in SQS_MESSAGE_VISIBILITY in your config file, whichever comes first. 
+([Read more about Message Visibility](SQS_QUEUE_information.md).) 
+After starting your fleet (and waiting long enough for at least one round of jobs to complete), you hope to see a linear decline in total messages with the number of hidden messages equal to the number of jobs being run (fleet size * tasks per machine * docker cores).
+
+![Blip in MessagesVisible/NotVisible](images/blip_in_messagesnotvisible.png)
+
+Sometimes you’ll see a blip where there is a rapid increase in the number of hidden messages (as pictured above). 
+This can happen if there is an error on a machine and the hard disk gets full - it rapidly pulls jobs and puts them back until the machine error is caught and rebooted. 
+This type of error shows in this widget as it happens.
+
+If your spot fleet loses capacity (see FulfilledCapacity), you may see a blip in MessagesVisible/NotVisible where the number of hidden messages rapidly decreases. 
+This appears in the widget the amount of time set in SQS_MESSAGE_VISIBILITY in your config file after the capacity loss when jobs that were started (i.e. hidden) but not completed return to visible status.
+
+The relative slope of your graph can also be informative. 
+For the run pictured at top, we discovered that a fraction of our jobs were erroring because the machines were running out of memory. 
+Midway through 7/12 we upped the memory of the machines in our fleet and you can see from that point on a greater slope as more jobs were finishing in the same amount of time (as fewer were failing to complete because of memory errors.)
+
+## Distinct Logs
+
+![Logs comparison](images/logs_comparison.png)
+
+This widget shows you the number of different specific jobs that start within your given time window by plotting the number of Cloudwatch logs that have your run command in them. 
+In this example, our run command is "cellprofiler -c".
+It is not necessarily informative on its own, but very helpful when compared with the following widget.
+
+## All logs
+This widget shows you the number of total times that jobs are started within your log group within the given time window. 
+Ideally, you want this number to match the number in the previous widget as it means that each job is starting in your software only once. 
+
+If this number is consistently larger than the previous widget’s number, it could mean that some of your jobs are erroring and you’ll need to figure out why (see MemoryUtilization and Show Errors or look directly in your Cloudwatch Logs for insights).
+
+## Show Errors
+![Show errors](images/expand_error_log.png)
+
+This widget shows you the log entry any time that it contains “Error”. 
+Ideally, this widget will remain empty. 
+If it is logging errors, you can toggle each row for more information - it will show the job that errored in @logStream and the actual error message in @message. 
+
+## Interacting with a Dashboard:
+
+Once you have your Dashboard created and full of widgets, you can adjust the timescale for which the widget is reporting metrics. 
+For any of the widgets you can set the absolute or relative time that the widget is showing by selecting the time scale from the upper right corner of the screen. 
+Zoom in to a particular time selection on a visible widget by drawing a box around that time on the widget itself (note that zooming in doesn’t change what’s plotted, just what part of the plot you can see so metrics like Show Errors won’t update with a zoom).
+
+Some widgets allow you to select/deselect certain metrics plotted in the widget. 
+To hide a metric without permanently removing it from the widget, simply click the X on the box next to the name of the metric in the legend.
+
+You can move the widgets around on your dashboard by hovering on the upper right or upper left corner of a widget until a 4-direction-arrow icon appears and then dragging and dropping the widget. 
+You can change the size of a widget by hovering on the lower right corner of the widget until a diagonal arrow icon appears and then dragging the widget to the desired size. 
+After making changes, make sure to select Save dashboard from the top menu so that they are maintained after refreshing the page.