Merge pull request #6 from NarrativeScience/QPT-31289/py2sfn-deploy-fix

QPT-31289/py2sfn-deploy-fix

Author: Boylancl

README.md

@@ -6,7 +6,7 @@ Manage workflow concurrency and job state using an external store.
 
 This orb allows you to:
 
-- Limit the number of concurrently running workflows. This is useful when you want to only deploy one batch of changes at a time or use AWS CloudFormation and need to wait for the previous deploy to finish.
+- Limit the number of concurrently running workflows. This is useful when you want to only allow one batch of changes at a time or use AWS CloudFormation and need to wait for the previous deploy to finish.
 - Squash commits that are deployed in a workflow when the workflow is allowed to proceed
 - Store and retreive data from a key-value store in jobs, even if they're run in parallel
 - Track the status of a workflow from the command line
@@ -98,7 +98,7 @@ username | string | GitHub username of commit author
 workflow_id | string | Workflow instance ID
 status | string | Localized secondary index. One of `QUEUED`, `RUNNING`, `SUCCESS`, `FAILED`. Starts out as `QUEUED`.
 
-The job then starts polling the table for the oldest item that doesn't have a status attribute of `SUCCESS` or `FAILED`. If that item has the same `workflow_id` as the job, that means the job is at the "front" of the queue and can continue; we set the item's status to `RUNNING` and the workflow transitions to the next job. It will wait in the queue for up to 4 hours before failing. When the workflow is allowed to continue, it can be said to have a "lock" on the deploy.
+The job then starts polling the table for the oldest item that doesn't have a status attribute of `SUCCESS` or `FAILED`. If that item has the same `workflow_id` as the job, that means the job is at the "front" of the queue and can continue; we set the item's status to `RUNNING` and the workflow transitions to the next job. It will wait in the queue for up to 4 hours before failing. When the workflow is allowed to continue, it can be said to have a "lock".
 
 ### Exiting the Queue: `exit-queue`
 
@@ -134,9 +134,9 @@ The deploy worklow has the capability to "squash commits", which replicates a fe
 
 __Why?__ The primary goal of this is to reduce the time it takes for developers to get their merged code into production. A secondary goal is to reduce the cost of a bunch of containers burning CircleCI credits while trying to acquire a lock on the workflow.
 
-__How does it work?__ When you merge, your commit [enters the queue](#entering-the-queue). However, if someone else merges and your commit is still waiting to proceed, it will detect that it's no longer the HEAD commit of the master branch and self-cancel. Since we build and deploy everything during the workflow, your changes will be included when that later commit starts running. Once a workflow passes the `wait-in-queue` job, it is considered to be in the "running" state and will not be squashed.
+__How does it work?__ When you merge, your commit [enters the queue](#entering-the-queue). However, if someone else merges and your commit is still waiting to proceed, it will detect that it's no longer last in the queue and self-cancel. Since we build and deploy everything during the workflow, your changes will be included when that later commit starts running. Once a workflow passes the `wait-in-queue` job, it is considered to be in the "running" state and will not be squashed.
 
-__What if I don't want my commit squashed?__ If you don't want this behavior for whatever reason, you can include the tag `[force deploy]` in your merged commit message.
+__What if I don't want my commit squashed?__ There are known cases in which a commit should not be squashed, in these cases add `[force deploy]` in your merged commit message. This is most common when a commit has database migrations included or the workflow has specific conditions that require commits to be executed without being squashed.
 
 __Note:__ As of now, if there's a failure in the deploy workflow, the Slack message sent to the channel will only include the author of that commit, i.e. it won't contain the list of authors of commits that were squashed. We can see how it plays out before deciding if this behavior should be added.
 

src/commands/wait-in-queue.yml

@@ -65,7 +65,7 @@ steps:
         mkdir -p /tmp/workspace
         echo "$WORKFLOW_KEY" > /tmp/workspace/workflow-key.json
 
-        PREV_COMMIT="$(git rev-parse HEAD^)"
+        PREV_COMMIT="$(git rev-parse $CIRCLE_SHA1^)"
 
         # Create a temporary file containing the new item value (JSON)
         ITEM="$(mktemp)"
@@ -87,6 +87,7 @@ steps:
         aws dynamodb put-item \
           --table-name "$DYNAMODB_TABLE_WORKFLOWS" \
           --item "file://$ITEM"
+        echo "Added commit ($CIRCLE_SHA1) to the queue"
 
         # Query the table to find the workflow with the single oldest committed_at
         # timestamp and with a status of running or queued
@@ -99,8 +100,7 @@ steps:
             --key-condition-expression '#key = :key' \
             --filter-expression '#status IN (:running, :queued)' \
             --expression-attribute-names '{"#key": "key", "#status": "status"}' \
-            --expression-attribute-values "$values" \
-            --max-items 1
+            --expression-attribute-values "$values"
         }
 
         function workflowForCommitExists() {
@@ -147,6 +147,7 @@ steps:
             # in the correct order.
             if [[ -n "$CHECK_PREVIOUS_COMMIT" ]]; then
               if ! workflowForCommitExists "$PREV_COMMIT"; then
+                echo "The previous commit: $PREV_COMMIT was not found! Will wait for $PREV_COMMIT to deploy"
                 exit 1
               fi
             fi
@@ -168,6 +169,34 @@ steps:
           )
         }
 
+        # Check if this workflow is at the end of the queue
+        function isWorkflowEndOfQueue() {
+          (
+            # Check to see if this commit is at the end of the queue or if another
+            # commit is pending that will deploy these changes.
+
+            local result="$(queryItems)"
+            if [[ $? -ne 0 ]]; then
+              echo "Failed to query for workflow items"
+              exit 1
+            fi
+
+            # Get the last workflow_id in the queue
+            local workflow_id="$(echo "$result" | jq -r .Items[-1].workflow_id.S -)"
+            local count="$(echo "$result" | jq -r .Count -)"
+            echo "There are $count items (including this one) currently in the queue"
+
+            if [[ "$workflow_id" == "$CIRCLE_WORKFLOW_ID" || "$count" == 0 ]]; then
+              echo "This workflow ($CIRCLE_WORKFLOW_ID) is the last in the queue."
+              exit 0
+            else
+              echo "This workflow ($CIRCLE_WORKFLOW_ID) is not at the end of the queue. Last Commit in the Queue is:"
+              echo "$(echo "$result" | jq .Items[-1] -)"
+              exit 1
+            fi
+          )
+        }
+
         # Check if the commit is still the HEAD of the branch
         function isCommitHeadOfBranch() {
           (
@@ -191,6 +220,8 @@ steps:
         shopt -s nocasematch
         if [[ "$MESSAGE" == *'<< parameters.do_not_cancel_workflow_if_tag_in_commit >>'* ]]; then
           SKIP_COMMIT_ALLOWED=0
+          echo "Skip is disabled" 
+          echo "This Commit ($CIRCLE_SHA1) will not self-cancel and run until it acquires the lock or times out"
         fi
         shopt -u nocasematch
         <</ parameters.do_not_cancel_workflow_if_tag_in_commit >>
@@ -223,18 +254,30 @@ steps:
             echo "Attempt: $n of $MAX_ATTEMPTS"
 
             if [[ "$SKIP_COMMIT_ALLOWED" == 1 ]]; then
-              if isCommitHeadOfBranch; then
-                echo "Commit $CIRCLE_SHA1 is still HEAD of the $CIRCLE_BRANCH branch"
+              # If this is not the last commit, then cancel
+              if isWorkflowEndOfQueue; then
+                echo "This commit ($CIRCLE_SHA1) is last in the queue; Waiting to acquire the lock"
               else
                 echo 'export CANCEL_JOB=1' >> $BASH_ENV
                 echo 'export WORKFLOW_LOCK_BUILD_STATUS=CANCELLED' >> $BASH_ENV
-                echo "Commit $CIRCLE_SHA1 is no longer HEAD of the $CIRCLE_BRANCH branch and will be skipped"
+                echo "A newer commit has been added to the queue and is expected to contain these changes"
+                echo "This workflow ($CIRCLE_WORKFLOW_ID) will self-cancel and the commit ($CIRCLE_SHA1) will be squashed into the next"
                 exit 0
               fi
             fi
 
             if isWorkflowFrontOfQueue; then
               updateWorkflowStatus
+              echo "This commit ($CIRCLE_SHA1) has acquired the lock"
+              
+              # Check if this is the head of the branch just to notify the user
+              if isCommitHeadOfBranch; then
+                echo "$CIRCLE_SHA1 is the head of $CIRCLE_BRANCH"
+              else
+                echo "$CIRCLE_SHA1 is NOT the head of $CIRCLE_BRANCH."
+                echo "This is okay and could be the case if squashing is disabled or your pipeline filters out some commits."
+              fi
+
               exit 0
             fi