Merge pull request #47 from stackhpc/full-deploy

Add a script to perform a complete multi-node deployment

Author: markgoddard

README.rst

@@ -189,8 +189,45 @@ Make sure to define `ssh_key_path` to point to the location of the SSH key in us
 VNI should be much smaller than the officially supported limit of 16,777,215 as we encounter errors when attempting to bring interfaces up that use a high VNI.
 You must set `vault_password_path`; this should be set to the path to a file containing the Ansible vault password.
 
-Deployment
-==========
+Deployment: The fast(er) way
+============================
+
+The `scripts/deploy.sh` script provides a fully automated deployment method
+that can be used to perform all steps from infrastructure deployment through to
+Tempest testing without user interaction. Any errors encountered will be
+reported and halt the deployment.
+
+This script makes use of the `ansible/deploy-openstack.yml` Ansible playbook
+that runs the `deploy-openstack.sh` script in a `tmux` session on the Ansible
+control host. The session is logged to `~/tmux.kayobe\:0.log` on the Ansible
+control host. Use `less -r ~/tmux.kayobe\:0.log` to view the logs in their
+original colourful glory.
+
+Note that this approach requires all Terraform, Ansible, Kayobe and OpenStack
+configuration to be provided in advance. For Kayobe and OpenStack
+configuration, this may be achieved by providing suitable branches for the
+kayobe-config and openstack-config repositories and referencing them in
+`ansible/vars/defaults.yml`.
+
+To tear down the cluster immediately after a successful deployment, combine
+with the `scripts/tear-down.sh` script.
+
+.. code-block:: console
+
+   ./scripts/deploy.sh && ./scripts/tear-down.sh -a -k
+
+Note that this will not tear down the cluster if deployment fails, allowing for
+debugging the issue and/or retrying.
+
+Deployment: The slow(er) way
+============================
+
+This section describes a more hands-on, interactive deployment method. It may
+be useful to gain a better understanding of how the deployment works, modify
+the deployment process in some way, or iterate on configuration.
+
+Terraform Deploy infrastructure using Terraform
+-----------------------------------------------
 
 Generate a plan:
 
@@ -207,7 +244,7 @@ Apply the changes:
 You should have requested a number of resources to be spawned on Openstack.
 
 Configure Ansible control host
-==============================
+------------------------------
 
 Run the configure-hosts.yml playbook to configure the Ansible control host.
 
@@ -223,7 +260,7 @@ This playbook sequentially executes 2 other playbooks:
 These playbooks are tagged so that they can be invoked or skipped using `tags` or `--skip-tags` as required.
 
 Deploy OpenStack
-================
+----------------
 
 Once the Ansible control host has been configured with a Kayobe/OpenStack configuration you can then begin the process of deploying OpenStack.
 This can be achieved by either manually running the various commands to configure the hosts and deploy the services or automated by using the generated `deploy-openstack.sh` script.

ansible/deploy-openstack.yml

@@ -0,0 +1,86 @@
+---
+- name: Deploy OpenStack
+  hosts: ansible_control
+  gather_facts: false
+  vars:
+    # 6 hours should be enough...
+    deployment_timeout_s: "{{ 6 * 60 * 60 }}"
+    lock_path: /tmp/deploy-openstack.lock
+    rc_path: /tmp/deploy-openstack.rc
+    tmux_session: kayobe
+    tmux_log_path: "~/tmux.{{ tmux_session }}:0.log"
+    connection_info: |
+      # SSH to Ansible control host
+      ssh {{ ansible_user }}@{{ ansible_host }}
+      # Either: Attach to the tmux session
+      tmux -t {{ tmux_session }} attach
+      # Or: Follow the log file
+      less -r {{ tmux_log_path }}
+  vars_files:
+    - vars/defaults.yml
+  tasks:
+    - name: Check if tmux session exists
+      command: tmux has-session -t {{ tmux_session }}
+      failed_when: false
+      register: session_check
+
+    - name: Create a new tmux window and log to a file
+      command: >-
+        tmux new -d -s {{ tmux_session }}\;
+        pipe-pane -t {{ tmux_session }} -o 'cat >> ~/tmux.#S:#P.log'
+      when: session_check.rc != 0
+
+    # deploy-openstack.sh uses a "lock" directory to ensure that only one
+    # instance can run concurrently.
+    - name: Check that no deployment is in progress
+      stat:
+        path: "{{ lock_path }}"
+      register: lock_stat
+
+    - name: Fail if a deployment is in progress
+      fail:
+        msg: |
+          Refusing to deploy because a deployment is currently in progress.
+          If you are sure this is not the case, remove the {{ lock_path }}
+          directory and run this playbook again.
+      when: lock_stat.stat.exists
+
+    - name: Run deploy-openstack.sh in tmux window
+      command: >-
+        tmux send -t {{ tmux_session }}.0 './deploy-openstack.sh' ENTER
+
+    - name: Show how to follow deployment progress
+      debug:
+        msg: |
+          Deployment of OpenStack has started.
+          To follow progress:
+
+          {{ connection_info }}
+
+    - name: Wait for deploy-openstack.sh to start
+      pause:
+        seconds: 30
+
+    - name: Wait for deployment to complete
+      stat:
+        path: "{{ lock_path }}"
+      register: lock_stat
+      until: not lock_stat.stat.exists
+      retries: "{{ (deployment_timeout_s | int / 10) | int }}"
+      delay: 10
+      failed_when: false
+
+    # deploy-openstack.sh writes an exit code to a file. 0 is success
+    - name: Check deployment result
+      slurp:
+        path: "{{ rc_path }}"
+      register: rc_slurp
+
+    - name: Fail if deployment was unsuccessful
+      fail:
+        msg: |
+          Deployment or testing of OpenStack was unsuccessful.
+          To see results:
+
+          {{ connection_info }}
+      when: rc_slurp.content | b64decode != "0"

scripts/deploy.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# This script performs a complete multi-node cluster deployment.
+
+set -eux
+set -o pipefail
+
+# Check for dependencies
+if ! type terraform; then
+  echo "Unable to find Terraform"
+  exit 1
+fi
+if ! type ansible; then
+  echo "Unable to find Ansible"
+  exit 1
+fi
+
+# Deploy infrastructure using Terraform.
+terraform plan
+terraform apply -auto-approve
+
+# Configure the Ansible control host.
+ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml
+
+# Deploy OpenStack.
+ansible-playbook -i ansible/inventory.yml ansible/deploy-openstack.yml

templates/deploy-openstack.tpl

@@ -24,6 +24,8 @@ declare -A config_directories=(
 )
 
 tempest_dir="$HOME/tempest-artifacts"
+lock_path=/tmp/deploy-openstack.lock
+rc_path=/tmp/deploy-openstack.rc
 
 function activate_virt_env () {
   set +u
@@ -37,6 +39,18 @@ function activate_kayobe_env () {
   set -u
 }
 
+# Use a "lock" directory to ensure that only one instance of this script can run concurrently.
+if mkdir "$lock_path"; then
+  trap "rmdir $lock_path" EXIT
+else
+  echo "Refusing to deploy because a deployment is currently in progress."
+  echo "If you are sure this is not the case, remove the $lock_path directory and run this script again."
+  exit 1
+fi
+
+# Write an exit code to a file to allow Ansible to report the result.
+echo 1 >$rc_path
+
 activate_virt_env "kayobe"
 activate_kayobe_env
 
@@ -174,3 +188,6 @@ if [[ $(wc -l < $tempest_dir/failed-tests) -ne 0 ]]; then
 fi
 
 echo "Tempest testing successful"
+
+# Report success.
+echo 0 >$rc_path