Merge pull request #64 from stackhpc/multinode.sh

Refactor deploy-openstack.sh into multinode.sh

Author: markgoddard

README.rst

@@ -25,6 +25,64 @@ This configuration is typically used with the `ci-multinode` environment in the
 <https://stackhpc-kayobe-config.readthedocs.io/en/stackhpc-2023.1/contributor/environments/ci-multinode.html>`__
 repository.
 
+What's in the box?
+==================
+
+This repository contains various items.
+
+Scripts
+-------
+
+* ``scripts/deploy.sh`` - end-to-end cluster deployment and testing.
+* ``scripts/tear-down.sh`` - tear down test cluster infrastructure.
+
+Terraform
+---------
+
+Terraform configuration deploys test cluster infrastructure on an OpenStack
+cloud. It provides outputs that can be used to populate Kayobe Configuration
+with the details of the test infrastructure.
+
+Ansible
+-------
+
+Ansible playbooks in the ``ansible/`` directory are provided to prepare and use
+the Ansible control host.
+
+#. ``configure-hosts.yml`` - sequentially executes 3 other playbooks:
+
+   #. ``wait-control-host.yml`` - Waits for the Ansible control host to become
+      accessible and ready for deployment. Tag: ``wait``
+   #. ``grow-control-host.yml`` - Applies LVM configuration to the control host
+      to ensure it has enough space to continue with the rest of the
+      deployment. Tag: ``lvm``
+   #. ``deploy-openstack-config.yml`` - Prepares the Ansible control host as a
+      Kayobe control host, cloning the Kayobe configuration and installing
+      virtual environments. Tag: ``deploy``
+
+   These playbooks are tagged so that they can be invoked or skipped using
+   `tags` or `--skip-tags` as required.
+
+#. ``deploy-openstack.yml`` - runs the ``multinode.sh deploy_full`` command in
+   a `tmux` session on the Ansible control host. The session is logged to
+   ``~/tmux.kayobe\:0.log`` on the Ansible control host.  Use ``less -r
+   ~/tmux.kayobe\:0.log`` to view the logs in their original colourful glory.
+
+#. ``fetch-logs.yml`` - fetches logs, diagnostics and tests results from the
+   cluster to the host runnin the playbook..
+
+Configuration variables for these playbooks are in
+``ansible/vars/defaults.yml``.
+
+multinode.sh
+------------
+
+The ``multinode.sh`` script is installed in ``/usr/local/bin/`` on the Ansible
+control host. It provides high-level automation of various aspects of OpenStack
+deployment, operations and testing. It accepts a single argument which is the
+command to perform. The supported commands may be listed by running it without
+arguments.
+
 Prerequisites
 =============
 
@@ -63,7 +121,7 @@ Initialise Terraform:
    terraform init
 
 Generate an SSH keypair. Note that `ED25519 keys are not currently supported by RHEL
-<https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/securing_networks/index#making-openssh-more-secure_assembly_using-secure-communications-between-two-systems-with-openssh>`__ 
+<https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/securing_networks/index#making-openssh-more-secure_assembly_using-secure-communications-between-two-systems-with-openssh>`__
 when using the FIPS security standard (as enabled by the CIS benchmark hardening
 scripts in kayobe-config). The public key will be registered in OpenStack as a
 keypair and authorised by the instances deployed by Terraform. The private and
@@ -187,7 +245,7 @@ access to the control host.
 Configure Ansible variables
 ===========================
 
-Review the vars defined within `ansible/vars/defaults.yml`. In here you can customise the version of kayobe, kayobe-config or openstack-config. 
+Review the vars defined within `ansible/vars/defaults.yml`. In here you can customise the version of kayobe, kayobe-config or openstack-config.
 Make sure to define `ssh_key_path` to point to the location of the SSH key in use by the nodes and also `vxlan_vni` which should be unique value between 1 to 100,000.
 VNI should be much smaller than the officially supported limit of 16,777,215 as we encounter errors when attempting to bring interfaces up that use a high VNI.
 You must set `vault_password_path`; this should be set to the path to a file containing the Ansible vault password.
@@ -201,7 +259,7 @@ Tempest testing without user interaction. Any errors encountered will be
 reported and halt the deployment.
 
 This script makes use of the `ansible/deploy-openstack.yml` Ansible playbook
-that runs the `deploy-openstack.sh` script in a `tmux` session on the Ansible
+that runs the `multinode.sh deploy_full` command in a `tmux` session on the Ansible
 control host. The session is logged to `~/tmux.kayobe\:0.log` on the Ansible
 control host. Use `less -r ~/tmux.kayobe\:0.log` to view the logs in their
 original colourful glory.
@@ -255,30 +313,24 @@ Run the configure-hosts.yml playbook to configure the Ansible control host.
 
    ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml
 
-This playbook sequentially executes 2 other playbooks:
-
-#. ``grow-control-host.yml`` - Applies LVM configuration to the control host to ensure it has enough space to continue with the rest of the deployment. Tag: ``lvm`` 
-#. ``deploy-openstack-config.yml`` - Prepares the Ansible control host as a Kayobe control host, cloning the Kayobe configuration and installing virtual environments. Tag: ``deploy``
-
-These playbooks are tagged so that they can be invoked or skipped using `tags` or `--skip-tags` as required.
-
 Deploy OpenStack
 ----------------
 
 Once the Ansible control host has been configured with a Kayobe/OpenStack configuration you can then begin the process of deploying OpenStack.
-This can be achieved by either manually running the various commands to configure the hosts and deploy the services or automated by using the generated `deploy-openstack.sh` script.
-`deploy-openstack.sh` should be available within the home directory on your Ansible control host provided you ran `deploy-openstack-config.yml` earlier.
+This can be achieved by either manually running the various commands to configure the hosts and deploy the services or automated by using the ``multinode.sh deploy_full`` command.
+``multinode.sh`` should be available within ``/usr/local/bin/`` on your Ansible control host provided you ran `deploy-openstack-config.yml` earlier.
 This script will go through the process of performing the following tasks:
 
    * kayobe control host bootstrap
    * kayobe seed host configure
    * kayobe overcloud host configure
    * cephadm deployment
+   * HashiCorp Vault deployment & certificate generation
    * kayobe overcloud service deploy
-   * openstack configuration
-   * tempest testing
+   * OpenStack configuration
+   * Tempest testing
 
-Tempest test results will be written to `~/tempest-artifacts`.
+Tempest test results will be written to ``~/tempest-artifacts``.
 
 If you choose to opt for the automated method you must first SSH into your Ansible control host.
 
@@ -292,11 +344,11 @@ Start a `tmux` session to avoid halting the deployment if you are disconnected.
 
    tmux
 
-Run the `deploy-openstack.sh` script.
+Run the `multinode.sh` script.
 
 .. code-block:: console
 
-   ~/deploy-openstack.sh
+   multinode.sh deploy_full
 
 Accessing OpenStack
 ===================
@@ -308,7 +360,7 @@ Using software such as sshuttle will allow for easy access.
 
    sshuttle -r $(terraform output -raw ssh_user)@$(terraform output -raw seed_access_ip_v4) 192.168.39.0/24
 
-You may also use sshuttle to proxy DNS via the multinode environment. Useful if you are working with Designate. 
+You may also use sshuttle to proxy DNS via the multinode environment. Useful if you are working with Designate.
 Important to node this will proxy all DNS requests from your machine to the first controller within the multinode environment.
 
 .. code-block:: console

ansible/deploy-openstack-config.yml

@@ -63,18 +63,57 @@
         state: present
       become: true
 
+    - name: Ensure uuid-runtime is present (Ubuntu)
+      ansible.builtin.package:
+        name: uuid-runtime
+        state: present
+      become: true
+      when: ansible_facts['distribution'] == "Ubuntu"
+
     - name: Ensure `source` directory exists
       ansible.builtin.file:
         path: "{{ src_directory }}"
         state: directory
         mode: "0755"
 
+    - name: Revert TLS changes to avoid git conflicts (upgrade)
+      ansible.builtin.shell:
+        cmd: >-
+          if [[ -f {{ kayobe_environment_path }}/kolla/globals-tls-config.yml ]]; then
+            sed -i 's/^kolla_enable_tls_internal: true/# kolla_enable_tls_internal: true/g' {{ kayobe_environment_path }}/kolla.yml
+          fi
+      when: upgrade | bool
+      vars:
+        kayobe_environment_path: "{{ src_directory }}/kayobe-config/etc/kayobe/environments/{{ kayobe_config_environment }}"
+
+    - name: Stash Kayobe Config changes (upgrade)
+      ansible.builtin.command:
+        cmd: git stash
+        chdir: "{{ src_directory }}/{{ kayobe_config_name }}"
+      when: upgrade | bool
+
     - name: Ensure Kayobe Config repository is present
       ansible.builtin.git:
         repo: "{{ kayobe_config_repo }}"
         version: "{{ kayobe_config_version }}"
         dest: "{{ src_directory }}/{{ kayobe_config_name }}"
-        update: false
+        update: "{{ upgrade | bool }}"
+
+    - name: Pop stashed Kayobe Config changes (upgrade)
+      ansible.builtin.command:
+        cmd: git stash pop
+        chdir: "{{ src_directory }}/{{ kayobe_config_name }}"
+      when: upgrade | bool
+
+    - name: Replace TLS changes to avoid git conflicts (upgrade)
+      ansible.builtin.shell:
+        cmd: >-
+          if [[ -f {{ kayobe_environment_path }}/kolla/globals-tls-config.yml ]]; then
+            sed -i 's/# kolla_enable_tls_internal: true/kolla_enable_tls_internal: true/g' {{ kayobe_environment_path }}/kolla.yml
+          fi
+      when: upgrade | bool
+      vars:
+        kayobe_environment_path: "{{ src_directory }}/kayobe-config/etc/kayobe/environments/{{ kayobe_config_environment }}"
 
     - name: Ensure hooks directory are present
       ansible.builtin.file:
@@ -103,7 +142,6 @@
         - { src: fix-networking.yml, dest: infra-vm-host-configure/pre.d/15-fix-networking.yml }
         - { src: configure-vxlan.yml, dest: infra-vm-host-configure/pre.d/20-configure-vxlan.yml }
 
-
     - name: Ensure Admin Overcloud Network file is present
       ansible.builtin.copy:
         src: "files/admin-oc-networks.yml"
@@ -116,6 +154,15 @@
         dest: "{{ src_directory }}/{{ kayobe_config_name }}/etc/kayobe/environments/{{ kayobe_config_environment }}/inventory/hosts"
         mode: "0644"
 
+    - name: Ensure multinode.sh script is present
+      ansible.builtin.copy:
+        src: "files/multinode.sh"
+        dest: "/usr/local/bin/multinode.sh"
+        mode: "0755"
+      become: true
+      tags:
+        - multinode.sh
+
     - name: Ensure root_domain is defined
       ansible.builtin.lineinfile:
         path: "{{ src_directory }}/{{ kayobe_config_name }}/etc/kayobe/environments/{{ kayobe_config_environment }}/inventory/group_vars/all/main.yml"
@@ -162,9 +209,15 @@
         repo: "{{ kayobe_repo }}"
         version: "{{ kayobe_version or 'stackhpc/' ~ openstack_release.stdout }}"
         dest: "{{ src_directory }}/{{ kayobe_name }}"
-        update: false
+        update: "{{ upgrade | bool }}"
       when: kayobe_src_required
 
+    - name: Ensure `venvs` directory is absent (upgrade)
+      ansible.builtin.file:
+        path: "{{ ansible_env.HOME }}/venvs"
+        state: absent
+      when: upgrade | bool
+
     - name: Ensure `venvs` directory exists
       ansible.builtin.file:
         path: "{{ ansible_env.HOME }}/venvs"
@@ -225,6 +278,21 @@
         dest: "{{ ansible_env.HOME }}/.ssh/id_rsa"
         mode: "0600"
 
+    - name: Ensure additional public keys are authorised
+      ansible.posix.authorized_key:
+        user: "{{ ansible_user }}"
+        key: "{{ item }}"
+      loop: "{{ extra_ssh_public_keys }}"
+
+    - name: "{% if upgrade | bool %}Upgrade{% else %}Bootstrap{% endif %} control host"
+      ansible.builtin.shell:
+        cmd: |-
+          set -eu
+          source {{ ansible_env.HOME }}/venvs/kayobe/bin/activate
+          source {{ src_directory }}/{{ kayobe_config_name }}/kayobe-env --environment {{ kayobe_config_environment }}
+          export KAYOBE_VAULT_PASSWORD="$(cat ~/vault.password)"
+          kayobe control host {% if upgrade | bool %}upgrade{% else %}bootstrap{% endif %}
+
     - name: Ensure OpenStack Config repository is present
       ansible.builtin.git:
         repo: "{{ openstack_config_repo }}"
@@ -272,13 +340,6 @@
         dest: "{{ src_directory }}/{{ openstack_config_name }}/ansible/inventory"
         mode: 0644
 
-    - name: Ensure `deploy-openstack.sh` script is present
-      ansible.builtin.copy:
-        src: "files/deploy-openstack.sh"
-        dest: "{{ ansible_env.HOME }}/deploy-openstack.sh"
-        mode: "755"
-      tags: deploy-openstack
-
     - name: Ensure docker repository is present
       ansible.builtin.command:
         cmd: dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

ansible/deploy-openstack.yml

@@ -3,6 +3,7 @@
   hosts: ansible_control
   gather_facts: false
   vars:
+    multinode_command: "deploy_full"
     # 6 hours should be enough...
     deployment_timeout_s: "{{ 6 * 60 * 60 }}"
     lock_path: /tmp/deploy-openstack.lock
@@ -30,7 +31,7 @@
         pipe-pane -t {{ tmux_session }} -o 'cat >> ~/tmux.#S:#P.log'
       when: session_check.rc != 0
 
-    # deploy-openstack.sh uses a "lock" directory to ensure that only one
+    # multinode.sh uses a "lock" directory to ensure that only one
     # instance can run concurrently.
     - name: Check that no deployment is in progress
       stat:
@@ -45,9 +46,9 @@
           directory and run this playbook again.
       when: lock_stat.stat.exists
 
-    - name: Run deploy-openstack.sh in tmux window
+    - name: "Run multinode.sh {{ multinode_command }} in tmux window"
       command: >-
-        tmux send -t {{ tmux_session }}.0 './deploy-openstack.sh' ENTER
+        tmux send -t {{ tmux_session }}.0 'multinode.sh {{ multinode_command }}' ENTER
 
     - name: Show how to follow deployment progress
       debug:
@@ -57,11 +58,11 @@
 
           {{ connection_info }}
 
-    - name: Wait for deploy-openstack.sh to start
+    - name: "Wait for multinode.sh {{ multinode_command }} to start"
       pause:
         seconds: 30
 
-    - name: Wait for deployment to complete
+    - name: "Wait for multinode.sh {{ multinode_command }} to complete"
       stat:
         path: "{{ lock_path }}"
       register: lock_stat
@@ -70,13 +71,13 @@
       delay: 10
       failed_when: false
 
-    # deploy-openstack.sh writes an exit code to a file. 0 is success
-    - name: Check deployment result
+    # multinode.sh writes an exit code to a file. 0 is success
+    - name: "Check multinode.sh {{ multinode_command }} result"
       slurp:
         path: "{{ rc_path }}"
       register: rc_slurp
 
-    - name: Fail if deployment was unsuccessful
+    - name: "Fail if multinode.sh {{ multinode_command }} was unsuccessful"
       fail:
         msg: |
           Deployment or testing of OpenStack was unsuccessful.