Add enter_maintenance and exit_maintenance roles

These roles can be used to place hosts into and remove them from
maintenance.

Co-Authored-By: Jack Hodgkiss <[email protected]>

Author: markgoddard

README.md

@@ -15,6 +15,8 @@ Roles:
 * [commands](roles/commands/README.md) for running arbitrary commands
 * [crush_rules](roles/crush_rules/README.md) for defining CRUSH rules
 * [ec_profiles](roles/ec_profiles/README.md) for defining EC profiles
+* [enter_maintenance](roles/enter_maintenance/README.md) for placing hosts into maintenance
+* [exit_maintenance](roles/exit_maintenance/README.md) for removing hosts from maintenance
 * [keys](roles/keys/README.md) for defining auth keys
 * [pools](roles/pools/README.md) for defining pools
 

roles/enter_maintenance/README.md

@@ -0,0 +1,21 @@
+# enter_maintenance
+
+This role places Ceph hosts into maintenance mode using `cephadm`.
+
+## Prerequisites
+
+This role should be executed on one host at a time. This can be achieved by
+adding `serial: 1` to a play.
+
+### Host prerequisites
+
+* The role assumes target hosts connection over SSH with user that has passwordless sudo configured.
+* Either direct Internet access or private registry with desired Ceph image accessible to all hosts is required.
+
+### Inventory
+
+This role assumes the existence of the following groups:
+
+* `mons`
+
+with at least one host in it - see the `cephadm` role for more details.

roles/enter_maintenance/tasks/enter.yml

@@ -0,0 +1,39 @@
+---
+- name: Check if host can enter maintenance mode
+  ansible.builtin.import_role:
+    name: stackhpc.cephadm.commands
+  vars:
+    cephadm_commands:
+      - "orch host ok-to-stop {{ ansible_facts.nodename }}"
+
+# Annoyingly, 'ceph orch host ok-to-stop' does not exit non-zero when
+# it is not OK to stop, so we need to check for specific messages.
+- name: Assert that it is safe to stop host
+  ansible.builtin.assert:
+    that:
+      # This one is seen for monitors
+      - "'It is NOT safe' not in cephadm_commands_result.results[0].stderr"
+      # This one is seen for OSDs
+      - "'unsafe to stop' not in cephadm_commands_result.results[0].stderr"
+    fail_msg: "{{ cephadm_commands_result.results[0].stderr }}"
+
+- name: Fail over Ceph manager
+  when: '"Cannot stop active Mgr daemon" in cephadm_commands_result.results[0].stderr'
+  block:
+    - name: Extract full name of active Ceph manager
+      ansible.builtin.set_fact:
+        active_ceph_mgr: "{{ cephadm_commands_result.results[0].stderr | split | last | replace(\"'\", '') }}"
+
+    - name: Ensure active manager has been switched to another node
+      ansible.builtin.import_role:
+        name: stackhpc.cephadm.commands
+      vars:
+        cephadm_commands:
+          - "mgr fail {{ active_ceph_mgr }}"
+
+- name: Ensure host is in maintenance mode
+  ansible.builtin.import_role:
+    name: stackhpc.cephadm.commands
+  vars:
+    cephadm_commands:
+      - "orch host maintenance enter {{ ansible_facts.nodename }}"

roles/enter_maintenance/tasks/main.yml

@@ -0,0 +1,26 @@
+---
+- name: Assert that execution is serialised
+  ansible.builtin.assert:
+    that:
+      - ansible_play_batch | length == 1
+    fail_msg: >-
+      Hosts must be placed into maintenance one at a time in order to first check
+      whether it is safe to stop them.
+
+- name: List hosts in maintenance
+  ansible.builtin.import_role:
+    name: stackhpc.cephadm.commands
+  vars:
+    cephadm_commands:
+      - "orch host ls --format json-pretty --host_status maintenance"
+
+# Entering maintenance fails if the host is already in maintenance.
+- name: Enter maintenance
+  ansible.builtin.include_tasks: enter.yml
+  when: ansible_facts.nodename not in cephadm_hosts_in_maintenance
+  vars:
+    cephadm_hosts_in_maintenance: >-
+      {{ cephadm_commands_result.results[0].stdout |
+         from_json |
+         map(attribute='hostname') |
+         list }}

roles/exit_maintenance/README.md

@@ -0,0 +1,21 @@
+# exit_maintenance
+
+This role removes Ceph hosts from maintenance mode using `cephadm`.
+
+## Prerequisites
+
+This role should be executed on one host at a time. This can be achieved by
+adding `serial: 1` to a play.
+
+### Host prerequisites
+
+* The role assumes target hosts connection over SSH with user that has passwordless sudo configured.
+* Either direct Internet access or private registry with desired Ceph image accessible to all hosts is required.
+
+### Inventory
+
+This role assumes the existence of the following groups:
+
+* `mons`
+
+with at least one host in it - see the `cephadm` role for more details.

roles/exit_maintenance/tasks/exit.yml

@@ -0,0 +1,7 @@
+---
+- name: Ensure host has exited maintenance mode
+  ansible.builtin.import_role:
+    name: stackhpc.cephadm.commands
+  vars:
+    cephadm_commands:
+      - "orch host maintenance exit {{ ansible_facts.nodename }}"

roles/exit_maintenance/tasks/main.yml

@@ -0,0 +1,25 @@
+---
+- name: Assert that execution is serialised
+  ansible.builtin.assert:
+    that:
+      - ansible_play_batch | length == 1
+    fail_msg: >-
+      Hosts must be removed from maintenance one at a time.
+
+- name: List hosts
+  ansible.builtin.import_role:
+    name: stackhpc.cephadm.commands
+  vars:
+    cephadm_commands:
+      - "orch host ls --format json-pretty"
+
+# Exiting maintenance fails if the host is not in maintenance or offline.
+- name: Exit maintenance
+  ansible.builtin.include_tasks: exit.yml
+  when: cephadm_host_status.status | lower in ["maintenance", "offline"]
+  vars:
+    cephadm_host_status: >-
+      {{ cephadm_commands_result.results[0].stdout |
+         from_json |
+         selectattr('hostname', 'equalto', ansible_facts.nodename) |
+         first }}