Merge branch 'stackhpc/yoga' into DWPD

Author: dougszumski

doc/source/configuration/index.rst

@@ -18,3 +18,4 @@ the various features provided.
    wazuh
    vault
    magnum-capi
+   security-hardening

doc/source/configuration/release-train.rst

@@ -107,6 +107,23 @@ apt repositories. This can be done on a host-by host basis by defining the
 variables as host or group vars under ``etc/kayobe/inventory/host_vars`` or
 ``etc/kayobe/inventory/group_vars``.
 
+For Ubuntu-based deployments, Pulp currently `lacks support
+<https://github.com/pulp/pulp_deb/issues/419>`_ for certain types of content,
+including i18n files and command-not-found indices. This breaks APT when the
+``command-not-found`` package is installed:
+
+.. code:: console
+
+   E: Failed to fetch https://pulp.example.com/pulp/content/ubuntu/jammy-security/development/dists/jammy-security/main/cnf/Commands-amd64   404  Not Found
+
+The ``purge-command-not-found.yml`` custom playbook can be used to uninstall
+the package, prior to running any other APT commands. It may be installed as a
+:kayobe-doc:`pre-hook <custom-ansible-playbooks.html#hooks>` to the ``host
+configure`` commands. Note that if used as a hook, this playbook matches all
+hosts, so will run against the seed, even when running ``overcloud host
+configure``. Depending on the stage of deployment, some hosts may be
+unreachable.
+
 For CentOS and Rocky Linux based systems, package manager configuration is
 provided by ``stackhpc_dnf_repos`` in ``etc/kayobe/dnf.yml``, which points to
 package repositories on the local Pulp server. To use this configuration, the

doc/source/configuration/security-hardening.rst

@@ -0,0 +1,44 @@
+==================
+Security Hardening
+==================
+
+CIS Benchmark Hardening
+-----------------------
+
+The roles from the `Ansible-Lockdown <https://github.com/ansible-lockdown>`_
+project are used to harden hosts in accordance with the CIS benchmark criteria.
+It won't get your benchmark score to 100%, but should provide a significant
+improvement over an unhardened system. A typical score would be 70%.
+
+The following operating systems are supported:
+
+- Rocky 8, RHEL 8, CentOS Stream 8
+- Ubuntu 22.04
+- Rocky 9
+
+Configuration
+--------------
+
+Some overrides to the role defaults are provided in
+``$KAYOBE_CONFIG_PATH/inventory/group_vars/overcloud/cis``. These may not be
+suitable for all deployments and so some fine tuning may be required. For
+instance, you may want different rules on a network node compared to a
+controller. It is best to consult the upstream role documentation for details
+about what each variable does. The documentation can be found here:
+
+- `Rocky 8, RHEL 8, CentOS Stream 8 <https://github.com/ansible-lockdown/RHEL8-CIS/tree/1.3.0>`__
+- `Ubuntu 22.04 <https://github.com/ansible-lockdown/UBUNTU22-CIS>`__
+- `Rocky 9 <https://github.com/ansible-lockdown/RHEL9-CIS>`__
+
+Running the playbooks
+---------------------
+
+As there is potential for unintended side effects when applying the hardening
+playbooks, the playbooks are not currently enabled by default. It is recommended
+that they are first applied to a representative staging environment to determine
+whether or not workloads or API requests are affected by any configuration changes.
+
+.. code-block:: console
+
+    kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cis.yml
+

doc/source/operations/rocky-linux-9.rst

@@ -67,6 +67,10 @@ Make the following changes to your Kayobe configuration:
 - Merge in the latest ``stackhpc-kayobe-config`` ``stackhpc/yoga`` branch.
 - Set ``os_distribution`` to ``rocky`` in ``etc/kayobe/globals.yml``.
 - Set ``os_release`` to ``"9"`` in ``etc/kayobe/globals.yml``.
+- Consider using a `prebuilt overcloud host image
+  <../configuration/host-images.html#pulling-host-images>`_ or building an
+  overcloud host image using the `standard configuration
+  <../configuration/host-images.html#building-host-images>`_.
 - If you are using Kayobe multiple environments, add the following into
   ``kayobe-config/etc/kayobe/environments/<env>/kolla/config/nova.conf``
   (as Kolla custom service config environment merging is not supported in
@@ -82,6 +86,30 @@ Make the following changes to your Kayobe configuration:
   This change does not need to be applied before migrating to Rocky Linux 9, but it should cause no harm to do so.
   Note that this will not affect existing VMs, only newly created VMs.
 
+Routing rules
+-------------
+
+Routing rules referencing tables by name may need adapting to be compatible with NetworkManager
+e.g:
+
+  .. code-block:: yaml
+
+      undercloud_prov_rules:
+        - from {{ internal_net_name | net_cidr }} table ironic-api
+
+will need to be updated to use numeric IDs:
+
+  .. code-block:: yaml
+
+      undercloud_prov_rules:
+        - from {{ internal_net_name | net_cidr }} table 1
+
+The error from NetworkManager was:
+
+  .. code-block:: shell
+
+      [1697192659.9611] keyfile: ipv4.routing-rules: invalid value for "routing-rule1": invalid value for "table"
+
 Prerequisites
 =============
 
@@ -602,7 +630,125 @@ Wazuh manager
 
 TODO
 
-In-place migrations
-===================
+In-place upgrades
+=================
 
-TODO
+Sometimes it is necessary to upgrade a system in-place.
+This may be the case for the seed hypervisor or Ansible control host which are often installed manually onto bare metal.
+This procedure is not officially recommended, and can be risky, so be sure to back up all critical data and ensure serial console access is available (including password login) in case of getting locked out.
+
+The procedure is performed in two stages:
+
+1. Migrate from CentOS Stream 8 to Rocky Linux 8
+2. Upgrade from Rocky Linux 8 to Rocky Linux 9
+
+Potential issues
+----------------
+
+Full procedure
+--------------
+
+- Inspect existing DNF packages and determine whether they are really required.
+
+- Use the `migrate2rocky.sh
+  <https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2rocky/migrate2rocky.sh>`__
+  script to migrate to Rocky Linux 8.
+
+- Disable all DNF modules - they're no longer used.
+
+  .. code-block:: console
+
+     sudo dnf module disable "*"
+
+- Migrate to NetworkManager. This can be done using a manual process or with Kayobe.
+
+  The manual process is as follows:
+
+  - Ensure that all network interfaces are managed by Network Manager:
+
+    .. code:: console
+
+       sudo sed -i -e 's/NM_CONTROLLED=no/NM_CONTROLLED=yes/g' /etc/sysconfig/network-scripts/*
+
+  - Enable and start NetworkManager:
+
+    .. code:: console
+
+       sudo systemctl enable NetworkManager
+       sudo systemctl start NetworkManager
+
+  - Migrate Ethernet connections to native NetworkManager configuration:
+
+    .. code:: console
+
+       sudo nmcli connection migrate
+
+  - Manually migrate non-Ethernet (bonds, bridges & VLAN subinterfaces) network interfaces to native NetworkManager.
+
+  - Look out for lost DNS configuration after migration to NetworkManager. This may be manually restored using something like this:
+
+    .. code:: console
+
+       nmcli con mod System\ brextmgmt.3003 ipv4.dns "10.41.4.4 10.41.4.5 10.41.4.6"
+
+  The following Kayobe process for migrating to NetworkManager has not yet been tested.
+
+  - Set ``interfaces_use_nmconnection: true`` as a host/group variable for the relevant hosts
+
+  - Run the appropriate host configure command. For example, for the seed hypervisor:
+
+    .. code:: console
+
+       kayobe seed hypervisor host configure -t network -kt none
+
+ - Make sure there are no funky udev rules left in
+   ``/etc/udev/rules.d/70-persistent-net.rules`` (e.g. from cloud-init run on
+   Rocky 9.1).
+
+  - Inspect networking configuration at this point, ideally reboot to validate correctness.
+
+- Upgrade to Rocky Linux 9
+
+  .. https://forums.rockylinux.org/t/dnf-warning-message-after-upgrade-from-rocky-8-to-rocky-9/8319/2
+
+  - Install Rocky Linux 9 repositories and GPG keys:
+
+    .. code:: console
+
+       sudo dnf install -y https://download.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/r/rocky-gpg-keys-9.2-1.6.el9.noarch.rpm \
+                           https://download.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/r/rocky-release-9.2-1.6.el9.noarch.rpm \
+                           https://download.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/r/rocky-repos-9.2-1.6.el9.noarch.rpm
+
+  - Remove the RedHat logos package:
+
+    .. code:: console
+
+       sudo rm -rf /usr/share/redhat-logos
+
+  - Synchronise all packages with current versions
+
+    .. code:: console
+
+       sudo dnf --releasever=9 --allowerasing --setopt=deltarpm=false distro-sync -y
+
+  - Rebuild RPB database:
+
+    .. code:: console
+
+       sudo rpm --rebuilddb
+
+  - Make a list of EL8 packages to remove:
+
+    .. code:: console
+
+       sudo rpm -qa | grep el8 > el8-packages
+
+  - Inspect the ``el8-packages`` list and ensure only expected packages are included.
+
+  - Remove the EL8 packages:
+
+    .. code:: console
+
+       cat el8-packages | xargs sudo dnf remove -y
+
+- You will need to re-create *all* virtualenvs afterwards due to system Python version upgrade.

etc/kayobe/ansible/cis.yml

@@ -4,12 +4,31 @@
   hosts: overcloud
   become: true
   tasks:
+    - name: Ensure the cron package is installed on ubuntu
+      package:
+        name: cron
+        state: present
+      when: ansible_facts.distribution == 'Ubuntu'
+
     - name: Remove /etc/motd
       # See remediation in:
       # https://github.com/wazuh/wazuh/blob/bfa4efcf11e288c0a8809dc0b45fdce42fab8e0d/ruleset/sca/centos/8/cis_centos8_linux.yml#L777
       file:
         path: /etc/motd
         state: absent
+      when: ansible_facts.os_family == 'RedHat' and ansible_facts.distribution_major_version == '8'
 
     - include_role:
         name: ansible-lockdown.rhel8_cis
+      when: ansible_facts.os_family == 'RedHat' and ansible_facts.distribution_major_version == '8'
+      tags: always
+
+    - include_role:
+        name: ansible-lockdown.rhel9_cis
+      when: ansible_facts.os_family == 'RedHat' and ansible_facts.distribution_major_version == '9'
+      tags: always
+
+    - include_role:
+        name: ansible-lockdown.ubuntu22_cis
+      when: ansible_facts.distribution == 'Ubuntu' and ansible_facts.distribution_major_version == '22'
+      tags: always

etc/kayobe/ansible/requirements.yml

@@ -15,6 +15,16 @@ roles:
   - name: ansible-lockdown.rhel8_cis
     src: https://github.com/ansible-lockdown/RHEL8-CIS
     version: 1.3.0
+  - name: ansible-lockdown.ubuntu22_cis
+    src: https://github.com/ansible-lockdown//UBUNTU22-CIS
+    #FIXME: Waiting for https://github.com/ansible-lockdown/UBUNTU22-CIS/pull/132
+    # to be in a tagged release
+    version: c91a1038fd218f727075d21b2d0880751322b162
+  - name: ansible-lockdown.rhel9_cis
+    src: https://github.com/ansible-lockdown/RHEL9-CIS
+    #FIXME: Waiting for https://github.com/ansible-lockdown/RHEL9-CIS/pull/54
+    # to be in a tagged release.
+    version: 3525cb6aab12a3d1e34aa8432ed77dd76be6a44a
   - name: wazuh-ansible
     src: https://github.com/stackhpc/wazuh-ansible
     version: stackhpc

etc/kayobe/cephadm.yml

@@ -12,7 +12,7 @@ cephadm_ceph_release: "{{ 'quincy' if (ansible_facts['distribution_release'] ==
 cephadm_image: "{{ stackhpc_docker_registry if stackhpc_sync_ceph_images | bool else 'quay.io' }}/ceph/ceph:{{ cephadm_image_tag }}"
 
 # Ceph container image tag.
-cephadm_image_tag: "{{ 'v17.2.6' if os_release == 'jammy' else 'v16.2.11' }}"
+cephadm_image_tag: "{{ 'v17.2.7' if os_release == 'jammy' else 'v16.2.14' }}"
 
 # Ceph custom repo workaround for Ubuntu Jammy as there are no official ceph repos for jammy.
 cephadm_custom_repos: "{{ ansible_facts['distribution_release'] == 'jammy' }}"