Improve pulp docs (#819)

* improve pulp and related docs

* fix docs spelling

* fix GHFM alerts not being rendered in lists

* fix lint errors

* fix markdown lint errors

* fix markdown prettier errors

* re-add lost changes following review on PR#812

* fix linter whitespace

Author: sjpb

ansible/adhoc/sync-pulp.yml

@@ -5,7 +5,5 @@
         name: pulp_site
         tasks_from: sync.yml
       vars:
-        pulp_site_target_arch: "x86_64"
-        pulp_site_target_distribution: "rocky"
         # default distribution to *latest* specified for baseos repo:
         pulp_site_target_distribution_version: "{{ dnf_repos_repos['baseos'].keys() | map('float') | sort | last }}"

docs/experimental/pulp.md

@@ -1,54 +1,108 @@
 # Pulp Server
 
-In order to ensure reproducible builds, the appliance can build images using repository mirrors from StackHPC's "Ark" Pulp server. The appliance can sync relevant repositories to a local Pulp server which will then be used instead of Ark.
+In order to ensure reproducibility, by default image builds use mirrors of DNF
+repositories hosted on StackHPC's "Ark" Pulp server. This page describes how to
+use a local Pulp server instead of Ark, which reduces network traffic and speeds
+up builds. The repositories on this local Pulp server are synchronised to Ark so
+that builds still use the same package snapshots.
 
-## Deploying/configuring Pulp Server
+It is also possible to use a local Pulp server to install packages during the
+`site.yml` playbook rather than during image builds, as described in [docs/operations.md](../operations.md#adding-additional-packages).
 
-### Deploying a Pulp server
+## Deploying and Configuring a Local Pulp Server
 
-A playbook is provided to install and configure a Pulp server on a given host. Admin credentials for this server are automatically generated through the `ansible/adhoc/generate-passwords.yml` playbook. To use this, create an inventory file
-defining a group `pulp_server` containing a single host, which requires at least 2 vCPUs and 4GB RAM. The group should be defined in your `site` environment's inventory so that a single Pulp server is shared between all environments and
-the same snapshots are tested in staging and production.
-Deploying and syncing Pulp has been tested on an RL9 host. The hostvar `ansible_host` should be defined, giving the IP address Ansible should use for SSH. For example, you can create an ini file at `environments/site/inventory/pulp` with the contents:
+The appliance can install and configure a local Pulp server on a specified host.
+This host should run RockyLinux 8 or 9 and have at least 2 vCPUs and 8GB RAM.
+Note upgrades etc. of this host will not be managed by the appliance. Access to
+Pulp content is not authenticated so this server should not be externally
+reachable.
 
-```ini
-[pulp_server]
-pulp_host ansible_host=<VM-ip-address>
-```
+> [!IMPORTANT]
+> Commands below should be run with the `staging` environment active, as all
+> Pulp syncs will be done from there.
 
-> [!WARNING]
-> The inventory hostname cannot conflict with group names i.e can't be called `pulp_site` or `pulp_server`.
+1. Define the host in a group `pulp_server` within the `site` inventory. This
+   means clusters in all environments use the same Pulp server, and the synced
+   DNF repository snapshots are tested in staging before use in production. E.g.:
 
-Once complete, it will print a message giving a value to set for `appliances_pulp_url` (see example config below), assuming the `ansible_host` address is also the address the cluster
-should use to reach the Pulp server.
+   ```ini
+   # environments/site/inventory/pulp:
+   [pulp_server]
+   pulp_host ansible_host=<VM-ip-address>
+   ```
 
-Note access to this server's content isn't authenticated so this assumes the `pulp_server` host is not externally reachable.
+   **NB:** The inventory hostname must not conflict with group names, i.e. it
+   cannot be `pulp_site` or `pulp_server`.
 
-### Using an existing Pulp server
+2. If adding Pulp to an existing deployment, ensure Pulp admin credentials
+   exist:
 
-An existing Pulp server can be used to host Ark repos by overriding `pulp_site_password` and `appliances_pulp_url` in the target environment. Note that this assumes the same configuration as the appliance deployed Pulp i.e no content authentication.
+   ```shell
+   ansible-vault decrypt environments/staging/inventory/group_vars/all/secrets.yml
+   ansible-playbook ansible/adhoc/generate-passwords.yml
+   ansible-vault encrypt environments/staging/inventory/group_vars/all/secrets.yml
+   ```
 
-## Syncing Pulp content with Ark
+3. Run the adhoc playbook to install and configure Pulp:
+
+   ```shell
+   ansible-playbook ansible/adhoc/deploy-pulp.yml
+   ```
+
+   Once complete, it will print a message giving a value to set for
+   `appliances_pulp_url`, assuming the inventory `ansible_host` address is
+   also the address the cluster should use to reach the Pulp server.
+
+4. Create group vars files defining `appliances_pulp_url` and dev credentials
+   for StackHPC's "Ark" Pulp server:
+
+   ```yaml
+   # environments/site/inventory/group_vars/all/pulp.yml:
+   appliances_pulp_url: "http://<pulp-host-ip>:8080"
+   pulp_site_upstream_username: your-ark-username
+   pulp_site_upstream_password: "{{ vault_pulp_site_upstream_password }}"
+   ```
+
+   ```yaml
+   # environments/site/inventory/group_vars/all/vault_pulp.yml:
+   vault_pulp_site_upstream_password: your-ark-password
+   ```
 
-If the `pulp_site` group is added to the Packer build groups, the local Pulp server will be synced with Ark on build. You must authenticate with Ark by overriding `pulp_site_upstream_username` and `pulp_site_upstream_password` with your vault encrypted Ark dev credentials. `dnf_repos_username` and `dnf_repos_password` must remain unset to access content from the local Pulp.
+   and vault-encrypt the latter:
 
-Content can also be synced by running `ansible/adhoc/sync-pulp.yml`. By default this syncs repositories for the latest version of Rocky supported by the appliance but this can be overridden by setting extra variables for `pulp_site_target_arch`, `pulp_site_target_distribution` and `pulp_site_target_distribution_version`.
+   ```shell
+   ansible-vault encrypt environments/site/inventory/group_vars/all/vault_pulp.yml
+   ```
+
+   If previously using Ark credentials directly e.g. for image builds, ensure
+   the variables `dnf_repos_username` and `dnf_repos_password` are no longer
+   set in any environment.
+
+5. Commit changes.
+
+## Using an existing Pulp server
+
+Alternatively, an existing Pulp server can be used to host Ark repos by
+setting `appliances_pulp_url` directly. Note that this assumes the same
+configuration as the appliance deployed Pulp i.e no content authentication.
+As above, the `dnf_repos_` variables must not be set in this configuration.
+
+## Syncing Pulp content with Ark
 
-## Example config in site variables
+The appliance can synchronise repositories on local Pulp server from Ark in
+two ways:
 
-```yaml
-# environments/site/inventory/group_vars/all/pulp_site.yml:
-appliances_pulp_url: "http://<pulp-host-ip>:8080"
-pulp_site_upstream_username: <Ark-username>
-pulp_site_upstream_password: <Ark-password>
-```
+1. If the `pulp_site` group is added to the Packer build groups, the local Pulp
+   server will be synced with Ark during image builds.
 
-## Installing packages from Pulp at runtime
+2. The sync can be manually be triggered by running:
 
-By default, system repos are overwritten to point at Pulp repos during [image builds,](../image-build.md) so using a site Pulp server will require a new fatimage. If you instead wish to install packages at runtime,
-you will need to add all host groups on which you will be installing packages to the `dnf_repos` group in `environments/site/inventory/groups` e.g:
+   ```shell
+   ansible-playbook ansible/adhoc/sync-pulp.yml
+   ```
 
-```yaml
-[dnf_repos:children]
-cluster
-```
+   By default this method syncs repositories for the latest version of RockyLinux
+   supported by the appliance. This can be overridden by setting
+   `pulp_site_target_distribution_version` to e.g. `'8.10'`, i.e the `Major.minor`
+   version of RockyLinux the site clusters are using. **NB:** This value
+   must be quoted to avoid an incorrect conversion to float.

docs/image-build.md

@@ -67,12 +67,12 @@ For either a site-specific fat-image build or an extra-build:
    - Normally the network must provide outbound internet access. However it
      does not need to provide access to resources used by the actual cluster
      nodes (e.g. Slurm control node, network filesystem servers etc.).
-   - The flavor used must have sufficent memory for the build tasks (usually
+   - The flavor used must have sufficient memory for the build tasks (usually
      8GB), but otherwise does not need to match the actual cluster node
      flavor(s).
    - By default, the build VM is volume-backed to allow control of the root
      disk size (and hence final image size), so the flavor's disk size does not
-     matter. The default volume size is not sufficent if enabling `cuda` and/or
+     matter. The default volume size is not sufficient if enabling `cuda` and/or
      `doca` and should be increased:
      ```terraform
      volume_size = 35 # GB
@@ -93,7 +93,7 @@ For either a site-specific fat-image build or an extra-build:
      All possible groups are listed in `environments/common/groups` but common
      options for this variable will be:
 
-     - For a fatimage build: `fatimage`: This is defined in `enviroments/site/inventory/groups`
+     - For a fatimage build: `fatimage`: This is defined in `environments/site/inventory/groups`
        and results in an update of all packages in the source image, plus
        installation of packages for default control, login and compute nodes.
 
@@ -137,15 +137,15 @@ For either a site-specific fat-image build or an extra-build:
 In summary, Packer creates an OpenStack VM, runs Ansible on that, shuts it down, then creates an image from the root disk.
 
 Many of the Packer variables defined in `openstack.pkr.hcl` control the definition of the build VM and how to SSH to it to run Ansible. These are generic OpenStack builder options
-and are not specific to the Slurm Appliance. Packer varibles can be set in a file at any convenient path; the build example above
+and are not specific to the Slurm Appliance. Packer variables can be set in a file at any convenient path; the build example above
 shows the use of the environment variable `$PKR_VAR_environment_root` (which itself sets the Packer variable
 `environment_root`) to automatically select a variable file from the current environment, but for site-specific builds
 using a path in a "parent" environment is likely to be more appropriate (as builds should not be environment-specific to allow testing before deployment to a production environment).
 
 What is Slurm Appliance-specific are the details of how Ansible is run:
 
 - The build VM is always added to the `builder` inventory group, which differentiates it from nodes in a cluster. This allows
-  Ansible variables to be set differently during Packer builds, e.g. to prevent services starting. The defaults for this are in `environments/common/inventory/group_vars/builder/`, which could be extended or overriden for site-specific fat image builds using `builder` groupvars for the relevant environment. It also runs some builder-specific code (e.g. to clean up the image).
+  Ansible variables to be set differently during Packer builds, e.g. to prevent services starting. The defaults for this are in `environments/common/inventory/group_vars/builder/`, which could be extended or overridden for site-specific fat image builds using `builder` groupvars for the relevant environment. It also runs some builder-specific code (e.g. to clean up the image).
 - The default fat image builds also add the build VM to the "top-level" `compute`, `control` and `login` groups. This ensures
   the Ansible specific to all of these types of nodes run. Note other inventory groups are constructed from these by `environments/common/inventory/groups file` - this is not builder-specific.
 - As noted above, for "extra" builds the additional groups can be specified directly. In this way an existing image can be extended with site-specific Ansible, without modifying the

docs/operations.md

@@ -67,25 +67,24 @@ This is a usually a two-step process:
 
 Deploying the additional nodes and applying these changes requires rerunning both OpenTofu and the Ansible site.yml playbook - follow [Deploying a Cluster](#deploying-a-cluster).
 
-## Package Repositories
-
-```yaml
-[squid:children]
-# Hosts to run squid proxy
-login
-```
+## Adding Additional Packages
 
-Note that many non-default roles include package installations from repositories which the appliance overwrites to point at snapshotted mirrors on a Pulp server (by default StackHPC's Ark server), which are
-disabled during runtime to prevent Ark credentials from being leaked. To enable this functionality, you must therefore either:
+The StackHPC images provided via [GitHub releases](https://github.com/stackhpc/ansible-slurm-appliance/releases)
+have all DNF repositories disabled, because for reproducibility these images are
+build using (authenticated) mirrors hosted on StackHPC's "Ark" Pulp server and
+the credentials are not provided as part of the appliance.
 
-- Create a site-specific fatimage (see [image build docs](image-build.md)) with the appropriate group added to the `inventory_groups` Packer variables.
-- If you instead wish roles to perform their installations during runtime, deploy a site Pulp server and sync it with with mirrors of the snapshots from the upstream Ark server (see [Pulp docs](experimental/pulp.md)).
+This means that when running the `site.yml` playbook, by default:
 
-In both cases, Ark credentials will be required.
+- Features which are not enabled by default, e.g., `freeipa_client`, cannot
+  install the packages they require.
+- It is not possible to install arbitrary packages using e.g. an `ansible.builtin.dnf`
+  task in a hook.
 
-## Adding Additional Packages
+The recommended way to resolve both of these issues is by carrying out a
+site-specific [image build](./image-build.md).
 
-By default, the following utility packages are installed during the StackHPC image build:
+By default, the following utility packages are installed in StackHPC images:
 
 - htop
 - nano
@@ -101,7 +100,7 @@ By default, the following utility packages are installed during the StackHPC ima
 
 Additional packages can be added during image builds by:
 
-1. Configuring an [image build](./image-build.md) to enable the
+1. Configuring the [image build](./image-build.md) to enable the
    `extra_packages` group:
 
    ```terraform
@@ -129,10 +128,13 @@ the OpenHPC installation guide (linked from the
 "user-facing" OpenHPC packages such as compilers, MPI libraries etc. include
 corresponding `lmod` modules.
 
-Packages _may_ also be installed during the site.yml, by adding the `cluster`
-group as a child of the `extra_packages` group. An error will occur if Ark
-credential are defined in this case, as they are readable by unprivileged users
-in the `.repo` files and a local Pulp mirror must be used instead.
+If a site-specific image build and cluster reimage is not possible (e.g. for
+an urgent patch), it is possible to install packages directly during the
+`site.yml` playbook by adding the `cluster` group as a child of the
+`extra_packages` group. An error will occur if Ark credentials are defined in
+this case, as they are readable by unprivileged users in the `.repo` files. A
+local Pulp mirror must be used instead, which also has the advantage of making
+this approach more reproducable.
 
 If additional repositories are required, these could be added/enabled as necessary in a play added to `environments/$SITE_ENV/hooks/{pre,post}.yml` as appropriate.
 Note such a play should NOT exclude the builder group, so that the repositories are also added to built images.
@@ -210,7 +212,7 @@ ansible-playbook ansible/adhoc/$PLAYBOOK
 Currently they include the following (see each playbook for links to documentation):
 
 - `hpctests.yml`: MPI-based cluster tests for latency, bandwidth and floating point performance.
-- `rebuild.yml`: Rebuild nodes with existing or new images (NB: this is intended for development not for reimaging nodes on an in-production cluster).
+- `rebuild.yml`: Rebuild nodes with existing or new images (NB: this is intended for development not for re-imaging nodes on an in-production cluster).
 - `restart-slurm.yml`: Restart all Slurm daemons in the correct order.
 - `update-packages.yml`: Update specified packages on cluster nodes (NB: not recommended for routine use).
 
@@ -220,4 +222,4 @@ The `ansible` binary [can be used](https://docs.ansible.com/ansible/latest/comma
 ansible [--become] <group/host> -m shell -a "<shell command>"
 ```
 
-This can be useful for debugging and development but any modifications made this way will be lost if nodes are rebuilt/reimaged.
+This can be useful for debugging and development but any modifications made this way will be lost if nodes are rebuilt/re-imaged.

environments/common/inventory/groups

@@ -229,7 +229,7 @@ doca
 # Host to deploy a Pulp server on and sync with mirrors of upstream Ark repositories. Should be a group containing a single VM provisioned
 # separately from the appliance. e.g
 # pulp_host ansible_host=<VM-ip-address>
-# Note the host name can't conflict with group names i.e can't be called `pulp` or `pulp_server`
+# Note the host name can't conflict with group names i.e can't be called `pulp_site` or `pulp_server`
 
 [raid]
 # Add `builder` to configure image for software raid

environments/site/inventory/groups

@@ -192,7 +192,7 @@ compute
 # Host to deploy a Pulp server on and sync with mirrors of upstream Ark repositories. Should be a group containing a single VM provisioned
 # separately from the appliance. e.g
 # pulp_host ansible_host=<VM-ip-address>
-# Note inventory host name cannot conflict with group names i.e can't be called `pulp` or `pulp_server`.
+# Note inventory host name cannot conflict with group names i.e can't be called `pulp_site` or `pulp_server`.
 
 [raid:children]
 # Add `builder` to configure image for software raid