updated operations guide for functionality requiring additional installs

Author: wtripp180901

docs/environments.md

@@ -14,7 +14,10 @@ All environments load the inventory from the `common` environment first, with th
 
 The ansible inventory for the environment is in `environments/<environment>/inventory/`. It should generally contain:
 - A `hosts` file. This defines the hosts in the appliance. Generally it should be templated out by the deployment automation so it is also a convenient place to define variables which depend on the deployed hosts such as connection variables, IP addresses, ssh proxy arguments etc.
-- A `groups` file defining ansible groups, which essentially controls which features of the appliance are enabled and where they are deployed. This repository generally follows a convention where functionality is defined using ansible roles applied to a group of the same name, e.g. `openhpc` or `grafana`. The meaning and use of each group is described in comments in `environments/common/inventory/groups`. As the groups defined there for the common environment are empty, functionality is disabled by default and must be enabled in a specific environment's `groups` file. Two template examples are provided in `environments/commmon/layouts/` demonstrating a minimal appliance with only the Slurm cluster itself, and an appliance with all functionality.
+- A `groups` file defining ansible groups, which essentially controls which features of the appliance are enabled and where they are deployed. This repository generally follows a convention where functionality is defined using ansible roles applied to a group
+of the same name, e.g. `openhpc` or `grafana`. The meaning and use of each group is described in comments in `environments/common/inventory/groups`. As the groups defined there for the common environment are empty, functionality is disabled by default and must be
+enabled in a specific environment's `groups` file. The `site` environment contains an ini file at `environments/site/inventory/groups` which enables groups for default appliance functionality across all environments. Additional groups should generally also be
+enabled in this file to avoid divergence between staging and production environments. Note that enabling some groups may require a site-specific image build and Ark credentials (see [operations guide](operations.md)).
 - Optionally, group variable files in `group_vars/<group_name>/overrides.yml`, where the group names match the functional groups described above. These can be used to override the default configuration for each functionality, which are defined in `environments/common/inventory/group_vars/all/<group_name>.yml` (the use of `all` here is due to ansible's precedence rules).
 
 Although most of the inventory uses the group convention described above there are a few special cases:

docs/experimental/pulp.md

@@ -16,7 +16,7 @@ pulp_host ansible_host=<VM-ip-address>
 ```
 
 > [!WARNING] 
-> The inventory hostname cannot conflict with group names i.e can't be called `pulp` or `pulp_server`.
+> The inventory hostname cannot conflict with group names i.e can't be called `pulp_site` or `pulp_server`.
 
 Once complete, it will print a message giving a value to set for `appliances_pulp_url` (see example config below), assuming the `ansible_host` address is also the address the cluster
 should use to reach the Pulp server.
@@ -28,7 +28,7 @@ An existing Pulp server can be used to host Ark repos by overriding `pulp_site_p
 
 ## Syncing Pulp content with Ark
 
-If the `pulp` group is added to the Packer build groups, the local Pulp server will be synced with Ark on build. You must authenticate with Ark by overriding `pulp_site_upstream_username` and `pulp_site_upstream_password` with your vault encrypted Ark dev credentials. `dnf_repos_username` and `dnf_repos_password` must remain unset to access content from the local Pulp.
+If the `pulp_site` group is added to the Packer build groups, the local Pulp server will be synced with Ark on build. You must authenticate with Ark by overriding `pulp_site_upstream_username` and `pulp_site_upstream_password` with your vault encrypted Ark dev credentials. `dnf_repos_username` and `dnf_repos_password` must remain unset to access content from the local Pulp.
 
 Content can also be synced by running `ansible/adhoc/sync-pulp.yml`. By default this syncs repositories for the latest version of Rocky supported by the appliance but this can be overridden by setting extra variables for `pulp_site_target_arch`, `pulp_site_target_distribution` and `pulp_site_target_distribution_version`.
 
@@ -40,3 +40,12 @@ appliances_pulp_url: "http://<pulp-host-ip>:8080"
 pulp_site_upstream_username: <Ark-username>
 pulp_site_upstream_password: <Ark-password>
 ```
+
+## Installing packages from Pulp at runtime
+By default, system repos are overwritten to point at Pulp repos during [image builds,](../image-build.md) so using a site Pulp server will require a new fatimage. If you instead wish to install packages at runtime,
+you will need to add all host groups on which you will be installing packages to the `dnf_repos` group in `environments/site/inventory/groups` e.g:
+
+```
+[dnf_repos:children]
+cluster
+```

docs/operations.md

@@ -9,7 +9,7 @@ All subsequent sections assume that:
 - Appropriate OpenStack credentials are available.
 - Any non-appliance controlled infrastructure is available (e.g. networks, volumes, etc.).
 - `$ENV` is your current, activated environment, as defined by e.g. `environments/production/`.
-- `$SITE_ENV` is the base site-specific environment, as defined by e.g. `environments/mysite/`.
+- `$SITE_ENV` is the base site-specific environment, as defined by `environments/site/`.
 - A string `some/path/to/file.yml:myvar` defines a path relative to the repository root and an Ansible variable in that file.
 - Configuration is generally common to all environments at a site, i.e. is made in `environments/$SITE_ENV` not `environments/$ENV`.
 
@@ -62,6 +62,24 @@ This is a usually a two-step process:
 
 Deploying the additional nodes and applying these changes requires rerunning both OpenTofu and the Ansible site.yml playbook - follow [Deploying a Cluster](#Deploying-a-Cluster).
 
+# Enabling additional functionality
+Roles in the appliance which are disabled by default can be enabled by adding the appropriate groups as children of the role's corresponding group in `environments/site/inventory/groups`. For example,
+to install a Squid proxy on nodes in the login group, you would modify the `squid` group definition in `environments/site/inventory/groups` to:
+
+```
+[squid:children]
+# Hosts to run squid proxy
+login
+```
+
+Note that many non-default roles include package installations from repositories which the appliance overwrites to point at snapshotted mirrors on a Pulp server (by default StackHPC's Ark server), which are
+disabled during runtime to prevent Ark credentials from being leaked. To enable this functionality, you must therefore either:
+
+- Create a site-specific fatimage (see [image build docs](image-build.md)) with the appropriate group added to the `inventory_groups` Packer variables.
+- If you instead wish roles to perform their installations during runtime, deploy a site Pulp server and sync it with with mirrors of the snapshots from the upstream Ark server (see [Pulp docs](experimental/pulp.md)).
+
+In both cases, Ark credentials will be required.
+
 # Adding Additional Packages
 By default, the following utility packages are installed during the StackHPC image build:
 - htop

environments/site/inventory/groups

@@ -158,6 +158,9 @@ compute
 # Note that this feature currently assumes all compute nodes are VMs, enabling
 # when the cluster contains baremetal compute nodes may lead to unexpected scheduling behaviour
 
+[pulp_site]
+# Add builder to this group to enable automatically syncing of pulp during image build
+
 [pulp_server]
 # Host to deploy a Pulp server on and sync with mirrors of upstream Ark repositories. Should be a group containing a single VM provisioned
 # separately from the appliance. e.g