docs: Bring various things up to date, fill in some gaps

markgoddard · markgoddard · commit ebf277024f8f · 2024-09-09T15:53:48.000+01:00
diff --git a/docs/architecture/index.md b/docs/architecture/index.md
@@ -36,16 +36,13 @@ There are three types of Pulp service in the release train architecture.
 Various different types of content are hosted by Pulp, including:
 
 * RPM package repositories ([Pulp RPM plugin](https://docs.pulpproject.org/pulp_rpm/))
-    * CentOS distribution packages
+    * Rocky Linux distribution packages
     * Third party packages
-* Container image repositories ([Pulp container plugin](https://docs.pulpproject.org/pulp_container/))
-    * Kolla container images
-
-We also anticipate supporting the following content:
-
 * Apt package repositories ([Pulp Deb plugin](https://docs.pulpproject.org/pulp_deb/))
     * Ubuntu distribution packages
     * Third party packages
+* Container image repositories ([Pulp container plugin](https://docs.pulpproject.org/pulp_container/))
+    * Kolla container images
 * File repositories ([Pulp file plugin](https://docs.pulpproject.org/pulp_file/))
     * Disk images
 
@@ -79,7 +76,7 @@ Build and test processes are provided with credentials that are authorised to pu
 At the top of the diagram above are the upstream sources.
 Some of these may be mirrored/synced into Ark, including:
 
-* OS distribution package repositories, e.g. CentOS Stream 8 BaseOS
+* OS distribution package repositories, e.g. Rocky Linux 9 BaseOS
 * Third party package repositories, e.g. Grafana
 
 The [Sync package repositories](https://github.com/stackhpc/stackhpc-release-train/actions/workflows/package-sync.yml) GitHub Actions workflow runs nightly and on demand, ensuring that we have regular versioned snapshots of these repositories.
@@ -95,7 +92,7 @@ There are a couple of repositories for which `mirror_complete` does not work, so
 
 Package repositories are versioned based on the date/time stamp at the beginning of the sync workflow, e.g. `20211122T102435`.
 This version string is used as the final component of the path at which the corresponding distribution is hosted.
-For example, a CentOS Stream 8 BaseOS snapshot may be hosted at https://ark.stackhpc.com/pulp/content/centos/8-stream/BaseOS/x86_64/os/20220105T044843/.
+For example, a Rocky Linux 9 BaseOS snapshot may be hosted at https://ark.stackhpc.com/pulp/content/rocky/9/BaseOS/x86_64/os/20240105T044843/.
 
 The rationale behind using a date/time stamp is that there is no sane way to version a large collection of content, such as a repository, in a way in which the version reflects changes in the content (e.g. SemVer).
 While the timestamp used is fairly arbitrary, it does at least provide a reasonable guarantee of ordering, and is easily automated.
@@ -115,36 +112,40 @@ All content in Ark that is required by the build and test processes is synced to
 
 ### Kolla container images
 
-Kolla container images are built via Kayobe, using a `builder` environment in [StackHPC Kayobe config](https://github.com/stackhpc/stackhpc-kayobe-config).
+Kolla container images are built via Kayobe, using a `ci-builder` environment in [StackHPC Kayobe config](https://github.com/stackhpc/stackhpc-kayobe-config).
 The configuration uses the package repositories in Ark when building containers.
-Currently this is run manually, but will eventually run as a CI job.
+Images are built using a manually triggered [GitHub Actions workflow](https://github.com/stackhpc/stackhpc-kayobe-config/actions/workflows/stackhpc-container-image-build.yml).
 The `stackhpc-dev` namespace in Ark contains [container push repositories](https://docs.pulpproject.org/pulp_container/workflows/push.html), which are pushed to using Kayobe.
-Currently this is rather slow due to a [Pulp bug](https://github.com/pulp/pulp_container/issues/494).
 
 The [Sync container repositories](https://github.com/stackhpc/stackhpc-release-train/actions/workflows/container-sync.yml) GitHub Actions workflow runs demand, syncing container repositories in test Pulp service with those in Ark.
 It also configures container image distributions to be private, since they are public by default.
 
-Kolla container images are versioned based on the OpenStack release name and the date/time stamp at the beginning of the build workflow, e.g. `wallaby-20211122T102435`.
+Kolla container images are versioned based on the OpenStack release name, OS distribution and the date/time stamp at the beginning of the build workflow, e.g. `2024.1-rocky-9-20240922T102435`.
 This version string is used as the image tag.
 Unlike package repositories, container image tags allow multiple versions to be present in a distribution of a container repository simultaneously.
 We therefore use separate namespaces for development (`stackhpc-dev`) and release (`stackhpc`).
 
 ### Disk images
 
-Disk images are currently not built by the release train.
+Overcloud host images are built via Kayobe, using the same ``ci-builder`` environment used to build Kolla container images.
+Overcloud images are built using a manually triggered [GitHub Actions workflow](https://github.com/stackhpc/stackhpc-kayobe-config/actions/workflows/overcloud-host-image-build.yml).
+They are pushed to a [Pulp file repository](https://pulpproject.org/pulp_file/) in Ark, and uploaded to the SMS lab Glance image service for all-in-one and multi-node testing.
+
+Overcloud images are versioned based on the OpenStack release name, and the date/time stamp at the beginning of the build workflow, e.g. `2024.1-20240922T102435`.
+This version string is included in the Pulp distribution `base_path` of the image, e.g. https://ark.stackhpc.com/pulp/content/kayobe-images/2023.1/rocky/9/2023.1-20240325T130221/overcloud-rocky-9.qcow2
 
 ## Testing
 
 Release train content is tested via a Kayobe deployment of OpenStack.
-An `aio` environment in [StackHPC Kayobe config](https://github.com/stackhpc/stackhpc-kayobe-config) provides a converged control/compute host for testing.
-Currently this is run manually, but will eventually run as a CI job.
+A `ci-aio` environment in [StackHPC Kayobe config](https://github.com/stackhpc/stackhpc-kayobe-config) provides a converged control/compute host for testing.
+Various all-in-one OpenStack test configurations are run against pull requests opened against the StackHPC Kayobe config repository.
 
 ## Promotion
 
 Whether content is mirrored from an upstream source or built locally, it is not immediately released.
 Promotion describes the process whereby release candidate content is made into a release that is available to clients.
 
-For package repositories, promotion does not affect how content is accessed, only who may access it.
+For package repositories and overcloud host images, promotion does not affect how content is accessed, only who may access it.
 Promotion involves changing the content guard for the distribution to be released from `development` to `release`.
 This makes the content accessible to clients using their client credentials.
 
@@ -177,7 +178,7 @@ This repository provides configuration and playbooks to:
 
 This configuration is in active development and is expected to evolve over the coming releases.
 
-Further documentation of this configuration is out of scope here, but is available in the [readme](https://github.com/stackhpc/stackhpc-kayobe-config/blob/stackhpc/wallaby/README.rst).
+Further documentation of this configuration is out of scope here, but is available in the [readme](https://github.com/stackhpc/stackhpc-kayobe-config/blob/stackhpc/2024.1/README.rst).
 
 ## Continuous Integration (CI) and automation
 
diff --git a/docs/index.md b/docs/index.md
@@ -19,15 +19,12 @@ This includes artifacts, metadata, and the configuration necessary to consume th
 The current scope of the release train covers OpenStack control plane deployment, and all the dependencies thereof.
 This includes:
 
-* Host Operating System (OS) repositories, e.g. CentOS Stream 8 BaseOS
+* Host Operating System (OS) repositories, e.g. Rocky Linux 9 BaseOS
 * Source code repositories
 * Kolla container images
 * Ironic Python Agent (IPA) deployment images
 * VM & bare metal disk images
 
-Initially, only CentOS Stream 8 is supported as a host and container OS.
-Ubuntu support will follow.
-
 In future, the scope may expand to cover other software, such as Kubernetes.
 
 ## Configuration
diff --git a/docs/operations/ark.md b/docs/operations/ark.md
@@ -0,0 +1,6 @@
+# Ark
+
+Ark is deployed on a single host that runs on Leafcloud.
+Data is stored in Leafcloud's object store.
+An [Ansible playbook](https://github.com/stackhpc/ansible-pulpcore-config) automates deployment of Pulp and its dependencies using the [Pulp OCI images](https://pulpproject.org/pulp-oci-images/) and [Compose configuration](https://pulpproject.org/pulp-oci-images/docs/admin/tutorials/quickstart/#podman-or-docker-compose) provided by the Pulp project.
+
diff --git a/docs/operations/github.md b/docs/operations/github.md
@@ -1,6 +1,15 @@
 # Operations - GitHub
 
-## GitHub Actions runners
+## GitHub Actions Runner Controller (ARC)
+
+Some GitHub Actions workflows run on public runners, while others require access to specific services or benefit from data locality.
+In the latter case we use GitHub's Actions Runner Controller (ARC) (not to be confused with our Ark...) to provide dynamically scaling containerised private GitHub Actions runners.
+The [ARC-Installer](https://github.com/stackhpc/ARC-Installer) repository contains scripts and Helm `values.yaml` configuration for registering the ARC controller services and runner scale sets.
+Any project that wishes to use runners on this cluster should define its runner scale set configuration in the ARC-Installer repository.
+
+## [LEGACY] GitHub Actions runners
+
+*We are no longer using this approach, but it is retained in case it is useful in future.*
 
 This Terraform configuration deploys a GitHub Actions runner VMs on an
 OpenStack cloud for the stackhpc-release-train repository.
diff --git a/docs/usage/access.md b/docs/usage/access.md
@@ -25,4 +25,6 @@ Configuration for content guards is in [ansible/inventory/group_vars/all/dev-pul
 
 ## Generating client credentials
 
-TODO
+Clients require credentials in order to access Ark.
+See the README in the [stackhpc-release-train-clients](https://github.com/stackhpc/stackhpc-release-train-clients) repository for how to generete and update them.
+It is also possible to use this repository to generate credentials with access to the unpromoted "development" content, but these should not be given to clients.
diff --git a/docs/usage/content-workflows.md b/docs/usage/content-workflows.md
@@ -175,16 +175,16 @@ After a successful container image build workflow, another workflow is triggered
 In the following example, the user specified a regular expression of `^skydive`, matching all of the Skydive images, and the `base` image that they depend on.
 
 ```
-REPOSITORY                                                     TAG                       IMAGE ID       CREATED         SIZE
-ark.stackhpc.com/stackhpc-dev/centos-source-skydive-agent      wallaby-20220811T091848   32f2b9299194   6 minutes ago   1.29GB
-ark.stackhpc.com/stackhpc-dev/centos-source-skydive-analyzer   wallaby-20220811T091848   35e4c1cda1a8   7 minutes ago   1.14GB
-ark.stackhpc.com/stackhpc-dev/centos-source-skydive-base       wallaby-20220811T091848   3bd5f3e50aa3   7 minutes ago   1.14GB
-ark.stackhpc.com/stackhpc-dev/centos-source-base               wallaby-20220811T091848   bd02fa0ec1d6   7 minutes ago   991MB
+REPOSITORY                                       TAG                       	  IMAGE ID       CREATED         SIZE
+ark.stackhpc.com/stackhpc-dev/skydive-agent      2023.1-rocky-9-20240811T091848   32f2b9299194   6 minutes ago   1.29GB
+ark.stackhpc.com/stackhpc-dev/skydive-analyzer   2023.1-rocky-9-20240811T091848   35e4c1cda1a8   7 minutes ago   1.14GB
+ark.stackhpc.com/stackhpc-dev/skydive-base       2023.1-rocky-9-20240811T091848   3bd5f3e50aa3   7 minutes ago   1.14GB
+ark.stackhpc.com/stackhpc-dev/base               2023.1-rocky-9-20240811T091848   bd02fa0ec1d6   7 minutes ago   991MB
 ```
 
-In this example, the base and Skydive images have been tagged `wallaby-20220811T091848`.
+In this example, the base and Skydive images have been tagged `2023.1-rocky-9-20240811T091848`.
 
-Instructions for building Kolla container images manually are provided in the [StackHPC kayobe config README](https://github.com/stackhpc/stackhpc-kayobe-config/blob/bf1396b8564b79344e4b6cfb934eab865ff1ad09/README.rst#L226).
+Instructions for building Kolla container images manually are provided in the [StackHPC kayobe config documentation](https://stackhpc-kayobe-config.readthedocs.io/en/stackhpc-2023.1/contributor/environments/ci-builder.html).
 
 ## Publishing container images
 
