Merge pull request #136 from epics-containers/august-update

August update

Author: gilesknap

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0
+    rev: v4.6.0
     hooks:
       - id: check-added-large-files
       - id: check-yaml

docs/explanations/introduction.md

@@ -52,9 +52,9 @@ It does not contain a startup script pr EPICS database, these are instance speci
 An IOC instance runs in a container runtime by loading two things:
 
 - The Generic IOC image passed to the container runtime.
-- The IOC instance configuration. This is mapped into the container at  runtime by mounting it into the filesystem. The mount point for this configuration is always `/epics/ioc/config`.
+- The IOC instance configuration. This is mapped into the container at  runtime by mounting it into the filesystem. The mount point for this configuration is usually the folder `/epics/ioc/config`.
 
-The configuration will bootstrap the unique properties of that instance. The following contents for the configuration are supported:
+The configuration will bootstrap the unique properties of that instance. The following contents for the configuration folder are supported:
 
 - ``ioc.yaml``: an **ibek** IOC description file which **ibek** will use to generate
   st.cmd and ioc.subst.
@@ -117,13 +117,13 @@ of the Chart within the cluster.
 It also supports registries for storing version history of Charts,
 much like docker.
 
-In this project we use Helm Charts to define and deploy IOC instances. IOCs are grouped into a {any}`services-repo`. Typical services repositories represent a beamline or accelerator technical area but any grouping is allowed. Each of these repositories holds the Helm Charts for its IOC Instances and any other services we require. Each IOC instance folder need only container:e
+In this project we use Helm Charts to define and deploy IOC instances. IOCs are grouped into a {any}`services-repo`. Typical services repositories represent a beamline or accelerator technical area but any grouping is allowed. Each of these repositories holds the Helm Charts for its IOC Instances and any other services we require. Each IOC instance folder need only contain:
 
-- a values.yaml file to override the default values in the repository's global Helm Chart
+- a values.yaml file to override the default values in the repository's global values.yaml.
 - a config folder as described in {any}`generic-iocs`.
-- a couple of boilerplate files that are the same for all IOCs.
+- a couple of boilerplate Helm files that are the same for all IOCs.
 
-**epics-containers** does not use helm registries for storing each IOC instance. Such registries only hold a zipped version of the Chart files, and this is redundant when we have a git repository holding the same information. Instead a single global helm chart represents the shared elements between all IOC instances and is stored in a helm registry. Each folder in the services repository is itself a helm chart that includes the global chart as a dependency.
+**epics-containers** does not use helm registries for storing each IOC instance. Such registries only hold a zipped version of the Chart files, and this is redundant when we have a git repository holding the same information. Instead a single global helm chart represents the shared elements between all IOC instances and is stored in a helm registry. Each folder in the services repository is itself a helm chart that includes that global chart as a dependency.
 
 ### Repositories
 
@@ -135,25 +135,26 @@ In the **epics-containers** examples all repositories are held in the same githu
 
 There are many alternative services for storing these repositories, both in the cloud and on premises. Below we list the choices we have tested during the proof of concept.
 
-The classes of repository are as follows:
+The most common classes of repository are as follows:
 
 ```{eval-rst}
-:Source Repository:
 
-  Holds the source code but also provides the Continuous Integration actions for testing, building and publishing to the image / helm repositories. These have been tested:
+:Generic IOC Source Repositories:
+
+  Define how a Generic IOC image is built, this does not typically include source code, but instead is a set of instructions for building the Generic IOC image by compiling source from a number of upstream support module repositories. Boilerplate IOC source code is also included in the Generic IOC source repository and can be customized if needed. These have been tested:
 
   - GitHub
   - GitLab (on premises)
 
-:Generic IOC Source Repositories:
+:Services Source Repositories:
 
-  Define how a Generic IOC image is built, this does not typically include source code, but instead is a set of instructions for building the Generic IOC image by compiling source from a number of upstream support module repositories. Boilerplate IOC source code is also included in the Generic IOC source repository and can be customized if needed.
+  Define the IOC instances and other services for a beamline, accelerator technical area or any other grouping strategy. These have been tested:
 
-:Services Source Repositories:
+  - GitHub
+  - GitLab (on premises)
 
-  Define the IOC instances and other services for a beamline, accelerator domain or any other grouping strategy.
 
-:An OCI Image Repository:
+:An OCI Image Registry:
 
   Holds the Generic IOC container images and their dependencies. Also used to hold the helm Charts that define the shared elements between all domains.
 
@@ -167,7 +168,7 @@ The classes of repository are as follows:
 ### Continuous Integration
 
 Our examples all use continuous integration to get from pushed source
-to the published images, IOC instances helm Charts and documentation.
+to the published images, IOC instances Helm Charts and documentation.
 
 This allows us to maintain a clean code base that is continually tested for
 integrity and also to maintain a direct relationship between source code version
@@ -184,36 +185,39 @@ There are these types of CI:
     - publishes the image to github packages (only if the commit is tagged)
       or other OCI registry
 
-:services source:
+:Services Source:
     - prepares a helm Chart from each IOC instance or other service definition
     - tests that the helm Chart is deployable (but does not deploy it)
     - locally launches each IOC instance and loads its configuration to
       verify that the configuration is valid (no system tests because this
       would require talking to real hardware instances).
 
-:documentation source:
+:Documentation Source:
     - builds the sphinx docs that you are reading now
     - publishes it to github.io pages with version tag or branch tag.
 
-:global helm Chart source:
+:Global Helm Chart Source:
     - ``ec-helm-chars`` repo only
     - packages a helm Chart from source
     - publishes it to github packages (only if the commit is tagged)
       or other OCI registry
 ```
 
+### Continuous Deployment
+
+ArgoCD is a Kubernetes controller that continuously monitors running applications and compares their current state with the desired state described in a git repository. If the current state does not match the desired state, ArgoCD will attempt to reconcile the two.
+
+For this purpose each services repository will have a companion deployment repository which tracks which version of each IOC in the services repository should currently be deployed to the cluster. This list of IOC versions is in a single YAML file and updating this file and pushing it to the deployment repository will trigger ArgoCD to update the cluster accordingly.
+
+In this fashion changes to IOC versions are tracked in git and it is easy to roll back to the same state as a given date because there is a complete record.
+
 ## Scope
 
-This project initially targets x86_64 Linux Soft IOCs and RTEMS IOC running
-on MVME5500 hardware. Soft IOCs that require access to hardware on the
-server (e.g. USB or PCIe) will be supported by mounting the hardware into
-the container (these IOCS will not support Kubernetes failover).
+This project initially targets x86_64 Linux Soft IOCs and RTEMS 'hard' IOCs running on MVME5500 hardware. Soft IOCs that require access to hardware on the server (e.g. USB or PCIe) will be supported by mounting the hardware into the container (these IOCS will not support Kubernetes failover).
 
-Other linux architectures could be added to the Kubernetes cluster. We have
-tested arm64 native builds and will add this as a supported architecture
-in the future.
+Other linux architectures could be added to the Kubernetes cluster. We have tested arm64 native builds and will add this as a supported architecture in the future.
 
-Python soft IOCs are also supported.
+Python soft IOCs are also supported. See <https://github.com/DiamondLightSource/pythonSoftIOC>
 
 GUI generation for engineering screens is supported via the PVI project. See <https://github.com/epics-containers/pvi>.
 

docs/explanations/repositories.md

@@ -3,25 +3,24 @@
 :::{note}
 **DLS Users** DLS is currently using these locations for assets:
 
-- Generic IOC Source: `https://github.com/epics-containers<Generic ioc>`
-- Beamline Source repos: `https://gitlab.diamond.ac.uk/controls/containers/beamline/<beamline>`
-- Accelerator Source repos: `https://gitlab.diamond.ac.uk/controls/containers/accelerator/<domain>`
-- Generic IOC Container Images: `ghcr.io/epics-containers/<Generic ioc>`
-- epics-containers Helm Charts: `https://github.com/orgs/epics-containers/packages?repo_name=ec-helm-charts`
+- Public Generic IOC Source:   <https://github.com/epics-containers/>
+- Private Generic IOC Source:  <https://gitlab.diamond.ac.uk/controls/containers/iocs>
+- Beamline Source repos:       <https://gitlab.diamond.ac.uk/controls/containers/beamline/>
+- Accelerator Source repos:    <https://gitlab.diamond.ac.uk/controls/containers/accelerator/>
+- Generic IOC Container Images:<ghcr.io/epics-containers/>
+- epics-containers Helm Charts:<https://github.com/orgs/epics-containers/packages?repo_name=ec-helm-charts>
 :::
 
 ## Where to Keep Source Code
 
 There are two main kinds of source repositories used in epics-containers:
 
 - Generic IOC Source
-- Beamline / Accelerator Domain IOC Instance Source
+- Beamline / Accelerator Services Repositories for IOC instances and other services.
 
 ### Generic IOC Source Repositories
 
-For Generic IOCs it is recommended that these be stored in public repositories
-on GitHub.  This allows the community to benefit from the work of others and
-also contribute to the development of the IOC.
+For public Generic IOC container images, the GitHub Container Registry is a good choice. It allows the containers to live at a URL related to the source repository that generated them. The default ioc-template comes with Github Actions that build the container and push it to the GitHub Container Registry.
 
 The intention is that a Generic IOC container image is a reusable component
 that can be used by multiple IOC instances in multiple domains. Because
@@ -36,32 +35,24 @@ Integration files for Generic IOCs work with GitHub actions, but also
 can work with DLS's internal GitLab instance (this could be adapted for
 other facilities' internal GitLab instances or alternative CI system).
 
-### IOC Instance Domain Repositories
+### IOC Services Repositories
 
 These repositories are very much specific to a particular domain or beamline
 in a particular facility. For this reason there is no strong reason to make
 them public, other than to share with others how you are using epics-containers.
 
-At DLS we have a private GitLab instance and we store our domain Repositories
+At DLS we have a private GitLab instance and we store our Services Repositories
 there.
 
 The CI for domain repos works both with GitHub actions and with DLS's internal
 GitLab instance (this could be adapted for
 other facilities' internal GitLab instances or alternative CI system).
 
-### BL45P
+### p45-services
 
-The test/example beamline at DLS for epics-containers is BL45P.
-The domain repository for this
-is at <https://github.com/epics-containers/bl45p>. This will always be
-kept in a public repository as it is a live example of a domain repo.
+The test/example beamline at DLS for epics-containers is p45.
+The domain repository for this is at <https://github.com/epics-containers/p45-services>. This will always be kept in a public repository as it is a live example of a domain repo.
 
-## Where to put Registries
+This beamline is deployed to Kubernetes at DLS using Argo CD continuous deployment. The repository containing the Argo CD apps that control the deployment is at <https://github.com/epics-containers/p45-deployment>
 
-### Generic IOC Container Images and Source Repos
 
-Usually GHCR but internal supported for license e.g. Nexus Repository Manager
-
-### IOC Instance Domain Repos
-
-Internal git registry or private GitHub registry.

docs/how-to/builder2ibek.support.md

@@ -29,7 +29,7 @@ into ibek support YAML for the `ibek-support` repository.
 
 .. code:: yaml
 
-   # yaml-language-server: $schema=https://github.com/epics-containers/ibek/releases/download/1.2.0/ibek.support.schema.json
+   # yaml-language-server: $schema=https://github.com/epics-containers/ibek/releases/download/3.0.1/ibek.support.schema.json
 
    module: lakeshore340
 

docs/how-to/debug.md

@@ -26,7 +26,7 @@ Add the phrase 'deliberate_error' to the top of the `ioc.yaml` file. Then try to
 ec -v deploy-local services/bl01t-ea-test-02
 ```
 
-You should see something like this (for docker users - podman users will see something similar):
+You should see something like this (for docker users - docker users will see something similar):
 
 <pre>$ ec -v deploy-local services/bl01t-ea-test-02
 <font color="#5F8787">docker --version</font>

docs/how-to/own_tools.md

@@ -17,6 +17,6 @@ epics-containers has been tested with
 - vscode
 - Github Codespaces
 
-If you prefer console based editors like vim or emacs, then you will get the best results by launching the development containers defined in the epics-containers using the devcontainer CLI as described here <https://containers.dev/supporting#devcontainer-cli>.
+If you prefer console based editors like neovim or emacs, then you will get the best results by launching the development containers defined in the epics-containers using the devcontainer CLI as described here <https://containers.dev/supporting#devcontainer-cli>.
 
 In addition you could install your editor inside the developer container by adding an apt-install command into the `epics-containers` user personalization file. See [details here](https://github.com/epics-containers/epics-containers.github.io/blob/3a87e808e1c7983430a30c5a4dcd0d0661895d60/.devcontainer/postCreateCommand#L23-L27)