Merge branch 'k8s-tutorials' into argocd-install

Author: gilesknap

docs/tutorials.md

@@ -21,6 +21,7 @@ tutorials/debug_generic_ioc
 tutorials/support_module
 tutorials/setup_k8s
 tutorials/setup_k8s_new_beamline
+tutorials/add_k8s_ioc
 tutorials/rtems_setup
 tutorials/rtems_ioc
 ```

docs/tutorials/add_k8s_ioc.md

@@ -0,0 +1,9 @@
+# Add an IOC to the Kubernetes Beamline
+
+In this tutorial we will add an additional IOC to the Kubernetes Simulation Beamline created in the previous tutorial.
+
+This IOC will be a Simulation Area Detector IOC, very like the one we made in {any}`create_ioc`.
+
+Here we will also take a look at the helm chart and resulting Kubernetes Manifest that is created as part of this process.
+
+TODO: WIP

docs/tutorials/setup_k8s.md

@@ -12,35 +12,21 @@ For this reason DLS users should skip this tutorial unless you have a spare linu
 
 ## Introduction
 
-This is a very easy set of instructions for setting up an experimental
-single-node Kubernetes cluster, ready to test deployment of EPICS IOCs.
+This is a very easy set of instructions for setting up an experimental single-node Kubernetes cluster, ready for a test deployment of EPICS IOCs.
 
 ## Bring Your Own Cluster
 
 If you already have a Kubernetes cluster then you can skip this section.
 and go straight to the next tutorial.
 
-IMPORTANT: you will require appropriate permissions on the cluster to work
-with epics-containers. In particular you will need to be able to create
-pods that run with network=host. This is to allow Channel Access traffic
-to be routed to and from the IOCs. You will also need to be able to create
-a namespace and a service account, although you could use an existing
-namespace and service account as long as it has network=host capability.
+IMPORTANT: you will require appropriate permissions on the cluster to work with epics-containers. In particular you will need to be able to create pods that run with network=host. This is to allow Channel Access traffic to be routed to and from the IOCs. You will also need to be able to create a namespace and a service account, although you could use an existing namespace and service account as long as it has network=host capability. The alternative to running with network=host is to run a ca-gateway in the cluster and expose the PVs to the IOCs via the gateway.
 
 Cloud based K8S offerings may not be appropriate because of the Channel Access
 routing requirement.
 
 ## Platform Choice
 
-These instructions have been tested on the following platforms. The simplest
-option is to use a linux distribution that is supported by k3s.
-
-```{eval-rst}
-========================== ============================================
-Ubuntu 22.04 and newer     any modern linux distro should also work
-Raspberry Pi OS 2021-05-07 See `raspberry`
-========================== ============================================
-```
+These instructions have been tested on Ubuntu 22.04; however, any modern Linux distribution that is supported by k3s and running on a modern x86 machine should also work.
 
 Note that K3S provides a good uninstaller that will clean up your system if you decide to back out. So there is no harm in trying it out.
 
@@ -99,7 +85,7 @@ mkdir ~/.kube
 sudo scp  /etc/rancher/k3s/k3s.yaml <YOUR_ACCOUNT>@<YOUR_WORKSTATION>:.kube/config
 ```
 
-If you do have separate workstation then edit the file .kube/config replacing 127.0.0.1 with your server's IP Address. For a single machine the file is leftas is.
+If you do have separate workstation then edit the file .kube/config replacing 127.0.0.1 with your server's IP Address. For a single machine the file is left as is.
 
 ### Install helm
 
@@ -168,7 +154,25 @@ To install the `argocd` cli tool:
 ```
 curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
 sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
-rm argocd-linux-amd64
+rm argocd-linux-amd64 
+```
+
+### Install persistent volume support
+
+As per <https://docs.k3s.io/storage/>, the "Longhorn" distributed block storage system can be set up in our cluster. This is done in order to get support for ReadWriteMany persistent volume claims, which is not supported by the out of the box "Local Path Provisioner".
+
+```bash
+# Install dependancies
+sudo apt-get update; sudo apt-get install -y open-iscsi nfs-common jq
+
+# Set up longhorn
+kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.0/deploy/longhorn.yaml
+
+# Monitor while Longhorn starts up
+kubectl get pods --namespace longhorn-system --watch
+
+# Confirm ready
+kubectl get storageclass
 ```
 
 ### Completed

docs/tutorials/setup_k8s_new_beamline.md

@@ -8,9 +8,9 @@ Helm is a package manager for Kubernetes that allows you to define a set of reso
 
 Previously our beamline repository contained a **services** folder.  Each subfolder of **services** contained a **compose.yaml** with details of the generic IOC container image, plus a **config** folder that provided an IOC instance definition.
 
-In the Kubernetes world, each folder under **services** will be an individually deployable Helm Chart. This means that instead of a **compose.yaml** file we will have a **Chart.yaml** which describes the dependencies of the chart and a **values.yaml** that describes some arguments to it. There is also a file **services/values.yaml** that describes the default arguments for all the charts in the repository.
+In the Kubernetes world the structure is very similar. Each folder under **services** will be an individually deployable Helm Chart. This means that instead of a **compose.yaml** file we will have a **Chart.yaml** which describes the dependencies of the chart and a **values.yaml** that describes some arguments to it. There is also a file **services/values.yaml** that describes the default arguments for all the charts in the repository.
 
-In this tutorial we will create a new beamline in a Kubernetes cluster. Here we assume that the cluster is already setup and that there is a namespace configured for use by the beamline. See the previous tutorial for how to set one up if you do not have this already.
+In this tutorial we will create a new simulation beamline in a Kubernetes cluster. Here we assume that the cluster is already setup and that there is a namespace configured for use by the beamline. See the previous tutorial for how to set one up if you do not have this already.
 
 :::{note}
 DLS users: you should use your personal namespace in the test cluster **Pollux**. Your personal namespace is named after your *fedid*
@@ -20,12 +20,11 @@ DLS users: you should use your personal namespace in the test cluster **Pollux**
 
 As before, we will use a copier template to create the new beamline repository. The steps are similar to the first tutorial {any}`create_beamline`.
 
-1. We are going to call the new beamline **bl03t** with the repository name **t03-services**. It will be created in the namespace **bl03t** on the local cluster that we created in the last tutorial OR your *fedid* namespace on the **Pollux** cluster if you are using the DLS cluster.
+1. We are going to call the new beamline **bl03t** with the repository name **t03-services**. It will be created in the namespace **t03-beamline** on the local cluster that we created in the last tutorial **OR** your *fedid* namespace on the **Pollux** cluster if you are using the DLS cluster.
 
     ```bash
     # make sure your Python virtual environment is active and copier is pip installed
     copier copy gh:epics-containers/services-template-helm t03-services
-    code t03-services
     ```
 
     Answer the copier template questions as follows for your own local cluster:
@@ -101,12 +100,12 @@ If you have brought your own cluster then you may need to edit the **environment
 
 ## Setup the epics containers CLI
 
-To deploy and manage IOC istances requires helm and kubectl command line tools. However we supply a simple wrapper for these tools that saves typing and helps with learning the commands. This is the `ec` command line tool. Go ahead and add the `ec` python package to your virtual environment.
+To deploy and manage IOC istances requires **helm** and **kubectl** command line tools. However we supply a simple wrapper for these tools that saves typing and helps with learning the commands. Go ahead and add the `ec-cli` python package to your virtual environment.
 
 ```bash
 # make sure your Python virtual environment is active, then:
 pip install ec-cli
-# make sure ec is not currently aliased to docker compose
+# make sure ec is not currently aliased to docker compose! (maybe that was a bad idea?)
 unalias ec
 # setup the environment for ec to know how to talk to the cluster
 # (make sure you are in the t03-services directory)
@@ -115,24 +114,23 @@ source ./environment.sh
 
 ## Deploy an Example IOC Instance
 
-The new repository has an example IOC that it comes with the template and is called t03-ea-test-01. It is a simple example IOC that is used for testing the deployment of IOCs to the cluster.
+The new repository has a simple example IOC that it comes with the template and is called t03-ea-test-01.
 
 For a new beamline we will also need to deploy the shared resources that all IOCs expect to find in their namespace, namely:
 - epics-pvcs: some persistent volumes (Kubernetes data stores) for the IOCs to store autosave files, GUI files and other data
 - epics-opis: an nginx web server that serves the IOC GUI files out of one of the above persistent volumes
 
-The ec tool can help with version tracking by deploying tagged version of services. So first lets go ahead and tag the current state of the repository.
+The ec tool can help with version tracking by deploying version of services from tagged commits in the git repo. So first lets go ahead and tag the current state of the repository.
 
 ```bash
 # make sure you are in the t03-services directory, then:
 git tag 2024.9.1
 git push origin 2024.9.1
 ```
 
-Now you can deploy the shared resources and the example IOC instance to the cluster. Using the version we just tagged. We will use the -v option which shows you the underlying commands that are being run.
+Now you can deploy the shared resources to the cluster, using the version we just tagged. We will use the -v option which shows the underlying commands that are being run.
 
 ```bash
-source environment.sh
 ec -v deploy epics-pvcs 2024.9.1
 ec -v deploy epic-opis 2024.9.1
 ```
@@ -152,10 +150,32 @@ ec ps
 You could also investigate the other commands that `ec` provides by running `ec --help`.
 
 :::{note}
+When things are not working as expected or you want to examine the resources you are deploying, you can use the `kubectl describe` command. If you prefer a more interactive approach, then look at the Kubernetes Dashboard.
+
+For a k3s local cluster refer to the notes on installing the dashboard in the previous tutorial. TODO add link when available.
+
 At DLS you can get to a Kubernetes Dashboard for your beamline via a landing page `https://pollux.diamond.ac.uk` for test beamlines on `Pollux` - remember to select the namespace from the dropdown in the top left.
 
 For production beamlines with dedicated clusters, you can find the landing page for example:
 `https://k8s-i22.diamond.ac.uk/` for BL22I.
-`https://k8s-b01-1.diamond.ac.uk/` for the 2nd branch of BL01B.
-in this case the namespace will be ixx-beamline.
+`https://k8s-b01-1.diamond.ac.uk/` for the 2nd branch of BL01C.
+in this case the namespace will be i22-beamline, b01-1-beamline, etc.
 :::
+
+## Verify that the IOC is working
+
+Right now you cannot see PVs from your IOC because it is running in a container network and channel access clients won't be able to contact it.
+
+For k3s users you can simply fix this by setting 'hostNetwork: true' in **services/values.yaml**. Then re-deploy the IOC instance (by pushing the change and making a new tag).
+
+DLS users do not have permission to run host network in their personal namespaces.
+
+The best solution is to use a channel access gateway to bridge the container network to the host network. We will do this in a later tutorial.
+
+For now you can check you IOC by launching a shell inside it's container and using the `caget` command. All IOC containers have the epics-base tools installed. Try the following commands to confirm that the IOC is running and that the PVs are accessible.
+
+```bash
+$ ec exec t03-ea-test-01
+root@t03-ea-test-01-0:/# caget T03:IBEK:A
+T03:IBEK:A                     2.54
+```