Merge pull request #26 from epics-containers/dev

Added first pass of RTEMS tutorials.

Author: gilesknap

.github/CONTRIBUTING.rst

@@ -26,4 +26,4 @@ The `Developer Guide`_ contains information on setting up a development
 environment, building docs and what standards the documentation
 should follow.
 
-.. _Developer Guide: https://diamondlightsource.github.io/epics-containers.github.io/main/developer/how-to/contribute.html
+.. _Developer Guide: https://epics-containers.github.io/main/developer/how-to/contribute.html

docs/user/index.rst

@@ -25,8 +25,8 @@ side-bar.
             tutorials/debug_generic_ioc
             tutorials/test_generic_ioc
             tutorials/support_module
-
-        +++
+            tutorials/rtems_setup
+            tutorials/rtems_ioc
 
         Tutorials for installation and typical usage. New users start here.
 

docs/user/tutorials/epics_devcontainer.rst

@@ -5,10 +5,13 @@ The EPICS Devcontainer
 Introduction
 ------------
 
-You can setup a single devcontainer for managing all of your IOCs. In
-`devcontainer` we launched a devcontainer for a single project. Here we
-will create a workspace that is managed by a devcontainer. This will allow
-you to manage multiple projects in a single devcontainer.
+For working with epics-containers we provide a developer container with
+all the tools you need to build and deploy EPICS IOCs already installed.
+In this tutorial we will install and configure this devcontainer.
+
+In `devcontainer` we demonstrated launching a container for a single project.
+Here we will create a workspace that will allow
+you to manage multiple projects within a single devcontainer.
 
 The base container is defined in https://github.com/epics-containers/dev-e7
 but we will use a customizable image derived from that. The customizable

docs/user/tutorials/rtems_ioc.rst

@@ -0,0 +1,214 @@
+RTEMS - Deploying an Example IOC
+================================
+
+The previous tutorials walked through how to create a generic linux soft
+IOC and how to deploy an IOC instance using that generic IOC.
+
+epics-containers also supports RTEMS 5 running on MVVME5500. We will
+now will look at the differences for this architecture. Further
+architectures will be supported in future.
+
+Each beamline or accelerator domain will require a server for
+serving the IOC binaries and instance files to the RTEMS devices. This
+needs to be set up for your test beamline before proceeding,
+see `rtems_setup`.
+
+Once you have the file server set up, deploying an IOC instance that uses
+an RTEMS Generic IOC is very similar to `deploy_example`.
+
+We will be adding
+a new IOC instance to the ``bl01t`` beamline that we created in the previous
+tutorials. You will need to have worked through the previous tutorials in
+order to complete this one.
+
+Preparing the RTEMS Boot loader
+-------------------------------
+
+To try this tutorial you will need a VME crate with an MVVME5500 processor card
+installed. You will also need access to the serial console over ethernet
+using a terminal server or similar.
+
+.. note::
+
+    **DLS Users** for details of setting up RTEMS on your VME crates see
+    this `internal link <https://confluence.diamond.ac.uk/pages/viewpage.action?spaceKey=CNTRLS&title=RTEMS>`_
+
+    The following crate is already running RTEMS and can be used for this
+    tutorial, but check with the accelerator controls team before using it:
+
+    :console: ts0001 7007
+    :crate monitor: ts0001 7008
+
+    It is likely already set up as per the example below.
+
+Use telnet to connect to the console of your target IOC. e.g.
+``telnet ts0001 7007``. We want to get to the MOTLoad prompt which should look
+like ``MVME5500>``. If you see an IOC Shell prompt instead hit ``Ctrl-D`` to
+exit and then ``Esc`` when you see
+``Boot Script - Press <ESC> to Bypass, <SPC> to Continue``
+
+Now you want to set the boot script to load the IOC binary from the network via
+TFTP and mount the instance files from the network via NFS. The command
+``gevShow`` will show you the current state of the global environment variables.
+e.g.
+
+.. code-block::
+
+    MVME5500> gevShow
+    mot-/dev/enet0-cipa=172.23.250.15
+    mot-/dev/enet0-snma=255.255.240.0
+    mot-/dev/enet0-gipa=172.23.240.254
+    mot-boot-device=/dev/em1
+    rtems-client-name=bl01t-ea-ioc-02
+    epics-script=172.23.168.203:/iocs:bl01t/bl01t-ea-ioc-02/config/st.cmd
+    mot-script-boot
+    dla=malloc 0x230000
+    tftpGet -d/dev/enet1 -fbl01t/bl01t-ea-ioc-02/bin/RTEMS-beatnik/ioc.boot -m255.255.240.0 -g172.23.240.254 -s172.23.168.203 -c172.23.250.15 -adla
+    go -a0095F000
+
+    Total Number of GE Variables =7, Bytes Utilized =427, Bytes Free =3165
+
+Now use ``gevEdit`` to change the global variables to the values you need.
+For this tutorial we will create an IOC called bl01t-ea-ioc-02 and for the
+example we assume the file server is on 172.23.168.203. For the details of
+setting up these parameters see your site documentation but the important
+values to change for this tutorial IOC would be:
+
+:rtems-client-name: bl01t-ea-ioc-02
+:epics-script: 172.23.168.203:/iocs:bl01t/bl01t-ea-ioc-02/config/st.cmd
+:mot-script-boot (2nd line): tftpGet -d/dev/enet1 -fbl01t/bl01t-ea-ioc-02/bin/RTEMS-beatnik/ioc.boot -m255.255.240.0 -g172.23.240.254 -s172.23.168.203 -c172.23.250.15 -adla
+
+Now your ``gevShow`` should look similar to the example above.
+
+Meaning of the parameters:
+
+:rtems-client-name: a name for the IOC crate
+:epics-script: an NFS address for the IOC's root folder
+:mot-script-boot: a TFTP address for the IOC's binary boot file
+
+Note that the IP parameters to the tftpGet command are respectively:
+net mask, gateway, server address, client address.
+
+
+Creating an RTEMS IOC Instance
+------------------------------
+
+We will be adding a new IOC instance to the ``bl01t`` beamline that we created in
+:doc:`create_beamline`. The first step is to make a copy of our existing IOC instance
+and make some modifications to it. We will call this new IOC instance
+``bl01t-ea-ioc-02``.
+
+.. code-block:: bash
+
+    cd bl01t
+    cp -r iocs/bl01t-ea-ioc-01 iocs/bl01t-ea-ioc-02
+    # don't need this file for the new IOC
+    rm iocs/bl01t-ea-ioc-02/config/extra.db
+
+We are going to make a very basic IOC with some hand coded database with
+a couple of simple records. Therefore the generic IOC that we use can just
+be ioc-template.
+
+Generic IOCs have multiple targets, they always have a
+``developer`` target which is used for building and debugging the generic IOC and
+a ``runtime`` target which is lightweight and usually used when running the IOC
+in the cluster. The matrix of targets also includes an architecture dimension,
+at present the ioc-template supports two architectures, ``linux`` and
+``rtems``, thus there are 4 targets in total as follows:
+
+- ghcr.io/epics-containers/ioc-template-linux-runtime
+- ghcr.io/epics-containers/ioc-template-linux-developer
+- ghcr.io/epics-containers/ioc-template-rtems-runtime
+- ghcr.io/epics-containers/ioc-template-rtems-developer
+
+We want to run the RTEMS runtime target on the cluster so this will appear
+at the top of the ``values.yaml`` file. In addition there are a number of
+environment variables required for the RTEMS target that we also specify in
+``values.yaml``.
+Edit the file
+``iocs/bl01t-ea-ioc-02/values.yaml`` to look like this:
+
+.. code-block:: yaml
+
+If you are not at DLS you will need to change the above to match the
+parameters of your RTEMS IOC. The environment variables are:
+
+:K8S_IOC_ADDRESS: The IP address of the IOC (mot-/dev/enet0-cipa above)
+:RTEMS_VME_CONSOLE_ADDR: Address of terminal server for console access
+:RTEMS_VME_CONSOLE_PORT: Port of terminal server for console access
+:RTEMS_VME_AUTO_REBOOT: true to reboot the hard IOC when the IOC container changes
+:RTEMS_VME_AUTO_PAUSE: true to pause/unpause when the IOC container stops/starts
+
+Edit the file ``iocs/bl01t-ea-ioc-02/Chart.yaml`` and change the 1st 4 lines
+to represent this new IOC (the rest of the file is boilerplate):
+
+.. code-block:: yaml
+
+    apiVersion: v2
+    name: bl01t-ea-ioc-02
+    description: |
+        example RTEMS IOC for bl01t
+
+For configuration we will create a simple database with a few of records and
+a basic startup script. Add the following files to the
+``iocs/bl01t-ea-ioc-02/config`` directory.
+
+.. code-block::  :caption: bl01t-ea-ioc-02.db
+
+    record(calc, "bl01t-ea-ioc-02:SUM") {
+        field(DESC, "Sum A and B")
+        field(CALC, "A+B")
+        field(SCAN, ".1 second")
+        field(INPA, "bl01t-ea-ioc-02:A")
+        field(INPB, "bl01t-ea-ioc-02:B")
+    }
+
+    record(ao, "bl01t-ea-ioc-02:A") {
+        field(DESC, "A voltage")
+        field(EGU,  "Volts")
+        field(VAL,  "0.0")
+    }
+
+    record(ao, "bl01t-ea-ioc-02:B") {
+        field(DESC, "B voltage")
+        field(EGU,  "Volts")
+        field(VAL,  "0.0")
+    }
+
+.. code-block::  :caption: st.cmd
+
+    # RTEMS Test IOC bl01t-ea-ioc-02
+
+    dbLoadDatabase "/iocs/bl01t/bl01t-ea-ioc-02/dbd/ioc.dbd"
+    ioc_registerRecordDeviceDriver(pdbbase)
+
+    # db files from the support modules are all held in this folder
+    epicsEnvSet(EPICS_DB_INCLUDE_PATH, "/iocs/bl01t/bl01t-ea-ioc-02/support/db")
+
+    # load our hand crafted database
+    dbLoadRecords("/iocs/bl01t/bl01t-ea-ioc-02/config/bl01t-ea-ioc-02.db")
+    # also make Database records for DEVIOCSTATS
+    dbLoadRecords(iocAdminSoft.db, "IOC=bl01t-ea-ioc-02")
+    dbLoadRecords(iocAdminScanMon.db, "IOC=bl01t-ea-ioc-02")
+
+    iocInit
+
+You now have a new helm chart in iocs/bl01t-ea-ioc-02 that describes an IOC
+instance for your RTEMS device. Recall that this is not literally where the IOC
+runs, it deploys a kubernetes pod that manages the RTEMS IOC. It does contain
+the IOC's configuration and the IOC's binary code, which it will copy to the
+file-server on startup.
+
+You are now ready to deploy the IOC instance to the cluster and test it out.
+
+
+Deploying an RTEMS IOC Instance
+-------------------------------
+
+TODO:
+
+Once you have the correct configuration in your RTEMS boot-loader and you have
+deployed the kubernetes IOC instance, you can restart the IOC with
+the ``reset`` command. This will cause it to reboot and it should pick
+up your binary from the network and start the IOC. You should see the
+iocShell fire up and run

docs/user/tutorials/rtems_setup.rst

@@ -0,0 +1,94 @@
+RTEMS - Creating a File Server
+==============================
+
+Introduction
+------------
+
+RTEMS IOCs are an example of a 'hard' IOC. Each IOC is a physical crate that
+contains a number of I/O cards and a processor card.
+
+For these types of
+IOC, the Kubernetes cluster runs a pod that represents the individual IOC.
+However, the IOC code itself runs on the processor card instead of the pod.
+The pod provides the following features:
+
+- Sets up the files to serve to the RTEMS OS
+- Provides a connection to the IOC console just like a linux IOC
+- Pauses, unpauses, restarts the IOC as necessary - thus the IOC is controlled
+  by the Kubernetes cluster in the same way as a linux IOC
+- Provides logging of the IOC console in the same way as linux IOCs
+- Monitors the IOC and restarts it if it crashes - using the same mechanism
+  as linux IOCs
+
+At present epics-containers supports the MVVME5500 processor card running
+RTEMS 5. The same model as described above can be used for other 'hard' IOC
+types in future.
+
+Create a File Server Service
+----------------------------
+
+When an RTEMS 5 IOC boots the bootloader loads the IOC binary from a TFTP
+address, this binary is then given access to a filesystem over NFS V2, this is
+where the IOC startup script and other configuration is loaded.
+
+Therefore we need a TFTP server and an NFS V2 server to serve the files to
+the IOC. For each EPICS domain a single service running in Kubernetes will
+supply a TFTP and NFS V2 server for all the IOCs in that domain.
+
+In the tutorial :doc:`create_beamline` we created a beamline repository that
+defines the IOC instances in the beamline ``bl01t``. The template project
+that we copied contains a folder called ``services/nfsv2-tftp``. The folder
+is a helm chart that will deploy a TFTP and NFS V2 server to Kubernetes.
+
+Before deploying the service we need to configure it. Make the following
+changes:
+
+- Change the ``name`` value in ``Chart.yaml`` to ``bl01t-ioc-files``
+- Change the ``loadBalancerIP`` value in ``values.yaml`` to a free IP address
+  in your cluster's Static Load Balancer range. This IP address will be used
+  to access the TFTP and NFS V2 servers from the IOC.
+
+.. note::
+
+  **DLS Users** The load balancer IP range on Pollux is
+  ``172.23.168.201-172.23.168.222``. Please use ``172.23.168.203``. The test
+  RTEMS crate is likely to already be set up to point at this address. There
+  are a limited number of addresses available, hence we have reserved a single
+  address for the training purposes.
+
+  Also note that ``bl01t`` is a shared resource so if there is already a
+  ``bl01t-ioc-files`` service running then you could just use the existing
+  service.
+
+You can verify if the service is already running using kubectl. The command
+shown below will list all the services in the ``bl01t`` namespace, and the
+example output shows that there is already a ``bl01t-ioc-files`` service
+using the IP address ``172.23.168.203``.
+
+.. code-block:: bash
+
+  $ kubectl get services -n bl01t
+  NAME                TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                                                                        AGE
+  bl01t-ioc-files     LoadBalancer   10.108.219.193   172.23.168.203   111:31491/UDP,2049:30944/UDP,20048:32277/UDP,69:32740/UDP                      32d
+
+Once you have made the changes to the helm chart you can deploy it to the
+cluster using the following command:
+
+.. code-block:: bash
+
+  cd bl01t
+  helm upgrade --install bl01t-ioc-files services/nfsv2-tftp -n bl01t
+
+Now if you run the ``kubectl get services`` command again you should see the
+new service.
+
+Once you have this service up and running you can leave it alone. It will
+serve the files to the IOCs using the IP address you configured over both
+TFTP and NFS V2. It uses a persistent volume to store the files and this
+persistent volume is shared with hard IOC pods so that they can place the
+files they need to serve to the IOC.
+
+See the next tutorial for how to deploy a hard IOC pod to the cluster.
+
+
+