Skip to content

Commit 102d075

Browse files
gilesknapgilesknap
authored
Merge pull request #16 from epics-containers/dev
Adding Tutorial 07
2 parents 0f81100 + c79cf0c commit 102d075

File tree

1 file changed

+351
-4
lines changed

1 file changed

+351
-4
lines changed

‎docs/user/tutorials/07_generic_ioc.rst

Lines changed: 351 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,355 @@
11
Working with Generic IOCs
22
=========================
33

4-
In this tutorial we will learn how modify a generic IOC container image and
5-
test our changes locally before updating an IOC instance and deploying it.
4+
In this tutorial we will learn how to create a generic IOC container image and
5+
test our changes locally before deploying it.
66

7-
TODO: cover creating a ADUrl based Area detector generic IOC based
8-
on the ADSimDetector one. (point at a public web cam)
7+
The example IOC used ADSimDetector, we will make a similar IOC that uses
8+
ADUrl to get images from a web cam.
9+
10+
Create a New Generic IOC project
11+
--------------------------------
12+
13+
By convention generic IOC projects are named ``ioc-XXX`` where ``XXX`` is the
14+
name of the primary support module used by the IOC. Here we will be building
15+
``ioc-adurl``.
16+
17+
Much like creating a new beamline we have a template project that can be used
18+
as the starting point for a new generic IOC. Again we will create this in
19+
your personal GitHub user space.
20+
21+
TODO: the following steps to create a new generic IOC project will be automated
22+
using an ``ec`` command.
23+
24+
#. Create a new, completely blank repository in your GitHub account
25+
called ``ioc-adurl``. To do this got to https://github.com/new
26+
and fill in the details as per the image below. Click
27+
``Create repository``.
28+
29+
#. Clone the template repo locally and rename from ioc-template to ioc-adurl
30+
31+
.. code-block:: bash
32+
33+
git clone [email protected]:epics-containers/ioc-template.git
34+
mv ioc-template ioc-adurl
35+
cd ioc-adurl
36+
37+
#. Add your new repo to your VSCode workspace and take a look at what you
38+
have.
39+
40+
From the VSCode menus: File->Add Folder to Workspace
41+
then select the folder ioc-adurl
42+
43+
#. Push the new repo back to a the new repo on github
44+
45+
.. code-block:: bash
46+
47+
git remote rm origin
48+
git remote add origin [email protected]:<YOUR USER NAME>/ioc-adurl.git
49+
git push origin main
50+
51+
Configure the ibek-defs Submodule
52+
---------------------------------
53+
54+
The ``ibek-defs`` submodule is used to share information about how to build
55+
support modules. It contains two kinds of files:
56+
57+
#. Patch Files: these are used to update a support module so that it will
58+
build correctly in the container environment. These should typically only
59+
add one or both of these files:
60+
61+
- configure/RELEASE.local
62+
- configure/CONFIG_SITE.linux-x86_64.Common
63+
64+
#. IBEK support module definitions: These are used to help IOCs build their
65+
iocShell boot scripts from YAML descriptions. They are not used in this
66+
tutorial as we are supplying a hand crafted boot script. For information
67+
on IBEK see https://github.com/epics-containers/ibek.
68+
69+
ibek-defs is curated for security reasons, therefore we need to work with
70+
a fork of it so we can add our own definitions for ADUrl. If you make changes
71+
to ibek-defs that are generally useful you can use a pull request to get them
72+
merged into the main repo.
73+
74+
Perform the following steps to create a fork and update the submodule:
75+
76+
- goto https://github.com/epics-containers/ibek-defs/fork
77+
- uncheck ``Copy the main branch only``
78+
- click ``Create Fork``
79+
- click on ``<> Code`` and COPY the ssh URL
80+
- cd to the ioc-adurl directory
81+
-
82+
.. code-block:: bash
83+
84+
git submodule set-url ibek-defs <PASTE URL HERE>
85+
git submodule init
86+
git submodule update
87+
cd ibek-defs
88+
git checkout tutorial
89+
cd ..
90+
91+
We are using the ``tutorial`` branch which has a snapshot of the ibek-defs state
92+
appropriate for this tutorial. Normally you would use the ``main`` branch and
93+
therefore omit ``git checkout tutorial``.
94+
95+
The git submodule allows us to share the ibek-defs definitions between all
96+
ioc-XXX projects but also allows each project to have its copy fixed to
97+
a particular commit (until updated with ``git pull``) see
98+
https://git-scm.com/book/en/v2/Git-Tools-Submodules for more information.
99+
100+
101+
Modify the Dockerfile
102+
---------------------
103+
104+
The heart of every ioc-XXX project is the Dockerfile. This is a text file
105+
that contains a set of instructions that are used to build a container image.
106+
See https://docs.docker.com/engine/reference/builder/ for details of how
107+
to make Dockerfiles.
108+
109+
All ioc-XXX projects will have the same pattern of Dockerfile instructions
110+
and will all be based upon the epics base images named:
111+
112+
- ghcr.io/epics-containers/epics-base-<ARCH>-<TARGET>
113+
114+
Where ARCH is currently ``linux`` or ``rtems`` and TARGET will always be ``developer``
115+
and ``runtime``. Support for further architectures will be added in the future.
116+
117+
The ``developer`` image contains all the tools needed to build support modules
118+
and is used for building and debugging the generic IOC. The ``runtime`` image
119+
is a minimal image that holds the minimum required to run the generic IOC.
120+
121+
The changes we will make to the template Dockerfile are as follows:
122+
123+
Add more support modules
124+
~~~~~~~~~~~~~~~~~~~~~~~~
125+
126+
After the make of ``busy`` add 3 more support module fetch and make steps
127+
like this:
128+
129+
.. code-block:: dockerfile
130+
131+
COPY ibek-defs/adsupport/ /ctools/adsupport/
132+
RUN python3 modules.py install ADSUPPORT R1-10 github.com/areaDetector/adsupport.git --patch adsupport/adsupport.sh
133+
RUN make -C ${SUPPORT}/adsupport -j $(nproc)
134+
135+
COPY ibek-defs/adcore/ /ctools/adcore/
136+
RUN python3 modules.py install ADCORE R3-12-1 github.com/areaDetector/adcore.git --patch adcore/adcore.sh
137+
RUN make -C ${SUPPORT}/adcore -j $(nproc)
138+
139+
COPY ibek-defs/adurl/ /ctools/adurl/
140+
RUN python3 modules.py install ADURL R2-3 github.com/areaDetector/adurl.git --patch adurl/adurl.sh
141+
RUN make -C ${SUPPORT}/adurl -j $(nproc)
142+
143+
This instructs the build to fetch the support module source code from GitHub
144+
for ADURL and its two dependencies ADSUPPORT and ADCORE. It also makes each
145+
module after fetching.
146+
147+
.. note::
148+
149+
You may think that there is a lot of duplication here but this is explicitly
150+
done to make the build cache more efficient and speed up development.
151+
For example we could copy everything out of the ibek-defs directory
152+
in a single command but then if I changed the ADURL patch file the
153+
build would have to re-fetch and re-make all the support modules.
154+
155+
Add System Dependencies
156+
~~~~~~~~~~~~~~~~~~~~~~~
157+
158+
If you tried to build the container image at this point you would find that
159+
it is missing the boost libraries which are required by areaDetector. You
160+
can use ``apt`` to install anything you need inside the container. Replace
161+
the commented out ``apt-get`` lines with:
162+
163+
.. code-block:: dockerfile
164+
165+
RUN apt-get update && apt-get upgrade -y && \
166+
apt-get install -y --no-install-recommends \
167+
libboost-all-dev
168+
169+
Add ibek-defs Patch file for ADURL
170+
----------------------------------
171+
172+
In the above we referred to a patch file for ADURL. Add this in the ``ibek-defs``
173+
folder by creating directory called ``ibek-defs/adurl`` and adding a file called
174+
``adurl.sh`` with the following contents:
175+
176+
.. code-block:: bash
177+
178+
#!/bin/bash
179+
180+
echo '
181+
CROSS_COMPILER_TARGET_ARCHS =
182+
183+
# Enable file plugins and source them all from ADSupport
184+
185+
WITH_GRAPHICSMAGICK = YES
186+
GRAPHICSMAGICK_EXTERNAL = NO
187+
188+
WITH_JPEG = YES
189+
JPEG_EXTERNAL = NO
190+
191+
WITH_PVA = YES
192+
WITH_BOOST = YES
193+
' > configure/CONFIG_SITE.linux-x86_64.Common
194+
195+
echo '
196+
# Generic RELEASE.local file that should work for all Support modules and IOCs
197+
198+
SUPPORT=NotYetSet
199+
AREA_DETECTOR=$(SUPPORT)
200+
include $(SUPPORT)/configure/RELEASE
201+
' > configure/RELEASE.local
202+
203+
This is a pretty standard patch file and most support modules will need
204+
something similar.
205+
It creates two files in the ADURL support module's configure folder as
206+
follows:
207+
208+
- ``CONFIG_SITE.linux-x86_64.Common`` - This tells the ADURL build
209+
to use the GraphicsMagick and JPEG libraries that are built by ADSUPPORT.
210+
For details of what to put in CONFIG_SITE for AreaDetector modules see
211+
`CONFIG_SITE.local`_.
212+
- ``RELEASE.local`` - This tells the ADURL build where to find
213+
the support modules that it depends on. epics-containers maintains a
214+
global release file that is used by all support modules and IOCs. It
215+
located at ``/repos/epics/support/configure/RELEASE``. Therefore we
216+
place a reference to this file in the RELEASE.local file. Whenever
217+
``python3 modules.py install`` is run it will update the global release
218+
file and also fixup any ``SUPPORT=`` lines in all ``configure/RELEASE*``
219+
files.
220+
221+
ADCore and ADSupport already have ibek-defs files as they were previously created
222+
when making ``ioc-adsimdetector``.
223+
224+
.. _CONFIG_SITE.local: https://areadetector.github.io/areaDetector/install_guide.html#edit-config-site-local-and-optionally-config-site-local-epics-host-arch
225+
226+
Update the IOC Makefile
227+
-----------------------
228+
229+
The IOC Makefile tells the IOC which modules to link against. We need to update
230+
it to pull in ADUrl and dependencies. Replace the file ``ioc/iocApp/src/Makefile``
231+
with the following:
232+
233+
.. code-block:: makefile
234+
235+
TOP = ../..
236+
include $(TOP)/configure/CONFIG
237+
238+
PROD_IOC = ioc
239+
DBD += ioc.dbd
240+
ioc_DBD += base.dbd
241+
ioc_DBD += devIocStats.dbd
242+
ioc_DBD += asyn.dbd
243+
ioc_DBD += busySupport.dbd
244+
ioc_DBD += ADSupport.dbd
245+
ioc_DBD += NDPluginSupport.dbd
246+
ioc_DBD += NDFileHDF5.dbd
247+
ioc_DBD += NDFileJPEG.dbd
248+
ioc_DBD += NDFileTIFF.dbd
249+
ioc_DBD += NDFileNull.dbd
250+
ioc_DBD += NDPosPlugin.dbd
251+
ioc_DBD += URLDriverSupport.dbd
252+
ioc_DBD += PVAServerRegister.dbd
253+
ioc_DBD += NDPluginPva.dbd
254+
255+
ioc_SRCS += ioc_registerRecordDeviceDriver.cpp
256+
257+
ioc_LIBS += ntndArrayConverter
258+
ioc_LIBS += nt
259+
ioc_LIBS += pvData
260+
ioc_LIBS += pvDatabase
261+
ioc_LIBS += pvAccessCA
262+
ioc_LIBS += pvAccessIOC
263+
ioc_LIBS += pvAccess
264+
ioc_LIBS += URLDriver
265+
ioc_LIBS += NDPlugin
266+
ioc_LIBS += ADBase
267+
ioc_LIBS += cbfad
268+
ioc_LIBS += busy
269+
ioc_LIBS += asyn
270+
ioc_LIBS += devIocStats
271+
ioc_LIBS += $(EPICS_BASE_IOC_LIBS)
272+
ioc_SRCS += iocMain.cpp
273+
274+
include $(TOP)/configure/RULES
275+
276+
TODO: in future the IBEK tool will generate the Makefile for you based on the
277+
ibek support YAML supplied with each module in ibek-defs.
278+
279+
280+
Build the Generic IOC
281+
---------------------
282+
283+
Now we can build the IOC. Run the following command from the ioc-adurl
284+
directory:
285+
286+
.. code-block:: bash
287+
288+
ec dev build
289+
290+
.. warning::
291+
292+
This will FAIL. There is a deliberate error which we will fix in the next
293+
step.
294+
295+
You should see this error::
296+
297+
../URLDriver.cpp:22:10: fatal error: Magick++.h: No such file or directory
298+
299+
Investigate the Build Failure
300+
-----------------------------
301+
302+
When a container build fails the container image is created up to the point
303+
where the last successful Dockerfile command was run. This means that we can
304+
investigate the build failure by running a shell in the container. ``ec``
305+
provides us with the following convenience command to do this:
306+
307+
.. code-block:: bash
308+
309+
ec dev debug-last
310+
311+
Now we have a prompt inside the part build container and can retry the failed
312+
command.
313+
314+
.. code-block:: bash
315+
316+
cd /repos/epics/support/adurl
317+
make
318+
319+
You should see the same error again.
320+
321+
322+
323+
324+
.. git submodule init
325+
.. git submodule update
326+
.. cd ibek ibek-defs TODO - do they need a fork of this??
327+
.. checkout main
328+
.. push --set-upstream origin main
329+
.. mkdir adurl
330+
331+
.. ec dev build
332+
333+
334+
.. copy steps from ADSimDetector
335+
.. copy makefile from ADSimDetector/ioc/iocApp/Makefile
336+
337+
.. Update this but discuss how we could have changed ADSupport to build GraphicsMagick
338+
.. configure/CONFIG_SITE.linux-x86_64.Common
339+
.. WITH_GRAPHICSMAGICK = YES
340+
.. GRAPHICSMAGICK_INCLUDE=/usr/include/GraphicsMagick
341+
342+
.. # THIS COULD GO INTO ADSUPPORT AND THEN WE DONT NEED INCLUDE OR apt-install
343+
.. # GRAPHICSMAGICK_EXTERNAL = NO
344+
345+
.. apt update
346+
.. apt install apt-file
347+
.. apt-file find Magick++.h
348+
.. add boost lib apt install
349+
.. AND libgraphicsmagick++1-dev
350+
.. change last step to adurl from ADSimDetector
351+
.. cp ibek-defs/adcore/adcore.sh ibek-defs/adurl/adurl.sh
352+
353+
354+
.. Once running:-
355+
.. caput -S BL01T-EA-TST-02:CAM:URL1

0 commit comments

Comments
 (0)