Skip to content

Commit 073b377

Browse files
gilesknapgilesknap
committed
update the debugging generic IOCs tutorial
1 parent e3a3912 commit 073b377

File tree

1 file changed

+106
-169
lines changed

1 file changed

+106
-169
lines changed

‎docs/user/tutorials/debug_generic_ioc.rst

Lines changed: 106 additions & 169 deletions
Original file line numberDiff line numberDiff line change
@@ -1,238 +1,175 @@
11
Debugging Generic IOC Builds
22
============================
33

4-
.. Warning::
5-
6-
This tutorial is out of date and will be updated soon.
7-
84
This tutorial is a continuation of `generic_ioc`. Here we will look into
9-
debugging failed builds and fix the issue we saw in the previous tutorial.
5+
debugging failed builds of Generic IOCs.
106

11-
This also comes under the category of type 2. change from the list
12-
at `ioc_change_types`.
7+
For the most part the recommended workflow is to always be working inside
8+
of a developer container. We always use a Generic IOC as the base for our
9+
developer containers. But what if the build of the Generic IOC fails, then
10+
you don't have a container to work in and need some other way to debug the
11+
build.
1312

14-
There are two ways to debug a failed build:
13+
There are two ways to debug such a failed build:
1514

1615
- Keep changing the Dockerfile and rebuilding the container until the build
1716
succeeds. This is the simplest approach and is often sufficient since our
1817
Dockerfile design maximizes the use of the build cache.
1918

2019
- Investigate the build failure by running a shell inside the
21-
partially-built container and
22-
using make. This is a good idea if you have to make fundamental changes
23-
such as installing a new system package. System package install happens
24-
at the start of the Dockerfile and would trigger a full rebuild when
25-
changed.
20+
partially-built container and retrying the failed command. This is particularly
21+
useful if you are fixing something early in the Dockerfile that causes a
22+
failure much later in the build. This type of failure is tedious to debug
23+
using the first approach above.
2624

27-
You already have the knowledge to apply the first approach. In this tutorial
28-
we will look debugging the build from *inside* the container.
25+
In this tutorial we will look debugging the build from *inside* the container.
2926

30-
Investigate the Build Failure
31-
-----------------------------
27+
Break the Build
28+
---------------
3229

33-
When a container build fails the container image is created up to the point
34-
where the last successful Dockerfile command was run. This means that we can
35-
investigate the build failure by running a shell in the container. ``ec``
36-
provides us with the following convenience command to do this:
30+
Let us break the build of our ioc-lakeshore340 project in the last
31+
tutorial. Open the the file
32+
``ioc-lakeshore340/ibek-support/StreamDevice/install.sh``.
33+
Comment out the apt-install line like this:
3734

3835
.. code-block:: bash
3936
40-
ec dev debug-last
37+
# ibek support apt-install --only=dev libpcre3-dev
4138
42-
Now we have a prompt inside the part-built container and can retry the failed
43-
command.
39+
Now rebuild the container - do this command from a new terminal *outside* of
40+
the devcontainer (make sure you have ``ec`` installed):
4441

4542
.. code-block:: bash
4643
47-
cd /workspaces/epics/support/adurl
48-
make
44+
cd ioc-lakeshore340 # where you cloned it
45+
ec dev build
4946
50-
You should see the same error again.
47+
First of all, notice the build cache. The build rapidly skips
48+
over all the steps until it gets to the StreamDevice support module. The,
49+
cache fails only when you get to ``COPY ibek-support/StreamDevice/ StreamDevice/``
50+
because a file in the source folder has changed.
5151

52-
A really good way to investigate this kind of error is with ``apt-file``
53-
which is a command line tool for searching Debian packages. apt-file is
54-
already installed in our devcontainer. So get another terminal open
55-
with the ``[E7]`` prompt and run the following commands:
52+
You should see the build fail with the following error:
5653

5754
.. code-block:: bash
5855
59-
apt-file search Magick++.h
56+
../RegexpConverter.cc:27:10: fatal error: pcre.h: No such file or directory
57+
27 | #include "pcre.h"
58+
| ^~~~~~~~
59+
compilation terminated.
6060
61-
graphicsmagick-libmagick-dev-compat: /usr/include/Magick++.h
62-
libgraphicsmagick++1-dev: /usr/include/GraphicsMagick/Magick++.h
63-
libmagick++-6-headers: /usr/include/ImageMagick-6/Magick++.h
61+
Investigate the Build Failure
62+
-----------------------------
6463

65-
The middle result looks most promising so we will install it (back *inside*
66-
the Generic IOC container now):
64+
When a container build fails the container image is created up to the point
65+
where the last successful Dockerfile command was run. This means that we can
66+
investigate the build failure by running a shell in the container. ``ec``
67+
provides us with the following convenience command to do this:
6768

6869
.. code-block:: bash
6970
70-
apt-get install -y libgraphicsmagick++1-dev
71-
72-
The reason using apt-file outside of the Generic IOC works is because
73-
the Generic IOC and the devcontainer are built upon the same version of
74-
Ubuntu and have the same packages available.
75-
76-
If we try the build again now it will still fail. We need to tell the
77-
AreaDetector build system where to find the new header file. This
78-
is documented here `CONFIG_SITE.local`_. The documentation says that we
79-
need to set the variable ``GRAPHICSMAGICK_INCLUDE`` in
80-
``CONFIG_SITE.linux-x86_64.Common``.
81-
We can see from the ``apt-file`` output that the header file is in
82-
``/usr/include/GraphicsMagick`` which is not a default include path.
83-
Therefore we need to edit this file inside our Generic IOC container:
84-
``/workspaces/epics/support/adurl/configure/CONFIG_SITE.linux-x86_64.Common``
85-
86-
.. _CONFIG_SITE.local: https://areadetector.github.io/areaDetector/install_guide.html#edit-config-site-local-and-optionally-config-site-local-epics-host-arch
87-
88-
89-
Making Changes Inside the Container
90-
-----------------------------------
91-
92-
You will find that the container includes busybox tools and there is a
93-
rudimentary version of ``vi`` installed. You could also install any editor
94-
you like from the Ubuntu repositories.
95-
96-
HOWEVER, there is a much easier way ...
97-
98-
When you launch containers with ``ec`` commands the ``/workspaces`` folder is
99-
synchronized to a local folder ioc-XXX/workspaces and that is mounted into the
100-
container. This means that you can edit files in the container using VSCode.
101-
The mounted repos folder also ensures that any changes you make inside the
102-
container are saved between invocations of the container.
103-
104-
See the image below to see how to navigate to
105-
``CONFIG_SITE.linux-x86_64.Common``
106-
in the ``adurl`` support module inside the ioc-adurl container.
107-
108-
.. figure:: ../images/repos_folder.png
109-
110-
VSCode file explorer showing the mounted repos folder
111-
112-
Using VSCode file explorer as pictured above, navigate to the file
113-
``CONFIG_SITE.linux-x86_64.Common`` and update the GRAPHICSMAGICK section to
114-
look like this:
71+
ec dev debug-last
11572
116-
.. code-block:: makefile
73+
Now we have a prompt inside the part-built container and can retry the failed
74+
command.
11775

118-
WITH_GRAPHICSMAGICK = YES
119-
GRAPHICSMAGICK_EXTERNAL = YES
120-
GRAPHICSMAGICK_INCLUDE = /usr/include/GraphicsMagick
76+
.. code-block:: bash
12177
122-
Now go back to the terminal and run ``make`` again. This time it should
123-
succeed.
78+
cd /workspaces/ioc-lakeshore340/ibek-support
79+
StreamDevice/install.sh 2.8.24
12480
125-
Applying Changes Made Inside the Container
126-
------------------------------------------
81+
You should see the same error again.
12782

128-
When you use the 'inside the container' approach to get the build working
129-
you still need to apply the changes you made 'outside' so that invoking
130-
container build will also succeed.
83+
This is a pretty common type of error
84+
when building a new support module. It implies that there is some dependency
85+
missing. There is a good chance this is a system dependency, in which case
86+
we want to search the Ubuntu repositories for the missing package.
13187

132-
:TIP: do NOT apply the below, the next heading supplies a better solution
133-
for this specific case.
88+
A really good way to investigate this kind of error is with ``apt-file``
89+
which is a command line tool for searching Debian packages. ``apt-file`` is
90+
not currently installed in the devcontainer. So you have two choices:
13491

135-
There are a few kinds of changes that need different approaches as follows:
92+
- Install it in the devcontainer - this is temporary and will be lost when
93+
the container is rebuilt. Ideal if you don't have install rights on your
94+
workstation.
13695

137-
:apt install:
96+
- Install it on your workstation - ideal if you have rights as you only need
97+
to install it once.
13898

139-
We did an apt install of ``libgraphicsmagick++1-dev``. Additional system
140-
package installs like this need to be added to the ``apt-get install``
141-
command at the top of the Dockerfile.
99+
TODO: consider adding apt-file to the base container developer target.
142100

143-
:CONFIG_SITE:
101+
Whether inside the container or in your workstation terminal, install
102+
``apt-file`` like this:
144103

145-
We edited ``CONFIG_SITE.linux-x86_64.Common``. This file is not part of
146-
the ADURL support module but was supplied by us from ibek-defs.
104+
.. code-block:: bash
147105
148-
:Patching:
106+
# drop the sudo from the start of the command if using podman
107+
sudo apt update
108+
sudo apt install apt-file
149109
150-
This should be avoided, but occasionally it may be necessary to patch other
151-
files in the support modules. This is just a variation of the CONFIG_SITE
152-
case above. You can place whatever script code you like in the
153-
``ioc-XXX/patch`` folder.
110+
Now we can search for the missing file:
154111

155-
:Support Module:
112+
.. code-block:: bash
156113
157-
Potentially we could have made changes to the ADUrl support module itself
158-
because we found a bug or wanted to add a feature. In this case we would
159-
push those changes back up to GitHub and get a release made so we
160-
could use the new version in our Dockerfile, This would in turn mean A
161-
change to the version number in the ``modules.py install ADURL``
162-
command. NOTE: the developer container we are using already holds clones
163-
of all the support modules so we could make changes in place and push them
164-
back.
114+
apt-file search pcre.h
165115
166-
An Easier Fix Using ADSupport
167-
-----------------------------
116+
There are a few results, but the most promising is:
168117

169-
Although we managed to fix the build by installing Graphics Magick, into the
170-
container there is an easier solution that is specific to areaDetector. The
171-
ADSupport module is capable of building most of the system dependencies that
172-
areaDetector needs. This has proved to be very useful in making containers
173-
because the curation of all of the compatible versions of these dependencies
174-
has already been done.
118+
libpcre3-dev: /usr/include/pcre.h
175119

176-
So the error we saw was due to us telling ADUrl to look for an 'internal'
177-
version of Graphics Magick built by ADSupport. However, we did not tell
178-
ADSupport to build Graphics Magick.
120+
Pretty much every time you are missing a header file you will find it in a
121+
system package with a name ending in ``-dev``.
179122

180-
So the simple fix to this is to add the following line to the
181-
``ioc-adurl/ibek-defs/adsupport/adsupport.sh`` file:
123+
Now we can install the missing package in the container and retry the build:
182124

183-
.. code-block:: makefile
125+
.. code-block:: bash
184126
185-
WITH_GRAPHICSMAGICK = YES
186-
GRAPHICSMAGICK_EXTERNAL = NO
127+
apt-get install -y libpcre3-dev
128+
StreamDevice/install.sh 2.8.24
187129
188-
Then rebuild the container:
130+
You should find the build succeeds. But this is not the whole story. There
131+
is another line in ``install.h`` that I added to make this work:
189132

190133
.. code-block:: bash
191134
192-
ec dev build
135+
ibek support add-config-macro ${NAME} PCRE_LIB /usr/lib/x86_64-linux-gnu
193136
194-
Note that the build skips quickly over the support modules until it gets
195-
to ADSupport. This is the build cache saving time.
196-
However this build will STILL FAIL, it turns out that building Graphics Magick
197-
does need one system library install.
137+
This added a macro to ``CONFIG_SITE.linux-x86_64.Common`` that tells the
138+
Makefiles to add an extra include path to the compiler command line. working
139+
out how to do this is a matter of taking a look in the Makefiles. But the
140+
nice thing is that you can experiment with things inside the container and
141+
get them working without having to keep rebuilding the container.
198142

199-
The final fix is to add ``libxext-dev`` to the ``apt-get install`` command in
200-
our Dockerfile. So that it looks like this:
143+
Note that ``ibek support add-config-macro`` is idempotent, so you can run it
144+
multiple times without getting repeated entries in the CONFIG. All ``ibek``
145+
commands behave this way as far as possible.
201146

202-
.. code-block:: bash
147+
Once you are happy with your manual changes you can make them permanent by
148+
adding to the install.sh or Dockerfile, then try a full rebuild.
203149

204-
RUN apt-get update && apt-get upgrade -y && \
205-
apt-get install -y --no-install-recommends \
206-
libboost-all-dev \
207-
libxext-dev
208-
209-
This is an example of a change that also requires a system package
210-
install for the runtime version of the container. Locate the second
211-
``apt-get install`` command in the Dockerfile and add ``libxext6`` so
212-
that it looks like this:
213-
214-
.. code-block:: bash
150+
Making Changes Inside the Container
151+
-----------------------------------
215152

216-
RUN apt-get update && apt-get upgrade -y && \
217-
apt-get install -y --no-install-recommends \
218-
libxext6 \
219-
&& rm -rf /var/lib/apt/lists/*
153+
You will find that the container includes busybox tools, vim and ifconfig.
154+
These should provide enough tools to investigate and fix most build problems.
155+
You are also
156+
free to use apt-get to install any other tools you need as demonstrated above.
157+
(type busybox to see the list of available tools).
220158

221-
You can remove the RTEMS specific runtime packages that came with ioc-template.
222-
Note that the ``rm -rf /var/lib/apt/lists/`` removes the apt cache and keeps
223-
the runtime image size down.
224159

225-
This build should now succeed. Unfortunately it has to rebuild the entire
226-
container from scratch because we changed the first command in the Dockerfile.
160+
Other ``ec dev`` Commands
161+
-------------------------
227162

228-
Wrapping Up
229-
-----------
163+
The ``ec dev`` namespace provides a number of other commands for working outside
164+
of developer containers. These were primarily developed before the use
165+
of developer containers matured. If you prefer not to use developer containers
166+
for any reason then take a look at the help as follows:
230167

231-
You now have a new Generic IOC that can be used to test the ADUrl plugin.
168+
.. code-block:: bash
232169
233-
The next tutorial will discuss how to test this IOC, including publishing
234-
the image to a container registry so that it can run in Kubernetes.
170+
ec dev --help
235171
172+
You should be able to perform most of the same tasks that the tutorials teach.
236173

237-
.. Once running:-
238-
.. caput -S $USER-EA-TST-02:CAM:URL1
174+
Don't forget that you can always use ``ec -v dev ...`` to get output showing
175+
the underlying docker/podman and git commands ``ec`` is using.

0 commit comments

Comments
 (0)