Skip to content

Commit 499c231

Browse files
authored
Merge pull request #118 from epics-containers/april-update
April update
2 parents 690ad22 + 3e74d6a commit 499c231

25 files changed

+385
-82
lines changed

.devcontainer/devcontainer.json

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
// Allow X11 apps to run inside the container
1010
"DISPLAY": "${localEnv:DISPLAY}"
1111
},
12+
"remoteUser": "${localEnv:EC_REMOTE_USER}",
1213
"customizations": {
1314
"vscode": {
1415
// Set *default* container specific settings.json values on container create.
@@ -33,14 +34,16 @@
3334
"upgradePackages": false
3435
}
3536
},
37+
// You can place any outside of the container before-launch commands here
38+
"initializeCommand": "bash .devcontainer/initializeCommand ${devcontainerId}",
39+
// One time global setup commands inside the container
40+
"postCreateCommand": "bash .devcontainer/postCreateCommand ${devcontainerId}",
3641
"runArgs": [
3742
// Allow the container to access the host X11 display and EPICS CA
3843
"--net=host",
3944
// Make sure SELinux does not disable with access to host filesystems like tmp
4045
"--security-opt=label=disable"
4146
],
4247
// Mount the parent as /workspaces so we can pip install peers as editable
43-
"workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind",
44-
// After the container is created, install the python project in editable form
45-
"postCreateCommand": "pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]' && pre-commit install"
48+
"workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind"
4649
}

.devcontainer/initializeCommand

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
3+
# custom initialization goes here - runs outside of the dev container
4+
# just before the container is launched but after the container is created
5+
6+
echo "devcontainerID ${1}"

.devcontainer/postCreateCommand

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/bin/bash
2+
3+
# Custom initialization goes here if needed.
4+
# Runs inside the dev container after the container is created
5+
6+
################################################################################
7+
# When using docker we will not be root inside the container
8+
# the following steps are then required
9+
################################################################################
10+
11+
if [[ $USER != "root" ]] ; then
12+
# make sure the non-root user can access the virtual environment
13+
sudo chown -R ${USER}:${USER} /venv
14+
fi
15+
16+
################################################################################
17+
# Shell customizations
18+
################################################################################
19+
20+
# pick a theme that does not cause completion corruption in zsh
21+
sed -i $HOME/.zshrc -e 's/ZSH_THEME="devcontainers"/ZSH_THEME="lukerandall"/'
22+
23+
# allow personalization of all devcontainers in this subdirectory
24+
# by placing a .devcontainer_rc file in the workspace root
25+
if [[ -f /workspaces/.devcontainer_rc ]] ; then
26+
source /workspaces/.devcontainer_rc
27+
fi
28+
29+
#After the container is created, install the python project in editable form
30+
pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]'
31+
pre-commit install

docs/conf.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,5 +177,5 @@
177177
html_show_copyright = False
178178

179179
# Logo
180-
html_logo = "images/dls-logo.svg"
180+
html_logo = "images/k8s-epics2.ico"
181181
html_favicon = html_logo

docs/explanations/changes.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
Change Management Manifesto
2+
===========================
3+
4+
Pledge
5+
------
6+
7+
This page represents a pledge to control breaking changes for users of `epics-containers`. The framework is still under development and there still may be breaking changes in future updates. However we now have a mechanism in place to allow users to adopt the framework, take advantage of the current features and then accept future updates in a controlled fashion.
8+
9+
From version 3.4.0 onwards we will endeavour to make changes in a controlled fashion that obeys SemVer 2.0.0 rules. We will also try to ensure that it is possible to apply updates in a gradual manner and not require a blanket update.
10+
11+
The [](../reference/changelog.md) will give details of any things to be aware of between versions, including minor version updates.
12+
13+
Dependency Matrix
14+
-----------------
15+
16+
Users of the framework will develop two kinds of repository:
17+
18+
| | |
19+
| --- | --- |
20+
| Beamline repo | A beamline or other grouping of IOC instance descriptions |
21+
| Generic IOC repo | A definition of a generic IOC container image for a particular class of device |
22+
23+
Both of these types of repository are initially created using a copier template. The copier template version will have a SemVer version number that determines which component versions it is compatible with.
24+
25+
For a discussion on how to update your projects to the latest version of the templates see [](../how-to/copier_update).
26+
27+
Any breaking changes to the framework will be made in a new Major version of the framework and hence a new Major version of the copier templates.
28+
29+
The following diagram shows the set of components that are involved in the framework and the relationships between them. The dependencies between these components are not a strict API and the diagram attempts to highlight the features of each component that affect other components.
30+
31+
The diagram may be used by developers of the framework to plan how changes will be made and to ensure that breaking changes are made in a controlled manner.
32+
33+
Users of the framework are only concerned with the top two boxes and these are always updated on an as-needed basis via the copier templates.
34+
35+
:::{figure} ../images/dependency_matrix.png
36+
`epics-containers` dependency matrix
37+
:::
38+
39+
All `ec` SemVer components will always have their major version bumped simultaneously. Likewise for `ibek` SemVer components. These are at versions 3.4.0 and 2.0.0 respectively at the time of writing.
40+
41+
42+
Updating user projects
43+
----------------------
44+
45+
A repository that was originally created using a copier template can be updated to a new version using the following command (assumes you have an active python venv with copier installed):
46+
47+
```bash
48+
copier update -r VERSION_NUMBER --trust .
49+
```
50+
51+
You can supply the VERSION_NUMBER of the template you want or omit the -r option to get the latest released version.
52+
53+
This will update your project in place. You should then inspect the changes using git (the source control pane in vscode is excellent for this purpose) and commit them to your repository.
54+
55+
When a beamline repository is updated, it is still possible to deploy old versions of its IOC instances, even with a major version difference. That is because the deploy mechanism makes a temporary clone of the beamline repository and deploys the instance described in that version.
56+
57+
User Project Versioning
58+
-----------------------
59+
60+
The documentation has recommended using DateVer for beamline repos and generic IOC repos. This is because SemVer is not really applicable to these. However, DateVer is not required and you are free to use any scheme you wish for these repositories.
61+
62+
It is easy to determine which template version and thus which `ec` SemVer version your repository was last updated from. Inspect the file `.copier_answers.yml` in the root of your repository. This file contains the version of the template that was used to create the repository in the field `_commit`.
63+
64+
65+
Types of Changes
66+
----------------
67+
68+
Changes to the framework are likely to be initiated in one the places described under the following headings. As far as possible such changes will be backwards compatible going forward, and if they are not then a major version release will be made.
69+
70+
### ibek
71+
72+
73+
Changes to the CLI commands inside of the container build/runtime are initiated in the `ibek` python module that lives inside every generic IOC. This can affect the support module build recipes in `ibek-support` and potentially the Dockerfile in Generic IOC projects.
74+
75+
### ec-helm-charts
76+
77+
Changes here affect how IOCs and other services are deployed into Kubernetes. These would likely affect beamline repositories as they contain the versions of Helm Charts used to deploy their instances. Potentially changes to these Charts may require an update to the edge-services-cli to support new features.
78+
79+
### ioc-template
80+
81+
The Generic IOC template is well established and stable. However, each time a new target architecture is added, this will need updates to the CI. We will be supporting Windows and ARM targets in future. These changes should certainly be backwards compatible and not affect existing projects.
82+
83+
84+
### ibek-support
85+
86+
`ibek-support` is a unique project in that it is a submodule of all Generic IOCs. It is expected that there will be constant change to this module as new support modules are added. However, such changes will almost entirely be adding new folders and not affect existing generic IOCs. We encourage users to fork this repository, add their own support modules and submit PRs back to the original so that a wide range of support modules can be shared (a branch rather than fork is preferred for internal developers).
87+
88+
If there is a need to change the CLI that ibek-support uses, then a new version of `ibek` will be released. Only generic IOCs that have been updated to pick up the new version of `ibek` would be able to use these changes. Because older generic IOCs will retain the old commit of the 'ibek-support' submodule, they will not be affected by the changes until they are updated.

docs/explanations/introduction.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ forks of these repositories.
3939

4040
#### Generic IOCs and instances
4141

42-
An important principal of the approach presented here is that an IOC container image represents a 'Generic' IOC. The Generic IOC image is used for all IOC instances that connect to a given class of device. For example the Generic IOC image here: [ghcr.io/epics-containers/ioc-adaravis-linux-runtime:2024.2.2 ](https://github.com/epics-containers/ioc-adaravis/pkgs/container/ioc-adaravis-linux-runtime) uses the AreaDetector driver ADAravis to connect to GigE cameras.
42+
An important principal of the approach presented here is that an IOC container image represents a 'Generic' IOC. The Generic IOC image is used for all IOC instances that connect to a given class of device. For example the Generic IOC image here: [ghcr.io/epics-containers/ioc-adaravisruntime:2024.2.2 ](https://github.com/epics-containers/ioc-adaravis/pkgs/container/ioc-adaravisruntime) uses the AreaDetector driver ADAravis to connect to GigE cameras.
4343

4444
An IOC instance runs in a container runtime by loading two things:
4545

docs/how-to/builder2ibek.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ builder beamline to epics-containers. i.e. those whose beamlines
66
have a BLxxY-BUILDER project.
77
:::
88

9-
TODO: this page is WIP and will be updated by Feb 2024.
9+
TODO: this page is WIP and will be updated by May 2024.
1010

1111
`builder2ibek` is a tool to convert DLS builder XML to ibek instance YAML.
1212
It is for working with converting IOC instances to epics-containers.

docs/how-to/builder2ibek.support.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ a DLS support module with builder support into an epics-containers
66
Generic IOC. i.e. support modules that have an `etc/builder.py` file.
77
:::
88

9-
TODO: this page is WIP and will be updated by Feb 2024.
9+
TODO: this page is WIP and will be updated by May 2024.
1010

1111
`builder2ibek.support` is a tool to convert DLS builder support modules
1212
into ibek support YAML for the `ibek-support` repository.

docs/how-to/copier_update.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
Updating User Repositories with Copier
2+
======================================
3+
4+
In the tutorials we created an example beamline repo and also an example generic IOC. In both cases we used the copier tool to create these projects from a template.
5+
6+
This tool can also be used to merge in the latest changes to the framework into your existing projects. This is done by running the `copier update` command in the root of your project.
7+
8+
Below are some details on how to use this tool.
9+
10+
Introduction
11+
------------
12+
13+
copier is a python package. To make use of it you require an activated python virtual environment with copier installed. If you don't already have one of these then the following commands will set one up for you:
14+
15+
```bash
16+
python3 -m venv venv
17+
source venv/bin/activate
18+
pip install copier
19+
```
20+
21+
NOTE: generic IOCs with a given major version number template should work with beamline repos with the same major version number template. When updating a beamline repo to a new major version, you may also need to update the generic IOCs it references. If this is the case it will be noted in [](../reference/changelog.md). Having said this, the two types of repo are reasonably well decoupled and we would aim to avoid this necessity. In which case update the beamline repo first and then the generic IOCs as and when new features are required.
22+
23+
Updating a Beamline Repository
24+
------------------------------
25+
26+
A beamline (or other grouping) repository is a collection of IOC instances and services that are deployed together. The beamline repository is created using the `ec-services-template` copier template. See [](../tutorials/create_beamline.md) for the tutorial on how to create a new beamline repository.
27+
28+
To update your beamline repository to the latest version of the templates you should run the following command in the root of your repository:
29+
30+
```bash
31+
copier update -r VERSION_NUMBER --trust .
32+
```
33+
34+
You can supply the VERSION_NUMBER of the template you want or omit the `-r` option to get the latest released version.
35+
36+
This will update your project in place. You should then inspect the changes using git (the source control pane in vscode is excellent for this purpose) and commit them to your repository. Once you are happy with the changes you can test them by re-deploying some of your IOC instances. When everything is working you can push the changes to your repository.
37+
38+
The template comes with an example IOC called xxxx-ea-test-01. You are free to delete this if you don't want it. However, we recommend keeping that IOC as it is a good reference for what changes might be needed in your own IOC instances.
39+
40+
For example, in version 3.4.0 the `ec-services-template` changed the way that the configmap is created for each IOC instance. This added a soft-link **templates** folder that points at **../../include/ioc/templates**. Looking at what changes happened in the example IOC will help you to understand what changes you might need to make in your own IOC instances. Copier migrations will attempt to make these changes for you but it is recommended to check that they have been done correctly.
41+
42+
43+
Updating a Generic IOC Repository
44+
---------------------------------
45+
46+
To see details of how to initially create a generic IOC repository see {any}`create_generic_ioc`.
47+
48+
To update your generic IOC repository to the latest version of the templates you should run the following command in the root of your repository:
49+
50+
```bash
51+
copier update -r VERSION_NUMBER --trust .
52+
```
53+
54+
Typically the only file that the user will have changed is the Dockerfile and typically merges will work well. It is still a good idea to validate the changes to the repo before committing them.
55+
56+
When Update fails
57+
-----------------
58+
59+
Copier cannot do the 'update' command if any of the following is true:
60+
61+
- Your repo pre-dates the use of copier
62+
- The copier template version that you last used is no longer available.
63+
64+
In this case you can still do an update using the 'copy' command - you just need to be more careful with the merge.
65+
66+
For the generic IOC case:
67+
68+
```bash
69+
copier copy gh:epics-containers/ioc-template --trust .
70+
```
71+
72+
For the beamline case:
73+
74+
```bash
75+
copier copy gh:epics-containers/ec-services-template --trust .
76+
```
77+
78+
In both cases you should select Y for each notice of a conflict, then resolve those conflicts in your editor.
79+
80+
Using copy will mean that you get asked the template questions again, but copier is smart enough to supply your previous answers as defaults (if your repo predates copier then you will need to answer all the questions for the first time).
81+
82+
The difference from 'update' is that you will need to do the merge yourself. For example in a generic IOC you will want to restore your support module instantiations in the middle of the Dockerfile, whereas the top and bottom of the Dockerfile should be updated by the template. Using vscode's SOURCE CONTROL pane is useful for this. The following image is an example of a Dockerfile merge:
83+
84+
:::{figure} ../images/dockerfile_merge.png
85+
merging a Dockerfile after a copy copy
86+
:::
87+
88+
In the above example the **copier copy** has made many changes to the repo. Because the only user supplied change is the list of support modules in the middle of the Dockerfile, the merge is simple. Using vscode you just need to click button indicated by the red arrow, save your changes, then:
89+
90+
```bash
91+
# from the root of the repo
92+
git add .
93+
git commit -m'update to template version 3.4.0'
94+
git push
95+
```
96+

docs/how-to/debug.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Are you sure ? [y/N]: y
4141
<font color="#5F8787">docker container create --name busybox -v bl01t-ea-test-02_config:/copyto busybox</font>
4242
<font color="#5F8787">docker cp /home/giles/tutorial/bl01t/services/bl01t-ea-test-02/config/ioc.yaml busybox:copyto</font>
4343
<font color="#5F8787">docker rm -f busybox</font>
44-
<font color="#5F8787">docker run -dit --net host --restart unless-stopped -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02 --name bl01t-ea-test-02 ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:2024.2.2</font>
44+
<font color="#5F8787">docker run -dit --net host --restart unless-stopped -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02 --name bl01t-ea-test-02 ghcr.io/epics-containers/ioc-adsimdetectorruntime:2024.2.2</font>
4545
76c2834dac805780b3329af91c332abb90fb2692a510c11b888b82e48f60b44f
4646
<font color="#5F8787">docker ps -f name=bl01t-ea-test-02 --format &apos;{{.Names}}&apos;</font>
4747
</pre>
@@ -59,7 +59,7 @@ Now you can tell `ec` to stop the IOC instance and then run it in a way that you
5959

6060
```bash
6161
ec stop bl01t-ea-test-02
62-
docker run --entrypoint bash -it --net host -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02 --name bl01t-ea-test-02-debug ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:2024.2.2
62+
docker run --entrypoint bash -it --net host -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02 --name bl01t-ea-test-02-debug ghcr.io/epics-containers/ioc-adsimdetectorruntime:2024.2.2
6363
```
6464

6565
You should now be in a shell inside the container. You can look at the files and run the IOC instance manually to see what the error is. You can re-run the IOC instance multiple times and you can even install your favourite editor or debugging tools.

0 commit comments

Comments
 (0)