You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Container Engine (CE) toolset is designed to enable computing jobs to
5
-
seamlessly run inside Linux application containers, thus providing support for
6
-
containerized user environments.
4
+
The Container Engine (CE) toolset is designed to enable computing jobs to seamlessly run inside Linux application containers, thus providing support for containerized user environments.
7
5
8
6
## Concept
9
7
10
-
Containers effectively encapsulate a software stack; however, to be useful in
11
-
HPC computing environments, they often require the customization of bind
12
-
mounts, environment variables, working directories, hooks, plugins, etc. To
13
-
simplify this process, the Container Engine (CE) toolset supports the
14
-
specification of user environments through Environment Definition Files.
8
+
Containers effectively encapsulate a software stack; however, to be useful in HPC computing environments, they often require the customization of bind mounts, environment variables, working directories, hooks, plugins, etc.
9
+
To simplify this process, the Container Engine (CE) toolset supports the specification of user environments through Environment Definition Files.
15
10
16
-
An Environment Definition File (EDF) is a text file in the [TOML
17
-
format](https://toml.io/en/) that declaratively and prescriptively represents
18
-
the creation of a computing environment based on a container image. Users can
19
-
create their own custom environments and share, edit, or build upon already
20
-
existing environments.
11
+
An Environment Definition File (EDF) is a text file in the [TOML format](https://toml.io/en/) that declaratively and prescriptively represents the creation of a computing environment based on a container image.
12
+
Users can create their own custom environments and share, edit, or build upon already existing environments.
21
13
22
-
The Container Engine (CE) toolset leverages its tight integration with the
23
-
Slurm workload manager to parse EDFs directly from the command line or batch
24
-
script and instantiate containerized user environments seamlessly and
25
-
transparently.
14
+
The Container Engine (CE) toolset leverages its tight integration with the Slurm workload manager to parse EDFs directly from the command line or batch script and instantiate containerized user environments seamlessly and transparently.
26
15
27
-
Through the EDF, container use cases can be abstracted to the point where end
28
-
users perform their workflows as if they were operating natively on the
29
-
computing system.
16
+
Through the EDF, container use cases can be abstracted to the point where end users perform their workflows as if they were operating natively on the computing system.
30
17
31
18
**Key Benefits**
32
19
33
-
**Freedom*: Container gives users full control of the user space. The user
34
-
can decide what to install without involving a sysadmin.
35
-
**Reproducibility*: Workloads consistently run in the same environment,
36
-
ensuring uniformity across job experimental runs.
37
-
**Portability*: The self-contained nature of containers simplifies the
38
-
deployment across architecture-compatible HPC systems.
39
-
**Seamless Access to HPC Resources*: CE facilitates native access to
40
-
specialized HPC resources like GPUs, interconnects, and other
41
-
system-specific tools crucial for performance
20
+
**Freedom*: Container gives users full control of the user space. The user can decide what to install without involving a sysadmin.
21
+
**Reproducibility*: Workloads consistently run in the same environment, ensuring uniformity across job experimental runs.
22
+
**Portability*: The self-contained nature of containers simplifies the deployment across architecture-compatible HPC systems.
23
+
**Seamless Access to HPC Resources*: CE facilitates native access to specialized HPC resources like GPUs, interconnects, and other system-specific tools crucial for performance.
42
24
43
25
## Quick Start
44
26
45
-
Let's set up a containerized Ubuntu 24.04 environment inside a scratch folder
46
-
(`${SCRATCH}`).
47
-
48
-
### Example EDF
27
+
Let's set up a containerized Ubuntu 24.04 environment on the scratch folder (`${SCRATCH}`).
28
+
Save this file below as `ubuntu.toml` in `${HOME}/.edf` directory (the default location of EDF files).
29
+
A more detailed explanation of each entry for the EDF can be seen in the [EDF reference][ref-ce-edf-reference].
If an EDF is located in the [EDF search path][ref-ce-edf-search-path], `--environment` also accepts the EDF filename without the `.toml` extension:
106
+
107
+
!!! example
108
+
```bash
109
+
$ srun --environment=debian cat /etc/os-release
110
+
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
111
+
NAME="Debian GNU/Linux"
112
+
VERSION_ID="12"
113
+
...
114
+
```
142
115
143
116
### Use from batch scripts
144
117
145
-
The recommended approach is to use `--environment` as part of the Slurm command
146
-
(e.g., `srun` or `salloc`). For example, when launching job steps:
118
+
The recommended approach is to use `--environment` as part of the Slurm command (e.g., `srun` or `salloc`):
147
119
148
-
```bash
149
-
#!/bin/bash
150
-
#SBATCH --job-name=edf-example
151
-
...
120
+
!!! example
121
+
```bash
122
+
#!/bin/bash
123
+
#SBATCH --job-name=edf-example
124
+
...
152
125
153
-
# Run job step
154
-
srun --environment=debian cat /etc/os-release
155
-
```
126
+
# Run job step
127
+
srun --environment=debian cat /etc/os-release
128
+
```
156
129
157
-
Alternatively, the `--environment` option can also be specified with an
158
-
`#SBATCH` option. However, the support status is still **experimental** and may
159
-
result in unexpected behaviors.
130
+
Alternatively, the `--environment` option can also be specified with an `#SBATCH` option.
131
+
However, the support status is still **experimental** and may result in unexpected behaviors.
160
132
161
133
!!! note
162
-
Specifying `--environment` with `#SBATCH` will put the entire batch script
163
-
inside the containerized environment, requiring the Slurm hook to use any
164
-
Slurm commands within the batch script (e.g., `srun` or `scontrol`). The
165
-
hook is controlled by the `ENROOT_SLURM_HOOK` environment variable and
166
-
activated by default on most vClusters.
134
+
Specifying `--environment` with `#SBATCH` will put the entire batch script inside the containerized environment, requiring the Slurm hook to use any Slurm commands within the batch script (e.g., `srun` or `scontrol`).
135
+
The hook is controlled by the `ENROOT_SLURM_HOOK` environment variable and activated by default on most vClusters.
167
136
168
137
[](){#ref-ce-edf-search-path}
169
138
### EDF search path
170
139
171
-
By default, the EDFs for each user are looked up in `$HOME/.edf`. The default
172
-
EDF search path can be changed through the `EDF_PATH` environment variable.
173
-
`EDF_PATH` must be a colon-separated list of absolute paths to directories,
174
-
where the CE searches each directory in order. If an EDF is located in the
175
-
search path, its name can be used in the `--environment` option without the
176
-
`.toml` extension. For example:
140
+
By default, the EDFs for each user are looked up in `$HOME/.edf`.
141
+
The default EDF search path can be changed through the `EDF_PATH` environment variable.
142
+
`EDF_PATH` must be a colon-separated list of absolute paths to directories, where the CE searches each directory in order.
143
+
If an EDF is located in the search path, its name can be used in the `--environment` option without the `.toml` extension.
By default, images defined in the EDF as remote registry references (e.g. a
199
-
Docker reference) are automatically pulled and locally cached. A cached image
200
-
would be preferred to pulling the image again in later usage.
166
+
By default, images defined in the EDF as remote registry references (e.g. a Docker reference) are automatically pulled and locally cached.
167
+
A cached image would be preferred to pulling the image again in later usage.
201
168
202
-
An image cache is automatically created at `.edf_imagestore` in the user's
203
-
scratch folder (i.e., `${SCRATCH}/.edf_imagestore`), under which cached images
204
-
are stored with the corresponding CPU architecture suffix (e.g., `x86` and
205
-
`aarch64`). Cached images may be subject to the automatic cleaning policy of
206
-
the scratch folder.
169
+
An image cache is automatically created at `.edf_imagestore` in the user's scratch folder (i.e., `${SCRATCH}/.edf_imagestore`), under which cached images are stored with the corresponding CPU architecture suffix (e.g., `x86` and `aarch64`).
170
+
Cached images may be subject to the automatic cleaning policy of the scratch folder.
207
171
208
-
Should users want to re-pull a cached image, they have to remove the
209
-
corresponding image in the cache.
172
+
Should users want to re-pull a cached image, they have to remove the corresponding image in the cache.
210
173
211
-
To choose an alternative image store path (e.g., to use a directory owned by a
212
-
group and not to an individual user), users can specify an image cache path
213
-
explicitly by defining the environment variable `EDF_IMAGESTORE`.
214
-
`EDF_IMAGESTORE` must be an absolute path to an existing folder.
174
+
To choose an alternative image store path (e.g., to use a directory owned by a group and not to an individual user), users can specify an image cache path explicitly by defining the environment variable `EDF_IMAGESTORE`.
175
+
`EDF_IMAGESTORE` must be an absolute path to an existing folder.
215
176
216
177
!!! note
217
-
If the CE cannot create a directory for the image cache, it operates in
218
-
cache-free mode, meaning that it pulls an ephemeral image before every
219
-
container launch and discards it upon termination.
178
+
If the CE cannot create a directory for the image cache, it operates in cache-free mode, meaning that it pulls an ephemeral image before every container launch and discards it upon termination.
0 commit comments