Skip to content

Commit 3071aa6

Browse files
portantendokos
portante
authored and
ndokos
committed
Initial pbench-agent container image layering
[Back-port] There are 3 layers: a "base" layer, intermediate layers on top of base providing "tools" and "workloads" independently, and an "all" layer that pulls them all together. The "base" container image is a full install of the pbench-agent RPM without and RPMs for tools or workloads, just the core components of the pbench-agent. The "tools" and "workloads" container images contain RPMs required for the operation of the various tools and workloads respectively. The "all" image is a convenient combination of the "tools" and "workloads" images into one image. Having the tools and workloads dependencies in separate images allows users to construct their own images when they have specific requirements without pulling in extra unwanted dependencies. The images will be tagged using the following image tags automatically when built: * `<git commit sha1>` * `<rpm version #>` A separate manual process to label the set of images as the `latest` is offered with the `make` target `latest`. We also `make` targets for labeling the latest "major" and "major.minor" versions as `v<major>-latest` and `v<major>.<minor>-latest` (respectively) with the `major-latest` and `major-minor-latest` targets. Future work is needed in order to reduce the size of the "base" image.
1 parent 798fca4 commit 3071aa6

File tree

12 files changed

+439
-0
lines changed

12 files changed

+439
-0
lines changed

‎README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,27 @@ Yes, we are using GitHub [Projects](https://github.com/distributed-system-analys
6161
Please find projects covering the [Agent](https://github.com/distributed-system-analysis/pbench/projects/2),
6262
[Server](https://github.com/distributed-system-analysis/pbench/projects/3), and a project that is named
6363
the same as the current [milestone](https://github.com/distributed-system-analysis/pbench/milestones).
64+
65+
## Pbench Release Tag Scheme (GitHub)
66+
We employ a simple major, minor, release, build (optional) scheme for tagging
67+
starting with the `v0.70.0` release (`v<Major>.<Minor>.<Release>[-<Build>]`).
68+
Prior to the v0.70.0 release, the scheme used was mostly `v<Major>.<Minor>`,
69+
where we only had minor releases (`Major = 0`).
70+
71+
The practice of using `-agent` or `-server` is also ending with the `v0.70.0`
72+
release.
73+
74+
### Container Image Tags
75+
This same GitHub "tag" scheme is used with tags applied to container images
76+
we build, with the following exceptions for tag names:
77+
78+
* `latest` - always points to the "latest" container image pushed to a
79+
repository
80+
81+
* `v<Major>-latest` - always points to the "latest" `Major` released
82+
image
83+
84+
* `v<Major>.<Minor>-latest` - always points to the "latest" release
85+
for `Major`.`Minor` released images
86+
87+
* `<SHA1 git hash>` (9 characters) - commit hash of the checked out code

‎agent/containers/images/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
*.Dockerfile
2+
*.repo
3+
*.yml
4+
*-tags.lis

‎agent/containers/images/Dockerfile.base.j2‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# {{ distro_image_name }} pbench-agent base image
2+
FROM docker.io/library/{{ distro_image }}
3+
4+
# Install the appropriate pbench repository file for {{ distro_image_name }}.
5+
COPY ./{{ pbench_repo_file }} /etc/yum.repos.d/pbench.repo
6+
7+
# Install the pbench-agent RPM, which should have all its dependencies enumerated;
8+
# ... and make sure we have a proper pbench-agent.cfg file in place;
9+
# ... and finally, ensure the proper pbench-agent environment variables are set up.
10+
RUN \
11+
{% if distro_image.startswith('centos') %}
12+
{{ pkgmgr }} install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-{{ distro_image.split(':', 1)[1] }}.noarch.rpm && \
13+
{% endif %}
14+
{% if distro_image == 'centos:7' %}
15+
{{ pkgmgr }} install -y centos-release-scl-rh centos-release-scl && \
16+
{{ pkgmgr }} install -y --enablerepo=centos-sclo-rh rh-python36 && \
17+
{% endif %}
18+
{{ pkgmgr }} install -y {% if distro_image == 'centos:8' %}--enablerepo PowerTools {% endif %}--enablerepo copr-pbench pbench-agent && \
19+
{{ pkgmgr }} -y clean all && \
20+
rm -rf /var/cache/yum && \
21+
cp -a /opt/pbench-agent/config/pbench-agent.cfg.example /opt/pbench-agent/config/pbench-agent.cfg

‎agent/containers/images/Dockerfile.layered.j2‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# {{ distro }} pbench-agent {{ kind }} image
2+
FROM pbench-agent-base-{{ distro }}:{{ tag }}
3+
4+
# Install all the RPMs required for this image.
5+
#
6+
# FIXME: this is not exhaustive, it does not include RPMs to support
7+
# Kubernetes or RHV environments.
8+
RUN {% if distro == 'centos-7' %}yum{% else %}dnf{% endif %} install -y {% if distro == 'centos-8' %}--enablerepo PowerTools {% endif %}--enablerepo copr-pbench {{ rpms }} && \
9+
{% if distro == 'centos-7' %}yum{% else %}dnf{% endif %} -y clean all && \
10+
rm -rf /var/cache/yum

‎agent/containers/images/Makefile‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Base Makefile for building all images and tagging them
2+
3+
# By default we only build images for x86_64 architectures.
4+
_ARCH = x86_64
5+
6+
# By default we use this user's account on Fedora COPR for where to
7+
# find the RPMs. You can override this using an environment variable
8+
# as appropriate.
9+
USER = ndokos
10+
11+
# By default we use Fedora COPR repos. You can override this default
12+
# using an environment variable as appropriate.
13+
URL_PREFIX = https://copr-be.cloud.fedoraproject.org/results/${USER}
14+
15+
# By default we use the pbench Quay.io organization for the image
16+
# repository. You can override this default using an environment
17+
# variable as appropriate.
18+
IMAGE_REPO = docker://quay.io/pbench
19+
20+
# Convenience reference to the repo template in the pbench tree.
21+
# Not intended to be overridden with an environment variable.
22+
_REPO_TEMPLATE = ../../ansible/pbench/agent/roles/pbench_repo_install/templates/etc/yum.repos.d/pbench.repo.j2
23+
24+
# The list of RPMs which provide the various tools we offer.
25+
# Not intended to be overridden with an environment variable.
26+
_TOOL_RPMS = pbench-sysstat perf blktrace bpftrace numactl pcp-system-tools strace kernel-tools tcpdump
27+
28+
# The list of RPMs for the default workloads we offer.
29+
# Not intended to be overridden with an environment variable.
30+
_WORKLOAD_RPMS = fio uperf
31+
32+
# Not intended to be overridden with an environment variable.
33+
_ALL_RPMS = ${_TOOL_RPMS} ${_WORKLOAD_RPMS}
34+
35+
# Default we only build images the following distributions:
36+
all: centos-8-all-tagged centos-7-all-tagged fedora-32-all-tagged fedora-31-all-tagged
37+
38+
#+
39+
# Tagging targets
40+
#-
41+
42+
# Add the "latest" tag to the local images.
43+
tag-latest: centos-8-tag-latest centos-7-tag-latest fedora-32-tag-latest fedora-31-tag-latest
44+
45+
# Add the "v<Major>-latest" tag to the local images.
46+
tag-major: centos-8-tag-major centos-7-tag-major fedora-32-tag-major fedora-31-tag-major
47+
48+
# Add the "v<Major>.<Minor>-latest" tag to the local images.
49+
tag-major-minor: centos-8-tag-major-minor centos-7-tag-major-minor fedora-32-tag-major-minor fedora-31-tag-major-minor
50+
51+
#+
52+
# Push targets
53+
#-
54+
55+
# Push images with "<git commit hash>" and "v<full RPM version>" tags.
56+
push: centos-8-push centos-7-push fedora-32-push fedora-31-push
57+
58+
# Push images with the "latest" tag.
59+
push-latest: centos-8-push-latest centos-7-push-latest fedora-32-push-latest fedora-31-push-latest
60+
61+
# Push images with the "v<Major>-latest" tag.
62+
push-major: centos-8-push-major centos-7-push-major fedora-32-push-major fedora-31-push-major
63+
64+
# Push images with the "v<Major>.<Minor>-latest" tag.
65+
push-major-minor: centos-8-push-major-minor centos-7-push-major-minor fedora-32-push-major-minor fedora-31-push-major-minor
66+
67+
#+
68+
# For the following rule patterns, the "%" represents the "distribution" name,
69+
# as derived from the "all" target's *-distro list.
70+
#-
71+
72+
%-all-tagged: %-all %-tags.lis
73+
./apply-tags pbench-agent-all-${@:-all-tagged=} ${@:-all-tagged=}-tags.lis
74+
75+
%-all: %-tools-tagged %-workloads-tagged %-all.Dockerfile
76+
./build-image all ${@:-all=} ${@:-all=}-tags.lis
77+
78+
%-all.Dockerfile: Dockerfile.layered.j2 %-tags.lis
79+
jinja2 Dockerfile.layered.j2 -D distro=${@:-all.Dockerfile=} -D tag="$$(grep -v -E '^v' ${@:-all.Dockerfile=}-tags.lis)" -D kind="all" -D rpms="${_ALL_RPMS}" > ./$@
80+
81+
%-tools-tagged: %-tools %-tags.lis
82+
./apply-tags pbench-agent-tools-${@:-tools-tagged=} ${@:-tools-tagged=}-tags.lis
83+
84+
%-tools: %-base-tagged %-tools.Dockerfile
85+
./build-image tools ${@:-tools=} ${@:-tools=}-tags.lis
86+
87+
%-tools.Dockerfile: Dockerfile.layered.j2 %-tags.lis
88+
jinja2 Dockerfile.layered.j2 -D distro=${@:-tools.Dockerfile=} -D tag="$$(grep -v -E '^v' ${@:-tools.Dockerfile=}-tags.lis)" -D kind="tools" -D rpms="${_TOOL_RPMS}" > ./$@
89+
90+
%-workloads-tagged: %-workloads %-tags.lis
91+
./apply-tags pbench-agent-workloads-${@:-workloads-tagged=} ${@:-workloads-tagged=}-tags.lis
92+
93+
%-workloads: %-base-tagged %-workloads.Dockerfile
94+
./build-image workloads ${@:-workloads=} ${@:-workloads=}-tags.lis
95+
96+
%-workloads.Dockerfile: Dockerfile.layered.j2 %-tags.lis
97+
jinja2 Dockerfile.layered.j2 -D distro=${@:-workloads.Dockerfile=} -D tag="$$(grep -v -E '^v' ${@:-workloads.Dockerfile=}-tags.lis)" -D kind="workloads" -D rpms="${_WORKLOAD_RPMS}" > ./$@
98+
99+
%-base-tagged: %-base
100+
./apply-tags pbench-agent-base-${@:-base-tagged=} ${@:-base-tagged=}-tags.lis
101+
102+
%-base: %-base.Dockerfile %-tags.lis
103+
./build-image base ${@:-base=} ${@:-base=}-tags.lis
104+
105+
#+
106+
# Push local images for the given tag and distribution.
107+
#-
108+
109+
%-push: %-tags.lis
110+
./push ${IMAGE_REPO} ${@:-push=}
111+
112+
%-push-latest: %-tags.lis
113+
./push ${IMAGE_REPO} ${@:-push-latest=} latest
114+
115+
%-push-major: %-tags.lis
116+
./push ${IMAGE_REPO} ${@:-push-major=} _major
117+
118+
%-push-major-minor: %-tags.lis
119+
./push ${IMAGE_REPO} ${@:-push-major-minor=} _minor
120+
121+
#+
122+
# Tag local images for the given distribution.
123+
#-
124+
125+
%-tag-latest: %-tags.lis
126+
./tagit ${@:-tag-latest=} latest
127+
128+
%-tag-major: %-tags.lis
129+
./tagit ${@:-tag-major=} major-latest
130+
131+
%-tag-major-minor: %-tags.lis
132+
./tagit ${@:-tag-major-minor=} major-minor-latest
133+
134+
# Build the tags file for the given distribution.
135+
%-tags.lis:
136+
./gen-tags-from-rpm "${URL_PREFIX}" "${@:-tags.lis=}" "${_ARCH}" > ${@}
137+
138+
#+
139+
# For the following rules, the various CentOS "base" images need a mapping
140+
# between the distribution name and the repo name, which for CentOS images
141+
# is "epel". And for both CentOS and Fedora, the distribution image
142+
# reference, the package manager (dnf vs yum), and image name, also require
143+
# mappings (e.g. centos-7 -> yum, centos:7, CentOS 7, fedora-32 -> dnf,
144+
# fedora:32, Fedora 32).
145+
#-
146+
centos-8-base.Dockerfile: Dockerfile.base.j2 epel-8-pbench.repo
147+
jinja2 Dockerfile.base.j2 -D pbench_repo_file=epel-8-pbench.repo -D pkgmgr=dnf -D distro_image=centos:8 -D distro_image_name="CentOS 8" -o $@
148+
149+
centos-7-base.Dockerfile: Dockerfile.base.j2 epel-7-pbench.repo
150+
jinja2 Dockerfile.base.j2 -D pbench_repo_file=epel-7-pbench.repo -D pkgmgr=yum -D distro_image=centos:7 -D distro_image_name="CentOS 7" -o $@
151+
152+
fedora-32-base.Dockerfile: Dockerfile.base.j2 fedora-32-pbench.repo
153+
jinja2 Dockerfile.base.j2 -D pbench_repo_file=fedora-32-pbench.repo -D pkgmgr=dnf -D distro_image=fedora:32 -D distro_image_name="Fedora 32" -o $@
154+
155+
fedora-31-base.Dockerfile: Dockerfile.base.j2 fedora-31-pbench.repo
156+
jinja2 Dockerfile.base.j2 -D pbench_repo_file=fedora-31-pbench.repo -D pkgmgr=dnf -D distro_image=fedora:31 -D distro_image_name="Fedora 31" -o $@
157+
158+
# Rule pattern dependencies on non-patterned targets have to be set up
159+
# separately for some reason.
160+
%.repo: ${_REPO_TEMPLATE}
161+
162+
%.repo: %.yml
163+
jinja2 ${_REPO_TEMPLATE} ${@:.repo=}.yml -o $@
164+
165+
%.yml: repo.yml.j2
166+
jinja2 repo.yml.j2 -D distro=${@:-pbench.yml=} -D url_prefix=${URL_PREFIX} -o $@
167+
168+
clean:
169+
rm -f *.Dockerfile *.repo *.yml *-tags.lis

‎agent/containers/images/README.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Build Requirements
2+
3+
Container image building requires both the `buildah` and `jinja2` CLI
4+
commands to be present.
5+
6+
On Fedora 31 and later systems, use the following command to install the
7+
required CLI interface RPMs:
8+
9+
sudo dnf install buildah python3-jinja2-cli
10+
11+
# Notes
12+
13+
* We currently only support building container images from previously
14+
built RPMs in an accessible yum/dnf set of repos
15+
16+
* We currently only support container images built from `x86_64`
17+
architecture RPMs
18+
19+
# How to Use
20+
21+
The first step is taken elsewhere, where one would build `pbench-agent`,
22+
and `pbench-sysstat` RPMs, build or find proper RPMs for `fio` & `uperf`,
23+
and place them in a single yum/dnf repository accessible via HTTPS. By
24+
default, we use Fedora COPR repos under the `ndokos` user (one can
25+
override the yum/dnf repos and user via the `URL_PREFIX` and `USER`
26+
environment variables).
27+
28+
Once the proper RPMs are available in the target repo, the default
29+
`Makefile` target, `all`, will build all the default images, and tag
30+
them with the pbench-agent RPM version and git commit hash ID. E.g.,
31+
when done, one might see output from `buildah images` that looks like:
32+
33+
```
34+
$ make all
35+
.
36+
. (lots of build output here ...)
37+
.
38+
$ buildah images
39+
.
40+
.
41+
.
42+
localhost/pbench-agent-base-centos-7 958aeba4 5499e5521f50 ...
43+
localhost/pbench-agent-base-centos-7 v0.69.3-1 5499e5521f50 ...
44+
localhost/pbench-agent-all-centos-8 958aeba4 9396f0337681 ...
45+
localhost/pbench-agent-all-centos-8 v0.69.3-1 9396f0337681 ...
46+
.
47+
.
48+
.
49+
```
50+
51+
There are make targets for each of the four supported distributions,
52+
CentOS 8 (`centos-8`), CentOS 7 (`centos-7`), Fedora 32 (`fedora-32`),
53+
and Fedora 31 (`fedora-31`). There are also make targets for each
54+
subset of the four images (all, base, tools, workloads) built for
55+
each distribution, e.g. `centos-8-tools`, `fedora-31-base`, etc.
56+
57+
Two tags are always applied to an image that is built, the `<git
58+
commit ID>` derived from the RPM version, and the version string of
59+
the RPM itself (without the trailing commit ID).
60+
61+
One can add additional local tags using the following targets when
62+
appropriate (these tags are not automatically applied at build time):
63+
64+
* `tag-latest` - adds the `latest` label to the images with the
65+
`<git commit ID>` as derived from the RPM version string of the
66+
pbench-agent RPM
67+
68+
* `tag-major` - adds the `v<Major>-latest` label to the images
69+
as derived from the RPM version string ...
70+
71+
* `tag-major-minor` -adds the `v<Major>.<Minor>-latest` label to
72+
the images ...
73+
74+
Finally, there are "push" targets to copy the locally built and
75+
tagged images to a non-local container image repository. By default
76+
we use `docker://quay.io/pbench` (you can override that via an
77+
environment variable). We have separate push targets to allow the
78+
administrator of the container image repository to label the images
79+
based on what they have built in relation to what has been published
80+
already. The push targets are:
81+
82+
* `push` - pushes all the images by their `<git commit ID>` tag,
83+
and their RPM version tag
84+
85+
* `push-latest` - pushes all the images by their `latest` tag
86+
87+
* `push-major` - pushes all the images by their `v<Major>-latest`
88+
tag
89+
90+
* `push-major-minor` - pushes all the images by their
91+
`v<Major>.<Minor>-latest` tag
92+
93+
NOTE WELL: Each separate tag for each image needs to be pushed to
94+
the non-local container image repository. This does NOT result in
95+
multiple image copies over the wire using up network bandwidth, as
96+
`buildah push` is smart enough push the actual image only once.

‎agent/containers/images/apply-tags‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash
2+
3+
imageName="${1}"
4+
tags_file="${2}"
5+
if [[ "${3}" == "latest" ]]; then
6+
# grep below for the tag line with "latest" in it.
7+
latest=""
8+
else
9+
# grep below for the tag line WITHOUT "latest" in it.
10+
latest="-v"
11+
fi
12+
13+
githash="$(grep -v -E "^v" ${tags_file})"
14+
15+
function apply_tag {
16+
buildah tag ${1}:${githash} ${1}:${2}
17+
}
18+
19+
for tag in $(grep -E "^v" ${tags_file} | grep ${latest} latest); do
20+
apply_tag ${imageName} ${tag}
21+
done

‎agent/containers/images/build-image‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
3+
image="${1}"
4+
distro="${2}"
5+
tags_file="${3}"
6+
7+
githash="$(grep -v -E "^v" ${tags_file})"
8+
9+
buildah bud -f ${distro}-${image}.Dockerfile -t pbench-agent-${image}-${distro}:${githash} .

‎agent/containers/images/gen-tags-from-rpm‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/bin/bash
2+
3+
url_prefix="${1}"
4+
dist="${2}"
5+
arch="${3}"
6+
7+
if [[ "${dist}" == "centos-8" ]]; then
8+
dist="epel-8"
9+
elif [[ "${dist}" == "centos-7" ]]; then
10+
dist="epel-7"
11+
fi
12+
13+
# Ensure we only consider the pbench-agent RPM's version string from the
14+
# target repo.
15+
version_string="$(yum list -q --repofrompath pbench,"${url_prefix}/pbench/${dist}-${arch}" pbench-agent.noarch 2>/dev/null | awk 'FNR==2{print $2}')"
16+
if [[ -z "${version_string}" ]]; then
17+
printf -- "Failed to fetch pbench-agent.noarch version string from '%s/pbench/%s-%s'\n" "${url_prefix}" "${dist}" "${arch}" >&2
18+
exit 1
19+
fi
20+
21+
# 0.69.3-1
22+
ver="${version_string%g*}"
23+
major="${ver%%.*}"
24+
major_minor="${ver%.*}"
25+
commit_id="${version_string#*g}"
26+
27+
printf -- "%s\n" "${commit_id}"
28+
printf -- "v%s\n" "${ver}"
29+
printf -- "v%s-latest v%s-latest\n" "${major}" "${major_minor}"
30+
31+
exit 0

0 commit comments

Comments
 (0)