Skip to content

Commit e9f4b8a

Browse files
dagrayviddagrayvid
committed
Add driver-toolkit to OpenShift docs
1 parent bf8fa0b commit e9f4b8a

File tree

5 files changed

+342
-0
lines changed

5 files changed

+342
-0
lines changed

‎_topic_map.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1913,6 +1913,9 @@ Topics:
19131913
- Name: The Node Feature Discovery Operator
19141914
File: psap-node-feature-discovery-operator
19151915
Distros: openshift-origin,openshift-enterprise
1916+
- Name: The Driver Toolkit
1917+
File: psap-driver-toolkit
1918+
Distros: openshift-origin,openshift-enterprise
19161919
- Name: Planning your environment according to object maximums
19171920
File: planning-your-environment-according-to-object-maximums
19181921
Distros: openshift-origin,openshift-enterprise

‎modules/psap-driver-toolkit-pulling.adoc‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/psap-driver-toolkit.adoc
4+
5+
[id="pulling-the-driver-toolkit"]
6+
= Pulling the Driver Toolkit container image
7+
8+
The `driver-toolkit` image is available from the link:https://registry.redhat.io/[Container images section of the Red Hat Ecosystem Catalog] and in the {product-title} release payload. The image corresponding to the most recent minor release of {product-title} will be tagged with the version number in the catalog. The image URL for a specific release can be found using the `oc adm` CLI command.
9+
10+
[id="pulling-the-driver-toolkit-from-registry"]
11+
== Pulling the Driver Toolkit container image from registry.redhat.io
12+
13+
Instructions for pulling the `driver-toolkit` image from registry.redhat.io with podman, or in OpenShift can be found on the link:https://catalog.redhat.com/software/containers/openshift4/driver-toolkit-rhel8/604009d6122bd89307e00865?container-tabs=gti[Red Hat Ecosystem Catalog].
14+
The driver-toolkit image for the latest minor release will be tagged with the minor release version on registry.redhat.io for example `registry.redhat.io/openshift4/driver-toolkit-rhel8:v4.8`.
15+
16+
[id="pulling-the-driver-toolkit-from-payload"]
17+
== Finding the Driver Toolkit image URL in the payload
18+
19+
.Prerequisites
20+
21+
* Obtain the image pull secret needed to perform an installation of {product-title}, from the link:https://cloud.redhat.com/openshift/install/pull-secret[Pull Secret] page on the {cloud-redhat-com} site.
22+
* Install the OpenShift CLI (`oc`).
23+
24+
.Procedure
25+
26+
. The image URL of the `driver-toolkit` corresponding to a certain release can be extracted from the release image using the `oc adm` command:
27+
+
28+
[source,terminal]
29+
----
30+
$ oc adm release info 4.8.0 --image-for=driver-toolkit
31+
----
32+
+
33+
.Example output
34+
[source,terminal]
35+
----
36+
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd84aee79606178b6561ac71f8540f404d518ae5deff45f6d6ac8f02636c7f4
37+
----
38+
39+
. This image can be pulled using a valid pull secret, such as the pull secret required to install {product-title}.
40+
41+
[source,terminal]
42+
----
43+
$ podman pull --authfile=path/to/pullsecret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<SHA>
44+
----

‎modules/psap-driver-toolkit-using.adoc‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/psap-driver-toolkit.adoc
4+
5+
[id="using-the-driver-toolkit"]
6+
= Using the Driver Toolkit
7+
8+
As an example, the Driver Toolkit can be used as the base image for building a very simple kernel module called simple-kmod.
9+
10+
[id="create-simple-kmod-image"]
11+
== Build and run the simple-kmod driver container on a cluster
12+
13+
.Prerequisites
14+
15+
* An {product-title} cluster
16+
* Install the OpenShift CLI (`oc`).
17+
* Log in as a user with `cluster-admin` privileges.
18+
19+
.Procedure
20+
21+
Create a namespace. For example:
22+
[source,terminal]
23+
-----
24+
$ oc new-project simple-kmod-demo
25+
-----
26+
27+
. The YAML defines an `ImageStream` for storing the `simple-kmod` driver container image, and a `BuildConfig` for building the container. Save this YAML as `0000-buildconfig.yaml.template`.
28+
+
29+
[source,yaml]
30+
----
31+
apiVersion: image.openshift.io/v1
32+
kind: ImageStream
33+
metadata:
34+
labels:
35+
app: simple-kmod-driver-container
36+
name: simple-kmod-driver-container
37+
namespace: simple-kmod-demo
38+
spec: {}
39+
---
40+
apiVersion: build.openshift.io/v1
41+
kind: BuildConfig
42+
metadata:
43+
labels:
44+
app: simple-kmod-driver-build
45+
name: simple-kmod-driver-build
46+
namespace: simple-kmod-demo
47+
spec:
48+
nodeSelector:
49+
node-role.kubernetes.io/worker: ""
50+
runPolicy: "Serial"
51+
triggers:
52+
- type: "ConfigChange"
53+
- type: "ImageChange"
54+
source:
55+
git:
56+
ref: "master"
57+
uri: "https://github.com/openshift-psap/kvc-simple-kmod.git"
58+
type: Git
59+
dockerfile: |
60+
FROM DRIVER_TOOLKIT_IMAGE
61+
62+
WORKDIR /build/
63+
64+
RUN yum -y install git make sudo gcc \
65+
&& yum clean all \
66+
&& rm -rf /var/cache/dnf
67+
68+
# Expecting kmod software version as an input to the build
69+
ARG KMODVER
70+
71+
# Grab the software from upstream
72+
RUN git clone https://github.com/openshift-psap/simple-kmod.git
73+
WORKDIR simple-kmod
74+
75+
# Prep and build the module
76+
RUN make buildprep KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}" kernel-core) KMODVER=${KMODVER} \
77+
&& make all KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}" kernel-core) KMODVER=${KMODVER} \
78+
&& make install KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}" kernel-core) KMODVER=${KMODVER}
79+
80+
# Add the helper tools
81+
WORKDIR /root/kvc-simple-kmod
82+
ADD Makefile .
83+
ADD simple-kmod-lib.sh .
84+
ADD simple-kmod-wrapper.sh .
85+
ADD simple-kmod.conf .
86+
RUN mkdir -p /usr/lib/kvc/ \
87+
&& mkdir -p /etc/kvc/ \
88+
&& make install
89+
90+
RUN systemctl enable kmods-via-containers@simple-kmod
91+
strategy:
92+
dockerStrategy:
93+
buildArgs:
94+
- name: KMODVER
95+
value: DEMO
96+
output:
97+
to:
98+
kind: ImageStreamTag
99+
name: simple-kmod-driver-container:demo
100+
----
101+
102+
. Substitute the correct driver toolkit image for the {product-title} version you are running in place of “DRIVER_TOOLKIT_IMAGE” with the following commands.
103+
+
104+
[source,terminal]
105+
----
106+
$ OCP_VERSION=$(oc get clusterversion/version -ojsonpath={.status.desired.version})
107+
----
108+
+
109+
[source,terminal]
110+
----
111+
$ DRIVER_TOOLKIT_IMAGE=$(oc adm release info $OCP_VERSION --image-for=driver-toolkit)
112+
----
113+
+
114+
[source,terminal]
115+
----
116+
$ sed "s#DRIVER_TOOLKIT_IMAGE#${DRIVER_TOOLKIT_IMAGE}#" 0000-buildconfig.yaml.template > 0000-buildconfig.yaml
117+
----
118+
+
119+
[NOTE]
120+
====
121+
The driver toolkit was introduced to {product-title} 4.6 as of version 4.6.30, in 4.7 as of version 4.7.11, and in 4.8.
122+
====
123+
124+
. Create the image stream and build config with
125+
+
126+
[source,terminal]
127+
----
128+
$ oc create -f 0000-buildconfig.yaml
129+
----
130+
131+
. Once the builder pod completes successfully, deploy the driver container image as a `DaemonSet`.
132+
133+
.. The driver container must run with the privileged security context in order to load the kernel modules on the host. The following YAML file contains the RBAC rules and the `DaemonSet` for running the driver container. Save this YAML as `1000-driver-container.yaml`.
134+
+
135+
[source,yaml]
136+
----
137+
apiVersion: v1
138+
kind: ServiceAccount
139+
metadata:
140+
name: simple-kmod-driver-container
141+
---
142+
apiVersion: rbac.authorization.k8s.io/v1
143+
kind: Role
144+
metadata:
145+
name: simple-kmod-driver-container
146+
rules:
147+
- apiGroups:
148+
- security.openshift.io
149+
resources:
150+
- securitycontextconstraints
151+
verbs:
152+
- use
153+
resourceNames:
154+
- privileged
155+
---
156+
apiVersion: rbac.authorization.k8s.io/v1
157+
kind: RoleBinding
158+
metadata:
159+
name: simple-kmod-driver-container
160+
roleRef:
161+
apiGroup: rbac.authorization.k8s.io
162+
kind: Role
163+
name: simple-kmod-driver-container
164+
subjects:
165+
- kind: ServiceAccount
166+
name: simple-kmod-driver-container
167+
userNames:
168+
- system:serviceaccount:simple-kmod-demo:simple-kmod-driver-container
169+
---
170+
apiVersion: apps/v1
171+
kind: DaemonSet
172+
metadata:
173+
name: simple-kmod-driver-container
174+
spec:
175+
selector:
176+
matchLabels:
177+
app: simple-kmod-driver-container
178+
template:
179+
metadata:
180+
labels:
181+
app: simple-kmod-driver-container
182+
spec:
183+
serviceAccount: simple-kmod-driver-container
184+
serviceAccountName: simple-kmod-driver-container
185+
containers:
186+
- image: image-registry.openshift-image-registry.svc:5000/simple-kmod-demo/simple-kmod-driver-container:demo
187+
name: simple-kmod-driver-container
188+
imagePullPolicy: Always
189+
command: ["/sbin/init"]
190+
lifecycle:
191+
preStop:
192+
exec:
193+
command: ["/bin/sh", "-c", "systemctl stop kmods-via-containers@simple-kmod"]
194+
securityContext:
195+
privileged: true
196+
nodeSelector:
197+
node-role.kubernetes.io/worker: ""
198+
----
199+
200+
.. Create the RBAC rules and daemon set:
201+
+
202+
[source,terminal]
203+
----
204+
$ oc create -f 1000-drivercontainer.yaml
205+
----
206+
207+
. Once the pods are running on the worker nodes, verify that the `simple_kmod` kernel module is loaded successfully on the host machines with `lsmod`.
208+
209+
.. Verify that the pods are running:
210+
+
211+
[source,terminal]
212+
----
213+
$ oc get pod -n simple-kmod-demo
214+
----
215+
+
216+
.Example output
217+
[source,terminal]
218+
----
219+
NAME READY STATUS RESTARTS AGE
220+
simple-kmod-driver-build-1-build 0/1 Completed 0 6m
221+
simple-kmod-driver-container-b22fd 1/1 Running 0 40s
222+
simple-kmod-driver-container-jz9vn 1/1 Running 0 40s
223+
simple-kmod-driver-container-p45cc 1/1 Running 0 40s
224+
----
225+
226+
.. Execute the `lsmod` command in the driver container pod:
227+
+
228+
[source,terminal]
229+
----
230+
$ oc exec -it pod/simple-kmod-driver-container-p45cc -- lsmod | grep simple
231+
----
232+
+
233+
.Example output
234+
[source,terminal]
235+
----
236+
simple_procfs_kmod 16384 0
237+
simple_kmod 16384 0
238+
----

‎modules/psap-driver-toolkit.adoc‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/psap-driver-toolkit.adoc
4+
5+
[id="about-driver-toolkit"]
6+
= About the Driver Toolkit
7+
8+
[discrete]
9+
== Background
10+
The Driver Toolkit is a container image in the {product-title} payload used as a base image on which you can build driver containers. The Driver Toolkit image contains the kernel packages commonly required as dependencies to build or install kernel modules, as well as a few tools needed in driver containers. The version of these packages will match the kernel version running on the {op-system-first} nodes in the corresponding {product-title} release.
11+
12+
Driver containers are container images used for building and deploying out-of-tree kernel modules and drivers on container operating systems like {op-system}. Kernel modules and drivers are software libraries running with a high level of privilege in the operating system kernel. They extend the kernel functionalities or provide the hardware-specific code required to control new devices. Examples include hardware devices like Field Programmable Gate Arrays (FPGA) or GPUs, and software-defined storage (SDS) solutions, such as Lustre parallel file systems, which require kernel modules on client machines. Driver containers are the first layer of the software stack used to enable these technologies on Kubernetes.
13+
14+
The list of kernel packages in the Driver Toolkit includes the following and their dependencies:
15+
16+
* `kernel-core`
17+
* `kernel-devel`
18+
* `kernel-headers`
19+
* `kernel-modules`
20+
* `kernel-modules-extra`
21+
22+
In addition, the Driver Toolkit also includes the corresponding real-time kernel packages:
23+
24+
* `kernel-rt-core`
25+
* `kernel-rt-devel`
26+
* `kernel-rt-modules`
27+
* `kernel-rt-modules-extra`
28+
29+
The Driver Toolkit also has several tools which are commonly needed to build and install kernel modules, including:
30+
31+
* `elfutils-libelf-devel`
32+
* `kmod`
33+
* `binutilskabi-dw`
34+
* `kernel-abi-whitelists`
35+
* dependencies for the above
36+
37+
[discrete]
38+
== Purpose
39+
Prior to the Driver Toolkit's existence, you could install kernel packages in a pod or build config on {product-title} using link:https://www.openshift.com/blog/how-to-use-entitled-image-builds-to-build-drivercontainers-with-ubi-on-openshift[entitled builds] or by installing from the kernel RPMs in the hosts `machine-os-content`. The Driver Toolkit simplifies the process by removing the entitlement step, and avoids the privileged operation of accessing the machine-os-content in a pod. The Driver Toolkit can also be used by partners who have access to pre-released {product-title} versions to prebuild driver-containers for their hardware devices for future {product-title} releases.
40+
41+
The Driver Toolkit is also used by the Special Resource Operator (SRO), which is currently available as a community Operator on OperatorHub. SRO supports out-of-tree and third-party kernel drivers and the support software for the underlying operating system. Users can create _recipes_ for SRO to build and deploy a driver container, as well as support software like a device plug-in, or metrics. Recipes can include a build config to build a driver container based on the Driver Toolkit, or SRO can deploy a prebuilt driver container.
42+

‎scalability_and_performance/psap-driver-toolkit.adoc‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[id="driver-toolkit"]
2+
= The Driver Toolkit
3+
include::modules/common-attributes.adoc[]
4+
:context: driver-toolkit
5+
6+
toc::[]
7+
8+
Learn about the Driver Toolkit and how you can use it as a base image for driver containers for enabling special software and hardware devices on Kubernetes.
9+
10+
include::modules/psap-driver-toolkit.adoc[leveloffset=+1]
11+
12+
include::modules/psap-driver-toolkit-pulling.adoc[leveloffset=+1]
13+
14+
include::modules/psap-driver-toolkit-using.adoc[leveloffset=+1]
15+

0 commit comments

Comments
 (0)