Skip to content

Commit 5f36791

Browse files
jab-rhjab-rh
authored
Merge pull request #58024 from StephenJamesSmith/TELCODOCS-992
Telcodocs 992: deploying modules and creating a multiloader image
2 parents 1a548d3 + c2cf044 commit 5f36791

10 files changed

+387
-3
lines changed

‎hardware_enablement/kmm-kernel-module-management.adoc

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@
44
include::_attributes/common-attributes.adoc[]
55
:context: kernel-module-management-operator
66

7-
87
toc::[]
98

109
Learn about the Kernel Module Management (KMM) Operator and how you can use it to deploy out-of-tree kernel modules and device plugins on {product-title} clusters.
@@ -16,6 +15,47 @@ include::modules/kmm-installation.adoc[leveloffset=+1]
1615
include::modules/kmm-installing-using-web-console.adoc[leveloffset=+2]
1716
include::modules/kmm-installing-using-cli.adoc[leveloffset=+2]
1817
include::modules/kmm-installing-older-versions.adoc[leveloffset=+2]
18+
include::modules/kmm-deploying-modules.adoc[leveloffset=+1]
19+
include::modules/kmm-creating-module-cr.adoc[leveloffset=+2]
20+
include::modules/kmm-security.adoc[leveloffset=+2]
21+
22+
[role="_additional-resources"]
23+
.Additional resources
24+
25+
* xref:../authentication/understanding-and-managing-pod-security-admission.adoc#understanding-and-managing-pod-security-admission[Understanding and managing pod security admission].
26+
27+
include::modules/kmm-example-module-cr.adoc[leveloffset=+2]
28+
include::modules/kmm-creating-moduleloader-image.adoc[leveloffset=+1]
29+
include::modules/kmm-running-depmod.adoc[leveloffset=+2]
30+
31+
[role="_additional-resources"]
32+
.Additional resources
33+
34+
* xref:../hardware_enablement/psap-driver-toolkit.adoc#driver-toolkit[Driver Toolkit].
35+
36+
include::modules/kmm-building-in-cluster.adoc[leveloffset=+2]
37+
38+
[role="_additional-resources"]
39+
.Additional resources
40+
41+
* xref:../cicd/builds/build-configuration.adoc#build-configuration[Build configuration resources].
42+
43+
include::modules/kmm-using-driver-toolkit.adoc[leveloffset=+2]
44+
45+
[role="_additional-resources"]
46+
.Additional resources
47+
48+
* xref:../hardware_enablement/psap-driver-toolkit.adoc#driver-toolkit[Driver Toolkit].
49+
50+
//Deploying kernel modules (Might just leave this short intro in the assembly and put further module below it)
51+
// * Running ModuleLoader images (CONCEPT, or could be included in the assembly with the intro)
52+
// * Using the device plugin (CONCEPT, or could be included in the assembly with the intro)
53+
// * Creating the Module Custom Resource (PROCEDURE? Seems like not a process the user does after reading it. Maybe a REFERENCE)
54+
// * Security and permissions (CONCEPT or REFERENCE)
55+
// * ServiceAccounts and SecurityContextConstraints (can include in Security and permissions)
56+
// * Pod Security Standards (can include in Security and permissions)
57+
// * Example Module CR (REFERENCE)
58+
1959
// Added for TELCODOCS-1065
2060
include::modules/kmm-using-signing-with-kmm.adoc[leveloffset=+1]
2161
include::modules/kmm-adding-the-keys-for-secureboot.adoc[leveloffset=+1]

‎modules/kmm-building-in-cluster.adoc

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="kmm-building-in-cluster_{context}"]
7+
8+
= Building in the cluster
9+
10+
KMM can build module loader images in the cluster. Follow these guidelines:
11+
12+
* Provide build instructions using the `build` section of a kernel mapping.
13+
* Copy the `Dockerfile` for your container image into a `ConfigMap` resource, under the `dockerfile` key.
14+
* Ensure that the `ConfigMap` is located in the same namespace as the `Module`.
15+
16+
KMM checks if the image name specified in the `containerImage` field exists. If it does, the build is skipped.
17+
18+
Otherwise, KMM creates a `Build` resource to build your image. After the image is built, KMM proceeds with the `Module` reconciliation. See the following example.
19+
20+
[source,yaml]
21+
----
22+
# ...
23+
- regexp: '^.+$'
24+
containerImage: "some.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
25+
build:
26+
buildArgs: <1>
27+
- name: ARG_NAME
28+
value: <some_value>
29+
secrets: <2>
30+
- name: <some_kubernetes_secret> <3>
31+
baseImageRegistryTLS:
32+
insecure: false <4>
33+
insecureSkipTLSVerify: false <5>
34+
dockerfileConfigMap: <6>
35+
name: <my_kmod_dockerfile>
36+
registryTLS:
37+
insecure: false <7>
38+
insecureSkipTLSVerify: false <8>
39+
----
40+
<1> Optional.
41+
<2> Optional.
42+
<3> Will be mounted in the build pod as `/run/secrets/some-kubernetes-secret`.
43+
<4> Optional: Avoid using this parameter. If set to `true`, the build will be allowed to pull the image in the Dockerfile `FROM` instruction using plain HTTP.
44+
<5> Optional: Avoid using this parameter. If set to `true`, the build will skip any TLS server certificate validation when pulling the image in the Dockerfile `FROM` instruction using plain HTTP.
45+
<6> Required.
46+
<7> Optional: Avoid using this parameter. If set to `true`, KMM will be allowed to check if the container image already exists using plain HTTP.
47+
<8> Optional: Avoid using this parameter. If set to `true`, KMM will skip any TLS server certificate validation when checking if the container image already exists.

‎modules/kmm-creating-module-cr.adoc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="kmm-creating-module-cr_{context}"]
7+
8+
= The Module custom resource definition
9+
10+
The `Module` custom resource definition (CRD) represents a kernel module that can be loaded on all or select nodes in the cluster, through a module loader image.
11+
A `Module` custom resource (CR) specifies one or more kernel versions with which it is compatible, and a node selector.
12+
13+
The compatible versions for a `Module` resource are listed under `.spec.moduleLoader.container.kernelMappings`.
14+
A kernel mapping can either match a `literal` version, or use `regexp` to match many of them at the same time.
15+
16+
The reconciliation loop for the `Module` resource runs the following steps:
17+
18+
. List all nodes matching `.spec.selector`.
19+
. Build a set of all kernel versions running on those nodes.
20+
. For each kernel version:
21+
.. Go through `.spec.moduleLoader.container.kernelMappings` and find the appropriate container image name. If the kernel mapping has `build` or `sign` defined and the container image does not already exist, run the build, the signing job, or both, as needed.
22+
.. Create a module loader daemon set with the container image determined in the previous step.
23+
.. If `.spec.devicePlugin` is defined, create a device plugin daemon set using the configuration specified under `.spec.devicePlugin.container`.
24+
. Run `garbage-collect` on:
25+
.. Existing daemon set resources targeting kernel versions that are not run by any node in the cluster.
26+
.. Successful build jobs.
27+
.. Successful signing jobs.

‎modules/kmm-creating-moduleloader-image.adoc

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="kmm-creating-moduleloader-image_{context}"]
7+
= Using a ModuleLoader image
8+
9+
Kernel Module Management (KMM) works with purpose-built module loader images.
10+
These are standard OCI images that must satisfy the following requirements:
11+
12+
* `.ko` files must be located in `+/opt/lib/modules/${KERNEL_VERSION}+`.
13+
* `modprobe` and `sleep` binaries must be defined in the `$PATH` variable.

‎modules/kmm-deploying-modules.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="kmm-deploy-kernel-modules_{context}"]
7+
= Kernel module deployment
8+
9+
For each `Module` resource, Kernel Module Management (KMM) can create a number of `DaemonSet` resources:
10+
11+
* One ModuleLoader `DaemonSet` per compatible kernel version running in the cluster.
12+
* One device plugin `DaemonSet`, if configured.
13+
14+
The module loader daemon set resources run ModuleLoader images to load kernel modules.
15+
A module loader image is an OCI image that contains the `.ko` files and both the `modprobe` and `sleep` binaries.
16+
17+
When the module loader pod is created, the pod runs `modprobe` to insert the specified module into the kernel.
18+
It then enters a sleep state until it is terminated.
19+
When that happens, the `ExecPreStop` hook runs `modprobe -r` to unload the kernel module.
20+
21+
If the `.spec.devicePlugin` attribute is configured in a `Module` resource, then KMM creates a link:https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/[device plugin]
22+
daemon set in the cluster.
23+
That daemon set targets:
24+
25+
* Nodes that match the `.spec.selector` of the `Module` resource.
26+
* Nodes with the kernel module loaded (where the module loader pod is in the `Ready` condition).

‎modules/kmm-example-module-cr.adoc

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: REFERENCE
6+
[id="kmm-example-cr_{context}"]
7+
8+
= Example Module CR
9+
10+
The following is an annotated `Module` example:
11+
12+
[source,yaml]
13+
----
14+
apiVersion: kmm.sigs.x-k8s.io/v1beta1
15+
kind: Module
16+
metadata:
17+
name: <my_kmod>
18+
spec:
19+
moduleLoader:
20+
container:
21+
modprobe:
22+
moduleName: <my_kmod> <1>
23+
dirName: /opt <2>
24+
firmwarePath: /firmware <3>
25+
parameters: <4>
26+
- param=1
27+
kernelMappings: <5>
28+
- literal: 6.0.15-300.fc37.x86_64
29+
containerImage: some.registry/org/my-kmod:6.0.15-300.fc37.x86_64
30+
- regexp: '^.+\fc37\.x86_64$' <6>
31+
containerImage: "some.other.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
32+
- regexp: '^.+$' <7>
33+
containerImage: "some.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
34+
build:
35+
buildArgs: <8>
36+
- name: ARG_NAME
37+
value: <some_value>
38+
secrets:
39+
- name: <some_kubernetes_secret> <9>
40+
baseImageRegistryTLS: <10>
41+
insecure: false
42+
insecureSkipTLSVerify: false <11>
43+
dockerfileConfigMap: <12>
44+
name: <my_kmod_dockerfile>
45+
sign:
46+
certSecret:
47+
name: <cert_secret> <13>
48+
keySecret:
49+
name: <key_secret> <14>
50+
filesToSign:
51+
- /opt/lib/modules/${KERNEL_FULL_VERSION}/<my_kmod>.ko
52+
registryTLS: <15>
53+
insecure: false <16>
54+
insecureSkipTLSVerify: false
55+
serviceAccountName: <sa_module_loader> <17>
56+
devicePlugin: <18>
57+
container:
58+
image: some.registry/org/device-plugin:latest <19>
59+
env:
60+
- name: MY_DEVICE_PLUGIN_ENV_VAR
61+
value: SOME_VALUE
62+
volumeMounts: <20>
63+
- mountPath: /some/mountPath
64+
name: <device_plugin_volume>
65+
volumes: <21>
66+
- name: <device_plugin_volume>
67+
configMap:
68+
name: <some_configmap>
69+
serviceAccountName: <sa_device_plugin> <22>
70+
imageRepoSecret: <23>
71+
name: <secret_name>
72+
selector:
73+
node-role.kubernetes.io/worker: ""
74+
----
75+
<1> Required.
76+
<2> Optional.
77+
<3> Optional: Copies `/firmware/*` into `/var/lib/firmware/` on the node.
78+
<4> Optional.
79+
<5> At least one kernel item is required.
80+
<6> For each node running a kernel matching the regular expression, KMM creates a `DaemonSet` resource running the image specified in `containerImage` with `${KERNEL_FULL_VERSION}` replaced with the kernel version.
81+
<7> For any other kernel, build the image using the Dockerfile in the `my-kmod` ConfigMap.
82+
<8> Optional.
83+
<9> Optional: A value for `some-kubernetes-secret` can be obtained from the build environment at `/run/secrets/some-kubernetes-secret`.
84+
<10> Optional: Avoid using this parameter. If set to `true`, the build is allowed to pull the image in the Dockerfile `FROM` instruction using plain HTTP.
85+
<11> Optional: Avoid using this parameter. If set to `true`, the build will skip any TLS server certificate validation when pulling the image in the Dockerfile `FROM` instruction using plain HTTP.
86+
<12> Required.
87+
<13> Required: A secret holding the public secureboot key with the key 'cert'.
88+
<14> Required: A secret holding the private secureboot key with the key 'key'.
89+
<15> Optional: Avoid using this parameter. If set to `true`, KMM will be allowed to check if the container image already exists using plain HTTP.
90+
<16> Optional: Avoid using this parameter. If set to `true`, KMM will skip any TLS server certificate validation when checking if the container image already exists.
91+
<17> Optional.
92+
<18> Optional.
93+
<19> Required: If the device plugin section is present.
94+
<20> Optional.
95+
<21> Optional.
96+
<22> Optional.
97+
<23> Optional: Used to pull module loader and device plugin images.

‎modules/kmm-running-depmod.adoc

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="kmm-running-depmod_{context}"]
7+
8+
= Running depmod
9+
10+
If your module loader image contains several kernel modules and if one of the modules depends on another module, it is best practice to run `depmod` at the end of the build process to generate dependencies and map files.
11+
12+
[NOTE]
13+
====
14+
You must have a Red Hat subscription to download the `kernel-devel` package.
15+
====
16+
17+
.Procedure
18+
19+
. To generate `modules.dep` and `.map` files for a specific kernel version, run `+depmod -b /opt ${KERNEL_VERSION}+`.
20+
21+
[id="example-dockerfile_{context}"]
22+
== Example Dockerfile
23+
24+
If you are building your image on {product-title}, consider using the Driver Tool Kit (DTK).
25+
26+
For further information, see link:https://cloud.redhat.com/blog/how-to-use-entitled-image-builds-to-build-drivercontainers-with-ubi-on-openshift[using an entitled build].
27+
28+
[source,yaml]
29+
----
30+
apiVersion: v1
31+
kind: ConfigMap
32+
metadata:
33+
name: kmm-ci-dockerfile
34+
data:
35+
dockerfile: |
36+
ARG DTK_AUTO
37+
FROM ${DTK_AUTO} as builder
38+
ARG KERNEL_VERSION
39+
WORKDIR /usr/src
40+
RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"]
41+
WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod
42+
RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_VERSION}/build make all
43+
FROM registry.redhat.io/ubi8/ubi-minimal
44+
ARG KERNEL_VERSION
45+
RUN microdnf install kmod
46+
COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_VERSION}/
47+
COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_VERSION}/
48+
RUN depmod -b /opt ${KERNEL_VERSION}
49+
----

‎modules/kmm-security.adoc

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_content-type: REFERENCE
6+
[id="kmm-security_{context}"]
7+
= Security and permissions
8+
9+
[IMPORTANT]
10+
====
11+
Loading kernel modules is a highly sensitive operation.
12+
After they are loaded, kernel modules have all possible permissions to do any kind of operation on the node.
13+
====
14+
15+
[id="serviceaccounts-and-securitycontextconstraint_{context}"]
16+
== ServiceAccounts and SecurityContextConstraints
17+
18+
Kernel Module Management (KMM) creates a privileged workload to load the kernel modules on nodes.
19+
That workload needs `ServiceAccounts` allowed to use the `privileged` `SecurityContextConstraint` (SCC) resource.
20+
21+
The authorization model for that workload depends on the namespace of the `Module` resource, as well as its spec.
22+
23+
* If the `.spec.moduleLoader.serviceAccountName` or `.spec.devicePlugin.serviceAccountName` fields are set, they are always used.
24+
* If those fields are not set, then:
25+
** If the `Module` resource is created in the operator's namespace (`openshift-kmm` by default), then KMM uses its default, powerful `ServiceAccounts` to run the daemon sets.
26+
** If the `Module` resource is created in any other namespace, then KMM runs the daemon sets as the namespace's `default` `ServiceAccount`. The `Module` resource cannot run a privileged workload unless you manually enable it to use the `privileged` SCC.
27+
28+
[IMPORTANT]
29+
====
30+
`openshift-kmm` is a trusted namespace.
31+
32+
When setting up RBAC permissions, remember that any user or `ServiceAccount` creating a `Module` resource in the `openshift-kmm` namespace results in KMM automatically running privileged workloads on potentially all nodes in the cluster.
33+
====
34+
35+
To allow any `ServiceAccount` to use the `privileged` SCC and therefore to run module loader or device plugin pods, use the following command:
36+
37+
[source,terminal]
38+
----
39+
$ oc adm policy add-scc-to-user privileged -z "${serviceAccountName}" [ -n "${namespace}" ]
40+
----
41+
42+
[id="pod-security-standards_{context}"]
43+
== Pod security standards
44+
45+
OpenShift runs a synchronization mechanism that sets the namespace Pod Security level automatically based on
46+
the security contexts in use. No action is needed.

0 commit comments

Comments
 (0)