Skip to content

Commit 4407caf

Browse files
michaelryanpetermichaelryanpeter
authored
Merge pull request #64355 from michaelryanpeter/opeco-4.14-mixed-arch
OSDOCS#4112[OPECO][4.14] Multi-arch support for Operator projects
2 parents 009f539 + 12831f8 commit 4407caf

8 files changed

+411
-0
lines changed

‎_topic_maps/_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1856,6 +1856,8 @@ Topics:
18561856
File: osdk-monitoring-prometheus
18571857
- Name: Configuring leader election
18581858
File: osdk-leader-election
1859+
- Name: Configuring support for multiple platforms
1860+
File: osdk-multi-arch-support
18591861
- Name: Object pruning utility
18601862
File: osdk-pruning-utility
18611863
- Name: Migrating package manifest projects to bundle format

‎modules/osdk-multi-arch-building-images.adoc‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/operator_sdk/osdk-multi-arch-support.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="osdk-multi-arch-building-images_{context}"]
7+
= Building a manifest list of the platforms your Operator supports
8+
9+
You can use the `make docker-buildx` command to build a manifest list of the platforms supported by your Operator and operands. A manifest list references specific image manifests for one or more architectures. An image manifest specifies the platforms that an image supports.
10+
11+
For more information, see link:https://specs.opencontainers.org/image-spec/image-index[OpenContainers Image Index Spec] or link:https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list[Image Manifest v2, Schema 2].
12+
13+
[IMPORTANT]
14+
====
15+
If your Operator project deploys an application or other workload resources, the following procedure assumes the application's multi-platform images are built during the application release process.
16+
====
17+
18+
.Prerequisites
19+
20+
* An Operator project built using the Operator SDK version {osdk_ver} or later
21+
* Docker installed
22+
23+
.Procedure
24+
25+
. Inspect the image manifests of your Operator and operands to find which platforms your Operator project can support. Run the following command to inspect an image manifest:
26+
+
27+
[source,terminal]
28+
----
29+
$ docker manifest inspect <image_manifest> <1>
30+
----
31+
<1> Specifies an image manifest, such as `redhat/ubi9:latest`.
32+
+
33+
The platforms that your Operator and operands mutually support determine the platform compatibility of your Operator project.
34+
+
35+
.Example output
36+
[source,json]
37+
----
38+
{
39+
"manifests": [
40+
{
41+
"digest": "sha256:c0669ef34cdc14332c0f1ab0c2c01acb91d96014b172f1a76f3a39e63d1f0bda",
42+
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
43+
"platform": {
44+
"architecture": "amd64",
45+
"os": "linux"
46+
},
47+
"size": 528
48+
},
49+
...
50+
{
51+
"digest": "sha256:30e6d35703c578ee703230b9dc87ada2ba958c1928615ac8a674fcbbcbb0f281",
52+
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
53+
"platform": {
54+
"architecture": "arm64",
55+
"os": "linux",
56+
"variant": "v8"
57+
},
58+
"size": 528
59+
},
60+
...
61+
----
62+
63+
. If the previous command does not output platform information, then the specified base image might be a single image instead of an image manifest. You can find which architectures an image supports by running the following command:
64+
+
65+
[source,terminal]
66+
----
67+
$ docker inspect <image>
68+
----
69+
70+
. For Go-based Operator projects, the Operator SDK explicitly references the `amd64` architecture in your project's Dockerfile. Make the following change
71+
to your Dockerfile to set an environment variable to the value specified by the platform flag:
72+
+
73+
.Example Dockerfile
74+
[source,docker]
75+
----
76+
FROM golang:1.19 as builder
77+
ARG TARGETOS
78+
ARG TARGETARCH
79+
...
80+
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager main.go <1>
81+
----
82+
<1> Change the `GOARCH` field from `amd64` to `$TARGETARCH`.
83+
84+
. Your Operator project's makefile defines the `PLATFORMS` environment variable. If your Operator's images do not support all of the platforms set by default, edit the variable to specify the supported platforms. The following example defines the supported platforms as `linux/arm64` and `linux/amd64`:
85+
+
86+
.Example makefile
87+
[source,make]
88+
----
89+
# ...
90+
PLATFORMS ?= linux/arm64,linux/amd64 <1>
91+
.PHONY: docker-buildx
92+
# ...
93+
----
94+
+
95+
<1> The following `PLATFORMS` values are set by default: `linux/arm64`, `linux/amd64`, `linux/s390x`, and `linux/ppc64le`.
96+
+
97+
When you run the `make docker buildx` command to generate a manifest list, the Operator SDK creates an image manifest for each of the platforms specified by the `PLATFORMS` variable.
98+
99+
. Run the following command from your Operator project directory to build your manager image. Running the command builds a manager image with multi-platform support and pushes the manifest list to your registry.
100+
+
101+
[source,terminal]
102+
----
103+
$ make docker-buildx \
104+
IMG=<image_registry>/<organization_name>/<repository_name>:<version_or_sha>
105+
----

‎modules/osdk-multi-arch-node-affinity.adoc‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/operator_sdk/osdk-multi-arch-support.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="osdk-multi-arch-node-affinity_{context}"]
7+
= About node affinity rules for multi-architecture compute machines and Operator workloads
8+
9+
You must set node affinity rules to ensure your Operator workloads can run on multi-architecture compute machines. Node affinity is a set of rules used by the scheduler to define a pod's placement. Setting node affinity rules ensures your Operator's workloads are scheduled to compute machines with compatible architectures.
10+
11+
If your Operator performs better on particular architectures, you can set preferred node affinity rules to schedule pods to machines with the specified architectures.
12+
13+
For more information, see "About clusters with multi-architecture compute machines" and "Controlling pod placement on nodes using node affinity rules".

‎modules/osdk-multi-arch-node-preference.adoc‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/operator_sdk/osdk-multi-arch-support.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="osdk-multi-arch-node-preference_{context}"]
7+
= Using preferred node affinity rules to configure support for multi-architecture compute machines for Operator projects
8+
9+
If your Operator performs better on particular architectures, you can configure preferred node affinity rules to schedule pods to nodes to the specified architectures.
10+
11+
.Prerequisites
12+
13+
* An Operator project created or maintained with Operator SDK {osdk_ver} or later.
14+
* A manifest list defining the platforms your Operator supports.
15+
* Required node affinity rules are set for your Operator project.
16+
17+
.Procedure
18+
19+
. Search your Operator project for Kubernetes manifests that define pod spec and pod template spec objects.
20+
+
21+
.Example Kubernetes manifest
22+
[source,yaml]
23+
----
24+
apiVersion: v1
25+
kind: Pod
26+
metadata:
27+
name: s1
28+
spec:
29+
containers:
30+
- name: <container_name>
31+
image: docker.io/<org>/<image_name>
32+
----
33+
34+
. Set your Operator's preferred node affinity rules in the Kubernetes manifests that define pod spec and pod template spec objects, similar to the following example:
35+
+
36+
.Example Kubernetes manifest
37+
[source,yaml]
38+
----
39+
apiVersion: v1
40+
kind: Pod
41+
metadata:
42+
name: s1
43+
spec:
44+
containers:
45+
- name: <container_name>
46+
image: docker.io/<org>/<image_name>
47+
affinity:
48+
nodeAffinity:
49+
preferredDuringSchedulingIgnoredDuringExecution: <1>
50+
- preference:
51+
matchExpressions: <2>
52+
- key: kubernetes.io/arch <3>
53+
operator: In <4>
54+
values:
55+
- amd64
56+
- arm64
57+
weight: 90 <5>
58+
----
59+
<1> Defines a preferred rule.
60+
<2> If you specify multiple `matchExpressions` associated with `nodeSelectorTerms`, then the pod can be scheduled onto a node only if all `matchExpressions` are satisfied.
61+
<3> Specifies the architectures defined in the manifest list.
62+
<4> Specifies an `operator`. The Operator can be `In`, `NotIn`, `Exists`, or `DoesNotExist`. For example, use the value of `In` to require the label to be in the node.
63+
<5> Specifies a weight for the node, valid values are `1`-`100`. The node with highest weight is preferred.

‎modules/osdk-multi-arch-node-reqs.adoc‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/operator_sdk/osdk-multi-arch-support.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="osdk-multi-arch-node-reqs_{context}"]
7+
= Using required node affinity rules to support multi-architecture compute machines for Operator projects
8+
9+
If you want your Operator to support multi-architecture compute machines, you must define your Operator's required node affinity rules.
10+
11+
.Prerequisites
12+
13+
* An Operator project created or maintained with Operator SDK {osdk_ver} or later.
14+
* A manifest list defining the platforms your Operator supports.
15+
16+
.Procedure
17+
18+
. Search your Operator project for Kubernetes manifests that define pod spec and pod template spec objects.
19+
+
20+
[IMPORTANT]
21+
====
22+
Because object type names are not declared in YAML files, look for the mandatory `containers` field in your Kubernetes manifests. The `containers` field is required when specifying both pod spec and pod template spec objects.
23+
24+
You must set node affinity rules in all Kubernetes manifests that define a pod spec or pod template spec, including objects such as `Pod`, `Deployment`, `DaemonSet`, and `StatefulSet`.
25+
====
26+
+
27+
.Example Kubernetes manifest
28+
[source,yaml]
29+
----
30+
apiVersion: v1
31+
kind: Pod
32+
metadata:
33+
name: s1
34+
spec:
35+
containers:
36+
- name: <container_name>
37+
image: docker.io/<org>/<image_name>
38+
----
39+
40+
. Set the required node affinity rules in the Kubernetes manifests that define pod spec and pod template spec objects, similar to the following example:
41+
+
42+
.Example Kubernetes manifest
43+
[source,yaml]
44+
----
45+
apiVersion: v1
46+
kind: Pod
47+
metadata:
48+
name: s1
49+
spec:
50+
containers:
51+
- name: <container_name>
52+
image: docker.io/<org>/<image_name>
53+
affinity:
54+
nodeAffinity:
55+
requiredDuringSchedulingIgnoredDuringExecution: <1>
56+
nodeSelectorTerms: <2>
57+
- matchExpressions: <3>
58+
- key: kubernetes.io/arch <4>
59+
operator: In
60+
values:
61+
- amd64
62+
- arm64
63+
- ppc64le
64+
- s390x
65+
- key: kubernetes.io/os <5>
66+
operator: In
67+
values:
68+
- linux
69+
----
70+
<1> Defines a required rule.
71+
<2> If you specify multiple `nodeSelectorTerms` associated with `nodeAffinity` types, then the pod can be scheduled onto a node if one of the `nodeSelectorTerms` is satisfied.
72+
<3> If you specify multiple `matchExpressions` associated with `nodeSelectorTerms`, then the pod can be scheduled onto a node only if all `matchExpressions` are satisfied.
73+
<4> Specifies the architectures defined in the manifest list.
74+
<5> Specifies the operating systems defined in the manifest list.
75+
76+
. Go-based Operator projects that use dynamically created workloads might embed pod spec and pod template spec objects in the Operator's logic.
77+
+
78+
If your project embeds pod spec or pod template spec objects in the Operator's logic, edit your Operator's logic similar to the following example. The following example shows how to update a `PodSpec` object by using the Go API:
79+
+
80+
[source,go]
81+
----
82+
Template: corev1.PodTemplateSpec{
83+
...
84+
Spec: corev1.PodSpec{
85+
Affinity: &corev1.Affinity{
86+
NodeAffinity: &corev1.NodeAffinity{
87+
RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
88+
NodeSelectorTerms: []corev1.NodeSelectorTerm{
89+
{
90+
MatchExpressions: []corev1.NodeSelectorRequirement{
91+
{
92+
Key: "kubernetes.io/arch",
93+
Operator: "In",
94+
Values: []string{"amd64","arm64","ppc64le","s390x"},
95+
},
96+
{
97+
Key: "kubernetes.io/os",
98+
Operator: "In",
99+
Values: []string{"linux"},
100+
},
101+
},
102+
},
103+
},
104+
},
105+
},
106+
},
107+
SecurityContext: &corev1.PodSecurityContext{
108+
...
109+
},
110+
Containers: []corev1.Container{{
111+
...
112+
}},
113+
},
114+
----
115+
+
116+
where:
117+
118+
`RequiredDuringSchedulingIgnoredDuringExecution`:: Defines a required rule.
119+
`NodeSelectorTerms`:: If you specify multiple `nodeSelectorTerms` associated with `nodeAffinity` types, then the pod can be scheduled onto a node if one of the `nodeSelectorTerms` is satisfied.
120+
`MatchExpressions`:: If you specify multiple `matchExpressions` associated with `nodeSelectorTerms`, then the pod can be scheduled onto a node only if all `matchExpressions` are satisfied.
121+
`kubernetes.io/arch`:: Specifies the architectures defined in the manifest list.
122+
`kubernetes.io/os`:: Specifies the operating systems defined in the manifest list.
123+
124+
[WARNING]
125+
====
126+
If you do not set node affinity rules and a container is scheduled to a compute machine with an incompatible architecture, the pod fails and triggers one of the following events:
127+
128+
`CrashLoopBackOff`:: Occurs when an image manifest's entry point fails to run and an `exec format error` message is printed in the logs.
129+
`ImagePullBackOff`:: Occurs when a manifest list does not include a manifest for the architecture where a pod is scheduled or the node affinity terms are set to the wrong values.
130+
====

‎modules/osdk-multi-arch-validate.adoc‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/operator_sdk/osdk-multi-arch-support.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="osdk-multi-arch-validate_{context}"]
7+
= Validating your Operator's multi-platform readiness
8+
9+
You can validate your Operator's multi-platform readiness by running the `bundle validate` command. The command verifies that your Operator project meets the following conditions:
10+
11+
* Your Operator's manager image supports the platforms labeled in the cluster service version (CSV) file.
12+
* Your Operator's CSV has labels for the supported platforms for Operator Lifecycle Manager (OLM) and OperatorHub.
13+
14+
.Procedure
15+
16+
* Run the following command to validate your Operator project for multiple architecture readiness:
17+
+
18+
[source,terminal]
19+
----
20+
$ operator-sdk bundle validate ./bundle \
21+
--select-optional name=multiarch
22+
----
23+
+
24+
.Example validation message
25+
[source,text]
26+
----
27+
INFO[0020] All validation tests have completed successfully
28+
----
29+
+
30+
.Example error message for missing CSV labels in the manager image
31+
[source,text]
32+
----
33+
ERRO[0016] Error: Value test-operator.v0.0.1: not all images specified are providing the support described via the CSV labels. Note that (SO.architecture): (linux.ppc64le) was not found for the image(s) [quay.io/example-org/test-operator:v1alpha1]
34+
ERRO[0016] Error: Value test-operator.v0.0.1: not all images specified are providing the support described via the CSV labels. Note that (SO.architecture): (linux.s390x) was not found for the image(s) [quay.io/example-org/test-operator:v1alpha1]
35+
ERRO[0016] Error: Value test-operator.v0.0.1: not all images specified are providing the support described via the CSV labels. Note that (SO.architecture): (linux.amd64) was not found for the image(s) [quay.io/example-org/test-operator:v1alpha1]
36+
ERRO[0016] Error: Value test-operator.v0.0.1: not all images specified are providing the support described via the CSV labels. Note that (SO.architecture): (linux.arm64) was not found for the image(s) [quay.io/example-org/test-operator:v1alpha1]
37+
----
38+
+
39+
.Example error message for missing OperatorHub flags
40+
[source,text]
41+
----
42+
WARN[0014] Warning: Value test-operator.v0.0.1: check if the CSV is missing the label (operatorframework.io/arch.<value>) for the Arch(s): ["amd64" "arm64" "ppc64le" "s390x"]. Be aware that your Operator manager image ["quay.io/example-org/test-operator:v1alpha1"] provides this support. Thus, it is very likely that you want to provide it and if you support more than amd64 architectures, you MUST,use the required labels for all which are supported.Otherwise, your solution cannot be listed on the cluster for these architectures
43+
----

‎operators/operator_sdk/osdk-bundle-validate.adoc‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,12 @@ include::modules/osdk-bundle-validate-tests.adoc[leveloffset=+1]
1717
* xref:../../operators/understanding/olm-packaging-format.adoc#olm-bundle-format_olm-packaging-format[Bundle format]
1818
1919
include::modules/osdk-bundle-validate-run.adoc[leveloffset=+1]
20+
21+
ifndef::openshift-rosa,openshift-dedicated[]
22+
include::modules/osdk-multi-arch-validate.adoc[leveloffset=+1]
23+
24+
[role="_additional-resources"]
25+
.Additional resources
26+
27+
* xref:../../operators/operator_sdk/osdk-multi-arch-support.adoc#osdk-multi-platform-support[Configuring Operator projects for multi-platform support]
28+
endif::[]

0 commit comments

Comments
 (0)