Skip to content

Commit cff2b26

Browse files
aireillyaireilly
committed
fix id
atomize large file to better reuse in different branches
1 parent c68f1c3 commit cff2b26

File tree

41 files changed

+1036
-960
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+1036
-960
lines changed

‎main.adoc‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,84 @@ include::modules/cnf-best-practices-openshift-operations.adoc[leveloffset=+2]
7373

7474
include::modules/cnf-best-practices-expectations-permissions.adoc[leveloffset=+2]
7575

76+
include::modules/cnf-best-practices-cloud-native-design-best-practices.adoc[leveloffset=+3]
77+
78+
include::modules/cnf-best-practices-high-level-cnf-expectations.adoc[leveloffset=+4]
79+
80+
include::modules/cnf-best-practices-pod-permissions.adoc[leveloffset=+4]
81+
82+
include::modules/cnf-best-practices-logging.adoc[leveloffset=+4]
83+
84+
include::modules/cnf-best-practices-monitoring.adoc[leveloffset=+4]
85+
86+
include::modules/cnf-best-practices-cpu-allocation.adoc[leveloffset=+4]
87+
88+
include::modules/cnf-best-practices-memory-allocation.adoc[leveloffset=+4]
89+
90+
include::modules/cnf-best-practices-pods.adoc[leveloffset=+4]
91+
92+
include::modules/cnf-best-practices-pod-interaction-configuration.adoc[leveloffset=+5]
93+
94+
include::modules/cnf-best-practices-pod-exit-status.adoc[leveloffset=+5]
95+
96+
include::modules/cnf-best-practices-graceful-termination.adoc[leveloffset=+5]
97+
98+
include::modules/cnf-best-practices-pod-resource-profiles.adoc[leveloffset=+5]
99+
100+
include::modules/cnf-best-practices-storage-emptydir.adoc[leveloffset=+5]
101+
102+
include::modules/cnf-best-practices-liveness-readiness-and-startup-probes.adoc[leveloffset=+5]
103+
104+
include::modules/cnf-best-practices-affinity-anti-affinity.adoc[leveloffset=+5]
105+
106+
include::modules/cnf-best-practices-upgrade-expectations.adoc[leveloffset=+5]
107+
108+
include::modules/cnf-best-practices-taints-and-tolerations.adoc[leveloffset=+5]
109+
110+
include::modules/cnf-best-practices-requests-limits.adoc[leveloffset=+5]
111+
112+
include::modules/cnf-best-practices-use-imagepullpolicy-if-not-present.adoc[leveloffset=+5]
113+
114+
include::modules/cnf-best-practices-automount-services-for-pods.adoc[leveloffset=+5]
115+
116+
include::modules/cnf-best-practices-disruption-budgets.adoc[leveloffset=+5]
117+
118+
include::modules/cnf-best-practices-no-naked-pods.adoc[leveloffset=+5]
119+
120+
include::modules/cnf-best-practices-image-tagging.adoc[leveloffset=+5]
121+
122+
include::modules/cnf-best-practices-one-process-per-container.adoc[leveloffset=+5]
123+
124+
include::modules/cnf-best-practices-init-containers.adoc[leveloffset=+5]
125+
126+
include::modules/cnf-best-practices-security-rbac.adoc[leveloffset=+4]
127+
128+
include::modules/cnf-best-practices-custom-role-to-access-application-crds.adoc[leveloffset=+4]
129+
130+
include::modules/cnf-best-practices-multus.adoc[leveloffset=+4]
131+
132+
include::modules/cnf-best-practices-multus-macvlan.adoc[leveloffset=+4]
133+
134+
include::modules/cnf-best-practices-sr-iov-interface-settings.adoc[leveloffset=+4]
135+
136+
include::modules/cnf-best-practices-attaching-the-vf-to-a-pod.adoc[leveloffset=+4]
137+
138+
include::modules/cnf-best-practices-discovering-sr-iov-devices-properties-from-the-application.adoc[leveloffset=+4]
139+
140+
include::modules/cnf-best-practices-numa-awareness.adoc[leveloffset=+4]
141+
142+
include::modules/cnf-best-practices-platform-upgrade.adoc[leveloffset=+4]
143+
144+
include::modules/cnf-best-practices-openshift-virtualization-kubevirt.adoc[leveloffset=+4]
145+
146+
include::modules/cnf-best-practices-vm-image-import-recommendations-cdi.adoc[leveloffset=+5]
147+
148+
include::modules/cnf-best-practices-working-with-large-vm-disk-images.adoc[leveloffset=+5]
149+
150+
include::modules/cnf-best-practices-operator-best-practices.adoc[leveloffset=+4]
151+
152+
include::modules/cnf-best-practices-cnf-operator-requirements.adoc[leveloffset=+5]
153+
76154
include::modules/cnf-best-practices-requirements-cnf-reqs.adoc[leveloffset=+2]
77155

78156
include::modules/cnf-best-practices-copyright.adoc[leveloffset=+1]

‎modules/cnf-best-practices-affinity-anti-affinity.adoc‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
[id="cnf-best-practices-affinity-anti-affinity"]
2+
= Affinity and anti-affinity
3+
4+
In OpenShift Container Platform pod affinity and pod anti-affinity allow you to constrain which nodes your pod are eligible to be scheduled based on the key/value labels on other pods. There are two types of affinity rules, required and preferred. Required rules must be met, whereas preferred rules are best effort.
5+
6+
These pod affinity/anti-affinity rules are set in the pod specification as `matchExpressions` to a `labelSelector`. See link:https://docs.openshift.com/container-platform/latest/nodes/scheduling/nodes-scheduler-pod-affinity.html[Placing pods relative to other pods using affinity and anti-affinity rules] for more information. The following example `Pod` CR illustrates pod affinity:
7+
8+
[source,yaml]
9+
----
10+
apiVersion: v1
11+
kind: Pod
12+
metadata:
13+
name: with-pod-affinity
14+
spec:
15+
affinity:
16+
podAffinity:
17+
requiredDuringSchedulingIgnoredDuringExecution:
18+
- labelSelector:
19+
matchExpressions:
20+
- key: security
21+
operator: In
22+
values:
23+
- S1
24+
topologyKey: failure-domain.beta.kubernetes.io/zone
25+
containers:
26+
- name: with-pod-affinity
27+
image: docker.io/ocpqe/hello-pod
28+
----
29+
30+
.CNF requirement
31+
[IMPORTANT]
32+
====
33+
Pods that need to be co-located on the same node need affinity rules. Pods that should not be
34+
co-located for resiliency purposes require anti-affinity rules.
35+
36+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#lifecycle-affinity-required-pods[lifecycle-affinity-required-pods]
37+
====
38+
39+
.CNF requirement
40+
[IMPORTANT]
41+
====
42+
Pods that perform the same microservice and could be disrupted if multiple members of the service are
43+
unavailable must implement affinity/anti-affinity group rules or spread the pods across nodes to prevent disruption in the event of node failures, patches, or upgrades.
44+
45+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#lifecycle-pod-high-availability[lifecycle-pod-high-availability]
46+
====
47+

‎modules/cnf-best-practices-attaching-the-vf-to-a-pod.adoc‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
[id="cnf-best-practices-attaching-the-vf-to-a-pod"]
2+
= Attaching the VF to a pod
3+
4+
Once the right network attachment definition is found, applying the `k8s.v1.cni.cncf.io/networks` annotation with the name of the network attachment definition to the pod will add the additional network interfaces in the pod namespace, as per the following example:
5+
6+
[source,yaml]
7+
----
8+
apiVersion: v1
9+
kind: Pod
10+
metadata:
11+
name: sample-pod
12+
annotations:
13+
k8s.v1.cni.cncf.io/networks: |-
14+
[
15+
{
16+
"name": "net1",
17+
"mac": "20:04:0f:f1:88:01",
18+
"ips": ["192.168.10.1/24", "2001::1/64"]
19+
}
20+
]
21+
----
22+

‎modules/cnf-best-practices-automount-services-for-pods.adoc‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
[id="cnf-best-practices-automount-services-for-pods"]
2+
= Automount services for pods
3+
4+
Pods which do not require API access should set the value of `automountServiceAccountToken` to false within the pod spec, for example:
5+
6+
[source,yaml]
7+
----
8+
apiVersion: v1
9+
kind: Pod
10+
metadata:
11+
name: my-pod
12+
spec:
13+
serviceAccountName: examplesvcacct
14+
automountServiceAccountToken: false
15+
----
16+
17+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#access-control-pod-automount-service-account-token[access-control-pod-automount-service-account-token]
18+

‎modules/cnf-best-practices-cloud-native-design-best-practices.adoc‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
[id="cnf-best-practices-cloud-native-design-best-practices"]
2+
= Cloud-native design best practices
3+
4+
The following best practices highlight some key principles of cloud-native application design.
5+
6+
Single purpose w/messaging interface::
7+
A container should address a single purpose with a well-defined (typically RESTful API) messaging interface. The motivation here is that such a container image is more reusable and more replaceable/upgradeable.
8+
9+
High observability::
10+
A container must provide APIs for the platform to observe the container health and act accordingly. These APIs include health checks (liveness and readiness), logging to stderr and stdout for log aggregation (by tools such as `Logstash` or `Filebeat`), and integrate with tracing and metrics-gathering libraries (such as `Prometheus` or `Metricbeat`).
11+
12+
Lifecycle conformance::
13+
A container must receive important events from the platform and conform/react to these events properly. For example, a container should catch SIGTERM or SIGKILL from the platform and shut down as quickly as possible. Other typically important events from the platform are PostStart to initialize before servicing requests and PreStop to release resources cleanly before shutting down.
14+
15+
See test cases link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#lifecycle-container-shutdown[lifecycle-container-shutdown], link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#lifecycle-container-startup[lifecycle-container-startup]
16+
17+
Image immutability::
18+
Container images are meant to be immutable; i.e. customized images for different environments should typically not be built. Instead, an external means for storing and retrieving configurations that vary across environments for the container should be used. Additionally, the container image should NOT dynamically install additional packages at runtime.
19+
20+
Process disposability::
21+
Containers should be as ephemeral as possible and ready to be replaced by another container instance at any point in time. There are many reasons to replace a container, such as failing a health check, scaling down the application, migrating the containers to a different host, platform resource starvation, or another issue.
22+
+
23+
This means that containerized applications must keep their state externalized or distributed and redundant. To store files or block level data, persistent volume claims should be used. For information such as user sessions, use of an external, low-latency, key-value store such as redis should be used. Process disposability also requires that the application should be quick in starting up and shutting down, and even be ready for a sudden, complete hardware failure.
24+
+
25+
Another helpful practice in implementing this principle is to create small containers. Containers in cloud-native environments may be automatically scheduled and started on different hosts. Having smaller containers leads to quicker start-up times because before being restarted, containers need to be physically copied to the host system.
26+
+
27+
A corollary of this practice is to "retry instead of crashing", for example, When one service in your application depends on another service, it should not crash when the other service is unreachable. For example, your API service is starting up and detects the database is unreachable. Instead of failing and refusing to start, you design it to retry the connection. While the database connection is down the API can respond with a 503 status code, telling the clients that the service is currently unavailable. This practice should already be followed by applications, but if you are working in a containerized environment where instances are disposable, then the need for it becomes more obvious.
28+
+
29+
Also related to this, by default containers are launched with shared images using COW filesystems which only exist as long as the container exists. Mounting Persistent Volume Claims enables a container to have persistent physical storage. Clearly defining the abstraction for what storage is persisted promotes the idea that instances are disposable.
30+
31+
.CNF requirement
32+
[IMPORTANT]
33+
====
34+
Application design should conform to cloud-native design principles to the maximum extent possible.
35+
====
36+

‎modules/cnf-best-practices-cnf-operator-requirements.adoc‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
[id="cnf-best-practices-cnf-operator-requirements"]
2+
= CNF Operator requirements
3+
4+
.CNF requirement
5+
[IMPORTANT]
6+
====
7+
Operators should be certified against the openshift version of the cluster they will be deployed on.
8+
9+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#affiliated-certification-operator-is-certified[affiliated-certification-operator-is-certified]
10+
====
11+
12+
.CNF requirement
13+
[IMPORTANT]
14+
====
15+
Operators must be compatible with our version of openshift
16+
17+
* See link:https://redhat-connect.gitbook.io/openshift-badges/badges/cloud-native-network-functions-cnf[Redhat Partner Guide for CNF Certification]
18+
19+
* See link:https://sdk.operatorframework.io/docs/best-practices/[Redhat Operator SDK & Best Practices], link:https://olm.operatorframework.io/docs/best-practices/[OLM Best Practices]
20+
21+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#platform-alteration-ocp-lifecycle[platform-alteration-ocp-lifecycle]
22+
====
23+
24+
.CNF requirement
25+
[IMPORTANT]
26+
====
27+
Operators must be in OLM bundle format (Operator Framework).
28+
29+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#operator-install-source[operator-install-source]
30+
====
31+
32+
.CNF requirement
33+
[IMPORTANT]
34+
====
35+
Must be able to function without the use of openshift routes or ingress objects.
36+
====
37+
38+
.CNF requirement
39+
[IMPORTANT]
40+
====
41+
All custom resources for operators require podspecs for both pod image override as well pod quotas.
42+
====
43+
44+
.CNF requirement
45+
[IMPORTANT]
46+
====
47+
Operators must not use daemonsets
48+
49+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#lifecycle-pod-owner-type[lifecycle-pod-owner-type]
50+
====
51+
52+
.CNF requirement
53+
[IMPORTANT]
54+
====
55+
The OLM operator CSV must support the "all namespaces" install method if the operator is upstream software. If the operator is a proprietary cnf operator it must support single namespaced installation. It is recommended for an operator to support all OLM install modes to ensure flexibility in our environment.
56+
====
57+
58+
.CNF requirement
59+
[IMPORTANT]
60+
====
61+
The operator must default to watch all namespaces if the target namespace is left NULL or empty string as this is how the OLM global-operators operator group functions.
62+
====
63+
64+
.CNF requirement
65+
[IMPORTANT]
66+
====
67+
All operator and operand images must be referenced using digest image tags "@sha256". Openshift "imagecontentsourcepolicy" objects (ICSP) only support mirror-by-digest at this time.
68+
====
69+
70+
.CNF requirement
71+
[IMPORTANT]
72+
====
73+
For general third party upstream operators (example: mongodb), the OLM package is recommended to be located within the Red Hat registries below to support our image mirror policy:
74+
75+
* `quay.io`
76+
77+
* `registry.redhat.io`
78+
79+
* `registry.connect.redhat.com`
80+
81+
* `registry.access.redhat.com`
82+
====
83+
84+
.CNF requirement
85+
[IMPORTANT]
86+
====
87+
Operators that are proprietary to a cnf application must ensure that their CRD's are unique, and will not conflict with other operators in the cluster.
88+
89+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#observability-crd-status[observability-crd-status]
90+
====
91+
92+
.CNF requirement
93+
[IMPORTANT]
94+
====
95+
If a cnf application requires a specific version of a third party non-proprietary operator for their app to function they will need to re-package the upstream third party operator and modify the api's so that it will not conflict with the globally installed operator version.
96+
====
97+
98+
.CNF requirement
99+
[IMPORTANT]
100+
====
101+
Successful operator installation and runtime must be validated in pre-deployment lab environments before being allowed to be deployed to production.
102+
103+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#operator-install-status-succeeded[operator-install-status-succeeded]
104+
====
105+
106+
.CNF requirement
107+
[IMPORTANT]
108+
====
109+
All required RBAC must be included in the OLM operator bundle so that it's managed by OLM.
110+
====
111+
112+
.CNF requirement
113+
[IMPORTANT]
114+
====
115+
It is not recommended for a cnf application to share a proprietary operator with another cnf application if that application does not share the same version lifecycle. If a cnf application does share an operator the CRDs must be backwards compatible.
116+
====

‎modules/cnf-best-practices-cpu-allocation.adoc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[id="cnf-best-practices-cpu-allocation"]
2+
= CPU allocation
3+
4+
It is important to note that when the OpenShift scheduler is placing pods, it first reviews the Pod CPU request and schedules it if there is a node that meets the requirements. It will then impose the CPU "Limits" to ensure the Pod doesn't consume more than the intended allocation. The limit can never be lower than the request.
5+
6+
NUMA Configuration:: OpenShift provides a topology manager which leverages the CPU manager and Device manager to help associate processes to CPUs. Topology manager handles NUMA affinity. This feature is available as of OpenShift 4.6. For some examples on how to leverage the topology manager and creating workloads that work in real time, see link:https://docs.openshift.com/container-platform/4.12/scalability_and_performance/cnf-numa-aware-scheduling.html[Scheduling NUMA-aware workloads] and link:https://docs.openshift.com/container-platform/4.12/scalability_and_performance/cnf-low-latency-tuning.html[Low latency tuning].
7+

‎modules/cnf-best-practices-custom-role-to-access-application-crds.adoc‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[id="cnf-best-practices-custom-role-to-access-application-crds"]
2+
= Custom role to access application CRDs
3+
4+
If an application requires installing/deploying CRDs (Custom Resource Definitions), the application must provide a role that allows necessary permissions to create CRs within the CRDs. The custom role to access CRDs must not create any permissions to access any other API resources than the CRDs.
5+
6+
.CNF requirement
7+
[IMPORTANT]
8+
====
9+
If an application creates CRDs; it must supply a role to access those CRDs and no other API resources/
10+
permissions.
11+
====
12+

‎modules/cnf-best-practices-discovering-sr-iov-devices-properties-from-the-application.adoc‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
[id="cnf-best-practices-discovering-sr-iov-devices-properties-from-the-application"]
2+
= Discovering SR-IOV devices properties from the application
3+
4+
All the properties of the interfaces are added to the pod's `k8s.v1.cni.cncf.io/network-status` annotation. The annotation is json-formatted and for each network object contains information such as IPs (where available), MAC address, PCI address. For example:
5+
6+
[source,yaml]
7+
----
8+
k8s.v1.cni.cncf.io/network-status: |-
9+
[{
10+
"name": "",
11+
"interface": "eth0",
12+
"ips": [
13+
"10.132.3.148"
14+
],
15+
"mac": "0a:58:0a:84:03:94",
16+
"default": true,
17+
"dns": {}
18+
}]
19+
----
20+
21+
[NOTE]
22+
====
23+
the IP information is not available if the driver specified is `vf-io`.
24+
====
25+
26+
The same annotation is available as a file content inside the pod, at the `/etc/podnetinfo/annotations` path. A convenience library is available to easily consume those informations from the application (bindings in C and Go).
27+
28+
For more information, see link:https://docs.openshift.com/container-platform/latest/networking/hardware_networks/about-sriov.html[About Single Root I/O Virtualization (SR-IOV) hardware networks].
29+

‎modules/cnf-best-practices-disruption-budgets.adoc‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[id="cnf-best-practices-disruption-budgets"]
2+
==== Disruption budgets
3+
4+
When managing the platform there are at least two types of disruptions that can occur. They are voluntary and involuntary. When dealing with voluntary disruptions a pod disruption budget can be set that determines how many replicas of the application must remain running at any given time. For example, consider the case where an administrator is shutting down a node for
5+
6+
maintenance and the node has to be drained. If there is a pod disruption budget set then OpenShift will respect that and ensure that the required number of pods are available by bringing up pods on different nodes before draining the current node.
7+
8+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#observability-pod-disruption-budget[observability-pod-disruption-budget]
9+

0 commit comments

Comments
 (0)