Skip to content

Commit 0735792

Browse files
mbaldessarimbaldessari
authored
Merge branch 'main' into main
2 parents 205b748 + b9be4d1 commit 0735792

File tree

15 files changed

+724
-3
lines changed

15 files changed

+724
-3
lines changed

‎.github/workflows/htmltest.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
name: HtmlTest Action
2-
on: [push, pull_request]
2+
on:
3+
schedule:
4+
- cron: '23 17 * * *'
35
jobs:
46
build:
57
name: Htmltest

‎content/blog/2024-11-07-clustergroup-sequencing.adoc‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,318 @@
1+
---
2+
date: 2024-11-07
3+
title: Additional Sequencing Capabilities in clusterGroup
4+
summary: How to sequence subscriptions in the Validated Patterns framework
5+
author: Martin Jackson
6+
blog_tags:
7+
- patterns
8+
- how-to
9+
- sequencing
10+
- subscriptions
11+
---
12+
:toc:
13+
14+
== Preamble
15+
16+
Ideally, all subscriptions installed in Kubernetes and OpenShift will envision
17+
all potential conflicts, and will deal gracefully with situations when other
18+
artifacts installed on the cluster require reconfiguration or different behavior.
19+
20+
Architecturally, we have always said that we prefer eventual consistency, by which
21+
we mean that in the presence of error conditions the software should be prepared to
22+
retry until it can achieve the state it wants to be in. In practice that means we
23+
should be able to install a number of artifacts at the same time, and the artifacts
24+
themselves should be able to achieve the declarative state expressed in their installation.
25+
But some software does not work this way (even if its stated goal and intention is to
26+
work this way) and it is advantageous to be able to impose order of events to create
27+
better situations for installation. For example, even well-crafted software can be
28+
subject to different kinds of timing problems, as we will illustrate below.
29+
30+
Because of this, we have introduced a set of capabilties to the Validated Patterns
31+
clusterGroup chart to enforce sequencing on the objects in a declarative and
32+
Kubernetes-native way.
33+
34+
In this blog post, we will explore the various options that are available as of
35+
today in the Validated Patterns framework for enforcing sequencing for subscriptions.
36+
Inside applications, Validated Patterns has supported these primitives since the first
37+
release of Medical Diagnosis, and will continue to do so.
38+
39+
Since the focus of Validated Patterns on OpenShift is the OpenShift GitOps Operator, these
40+
capabilities rely on the use of resource hooks, described in the upstream docs https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/[here].
41+
42+
Within the framework, we support resource hooks in three ways:
43+
44+
1. Supporting annotations directly on subscription objects at the clusterGroup level
45+
2. Exposing a new optional sequenceJob attribute on subscriptions that uses resource hooks
46+
3. Exposing a new top-level clusterGroup object, extraObjects, that allows users full control of creating their own
47+
resource hooks.
48+
49+
These features are available in version 0.9.9 and later of the Validated Patterns clustergroup chart.
50+
51+
== Race Conditions: The Problem
52+
53+
Timing issues are one of the key problems of distributed systems. One of the biggest categories of timing problems
54+
is https://en.wikipedia.org/wiki/Race_condition[race conditions]. In our context, let's say subscription A reacts to a
55+
condition in subscription B. Subscription A only checks on that condition during installation time. Thus,
56+
subscription A's final state depends on when, exactly, subscription B was installed. Even if A and B were both installed
57+
at the same time, the normal variance of things like how quickly the software was downloaded could result in different
58+
parts of the installation being run at different times, and potentially different results.
59+
60+
The ideal solution to this problem is for subscription B to always be watching for the presence of subscription A, and
61+
reconfiguring itself if it sees subscription A being installed. But if subscription B does not want to do this, or
62+
for some reason cannot do this, or even if a better fix is committed but not yet available, we want to have a set of
63+
practical solutions in the Validated Patterns framework.
64+
65+
So, in the absence of an ideal fix - where all subscriptions are prepared to deal with all possible race outcomes -
66+
a very effective way of working around the problem is enforce ordering or sequencing of actions.
67+
68+
== The Specific Race Condition that led to this feature
69+
70+
The specific case that gave rise to the development of this feature is a race condition between OpenShift Data
71+
Foundation (ODF) and OpenShift Virtualization (OCP-Virt), which will be fixed in a future release of OCP-Virt. The
72+
condition results when OCP-Virt, on installation, discovers a default storageclass, but then subsequently ODF is
73+
installed, which OCP-Virt has specific optimizations for, related to how images are managed for VM guests to be
74+
created. One way to workaround the race condition is to ensure that ODF completes creating its storageclasses before
75+
the OCP-Virt subscription is allowed to install.
76+
77+
== Sync-Waves and ArgoCD/OpenShift GitOps Resource Hooks
78+
79+
The way that resource hooks are designed to work is by giving ordering hints, so that ArgoCD knows what order to
80+
apply resources in. The mechanism is described in the ArgoCD upstream docs https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/[here]. When sync-waves are in use, all resouces in the same sync-wave have to be "healthy" before
81+
resources in the numerically next sync-wave are synced. This mechanism gives us a way of having ArgocD help us enforce
82+
order with objects that it manages.
83+
84+
== Solution 1: Sync-Waves for Subscriptions (and/or Applications) in clusterGroup
85+
86+
The Validated Patterns framework now allows Kubernetes https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/[annotations] to be added directly to subscription objects and application objects in the clusterGroup.
87+
ArgoCD uses annotations for Resource Hooks. The clustergoup chart now passes any annotations attached to subscriptions
88+
through to the subscription object(s) that the clustergroup chart creates. For example:
89+
90+
[source,yaml]
91+
----
92+
openshift-virtualization:
93+
name: kubevirt-hyperconverged
94+
namespace: openshift-cnv
95+
channel: stable
96+
annotations:
97+
argocd.argoproj.io/sync-wave: "10"
98+
99+
openshift-data-foundation:
100+
name: odf-operator
101+
namespace: openshift-storage
102+
annotations:
103+
argocd.argoproj.io/sync-wave: "5"
104+
----
105+
106+
will result in a subscription object that includes the annotations:
107+
108+
[source,yaml]
109+
----
110+
apiVersion: operators.coreos.com/v1alpha1
111+
kind: Subscription
112+
metadata:
113+
annotations:
114+
argocd.argoproj.io/sync-wave: "10"
115+
labels:
116+
app.kubernetes.io/instance: ansible-edge-gitops-hub
117+
operators.coreos.com/kubevirt-hyperconverged.openshift-cnv: ""
118+
name: kubevirt-hyperconverged
119+
namespace: openshift-cnv
120+
spec:
121+
channel: stable
122+
installPlanApproval: Automatic
123+
name: kubevirt-hyperconverged
124+
source: redhat-operators
125+
sourceNamespace: openshift-marketplace
126+
127+
apiVersion: operators.coreos.com/v1alpha1
128+
kind: Subscription
129+
metadata:
130+
annotations:
131+
argocd.argoproj.io/sync-wave: "5"
132+
labels:
133+
app.kubernetes.io/instance: ansible-edge-gitops-hub
134+
operators.coreos.com/odf-operator.openshift-storage: ""
135+
name: odf-operator
136+
namespace: openshift-storage
137+
spec:
138+
installPlanApproval: Automatic
139+
name: odf-operator
140+
source: redhat-operators
141+
sourceNamespace: openshift-marketplace
142+
----
143+
144+
With this configuration, any objects created with sync-waves lower than "10" must be healthy before the objects in
145+
sync-wave "10" sync. In particular, the odf-operator subscription must be healthy before the kubevirt-hyperconverged
146+
subscription will sync. Similarly, if we defined objects with higher sync-waves than "10", all the resources with
147+
sync-waves higher than "10" will wait until the resources in "10" are healthy. If the subscriptions in question wait
148+
until their components are healthy before reporting they are healthy themselves, this might be all you need to do.
149+
In the case of this particular issue, it was not enough. But because all sequencing in ArgoCD requires the use of
150+
sync-wave annotations, adding the annotation to the subscription object will be necessary for using the other
151+
solutions.
152+
153+
The sequencing of applications would work the same way, with the same format for adding annotations to the application
154+
stanzas in the clustergroup.
155+
156+
== Solution 2: The `sequenceJob` attribute for Subscriptions in clusterGroup
157+
158+
In this situation, we have a subscription that installs an operator, but it is not enough for just the subscriptions
159+
to be in sync-waves. This is because the subscriptions install operators, and it is the action of the operators
160+
themselves that we have to sequence. In many of these kinds of situations, we can sequence the action by looking for
161+
the existence of a single resource. The new `sequenceJob` construct in subscriptions allows for this kind of
162+
relationship by creating a Job at the same sync-wave precedence as the subscription, and looking for the existence
163+
of a single arbitrary resource in an arbitrary namespace. The Job then waits for that resource to appear, and when
164+
it does, it will be seen as "healthy" and will allow future sync-waves to proceed.
165+
166+
In this example, the ODF operator needs to have created a storageclass so that the OCP-Virt operators can use it as
167+
virtualization storage. If it does not find the kind of storage it wants, it will use the default storageclass
168+
instead, which may lead to inconsistencies in behavior. We can have the Validated Patterns framework create a
169+
mostly boilerplate job to look for the needed resource this way:
170+
171+
[source,yaml]
172+
----
173+
openshift-virtualization:
174+
name: kubevirt-hyperconverged
175+
namespace: openshift-cnv
176+
channel: stable
177+
annotations:
178+
argocd.argoproj.io/sync-wave: "10"
179+
180+
openshift-data-foundation:
181+
name: odf-operator
182+
namespace: openshift-storage
183+
sequenceJob:
184+
resourceType: sc
185+
resourceName: ocs-storagecluster-ceph-rbd
186+
annotations:
187+
argocd.argoproj.io/sync-wave: "5"
188+
----
189+
190+
Note the addition of the `sequenceJob` section in the odf-operator subscription block. This structure will result
191+
in the following Job being created alongside the subscriptions:
192+
193+
[source,yaml]
194+
----
195+
apiVersion: batch/v1
196+
kind: Job
197+
metadata:
198+
annotations:
199+
argocd.argoproj.io/hook: Sync
200+
argocd.argoproj.io/sync-wave: "5"
201+
labels:
202+
app.kubernetes.io/instance: ansible-edge-gitops-hub
203+
name: odf-operator-sequencejob
204+
namespace: openshift-operators
205+
spec:
206+
backoffLimit: 6
207+
completionMode: NonIndexed
208+
completions: 1
209+
manualSelector: false
210+
parallelism: 1
211+
podReplacementPolicy: TerminatingOrFailed
212+
selector:
213+
matchLabels:
214+
batch.kubernetes.io/controller-uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
215+
suspend: false
216+
template:
217+
metadata:
218+
creationTimestamp: null
219+
labels:
220+
batch.kubernetes.io/controller-uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
221+
batch.kubernetes.io/job-name: odf-operator-sequencejob
222+
controller-uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
223+
job-name: odf-operator-sequencejob
224+
spec:
225+
containers:
226+
- command:
227+
- /bin/bash
228+
- -c
229+
- |
230+
while [ 1 ];
231+
do
232+
oc get sc ocs-storagecluster-ceph-rbd && break
233+
echo "sc ocs-storagecluster-ceph-rbd not found, waiting..."
234+
sleep 5
235+
done
236+
echo "sc ocs-storagecluster-ceph-rbd found, exiting..."
237+
exit 0
238+
image: quay.io/hybridcloudpatterns/imperative-container:v1
239+
imagePullPolicy: IfNotPresent
240+
name: odf-operator-sequencejob
241+
resources: {}
242+
terminationMessagePath: /dev/termination-log
243+
terminationMessagePolicy: File
244+
dnsPolicy: ClusterFirst
245+
restartPolicy: OnFailure
246+
schedulerName: default-scheduler
247+
securityContext: {}
248+
terminationGracePeriodSeconds: 30
249+
----
250+
251+
Since the job is created in sync-wave "5" (which it inherits from the subscription it is attached to by default, though
252+
you can specify a different sync-wave if you prefer), this job must complete before sync-wave "10" starts. So the
253+
storageclass `ocs-storagecluster-ceph-rbd` must exist before OCP-Virt starts deploying, ensuring that it will be able
254+
to "see" and use that storageclass as its default virtualization storage class.
255+
256+
Each subscription is permitted one sequenceJob. Each sequenceJob may have the following attributes:
257+
258+
* *syncWave*: Defaults to the subscription's syncwave from annotations.
259+
* *resourceType*: Resource kind for the resource to watch for.
260+
* *resourceName*: Name of the resource to watch for.
261+
* *resourceNamespace*: Namespace to watch for the resourceType and resourceName in.
262+
* *hookType*: Any of the permissible ArgoCD Resource Hook types. Defaults to "Sync".
263+
* *image*: Image of the container to use for the job. Defaults to the Validated Patterns imperative image.
264+
* *command*: Command to run inside the container, if the default is not suitable. This also enables you to specify multiple resources to watch for in the same job, or to look for a different condition altogether.
265+
* *disabled*: Set this to true in an override if you wish to disable the sequenceJob for some reason (such as running on
266+
a different version of OpenShift or running on a different cloud platform).
267+
268+
If the sequenceJob is not sufficient for your sequencing needs, we have a more generic interface that you can use
269+
that places no restrictions on the objects you can add, so you can use it to create different kinds of conditions.
270+
271+
== Solution 3: The `extraObjects` attribute in clusterGroup
272+
273+
The most open-ended solution to the sequencing problem involves defining arbitrary objects under the `extraObjects`
274+
key for the clustergroup. Here is how you could do that using the example we have been using so far:
275+
276+
[yaml,source]
277+
----
278+
extraObjects:
279+
wait-for-virt-storageclass:
280+
apiVersion: batch/v1
281+
kind: Job
282+
metadata:
283+
name: wait-for-virt-storageclass
284+
annotations:
285+
argocd.argoproj.io/hook: Sync
286+
argocd.argoproj.io/sync-wave: "5"
287+
spec:
288+
parallelism: 1
289+
completions: 1
290+
template:
291+
spec:
292+
restartPolicy: OnFailure
293+
containers:
294+
- name: wait-for-storage-class
295+
image: quay.io/hybridcloudpatterns/imperative-container:v1
296+
command:
297+
- /bin/bash
298+
- -c
299+
- |
300+
while [ 1 ];
301+
do
302+
oc get sc ocs-storagecluster-ceph-rbd && break
303+
echo "Storage class ocs-storagecluster-ceph-rbd not found, waiting..."
304+
sleep 5
305+
done
306+
echo "Storage class ocs-storagecluster-ceph-rbd found, exiting"
307+
exit 0
308+
----
309+
310+
Note that each extraObject has a key and value, and the value will be passed almost unaltered as a Kubernetes manifest.
311+
The special key `disabled` can be used to disable a specific, named extraObject from being created in subsequent
312+
overrides.
313+
314+
== Conclusion
315+
316+
Here is hoping that you do not have sequencing problems to solve in your OpenShift deployments. But if you do, we
317+
hope you will find this feature in Validated Patterns useful. Please let us know, one way or the other, or if you
318+
find other uses, especially for the `extraObjects` feature.

‎content/patterns/ansible-edge-gitops-kasten/_index.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
---
22
title: OpenShift Virtualization Data Protection with Veeam Kasten
3-
date: 2024-11-12
43
tier: sandbox
54
summary: This pattern uses OpenShift Virtualization to simulate an edge environment for VMs, protected by Veeam Kasten.
65
rh_products:
@@ -28,7 +27,7 @@ links:
2827

2928
This example extends the standard [Ansible Edge GitOps pattern](https://validatedpatterns.io/patterns/ansible-edge-gitops/) to include automated deployment and configuration of [Veeam Kasten](https://www.veeam.com/products/cloud/kubernetes-data-protection.html), the #1 Kubernetes data protection and mobility solution.
3029

31-
The base pattern explores the possibilities of using an Ansible Automation Platform deployment to manage the configuration of OpenShift Virtualization VMs at the edge. As VMs are inherently stateful workloads, a GitOps approach alone is not sufficient to recover an environment in the event of accidental data loss, malware attack, or infrastructure failure - especially in edge environments where infrastructure may be less resilient or subject to harsh environments.
30+
This pattern uses **Red Hat OpenShift Virtualization** (the productization of KubeVirt) to provision VMs alongside Kubernetes-native workloads on the cluster. As VMs are inherently stateful workloads, a GitOps approach alone is not sufficient to recover an environment in the event of accidental data loss, malware attack, or infrastructure failure - especially in edge environments where infrastructure may be less resilient or subject to harsh environments. This example extends the standard [Ansible Edge GitOps pattern](https://validatedpatterns.io/patterns/ansible-edge-gitops/) to include automated deployment and configuration of [Veeam Kasten](https://www.veeam.com/products/cloud/kubernetes-data-protection.html), the #1 Kubernetes data protection and mobility solution.
3231

3332
### Solution elements
3433

0 commit comments

Comments
 (0)