Skip to content

Commit ec4f1eb

Browse files
amoralejopenshift-merge-bot[bot]
amoralej
authored and
openshift-merge-bot[bot]
committed
Make deployment jobs compatible with standalone and integrated mode
We are working in integrating the watcher-operator in the openstack-operator following the same approach that other openstack services operators are doing. To make the transition smooth, I am proposing to make the jobs compatible with both standalone and integrated mode by implementing following changes: - Now, the content provider jobs should be rebuilding the watcher-operator and the openstack-operator. For the integration mode, that should make that, when the openstack-operator deployes the watcher-operator it will run using the patched version of it. - I'm adding logic to find out if the watcher-operator is already deployed when running the deploy_watcher_service.yaml playbook. If the Watcher CRDs are already installed, it means the watcher operator has been installed by the openstack-operator and there is no need to install it standalone. - The patch also adds a new makefile target stop_watcher_integrated to stop the watcher-operator when running in integrated mode. This is a requirement to install it in standalone mode after it is integrated and we need it for kuttl tests and in the development workflow. - I'm also adding logic to find out if the watcher is integrated in the openstackcontrolplane. If so, watcher is deployed by patching the existing openstackcontrolplane. Otherwise, it creates a Watcher CR as it is currently doing. - Finally, we need to cover the cases where we deploy Watcher using a Watcher CR but there are no matating webhooks to include the container images automatically (that will be managed at the openstackcontrolplane level when fully integrated). - For the kuttl tests we always want to deploy the watcher-operator in standalone mode so that we have the watcher webhooks running. Note that, at some point we will be able to improve this approach by applying a kustomization to the controlplane in pre_deploy, but at the pre_deploy is not possible to discover if the watcher is already integrated in the openstackcontrolplane CRD, so we can keep using this post_deployment approach until the integration is finished and propagated to all the branches and environments. Jira: OSPRH-16242
1 parent e8c99c4 commit ec4f1eb

File tree

7 files changed

+250
-31
lines changed

7 files changed

+250
-31
lines changed

‎.zuul.yaml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,7 @@
129129
deploy_watcher_service_extra_vars:
130130
watcher_catalog_image: "{{ cifmw_operator_build_output['operators']['watcher-operator'].image_catalog }}"
131131
deploy_watcher_service: false
132+
force_watcher_standalone: true
132133
cifmw_extras:
133134
- "@{{ ansible_user_dir }}/{{ zuul.projects['github.com/openstack-k8s-operators/watcher-operator'].
134135
src_dir }}/ci/scenarios/kuttl.yml"
@@ -163,7 +164,6 @@
163164
pre-run:
164165
- ci/playbooks/copy_container_files.yaml
165166
vars:
166-
cifmw_operator_build_meta_build: false
167167
cifmw_bop_openstack_release: epoxy
168168
cifmw_bop_dlrn_baseurl: "https://trunk.rdoproject.org/centos9-epoxy"
169169
cifmw_repo_setup_branch: epoxy
@@ -224,7 +224,6 @@
224224
parent: openstack-meta-content-provider
225225
vars:
226226
cifmw_bop_openstack_release: master
227-
cifmw_operator_build_meta_build: false
228227
cifmw_bop_dlrn_baseurl: "https://trunk.rdoproject.org/centos10-master"
229228
cifmw_repo_setup_branch: master
230229
cifmw_build_containers_registry_namespace: podified-master-centos10

‎Makefile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -398,8 +398,16 @@ OPENSTACK_NAMESPACE ?= openstack
398398

399399
.PHONY: watcher_deploy
400400
watcher_deploy: ## Deploy watcher service
401+
# In some cases, when we do not have the mutating webhooks enabled, we need to have images URLs
402+
# in the watcher CR and get it replaced by the url of containers to be tested
403+
$(eval TEMPDIR=$(shell mktemp -d))
404+
cp ${WATCHER_SAMPLE_CR_PATH} ${TEMPDIR}
405+
sed -i "s|WATCHER_API_CI_IMAGE|${WATCHER_API_CI_IMAGE}|g" ${TEMPDIR}/*
406+
sed -i "s|WATCHER_APPLIER_CI_IMAGE|${WATCHER_APPLIER_CI_IMAGE}|g" ${TEMPDIR}/*
407+
sed -i "s|WATCHER_DECISION_ENGINE_CI_IMAGE|${WATCHER_DECISION_ENGINE_CI_IMAGE}|g" ${TEMPDIR}/*
401408
oc apply -f ${WATCHER_SAMPLE_CR_PATH} -n ${OPENSTACK_NAMESPACE}
402409
oc wait watcher watcher --for condition=Ready --timeout=600s -n ${OPENSTACK_NAMESPACE}
410+
rm -rf ${TEMPDIR}
403411

404412
.PHONY: watcher_deploy_cleanup
405413
watcher_deploy_cleanup: ## Undeploy watcher service
@@ -416,6 +424,10 @@ KUTTL_SUITE ?= default
416424
KUTTL_NAMESPACE ?= watcher-kuttl-$(KUTTL_SUITE)
417425
KUTTL_SUITE_DIR ?= tests/kuttl/test-suites/$(KUTTL_SUITE)
418426

427+
.PHONY: stop_watcher_integrated
428+
stop_watcher_integrated:
429+
bash hack/stop_integrated_watcher.sh
430+
419431
.PHONY: kuttl-test-prep
420432
kuttl-test-prep:
421433
oc apply -k $(KUTTL_SUITE_DIR)/deps/ --timeout=120s

‎ci/ctlplane_watcher_patch.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
spec:
2+
watcher:
3+
enabled: true
4+
decisionengineServiceTemplate:
5+
customServiceConfig: |
6+
[watcher_cluster_data_model_collectors.compute]
7+
period = 60
8+
[watcher_cluster_data_model_collectors.storage]
9+
period = 60

‎ci/full_watcher_v1beta1_watcher_tlse.yaml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
apiVersion: cert-manager.io/v1
3+
kind: Certificate
4+
metadata:
5+
name: watcher-internal-svc
6+
spec:
7+
dnsNames:
8+
- watcher-internal.openstack.svc
9+
- watcher-internal.openstack.svc.cluster.local
10+
duration: 43800h0m0s
11+
issuerRef:
12+
group: cert-manager.io
13+
kind: Issuer
14+
name: rootca-internal
15+
secretName: cert-watcher-internal-svc
16+
usages:
17+
- key encipherment
18+
- digital signature
19+
- server auth
20+
---
21+
apiVersion: cert-manager.io/v1
22+
kind: Certificate
23+
metadata:
24+
name: watcher-public-route
25+
spec:
26+
dnsNames:
27+
- watcher-public-openstack.apps-crc.testing
28+
issuerRef:
29+
group: cert-manager.io
30+
kind: Issuer
31+
name: rootca-public
32+
secretName: cert-watcher-public-route
33+
usages:
34+
- key encipherment
35+
- digital signature
36+
- server auth
37+
---
38+
apiVersion: cert-manager.io/v1
39+
kind: Certificate
40+
metadata:
41+
name: watcher-public-svc
42+
spec:
43+
dnsNames:
44+
- watcher-public.openstack.svc
45+
- watcher-public.openstack.svc.cluster.local
46+
duration: 43800h0m0s
47+
issuerRef:
48+
group: cert-manager.io
49+
kind: Issuer
50+
name: rootca-public
51+
secretName: cert-watcher-public-svc
52+
usages:
53+
- key encipherment
54+
- digital signature
55+
- server auth
56+
---
57+
apiVersion: watcher.openstack.org/v1beta1
58+
kind: Watcher
59+
metadata:
60+
name: watcher
61+
spec:
62+
apiContainerImageURL: WATCHER_API_CI_IMAGE
63+
applierContainerImageURL: WATCHER_APPLIER_CI_IMAGE
64+
decisionengineContainerImageURL: WATCHER_DECISION_ENGINE_CI_IMAGE
65+
dbPurge: {}
66+
databaseInstance: "openstack"
67+
apiOverride:
68+
tls:
69+
secretName: cert-watcher-public-route
70+
apiServiceTemplate:
71+
override:
72+
service:
73+
public:
74+
endpointURL: https://watcher-public-openstack.apps-crc.testing
75+
tls:
76+
caBundleSecretName: "combined-ca-bundle"
77+
api:
78+
internal:
79+
secretName: cert-watcher-internal-svc
80+
public:
81+
secretName: cert-watcher-public-svc
82+
decisionengineServiceTemplate:
83+
customServiceConfig: |
84+
[watcher_cluster_data_model_collectors.compute]
85+
period = 60
86+
[watcher_cluster_data_model_collectors.storage]
87+
period = 60

‎ci/playbooks/deploy_watcher_service.yaml‎

Lines changed: 122 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,26 @@
44
- name: Deploy Watcher service
55
hosts: "{{ cifmw_target_hook_host | default('localhost') }}"
66
gather_facts: false
7+
vars:
8+
watcher_cr_file_full: "ci/full_watcher_v1beta1_watcher_tlse.yaml"
79
environment:
810
KUBECONFIG: "{{ cifmw_openshift_kubeconfig }}"
911
PATH: "{{ cifmw_path }}"
12+
CTLPLANE_PATCH: "{{ watcher_ctlplane_patch | default( '{{ watcher_repo }}/ci/ctlplane_watcher_patch.yaml' ) }}"
1013
tasks:
14+
# If the watcher-operator installation is already included in the openstack-operator we don't need to install it as
15+
# an standalone operator.
16+
- name: Check if Watcher API resources are available
17+
ansible.builtin.shell:
18+
cmd:
19+
oc api-resources --api-group=watcher.openstack.org -o name | grep -q watcher
20+
failed_when: false
21+
register: watcher_api_resources
22+
23+
- name: add watcher_installed_integrated fact, true if Watcher API resources exist
24+
set_fact:
25+
watcher_installed_integrated: "{{ watcher_api_resources.rc == 0 }}"
26+
1127
- name: Fetch dlrn md5_hash from DLRN repo
1228
when: fetch_dlrn_hash | default(true) | bool
1329
ansible.builtin.uri:
@@ -18,34 +34,111 @@
1834
retries: 6
1935
delay: 5
2036

21-
- name: Install Watcher Operator
22-
vars:
23-
_tag: "{{ latest_dlrn_tag.content | default(watcher_services_tag) | default('current-podified') }}"
24-
# When there is no Depends-On from opendev, then content_provider_os_registry_url will return null
25-
# value to child job. In that case, we need to set default to quay registry.
26-
_registry_url: >-
27-
{%- if watcher_registry_url is defined -%}
28-
{{ watcher_registry_url }}
29-
{%- elif content_provider_os_registry_url is defined and content_provider_os_registry_url == 'null' -%}
30-
quay.io/podified-master-centos9
31-
{%- else -%}
32-
{{ content_provider_os_registry_url | default('quay.io/podified-master-centos9') }}
33-
{%- endif -%}
34-
cifmw.general.ci_script:
35-
output_dir: "{{ cifmw_basedir }}/artifacts"
36-
chdir: "{{ watcher_repo }}"
37-
script: make watcher
38-
extra_args:
39-
CATALOG_IMAGE: "{{ watcher_catalog_image | default('quay.io/openstack-k8s-operators/watcher-operator-index:latest') }}"
40-
WATCHER_API_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-api:{{ _tag }}"
41-
WATCHER_DECISION_ENGINE_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-decision-engine:{{ _tag }}"
42-
WATCHER_APPLIER_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-applier:{{ _tag }}"
37+
- name: Install standalone watcher-operator if not integrated or force_watcher_standalone is true
38+
when: (not watcher_installed_integrated) or (force_watcher_standalone | default(false) | bool)
39+
block:
40+
- name: Stop watcher-operator if it is running in integrated mode
41+
when: watcher_installed_integrated
42+
cifmw.general.ci_script:
43+
output_dir: "{{ cifmw_basedir }}/artifacts"
44+
chdir: "{{ watcher_repo }}"
45+
script: make stop_watcher_integrated
46+
47+
- name: Install Watcher Operator in standalone mode
48+
vars:
49+
_tag: "{{ latest_dlrn_tag.content | default(watcher_services_tag) | default('current-podified') }}"
50+
# When there is no Depends-On from opendev, then content_provider_os_registry_url will return null
51+
# value to child job. In that case, we need to set default to quay registry.
52+
_registry_url: >-
53+
{%- if watcher_registry_url is defined -%}
54+
{{ watcher_registry_url }}
55+
{%- elif content_provider_os_registry_url is defined and content_provider_os_registry_url == 'null' -%}
56+
quay.io/podified-master-centos9
57+
{%- else -%}
58+
{{ content_provider_os_registry_url | default('quay.io/podified-master-centos9') }}
59+
{%- endif -%}
60+
cifmw.general.ci_script:
61+
output_dir: "{{ cifmw_basedir }}/artifacts"
62+
chdir: "{{ watcher_repo }}"
63+
script: make watcher
64+
extra_args:
65+
CATALOG_IMAGE: "{{ watcher_catalog_image | default('quay.io/openstack-k8s-operators/watcher-operator-index:latest') }}"
66+
WATCHER_API_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-api:{{ _tag }}"
67+
WATCHER_DECISION_ENGINE_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-decision-engine:{{ _tag }}"
68+
WATCHER_APPLIER_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-applier:{{ _tag }}"
69+
70+
# Once the integration in the openstack controlplane is finished and merged everywhere we will be
71+
# able to deploy watcher using kustomize in a pre_deploy hook. Until then we need to discover
72+
# if the integration is finished and that can not be done in pre_deploy hook as it runs before
73+
# make openstack_init so I am implementing it in this playbook which runs in post_deploy
74+
- name: Check if a Watcher CR is already created
75+
ansible.builtin.shell:
76+
cmd:
77+
oc get watcher -A -o name | grep -q watcher
78+
failed_when: false
79+
register: watcher_deployed
80+
81+
- name: Add watcher_is_deployed fact, true if a Watcher CR exist, otherwise false
82+
set_fact:
83+
watcher_is_deployed: "{{ watcher_deployed.rc == 0 }}"
84+
85+
- name: Check if Watcher is fully integrated in the OpenStackControlPlane
86+
ansible.builtin.shell:
87+
cmd:
88+
oc explain openstackcontrolplane.spec|grep -q -w "watcher"
89+
failed_when: false
90+
register: watcher_controlplane_spec
91+
92+
- name: Add watcher_in_ctlplane fact, true if Watcher is in the controlplane CRD
93+
set_fact:
94+
watcher_in_ctlplane: "{{ watcher_controlplane_spec.rc == 0 }}"
4395

4496
- name: Deploy Watcher service
45-
when: deploy_watcher_service | default('true') | bool
46-
cifmw.general.ci_script:
47-
output_dir: "{{ cifmw_basedir }}/artifacts"
48-
chdir: "{{ watcher_repo }}"
49-
script: make watcher_deploy
50-
extra_args:
51-
WATCHER_SAMPLE_CR_PATH: "{{ watcher_cr_file | default('ci/watcher_v1beta1_watcher.yaml') }}"
97+
when:
98+
- deploy_watcher_service | default('true') | bool
99+
- not watcher_is_deployed
100+
block:
101+
# When not using the standalone mode in the operator but watcher is not
102+
# in the openstackcontrolplane we do not have mutating webhooks so we need
103+
# to override the Watcher CR to use one that includes the container image URLs
104+
- name: Set watcher_cr_file to the one with container images
105+
when:
106+
- not watcher_in_ctlplane
107+
- watcher_installed_integrated
108+
- not force_watcher_standalone | default(false) | bool
109+
set_fact:
110+
watcher_cr_file_override: "{{ watcher_cr_file_full }}"
111+
112+
- name: Deploy Watcher service using standalone Watcher CR
113+
when: not watcher_in_ctlplane
114+
vars:
115+
_tag: "{{ latest_dlrn_tag.content | default(watcher_services_tag) | default('current-podified') }}"
116+
# When there is no Depends-On from opendev, then content_provider_os_registry_url will return null
117+
# value to child job. In that case, we need to set default to quay registry.
118+
_registry_url: >-
119+
{%- if watcher_registry_url is defined -%}
120+
{{ watcher_registry_url }}
121+
{%- elif content_provider_os_registry_url is defined and content_provider_os_registry_url == 'null' -%}
122+
quay.io/podified-master-centos9
123+
{%- else -%}
124+
{{ content_provider_os_registry_url | default('quay.io/podified-master-centos9') }}
125+
{%- endif -%}
126+
cifmw.general.ci_script:
127+
output_dir: "{{ cifmw_basedir }}/artifacts"
128+
chdir: "{{ watcher_repo }}"
129+
script: make watcher_deploy
130+
extra_args:
131+
WATCHER_SAMPLE_CR_PATH: "{{ watcher_cr_file_override | default( watcher_cr_file | default('ci/watcher_v1beta1_watcher.yaml')) }}"
132+
WATCHER_API_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-api:{{ _tag }}"
133+
WATCHER_DECISION_ENGINE_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-decision-engine:{{ _tag }}"
134+
WATCHER_APPLIER_CI_IMAGE: "{{ _registry_url }}/openstack-watcher-applier:{{ _tag }}"
135+
136+
- name: Deploy Watcher service by patching an existing OpenStackControlplane
137+
when: watcher_in_ctlplane
138+
ansible.builtin.shell:
139+
cmd: |
140+
set -ex
141+
CTLPLANE=$(oc get openstackcontrolplane -n openstack -o jsonpath="{range .items[*]}{@.metadata.name}{end}")
142+
oc patch openstackcontrolplane $CTLPLANE -n openstack --type merge --patch-file $CTLPLANE_PATCH
143+
oc wait openstackcontrolplane $CTLPLANE -n openstack --for condition=Ready --timeout=600s
144+
oc wait watcher watcher -n openstack --for condition=Ready --timeout=600s

‎ci/scenarios/kuttl.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,6 @@ post_install_operators_kuttl_from_operator:
2222
deploy_watcher_service_extra_vars | default({}) |
2323
combine({ 'watcher_repo': watcher_repo })
2424
}}
25+
26+
# Enable observability operator
27+
cifmw_deploy_obs: true

‎hack/stop_integrated_watcher.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
set -x
3+
4+
WATCHER_CSV_NAME="$(oc get csv -n openstack-operators -l operators.coreos.com/watcher-operator.openstack-operators -o name)"
5+
6+
if [ -z ${WATCHER_CSV_NAME} ]; then
7+
OPENSTACK_CSV_NAME="$(oc get csv -n openstack-operators -l operators.coreos.com/openstack-operator.openstack-operators -o name)"
8+
if [ -n "{OPENSTACK_CSV_NAME}" ]; then
9+
oc patch "${OPENSTACK_CSV_NAME}" -n openstack-operators --type=json -p="[{'op': 'replace', 'path': '/spec/install/spec/deployments/0/spec/replicas', 'value': 0}]"
10+
oc scale --replicas=0 -n openstack-operators deploy/watcher-operator-controller-manager
11+
else
12+
echo "Openstack operator is not installed"
13+
fi
14+
else
15+
echo "Watcher operator installed in standalone mode"
16+
fi

0 commit comments

Comments
 (0)