From e9f00b11de1539f54bad4ecd508adb7db7c45c89 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Mon, 14 Oct 2024 11:39:46 +0200 Subject: [PATCH 01/16] Improve validate-origin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit If we set the main.git.upstreamRepoURL then we need to check that one for the presence of the branch and not whatever is configured as the origin remote. Tested as follows: - Working upstreamRepoURL: ❯ yq -r '.main.git.repoUpstreamURL // (.main.git.repoUpstreamURL = "")' values-global.yaml https://github.com/mbaldessari/industrial-edge ❯ make validate-origin make -f common/Makefile validate-origin make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/industrial-edge' Checking repository: Upstream URL set to: https://github.com/mbaldessari/industrial-edge https://github.com/mbaldessari/industrial-edge - branch 'for-max-s3-kafka-fix': OK make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/industrial-edge' - upstreamRepoURL without the branch: ❯ yq -r '.main.git.repoUpstreamURL // (.main.git.repoUpstreamURL = "")' values-global.yaml https://github.com/mamurak/industrial-edge ❯ make validate-origin make -f common/Makefile validate-origin make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/industrial-edge' Checking repository: Upstream URL set to: https://github.com/mamurak/industrial-edge https://github.com/mamurak/industrial-edge - branch 'for-max-s3-kafka-fix': NOT FOUND make[1]: *** [common/Makefile:138: validate-origin] Error 1 make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/industrial-edge' make: *** [Makefile:24: validate-origin] Error 2 - No repoUpstreamURL set: ❯ yq -r '.main.git.repoUpstreamURL // (.main.git.repoUpstreamURL = "")' values-global.yaml ❯ ❯ make validate-origin make -f common/Makefile validate-origin make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/industrial-edge' Checking repository: https://github.com/mbaldessari/industrial-edge.git - branch 'for-max-s3-kafka-fix': OK make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/industrial-edge' --- Makefile | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index a5c102b8..ceec9745 100644 --- a/Makefile +++ b/Makefile @@ -129,12 +129,22 @@ token-kubeconfig: ## Create a local ~/.kube/config with password (not usually ne # We only check the remote ssh git branch's existance if we're not running inside a container # as getting ssh auth working inside a container seems a bit brittle +# If the main repoUpstreamURL field is set, then we need to check against +# that and not target_repo .PHONY: validate-origin validate-origin: ## verify the git origin is available @echo "Checking repository:" - @echo -n " $(TARGET_REPO) - branch '$(TARGET_BRANCH)': " - @git ls-remote --exit-code --heads $(TARGET_REPO) $(TARGET_BRANCH) >/dev/null &&\ - echo "OK" || (echo "NOT FOUND"; exit 1) + $(eval UPSTREAMURL := $(shell yq -r '.main.git.repoUpstreamURL // (.main.git.repoUpstreamURL = "")' values-global.yaml)) + @if [ -z "$(UPSTREAMURL)" ]; then\ + echo -n " $(TARGET_REPO) - branch '$(TARGET_BRANCH)': ";\ + git ls-remote --exit-code --heads $(TARGET_REPO) $(TARGET_BRANCH) >/dev/null &&\ + echo "OK" || (echo "NOT FOUND"; exit 1);\ + else\ + echo "Upstream URL set to: $(UPSTREAMURL)";\ + echo -n " $(UPSTREAMURL) - branch '$(TARGET_BRANCH)': ";\ + git ls-remote --exit-code --heads $(UPSTREAMURL) $(TARGET_BRANCH) >/dev/null &&\ + echo "OK" || (echo "NOT FOUND"; exit 1);\ + fi .PHONY: validate-cluster validate-cluster: ## Do some cluster validations before installing From 8dfad491b7b616a0866d752e361935d526335382 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Tue, 15 Oct 2024 19:25:46 +0200 Subject: [PATCH 02/16] Make the make install output less daunting The current output is a bit daunting for first-time users as it outputs things like the following a few times: customresourcedefinition.apiextensions.k8s.io/patterns.gitops.hybrid-cloud-patterns.io created configmap/patterns-operator-config created subscription.operators.coreos.com/patterns-operator created error: resource mapping not found for name: "rhoai-patterns-demo" namespace: "openshift-operators" from "STDIN": no matches for kind "Pattern" in version "gitops.hybrid-cloud-patterns.io/v1alpha1" ensure CRDs are installed first Let's switch to something a bit more user-friendly: make -f common/Makefile operator-deploy make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' Checking repository: https://github.com/mbaldessari/multicloud-gitops.git - branch 'luis-demo': OK Checking cluster: cluster-info: OK storageclass: WARNING: No storageclass found Installing pattern: Done ... Do some magic with file descriptors so we still manage to capture the helm template stderr and the oc apply stdout+stderr and output them at the end in case of failure. In such cases the output will be something like the following: Installing pattern: .....Installation failed [5/5]. Error: Pulled: quay.io/hybridcloudpatterns/pattern-install:0.0.3 Digest: sha256:dd2d35d462b75aa8358ff278757dca0ee3c878cadafa64df8c68f880b59569ef E1015 18:41:31.585465 196315 memcache.go:265] couldn't get current server API group list: Get "https://api.sno3.ocplab.ocp:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by un known authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer") ... Suggested-by: Luis Tomas Bolivar --- Makefile | 6 +----- scripts/deploy-pattern.sh | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 5 deletions(-) create mode 100755 scripts/deploy-pattern.sh diff --git a/Makefile b/Makefile index a5c102b8..44a42228 100644 --- a/Makefile +++ b/Makefile @@ -68,11 +68,7 @@ preview-%: .PHONY: operator-deploy operator-deploy operator-upgrade: validate-prereq validate-origin validate-cluster ## runs helm install - @set -e -o pipefail - # Retry five times because the CRD might not be fully installed yet - for i in {1..5}; do \ - helm template --include-crds --name-template $(NAME) $(PATTERN_INSTALL_CHART) $(HELM_OPTS) | oc apply -f- && break || sleep 10; \ - done + @common/scripts/deploy-pattern.sh $(NAME) $(PATTERN_INSTALL_CHART) $(HELM_OPTS) .PHONY: uninstall uninstall: ## runs helm uninstall diff --git a/scripts/deploy-pattern.sh b/scripts/deploy-pattern.sh new file mode 100755 index 00000000..7393d9b6 --- /dev/null +++ b/scripts/deploy-pattern.sh @@ -0,0 +1,26 @@ +#!/bin/bash +set -o pipefail + +RUNS=5 +# Retry five times because the CRD might not be fully installed yet +echo -n "Installing pattern: " +for i in $(seq 1 ${RUNS}); do \ + exec 3>&1 4>&2 + OUT=$( { helm template --include-crds --name-template $* 2>&4 | oc apply -f- 2>&4 1>&3; } 4>&1 3>&1) + exec 3>&- 4>&- + ret=$? + if [ ${ret} -eq 0 ]; then + break; + else + echo -n "." + sleep 10 + fi +done + +# All the runs failed +if [ ${i} -eq ${RUNS} ]; then + echo "Installation failed [${i}/${RUNS}]. Error:" + echo "${OUT}" + exit 1 +fi +echo "Done" From c9f1fddbc7db2d2093c0e27dbab52b80767904d5 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Tue, 22 Oct 2024 10:00:14 +0200 Subject: [PATCH 03/16] Record the exit code at the right time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Otherwise it records the exit code of the exec call which is always 0. This now makes things error out correctly whereas previously it did not: ❯ ./pattern.sh make install make -f common/Makefile operator-deploy make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' Checking repository: https://github.com/mbaldessari/multicloud-gitops.git - branch 'debug': OK Checking cluster: cluster-info: OK storageclass: OK Installing pattern: .....Installation failed [5/5]. Error: Pulled: quay.io/rhn_support_mbaldess/pattern-install:0.0.4 Digest: sha256:6a5431d257f4bf05d6ac91bca4f4aa010ecb1ee60d2e03c1d0984835d210a0a8 customresourcedefinition.apiextensions.k8s.io/patterns.gitops.hybrid-cloud-patterns.io unchanged configmap/patterns-operator-config unchanged resource mapping not found for name: "patterns-operator" namespace: "openshift-operators" from "STDIN": no matches for kind "SubscriptionNotexistant" in version "operators.coreos.com/v1alpha1" ensure CRDs are installed first Error from server (NotFound): error when creating "STDIN": namespaces "openshift-operators-notexistant" not found make[1]: *** [common/Makefile:71: operator-deploy] Error 1 make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' make: *** [Makefile:12: operator-deploy] Error 2 --- scripts/deploy-pattern.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/deploy-pattern.sh b/scripts/deploy-pattern.sh index 7393d9b6..56837366 100755 --- a/scripts/deploy-pattern.sh +++ b/scripts/deploy-pattern.sh @@ -7,8 +7,8 @@ echo -n "Installing pattern: " for i in $(seq 1 ${RUNS}); do \ exec 3>&1 4>&2 OUT=$( { helm template --include-crds --name-template $* 2>&4 | oc apply -f- 2>&4 1>&3; } 4>&1 3>&1) - exec 3>&- 4>&- ret=$? + exec 3>&- 4>&- if [ ${ret} -eq 0 ]; then break; else From 9a24f6fcfc28227ada119bf4d1d389808e146fa9 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Mon, 25 Nov 2024 12:08:02 +0100 Subject: [PATCH 04/16] Fix path when invoking the qe run_test.sh script --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index fef82fd1..20e543fc 100644 --- a/Makefile +++ b/Makefile @@ -214,7 +214,7 @@ argo-healthcheck: ## Checks if all argo applications are synced .PHONY: qe-tests qe-tests: ## Runs the tests that QE runs @set -e; if [ -f ./tests/interop/run_tests.sh ]; then \ - ./tests/interop/run_tests.sh; \ + pushd ./tests/interop; ./run_tests.sh; popd; \ else \ echo "No ./tests/interop/run_tests.sh found skipping"; \ fi From 0d9d51ad7198e22edbf23cfddcac7f8835aac31e Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Thu, 19 Dec 2024 13:47:14 +0100 Subject: [PATCH 05/16] Add a warning when pattern folder and pattern name differ Co-Authored-By: Akos Eros --- Makefile | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Makefile b/Makefile index 20e543fc..37984d1e 100644 --- a/Makefile +++ b/Makefile @@ -164,6 +164,13 @@ validate-schema: ## validates values files against schema in common/clustergroup .PHONY: validate-prereq validate-prereq: ## verify pre-requisites + $(eval GLOBAL_PATTERN := $(shell yq -r .global.pattern values-global.yaml)) + @if [ $(NAME) != $(GLOBAL_PATTERN) ]; then\ + echo "";\ + echo "WARNING: folder directory is \"$(NAME)\" and global.pattern is set to \"$(GLOBAL_PATTERN)\"";\ + echo "this can create problems. Please make sure they are the same!";\ + echo "";\ + fi @if [ ! -f /run/.containerenv ]; then\ echo "Checking prerequisites:";\ for t in $(EXECUTABLES); do if ! which $$t > /dev/null 2>&1; then echo "No $$t in PATH"; exit 1; fi; done;\ From 0cc13f6d9f41330516118c7fc4650b54e195a130 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Thu, 19 Dec 2024 13:48:16 +0100 Subject: [PATCH 06/16] Drop some old and unused prereq test --- Makefile | 2 -- 1 file changed, 2 deletions(-) diff --git a/Makefile b/Makefile index 37984d1e..1be6bbf3 100644 --- a/Makefile +++ b/Makefile @@ -173,8 +173,6 @@ validate-prereq: ## verify pre-requisites fi @if [ ! -f /run/.containerenv ]; then\ echo "Checking prerequisites:";\ - for t in $(EXECUTABLES); do if ! which $$t > /dev/null 2>&1; then echo "No $$t in PATH"; exit 1; fi; done;\ - echo " Check for '$(EXECUTABLES)': OK";\ echo -n " Check for python-kubernetes: ";\ if ! ansible -m ansible.builtin.command -a "{{ ansible_python_interpreter }} -c 'import kubernetes'" localhost > /dev/null 2>&1; then echo "Not found"; exit 1; fi;\ echo "OK";\ From 5063467592e97e9a5cbd10e5399a9f0c23a6dee3 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Tue, 7 Jan 2025 17:56:28 +0100 Subject: [PATCH 07/16] Fix validated-schema make target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since the slimming of common this won't work anymore: ❯ make validate-schema make -f common/Makefile validate-schema make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' Validating clustergroup schema of: ./values-global.yamlError: repo common not found make[1]: *** [common/Makefile:162: validate-schema] Error 1 make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' make: *** [Makefile:12: validate-schema] Error 2 Fix this to use the oci helm chart: ❯ make validate-schema make -f common/Makefile validate-schema make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' Validating clustergroup schema of: ./values-global.yamlPulled: quay.io/hybridcloudpatterns/clustergroup:0.9.13 Digest: sha256:725af54c0a5ad8c2235676bbff2785ece62c9929ab58aaf33837aa3f19708ce6 ./values-group-one.yamlPulled: quay.io/hybridcloudpatterns/clustergroup:0.9.13 Digest: sha256:725af54c0a5ad8c2235676bbff2785ece62c9929ab58aaf33837aa3f19708ce6 ./values-hub.yamlPulled: quay.io/hybridcloudpatterns/clustergroup:0.9.13 Digest: sha256:725af54c0a5ad8c2235676bbff2785ece62c9929ab58aaf33837aa3f19708ce6 make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops' --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 1be6bbf3..53e15d3f 100644 --- a/Makefile +++ b/Makefile @@ -159,7 +159,7 @@ validate-cluster: ## Do some cluster validations before installing validate-schema: ## validates values files against schema in common/clustergroup $(eval VAL_PARAMS := $(shell for i in ./values-*.yaml; do echo -n "$${i} "; done)) @echo -n "Validating clustergroup schema of: " - @set -e; for i in $(VAL_PARAMS); do echo -n " $$i"; helm template common/clustergroup $(HELM_OPTS) -f "$${i}" >/dev/null; done + @set -e; for i in $(VAL_PARAMS); do echo -n " $$i"; helm template oci://quay.io/hybridcloudpatterns/clustergroup $(HELM_OPTS) -f "$${i}" >/dev/null; done @echo .PHONY: validate-prereq From 250c154582b91a443450bc46c7d347f7900cb23e Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Tue, 14 Jan 2025 20:40:19 +0100 Subject: [PATCH 08/16] Wait some more before giving up Yukin observed a case on a baremetal server where the install failed with: make -f common/Makefile operator-deploy make[1]: Entering directory '/home/fedora/validated_patterns/multicloud-gitops' Checking repository: https://github.com/validatedpatterns-workspace/multicloud-gitops - branch 'qe_test-18760': OK Checking cluster: cluster-info: OK storageclass: OK Installing pattern: ....Installation failed [5/5]. Error: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/fedora/rhvpsno2-intel-18760/auth/kubeconfig WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/fedora/rhvpsno2-intel-18760/auth/kubeconfig Pulled: quay.io/hybridcloudpatterns/pattern-install:0.0.7 Digest: sha256:b845f86c735478cfd44b0b43842697851cec64737c737bd18a872fa86bb0484d customresourcedefinition.apiextensions.k8s.io/patterns.gitops.hybrid-cloud-patterns.io unchanged configmap/patterns-operator-config unchanged pattern.gitops.hybrid-cloud-patterns.io/multicloud-gitops created subscription.operators.coreos.com/patterns-operator unchanged make[1]: *** [common/Makefile:71: operator-deploy] Error 1 make[1]: Leaving directory '/home/fedora/validated_patterns/multicloud-gitops' make: *** [Makefile:12: operator-deploy] Error 2 In fact the install proceeded just okay, we just gave up too early. Let's double the amount of times we wait for this and also increase the wait in between tries by 5 seconds. Hopefully this should cover these edge cases. --- scripts/deploy-pattern.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/deploy-pattern.sh b/scripts/deploy-pattern.sh index 56837366..61074fe1 100755 --- a/scripts/deploy-pattern.sh +++ b/scripts/deploy-pattern.sh @@ -1,7 +1,8 @@ #!/bin/bash set -o pipefail -RUNS=5 +RUNS=10 +WAIT=15 # Retry five times because the CRD might not be fully installed yet echo -n "Installing pattern: " for i in $(seq 1 ${RUNS}); do \ @@ -13,7 +14,7 @@ for i in $(seq 1 ${RUNS}); do \ break; else echo -n "." - sleep 10 + sleep "${WAIT}" fi done From cc36e1f879f7d47ed9c8ae9be855d04fc3f98449 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Wed, 22 Jan 2025 17:14:48 +0100 Subject: [PATCH 09/16] Fix argo-healthcheck target If ACM is installed the search for `applications` matches the ACM one and not the argo one. --- Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 53e15d3f..fc10236e 100644 --- a/Makefile +++ b/Makefile @@ -193,16 +193,16 @@ validate-prereq: ## verify pre-requisites .PHONY: argo-healthcheck argo-healthcheck: ## Checks if all argo applications are synced @echo "Checking argo applications" - $(eval APPS := $(shell oc get applications -A -o jsonpath='{range .items[*]}{@.metadata.namespace}{","}{@.metadata.name}{"\n"}{end}')) + $(eval APPS := $(shell oc get applications.argoproj.io -A -o jsonpath='{range .items[*]}{@.metadata.namespace}{","}{@.metadata.name}{"\n"}{end}')) @NOTOK=0; \ for i in $(APPS); do\ n=`echo "$${i}" | cut -f1 -d,`;\ a=`echo "$${i}" | cut -f2 -d,`;\ - STATUS=`oc get -n "$${n}" application/"$${a}" -o jsonpath='{.status.sync.status}'`;\ + STATUS=`oc get -n "$${n}" applications.argoproj.io/"$${a}" -o jsonpath='{.status.sync.status}'`;\ if [[ $$STATUS != "Synced" ]]; then\ NOTOK=$$(( $${NOTOK} + 1));\ fi;\ - HEALTH=`oc get -n "$${n}" application/"$${a}" -o jsonpath='{.status.health.status}'`;\ + HEALTH=`oc get -n "$${n}" applications.argoproj.io/"$${a}" -o jsonpath='{.status.health.status}'`;\ if [[ $$HEALTH != "Healthy" ]]; then\ NOTOK=$$(( $${NOTOK} + 1));\ fi;\ From e3e7add390b9e8c071be6ea8028fd790a1f2d052 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Tue, 4 Feb 2025 21:25:55 +0100 Subject: [PATCH 10/16] Push TARGET_SITE into the in-container variables This way we can override the TARGET_SITE when invoking pattern.sh --- scripts/pattern-util.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/pattern-util.sh b/scripts/pattern-util.sh index 1fcaee76..d04ea05d 100755 --- a/scripts/pattern-util.sh +++ b/scripts/pattern-util.sh @@ -85,6 +85,7 @@ podman run -it --rm --pull=newer \ -e EXTRA_HELM_OPTS \ -e EXTRA_PLAYBOOK_OPTS \ -e TARGET_ORIGIN \ + -e TARGET_SITE \ -e NAME \ -e TOKEN_SECRET \ -e TOKEN_NAMESPACE \ From 0746c6e77c8d5c405e99dd10b1ae9fffedda1d9c Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Fri, 7 Feb 2025 11:49:02 +0100 Subject: [PATCH 11/16] Do not bind mount /etc/pki blindly Since ubuntu sometimes has /etc/pki/fwupd with little else in there, let's just bind mount /etc/pki when /etc/pki/tls exists. This keeps Fedora-based distros running and should fix this specific corner case observed on ubuntu. Co-Authored-By: Akos Eros Closes: https://github.com/validatedpatterns/medical-diagnosis/issues/130 --- scripts/pattern-util.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/pattern-util.sh b/scripts/pattern-util.sh index d04ea05d..27b30229 100755 --- a/scripts/pattern-util.sh +++ b/scripts/pattern-util.sh @@ -64,8 +64,10 @@ fi # if we are using podman machine then we do not bind mount anything (for now!) REMOTE_PODMAN=$(podman system connection list -q | wc -l) if [ $REMOTE_PODMAN -eq 0 ]; then # If we are not using podman machine we check the hosts folders - # Use /etc/pki by default and try a couple of fallbacks if it does not exist - if [ -d /etc/pki ]; then + # We check /etc/pki/tls because on ubuntu /etc/pki/fwupd sometimes + # exists but not /etc/pki/tls and we do not want to bind mount in such a case + # as it would find no certificates at all. + if [ -d /etc/pki/tls ]; then PKI_HOST_MOUNT_ARGS="-v /etc/pki:/etc/pki:ro" elif [ -d /etc/ssl ]; then PKI_HOST_MOUNT_ARGS="-v /etc/ssl:/etc/ssl:ro" From 98744fb1316eb40135767af2a8e0ea7154851c72 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Sun, 9 Feb 2025 18:52:05 +0100 Subject: [PATCH 12/16] Add a CI job for pattern.sh --- .github/workflows/pattern-sh-ci.yml | 41 +++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 .github/workflows/pattern-sh-ci.yml diff --git a/.github/workflows/pattern-sh-ci.yml b/.github/workflows/pattern-sh-ci.yml new file mode 100644 index 00000000..9aa4cf4e --- /dev/null +++ b/.github/workflows/pattern-sh-ci.yml @@ -0,0 +1,41 @@ +name: Run Bash Script on Multiple Distributions + +on: + push: + branches: + - main + pull_request: + +jobs: + run-script: + name: Run Bash Script + strategy: + matrix: + # Fedora is not an option yet + os: [ubuntu-latest, ubuntu-22.04] + runs-on: ${{ matrix.os }} + + steps: + - name: Checkout Repository + uses: actions/checkout@v4 + + - name: Install Podman on Ubuntu + if: contains(matrix.os, 'ubuntu') + run: | + sudo apt-get update + sudo apt-get install -y podman + + # Currently we do not do MacOSX as it is not free, maybe in the future + # - name: Install Podman on macOS + # if: contains(matrix.os, 'macos') + # run: | + # brew install podman + # podman machine init + # podman machine start + + - name: Verify Podman Installation + run: podman --version + + - name: Run pattern.sh script + run: | + ./scripts/pattern-util.sh make validate-origin From 7c950f678f1dd71acffa80b69dfba212a696eb04 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Mon, 10 Feb 2025 09:15:27 +0100 Subject: [PATCH 13/16] Allow TARGET_BRANCH to be overridden --- .github/workflows/pattern-sh-ci.yml | 1 + Makefile | 2 +- scripts/pattern-util.sh | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pattern-sh-ci.yml b/.github/workflows/pattern-sh-ci.yml index 9aa4cf4e..92eadfd5 100644 --- a/.github/workflows/pattern-sh-ci.yml +++ b/.github/workflows/pattern-sh-ci.yml @@ -38,4 +38,5 @@ jobs: - name: Run pattern.sh script run: | + export TARGET_BRANCH=main ./scripts/pattern-util.sh make validate-origin diff --git a/Makefile b/Makefile index fc10236e..4e977e10 100644 --- a/Makefile +++ b/Makefile @@ -18,7 +18,7 @@ TARGET_ORIGIN ?= origin # This is because we expect to use tokens for repo authentication as opposed to SSH keys TARGET_REPO=$(shell git ls-remote --get-url --symref $(TARGET_ORIGIN) | sed -e 's/.*URL:[[:space:]]*//' -e 's%^git@%%' -e 's%^https://%%' -e 's%:%/%' -e 's%^%https://%') # git branch --show-current is also available as of git 2.22, but we will use this for compatibility -TARGET_BRANCH=$(shell git rev-parse --abbrev-ref HEAD) +TARGET_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD) UUID_FILE ?= ~/.config/validated-patterns/pattern-uuid UUID_HELM_OPTS ?= diff --git a/scripts/pattern-util.sh b/scripts/pattern-util.sh index 27b30229..8258d469 100755 --- a/scripts/pattern-util.sh +++ b/scripts/pattern-util.sh @@ -88,6 +88,7 @@ podman run -it --rm --pull=newer \ -e EXTRA_PLAYBOOK_OPTS \ -e TARGET_ORIGIN \ -e TARGET_SITE \ + -e TARGET_BRANCH \ -e NAME \ -e TOKEN_SECRET \ -e TOKEN_NAMESPACE \ From 46af2026b3c8221b143580826bc6c3f800c449f0 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Mon, 10 Feb 2025 09:17:50 +0100 Subject: [PATCH 14/16] Restrict GH actions only to certain files --- .github/workflows/pattern-sh-ci.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/pattern-sh-ci.yml b/.github/workflows/pattern-sh-ci.yml index 92eadfd5..ed0e6a07 100644 --- a/.github/workflows/pattern-sh-ci.yml +++ b/.github/workflows/pattern-sh-ci.yml @@ -2,9 +2,15 @@ name: Run Bash Script on Multiple Distributions on: push: + paths: + - "scripts/**" + - "Makefile" branches: - main pull_request: + paths: + - "scripts/**" + - "Makefile" jobs: run-script: From 39882c74be9689498d4b945f670a1efc5b32247c Mon Sep 17 00:00:00 2001 From: Drew Minnear Date: Tue, 25 Feb 2025 13:27:12 -0500 Subject: [PATCH 15/16] propagate the env var EXTRA_PLAYBOOK_OPTS to our ansible-playbook commands Currently, we pass the env var EXTRA_PLAYBOOK_OPTS into our utility container when running the `pattern-util.sh` script, however, we do not use it anywhere. This commit adds propagation of the env var to the `ansible-playbook` commands which could make use of it. As an example, you could set ```sh export EXTRA_PLAYBOOK_OPTS="-vvv" ``` which would enable verbose logging for any of the ansible playbooks when we run `./pattern.sh make ` in any of our pattern repos. --- Makefile | 6 +++++- scripts/display-secrets-info.sh | 4 +++- scripts/load-k8s-secrets.sh | 4 +++- scripts/process-secrets.sh | 4 +++- scripts/vault-utils.sh | 4 +++- scripts/write-token-kubeconfig.sh | 4 +++- 6 files changed, 20 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 4e977e10..84f6afc4 100644 --- a/Makefile +++ b/Makefile @@ -8,6 +8,10 @@ endif # the command line. I.e. we can set things without having to tweak values files EXTRA_HELM_OPTS ?= +# This variable can be set in order to pass additional ansible-playbook arguments from the +# the command line. I.e. we can set -vvv for more verbose logging +EXTRA_PLAYBOOK_OPTS ?= + # INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:394248 # or # INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:394248,registry-proxy.engineering.redhat.com/rh-osbs/iib:394249 @@ -111,7 +115,7 @@ secrets-backend-none: ## Edits values files to remove secrets manager + ESO .PHONY: load-iib load-iib: ## CI target to install Index Image Bundles @set -e; if [ x$(INDEX_IMAGES) != x ]; then \ - ansible-playbook rhvp.cluster_utils.iib_ci; \ + ansible-playbook $(EXTRA_PLAYBOOK_OPTS) rhvp.cluster_utils.iib_ci; \ else \ echo "No INDEX_IMAGES defined. Bailing out"; \ exit 1; \ diff --git a/scripts/display-secrets-info.sh b/scripts/display-secrets-info.sh index d9915855..ca0069e2 100755 --- a/scripts/display-secrets-info.sh +++ b/scripts/display-secrets-info.sh @@ -23,4 +23,6 @@ fi PATTERN_NAME=$(basename "`pwd`") -ansible-playbook -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" -e secrets_backing_store="${SECRETS_BACKING_STORE}" -e override_no_log=false "rhvp.cluster_utils.display_secrets_info" +EXTRA_PLAYBOOK_OPTS="${EXTRA_PLAYBOOK_OPTS:-}" + +ansible-playbook -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" -e secrets_backing_store="${SECRETS_BACKING_STORE}" -e hide_sensitive_output=false ${EXTRA_PLAYBOOK_OPTS} "rhvp.cluster_utils.display_secrets_info" diff --git a/scripts/load-k8s-secrets.sh b/scripts/load-k8s-secrets.sh index 9219f92f..707e51a7 100755 --- a/scripts/load-k8s-secrets.sh +++ b/scripts/load-k8s-secrets.sh @@ -13,4 +13,6 @@ PATTERNPATH=$(dirname "${COMMONPATH}") PATTERN_NAME=${1:-$(basename "`pwd`")} -ansible-playbook -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" "rhvp.cluster_utils.k8s_secrets" +EXTRA_PLAYBOOK_OPTS="${EXTRA_PLAYBOOK_OPTS:-}" + +ansible-playbook -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" ${EXTRA_PLAYBOOK_OPTS} "rhvp.cluster_utils.k8s_secrets" diff --git a/scripts/process-secrets.sh b/scripts/process-secrets.sh index 47eff7fa..a0d34f88 100755 --- a/scripts/process-secrets.sh +++ b/scripts/process-secrets.sh @@ -14,4 +14,6 @@ PATTERNPATH=$(dirname "${COMMONPATH}") PATTERN_NAME=${1:-$(basename "`pwd`")} SECRETS_BACKING_STORE="$($SCRIPTPATH/determine-secretstore-backend.sh)" -ansible-playbook -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" -e secrets_backing_store="${SECRETS_BACKING_STORE}" "rhvp.cluster_utils.process_secrets" +EXTRA_PLAYBOOK_OPTS="${EXTRA_PLAYBOOK_OPTS:-}" + +ansible-playbook -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" -e secrets_backing_store="${SECRETS_BACKING_STORE}" ${EXTRA_PLAYBOOK_OPTS} "rhvp.cluster_utils.process_secrets" diff --git a/scripts/vault-utils.sh b/scripts/vault-utils.sh index b014e5a4..2f766499 100755 --- a/scripts/vault-utils.sh +++ b/scripts/vault-utils.sh @@ -25,4 +25,6 @@ if [ -z ${TASK} ]; then exit 1 fi -ansible-playbook -t "${TASK}" -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" "rhvp.cluster_utils.vault" +EXTRA_PLAYBOOK_OPTS="${EXTRA_PLAYBOOK_OPTS:-}" + +ansible-playbook -t "${TASK}" -e pattern_name="${PATTERN_NAME}" -e pattern_dir="${PATTERNPATH}" ${EXTRA_PLAYBOOK_OPTS} "rhvp.cluster_utils.vault" diff --git a/scripts/write-token-kubeconfig.sh b/scripts/write-token-kubeconfig.sh index 7544fac2..e7913e53 100755 --- a/scripts/write-token-kubeconfig.sh +++ b/scripts/write-token-kubeconfig.sh @@ -13,4 +13,6 @@ SCRIPTPATH=$(dirname "${SCRIPT}") COMMONPATH=$(dirname "${SCRIPTPATH}") PATTERNPATH=$(dirname "${COMMONPATH}") -ansible-playbook -e pattern_dir="${PATTERNPATH}" -e kubeconfig_file="${OUTPUTFILE}" "rhvp.cluster_utils.write-token-kubeconfig" +EXTRA_PLAYBOOK_OPTS="${EXTRA_PLAYBOOK_OPTS:-}" + +ansible-playbook -e pattern_dir="${PATTERNPATH}" -e kubeconfig_file="${OUTPUTFILE}" ${EXTRA_PLAYBOOK_OPTS} "rhvp.cluster_utils.write-token-kubeconfig" From ba9feab00e2b077c4db567feae4f9b5588333f64 Mon Sep 17 00:00:00 2001 From: Drew Minnear Date: Thu, 27 Feb 2025 09:58:09 -0500 Subject: [PATCH 16/16] update how to make common subtree A few small changes in this commit: * Update README to reference the `make-common-subtree` script in common rather than MCG repo * Update README and `make-common-subtree` script to use same default remote name for common subtree that we use in our `update-common-everywhere` script. * Update file name for the script to use dashes rather than underscores for consistency * Update the name of our GH org to `validatedpatterns` --- README.md | 10 +++++----- .../{make_common_subtree.sh => make-common-subtree.sh} | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) rename scripts/{make_common_subtree.sh => make-common-subtree.sh} (91%) diff --git a/README.md b/README.md index 41223529..b36bc1a4 100644 --- a/README.md +++ b/README.md @@ -34,16 +34,16 @@ main: ## Start Here This repository is never used as standalone. It is usually imported in each pattern as a subtree. -In order to import the common/ the very first time you can use -`https://github.com/validatedpatterns/multicloud-gitops/blob/main/common/scripts/make_common_subtree.sh` +In order to import the common subtree the very first time you can use the script +[make_common_subtree.sh](scripts/make-common-subtree.sh). In order to update your common subtree inside your pattern repository you can either use `https://github.com/validatedpatterns/utilities/blob/main/scripts/update-common-everywhere.sh` or -do it manually by doing the following: +do it manually with the following commands: ```sh -git remote add -f upstream-common https://github.com/validatedpatterns/common.git -git merge -s subtree -Xtheirs -Xsubtree=common upstream-common/main +git remote add -f common-upstream https://github.com/validatedpatterns/common.git +git merge -s subtree -Xtheirs -Xsubtree=common common-upstream/main ``` ## Secrets diff --git a/scripts/make_common_subtree.sh b/scripts/make-common-subtree.sh similarity index 91% rename from scripts/make_common_subtree.sh rename to scripts/make-common-subtree.sh index a5e406d8..196a4c8d 100755 --- a/scripts/make_common_subtree.sh +++ b/scripts/make-common-subtree.sh @@ -7,7 +7,7 @@ if [ "$1" = "-h" ]; then echo "$0 " echo echo "Run without arguments, the script would run as if these arguments had been passed:" - echo "$0 https://github.com/hybrid-cloud-patterns/common.git main common-subtree" + echo "$0 https://github.com/validatedpatterns/common.git main common-upstream" echo echo "Please ensure the git subtree command is available. On RHEL/Fedora, the git subtree command" echo "is in a separate package called git-subtree" @@ -41,7 +41,7 @@ fi if [ "$1" ]; then subtree_repo=$1 else - subtree_repo=https://github.com/hybrid-cloud-patterns/common.git + subtree_repo=https://github.com/validatedpatterns/common.git fi if [ "$2" ]; then @@ -53,7 +53,7 @@ fi if [ "$3" ]; then subtree_remote=$3 else - subtree_remote=common-subtree + subtree_remote=common-upstream fi git diff --quiet || (echo "This script must be run on a clean working tree" && exit 1)