K8SPSMDB-885 - E2E tests - start mongod major upgrade from the lowest version (#1194)

cap1984 · nmarukovich · egegunes · web-flow · commit e6e206b5ff8e · 2023-09-11T10:08:25.000+02:00
* K8SPSMDB-794 release 1.14.0 * added missed tests * Update deploy/cr.yaml Co-authored-by: Ege Güneş <ege.gunes@percona.com> * update init image * update default mongo version * Add backup type into restore example for backupSource (#1170) * K8SPSMDB-769 update rs-shard-migration (#1172) * update rs-shard-migration * Update e2e-tests/rs-shard-migration/run Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * K8SPSMDB-873: Change HOME dir for mongosh (#1171) * K8SPSMDB-853: fix `version_test` unit-test (#1169) https://jira.percona.com/browse/K8SPSMDB-853 * Change HOME dir for mongosh. * K8SPSMDB-873 Force interpreter to incorporate variable in CLI call Bash tries to execute HOME=/tmp as a regular binary thus we need to trick it a bit. * Update build/ps-entry.sh Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: Andrii Dema <a.dema@jazzserve.com> Co-authored-by: Ivan Pylypenko <ivan.pylypenko@percona.com> Co-authored-by: Viacheslav Sarzhan <slava.sarzhan@percona.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix vault deployment for OpenShift (#1174) * fix vault deployment for openshift * Remove the unknown field * Adapt vault installation for Openshift --------- Co-authored-by: Ivan Pylypenko <ivan.pylypenko@percona.com> * Fix for version-service semver channel name (#1182) * Fix for version-service semver channel name * Increase timeout a bit for openshift * Update some obsolete code * Fixing compare files for openshift (#1183) * Fixing compare files for openshift * Add some verification checks * Enhance debug processing section (#1186) * Use separate namespace for getting mongod version (#1184) * Use separete namespace for getting mongod version * Enabling debug processing and test files * K8SPSMDB-794 - Fix diff in demand-backup-eks-credentials test (#1187) * Update e2e-tests/functions Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * K8SPSMDB-878 - Fix EKS tests (#1188) * K8SPSMDB-878 - Fix upgrade-consistency test to check cluster readyness * K8SPSMDB-794 - Fix runtimeclass for EKS 1.24 * K8SPSMDB-713: Retry PBM describe-restore on failure (#1189) * K8SPSMDB-853: fix `version_test` unit-test (#1169) https://jira.percona.com/browse/K8SPSMDB-853 * Retry pbm describe-restore if necessary. --------- Co-authored-by: Andrii Dema <a.dema@jazzserve.com> * K8SPSMDB-881 Force error exit on restore error (#1190) * Force error exit on restore error we should not wait any longer if the restore error happened * K8SPSMDB-881 Exit on backup error as well * fix cross-site-sharded (#1192) * K8SPSMDB-885 Always start from the lowest mongod version * K8SPSMDB-885 Make sure backup image is provided by version-service A dismal bug lead to a false conclusions regarding pbm functionality. It was caused by depending on static main-backup image instead the one from IMAGE_BACKUP. Also additional check for FCV inside mongo has been added * Update e2e-tests/mongod-major-upgrade-sharded/run Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update e2e-tests/mongod-major-upgrade/run Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update e2e-tests/mongod-major-upgrade-sharded/run Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update e2e-tests/mongod-major-upgrade/run Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * K8SPSMDB-885 Refactoring a bit Making sure only dynamic vs generation happen Enhance FCV value getting * K8SPSMDB-885 Add lines missing after resoving merge conflict * Fix mongod-major-upgrade test * Fix mongod-major-upgrade-sharded test --------- Co-authored-by: Natalia Marukovich <natalia.marukovich@percona.com> Co-authored-by: Natalia Marukovich <nmarukovich@gmail.com> Co-authored-by: Ege Güneş <ege.gunes@percona.com> Co-authored-by: Tomislav Plavcic <tomislav.plavcic@percona.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Inel Pandzic <inel.pandzic@percona.com> Co-authored-by: Andrii Dema <a.dema@jazzserve.com> Co-authored-by: Viacheslav Sarzhan <slava.sarzhan@percona.com> Co-authored-by: Tomislav Plavcic <tplavcic@gmail.com>
diff --git a/e2e-tests/functions b/e2e-tests/functions
@@ -1203,13 +1203,7 @@ check_crd_for_deletion() {
 function get_mongod_ver_from_image() {
 	local image=${1}
 
-	local pod_name=${RANDOM}
-	kubectl_bin -n default run ${pod_name} --image=${image} --restart=Never --command -- sleep infinity >/dev/null
-	kubectl_bin -n default wait --for=condition=Ready pod/${pod_name} >/dev/null
-
-	local version_info=$(kubectl_bin -n default exec ${pod_name} -- mongod --version)
-	kubectl_bin -n default delete pod/${pod_name} --grace-period=0 --force >/dev/null
-	version_info=$(echo ${version_info} | head -n2 | $sed -r 's/^.*db version v(([0-9]+\.){2}[0-9]+-[0-9]+).*$/\1/g')
+	version_info=$(run_simple_cli_inside_image ${image} 'mongod --version' | $sed -r 's/^.*db version v(([0-9]+\.){2}[0-9]+-[0-9]+).*$/\1/g')
 
 	if [[ ! ${version_info} =~ ^([0-9]+\.){2}[0-9]+-[0-9]+$ ]]; then
 		printf "No mongod version obtained from %s. Exiting" ${image}
@@ -1218,6 +1212,52 @@ function get_mongod_ver_from_image() {
 	echo ${version_info}
 }
 
+function get_pbm_version() {
+	local image=${1}
+
+	local version_info=$(run_simple_cli_inside_image ${image} 'pbm-agent version' | $sed -r 's/^Version:\ (([0-9]+\.){2}[0-9]+)\ .*/\1/g')
+
+	if [[ ! ${version_info} =~ ^([0-9]+\.){2}[0-9]+$ ]]; then
+		printf "No pbm version obtained from %s. Exiting" ${image}
+		exit 1
+	fi
+	echo ${version_info}
+}
+
+function run_simple_cli_inside_image() {
+	local image=${1}
+	local cli=${2}
+
+	local pod_name=${RANDOM}
+	kubectl_bin -n default run ${pod_name} --image=${image} --restart=Never --command -- sleep infinity >/dev/null
+	kubectl_bin -n default wait --for=condition=Ready pod/${pod_name} >/dev/null
+	local output=$(kubectl_bin -n default exec ${pod_name} -- ${cli})
+	kubectl_bin -n default delete pod/${pod_name} --grace-period=0 --force >/dev/null
+	echo ${output}
+}
+
+function generate_vs_json() {
+	local template_path=${1}
+	local target_path=${2}
+
+	local version_service_source=$(jq '.versions[0].operator="'${OPERATOR_VERSION}'"' ${template_path})
+
+	for image_mongod in ${IMAGE_MONGOD_CHAIN[@]}; do
+		current_mongod_version=$(get_mongod_ver_from_image ${image_mongod})
+
+		version_service_source=$(echo ${version_service_source} \
+			| jq '.versions[0].matrix.mongod += {"'${current_mongod_version}'": {"image_path":"'${image_mongod}'","status":"recommended"}}')
+	done
+
+	version_service_source=$(echo ${version_service_source} \
+		| jq '.versions[0].matrix.backup += {"'$(get_pbm_version ${IMAGE_BACKUP})'": {"image_path":"'${IMAGE_BACKUP}'","status":"recommended"}}')
+
+	version_service_source=$(echo ${version_service_source} \
+		| jq '.versions[0].matrix.operator += {"'${OPERATOR_VERSION}'": {"image_path":"'${IMAGE}'","status":"recommended"}}')
+
+	echo ${version_service_source} | jq '.' >${target_path}
+}
+
 check_passwords_leak() {
 	secrets=$(kubectl_bin get secrets -o json | jq -r '.items[].data | to_entries | .[] | select(.key | (contains("_PASSWORD"))) | .value')
 	echo secrets=$secrets
diff --git a/e2e-tests/mongod-major-upgrade-sharded/compare/find-2.json b/e2e-tests/mongod-major-upgrade-sharded/compare/find-2.json
@@ -1,4 +1,4 @@
 switched to db myApp
-{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
 { "_id" : , "x" : 100502 }
 bye
diff --git a/e2e-tests/mongod-major-upgrade-sharded/compare/find-3.json b/e2e-tests/mongod-major-upgrade-sharded/compare/find-3.json
@@ -1,4 +1,5 @@
 switched to db myApp
-{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
+{ "_id" : , "x" : 100502 }
 { "_id" : , "x" : 100503 }
 bye
diff --git a/e2e-tests/mongod-major-upgrade-sharded/compare/find.json b/e2e-tests/mongod-major-upgrade-sharded/compare/find.json
@@ -1,3 +1,3 @@
 switched to db myApp
-{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
 bye
diff --git a/e2e-tests/mongod-major-upgrade-sharded/conf/backup-minio.yml b/e2e-tests/mongod-major-upgrade-sharded/conf/backup-minio.yml
diff --git a/e2e-tests/mongod-major-upgrade-sharded/conf/operator.main.psmdb-operator.json b/e2e-tests/mongod-major-upgrade-sharded/conf/operator.main.psmdb-operator.json
@@ -15,20 +15,8 @@
           }
         },
         "backup": {
-          "2.0.3": {
-            "image_path": "perconalab/percona-server-mongodb-operator:main-backup",
-            "image_hash": "01388bcc310de0190cad0c4cfbe8cf8edc07c5c7ea057edec137e8f58931696b",
-            "status": "recommended",
-            "critical": false
-          }
         },
         "operator": {
-          "1.14.0": {
-            "image_path": "perconalab/percona-server-mongodb-operator:main",
-            "image_hash": "e9ed11994cef3f7ab33e126484d5d5991cccc00b54d066183d1c7abe8e29b802",
-            "status": "recommended",
-            "critical": false
-          }
         }
       }
     }
diff --git a/e2e-tests/mongod-major-upgrade-sharded/conf/restore.yml b/e2e-tests/mongod-major-upgrade-sharded/conf/restore.yml
diff --git a/e2e-tests/mongod-major-upgrade-sharded/run b/e2e-tests/mongod-major-upgrade-sharded/run
@@ -7,21 +7,6 @@ test_dir=$(realpath $(dirname $0))
 . ${test_dir}/../functions
 set_debug
 
-function generate_vs_json() {
-	local template_path=${1}
-	local target_path=${2}
-
-	local version_service_source=$(jq '.versions[0].operator="'${OPERATOR_VERSION}'"' ${template_path})
-
-	for image_mongod in ${IMAGE_MONGOD_CHAIN[@]}; do
-		current_mongod_version=$(get_mongod_ver_from_image ${image_mongod})
-
-		version_service_source=$(echo ${version_service_source} \
-			| jq '.versions[0].matrix.mongod += {"'${current_mongod_version}'": {"image_path":"'${image_mongod}'","status":"recommended"}}')
-	done
-	echo ${version_service_source} | jq '.' >${target_path}
-}
-
 function main() {
 	rbac="rbac"
 	if [ -n "$OPERATOR_NS" ]; then
@@ -49,27 +34,16 @@ function main() {
 	version_matrix=$(jq '.[] | .[] |.matrix.mongod' ${tmp_dir}/operator.${OPERATOR_VERSION}.psmdb-operator.json)
 
 	current_mongod_version=$(get_mongod_ver_from_image ${IMAGE_MONGOD})
-	version_matrix=$(echo ${version_matrix} | jq '. += {"'${current_mongod_version}'":{"image_path":"'${IMAGE_MONGOD}'"}}')
+	version_matrix=$(echo ${version_matrix} | jq '. += {"'${current_mongod_version}'":{"image_path":"'${IMAGE_MONGOD}'","status": "recommended"}}')
 
 	version_matrix=$(echo ${version_matrix} | jq 'to_entries
 												 | sort_by( .key | split("[[:punct:]]";"g") | map(tonumber) )
 												 | map({(.key): .value}) ')
 
-	current_version_index=$(echo ${version_matrix} | jq 'map(keys[] == "'${current_mongod_version}'") | index(true)')
-
-	if [[ $((current_version_index + 1)) == $(echo ${version_matrix} | jq length) ]]; then
-		desc 'IMAGE_MONGOD '${IMAGE_MONGOD}' has '${current_mongod_version}' is the last in upgrade chain. Exiting...'
-		destroy ${namespace}
-		exit 0
-	fi
-
-	version_matrix=$(echo ${version_matrix} | jq '.['$((current_version_index + 1))':]')
-
-	versions_to_verify=$(echo ${version_matrix} | jq -r '.[] | keys | .[] | split(".") | .[:2] | join(".")' | uniq | grep -v ${current_mongod_version:0:3})
-
+	versions_to_verify=$(echo ${version_matrix} | jq -r '.[] | keys | .[] | split(".") | .[:2] | join(".")' | uniq | tail -n +2)
 	cluster="some-name"
-	desc 'Starting the cluster with IMAGE_MONGOD '${IMAGE_MONGOD}''
-
+	desc 'Starting the cluster with IMAGE_MONGOD '$(echo ${version_matrix} | jq '.[0] | keys | .[0]')''
+	export IMAGE_MONGOD=$(echo ${version_matrix} | jq -r 'to_entries | .[0].value | .[].image_path')
 	apply_cluster "${test_dir}/conf/${cluster}.yml"
 
 	desc 'check if all 3 Pods started'
@@ -84,35 +58,51 @@ function main() {
 		"userAdmin:userAdmin123456@${cluster}-mongos.${namespace}"
 	sleep 2
 	run_mongos \
-		'use myApp\n db.test.insert({ x: 100500 })' \
+		'use myApp\n db.test.insert({ x: 100501 })' \
 		"myApp:myPass@${cluster}-mongos.${namespace}"
 	compare_mongos_cmd "find" "myApp:myPass@${cluster}-mongos.${namespace}"
 
 	desc 'Starting to follow mongod upgrade images chain'
-
 	target_generation=2
 	for version in ${versions_to_verify[@]}; do
+		desc "Testing upgrade to version: ${version}"
 		kubectl_bin patch psmdb/${cluster%%-rs0} \
 			--type=json \
 			-p='[
 					{"op":"replace","path":"/spec/upgradeOptions/apply","value": "'${version}'-recommended"},
-					{"op":"replace","path":"/spec/upgradeOptions/setFCV","value": true}
 				]'
 
-		sleep 60 # waiting for update loop to fire
+		sleep 70 # waiting for update loop to fire
 		wait_for_running "${cluster}-cfg" 3
 		wait_for_running "${cluster}-rs0" 3
 		wait_for_running "${cluster}-mongos" 3
 		wait_cluster_consistency "${cluster}"
 
 		compare_generation ${target_generation} "statefulset" "${cluster}-cfg"
 		compare_generation ${target_generation} "statefulset" "${cluster}-rs0"
-		compare_generation 2 "statefulset" "${cluster}-mongos" # generation is 2 because mongos gets deleted on restore
+		compare_generation ${target_generation} "statefulset" "${cluster}-mongos"
+
+		kubectl_bin patch psmdb/${cluster%%-rs0} \
+			--type=json \
+			-p='[
+					{"op":"replace","path":"/spec/upgradeOptions/setFCV","value": true}
+				]'
+		sleep 10
+		wait_cluster_consistency "${cluster}"
+
+		currentFCV=$(run_mongos 'JSON.stringify(db.adminCommand({getParameter:1,featureCompatibilityVersion:1}))' "clusterAdmin:clusterAdmin123456@${cluster}-cfg.${namespace}" \
+			| grep -E '^\{.*\}$' | jq -r '.featureCompatibilityVersion.version')
+
+		if [[ ${currentFCV} != ${version} ]]; then
+			echo "FCV at the moment is ${currentFCV} and is not set to ${version} as it should. Exiting..."
+			exit 1
+		fi
 
 		run_mongos \
 			'use myApp\n db.test.insert({ x: 10050'${target_generation}' })' \
 			"myApp:myPass@${cluster}-mongos.${namespace}"
 		compare_mongos_cmd "find" "myApp:myPass@${cluster}-mongos.${namespace}" "-${target_generation}"
+
 		target_generation=$((target_generation + 1))
 	done
 
diff --git a/e2e-tests/mongod-major-upgrade/compare/find-2.json b/e2e-tests/mongod-major-upgrade/compare/find-2.json
@@ -1,4 +1,4 @@
 switched to db myApp
-{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
 { "_id" : , "x" : 100502 }
 bye
diff --git a/e2e-tests/mongod-major-upgrade/compare/find-3.json b/e2e-tests/mongod-major-upgrade/compare/find-3.json
@@ -1,4 +1,5 @@
 switched to db myApp
-{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
+{ "_id" : , "x" : 100502 }
 { "_id" : , "x" : 100503 }
 bye
diff --git a/e2e-tests/mongod-major-upgrade/compare/find.json b/e2e-tests/mongod-major-upgrade/compare/find.json
@@ -1,3 +1,3 @@
 switched to db myApp
-{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
 bye
diff --git a/e2e-tests/mongod-major-upgrade/conf/backup-minio.yml b/e2e-tests/mongod-major-upgrade/conf/backup-minio.yml
diff --git a/e2e-tests/mongod-major-upgrade/conf/operator.main.psmdb-operator.json b/e2e-tests/mongod-major-upgrade/conf/operator.main.psmdb-operator.json
@@ -15,20 +15,8 @@
           }
         },
         "backup": {
-          "2.0.3": {
-            "image_path": "perconalab/percona-server-mongodb-operator:main-backup",
-            "image_hash": "01388bcc310de0190cad0c4cfbe8cf8edc07c5c7ea057edec137e8f58931696b",
-            "status": "recommended",
-            "critical": false
-          }
         },
         "operator": {
-          "1.14.0": {
-            "image_path": "perconalab/percona-server-mongodb-operator:main",
-            "image_hash": "e9ed11994cef3f7ab33e126484d5d5991cccc00b54d066183d1c7abe8e29b802",
-            "status": "recommended",
-            "critical": false
-          }
         }
       }
     }
diff --git a/e2e-tests/mongod-major-upgrade/conf/restore.yml b/e2e-tests/mongod-major-upgrade/conf/restore.yml
diff --git a/e2e-tests/mongod-major-upgrade/run b/e2e-tests/mongod-major-upgrade/run

Original file line number	Diff line number	Diff line change
`@@ -15,20 +15,8 @@`
`15`	`15`	`}`
`16`	`16`	`},`
`17`	`17`	`"backup": {`
`18`		`- "2.0.3": {`
`19`		`- "image_path": "perconalab/percona-server-mongodb-operator:main-backup",`
`20`		`- "image_hash": "01388bcc310de0190cad0c4cfbe8cf8edc07c5c7ea057edec137e8f58931696b",`
`21`		`- "status": "recommended",`
`22`		`- "critical": false`
`23`		`- }`
`24`	`18`	`},`
`25`	`19`	`"operator": {`
`26`		`- "1.14.0": {`
`27`		`- "image_path": "perconalab/percona-server-mongodb-operator:main",`
`28`		`- "image_hash": "e9ed11994cef3f7ab33e126484d5d5991cccc00b54d066183d1c7abe8e29b802",`
`29`		`- "status": "recommended",`
`30`		`- "critical": false`
`31`		`- }`
`32`	`20`	`}`
`33`	`21`	`}`
`34`	`22`	`}`