Skip to content

Commit 8e5b30c

Browse files
lsierantlsierant
authored
Improvements to shell scripts (#48)
# Summary This PR contains various UX, code and performance improvements to our most commonly used shell scripts. * added printing with a prefix (pipe to `prepend`) to see from where the output comes (especially useful when running commands in parallel, e.g. creating kind clusters) * prefix printing wasn't added to every script: mostly to the most commonly used: pre-commit, prepare_local_e2e_run, evg_host.sh, recreate_kind_cluster.sh * introduced shell debug pattern `test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x` allowing to enable verbose logging on demand * `recreate_kind_clusters.sh`: * improved docker cleanup * extracted registry and docker related operations from setup_kind_cluster.sh to allow for parallel execution * running recreate of all 5 kind clusters in parallel * `setup_kind_cluster.sh`: * installing metrics server by default * `evg_host.sh` * improved setting fs limits * added retry with sleep to `tunnel` command to automatically retry ## Proof of Work 1. Parallel execution is properly prefixed now: ``` kind: Creating cluster "kind" ... kind: • Ensuring node image (docker.io/kindest/node:v1.32.2) 🖼 ... e2e-cluster-2: DEBUG: docker/images.go:67] Pulling image: docker.io/kindest/node:v1.32.2@sha256:f226345927d7e348497136874b6d207e0b32cc52154ad8323129352923a3142f ... e2e-cluster-1: DEBUG: docker/images.go:67] Pulling image: docker.io/kindest/node:v1.32.2@sha256:f226345927d7e348497136874b6d207e0b32cc52154ad8323129352923a3142f ... e2e-cluster-3: DEBUG: docker/images.go:67] Pulling image: docker.io/kindest/node:v1.32.2@sha256:f226345927d7e348497136874b6d207e0b32cc52154ad8323129352923a3142f ... e2e-operator: DEBUG: docker/images.go:67] Pulling image: docker.io/kindest/node:v1.32.2@sha256:f226345927d7e348497136874b6d207e0b32cc52154ad8323129352923a3142f ... kind: DEBUG: docker/images.go:67] Pulling image: docker.io/kindest/node:v1.32.2@sha256:f226345927d7e348497136874b6d207e0b32cc52154ad8323129352923a3142f ... e2e-cluster-2: ✓ Ensuring node image (docker.io/kindest/node:v1.32.2) 🖼 e2e-cluster-2: • Preparing nodes 📦 ... e2e-cluster-3: ✓ Ensuring node image (docker.io/kindest/node:v1.32.2) 🖼 e2e-cluster-3: • Preparing nodes 📦 ... e2e-operator: ✓ Ensuring node image (docker.io/kindest/node:v1.32.2) 🖼 e2e-operator: • Preparing nodes 📦 ... ``` 2. [Green EVG](https://spruce.mongodb.com/version/680e93d22006d40007d38921/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC) 3. 30% speedup in creating kind clusters in EVG: <img width="871" alt="image" src="https://github.com/user-attachments/assets/87d9f432-1dac-4576-a3c5-4d560b890ee6" /> vs <img width="900" alt="image" src="https://github.com/user-attachments/assets/049328b9-87c0-497f-a557-9cfdd962c507" /> ## Checklist - [ ] Have you linked a jira ticket and/or is the ticket in the title? - [ ] Have you checked whether your jira ticket required DOCSP changes? - [ ] Have you checked for release_note changes? ## Reminder (Please remove this when merging) - Please try to Approve or Reject Changes the PR, keep PRs in review as short as possible - Our Short Guide for PRs: [Link](https://docs.google.com/document/d/1T93KUtdvONq43vfTfUt8l92uo4e4SEEvFbIEKOxGr44/edit?tab=t.0) - Remember the following Communication Standards - use comment prefixes for clarity: * **blocking**: Must be addressed before approval. * **follow-up**: Can be addressed in a later PR or ticket. * **q**: Clarifying question. * **nit**: Non-blocking suggestions. * **note**: Side-note, non-actionable. Example: Praise * --> no prefix is considered a question
1 parent 13431e8 commit 8e5b30c

14 files changed

+395
-293
lines changed

‎.evergreen-functions.yml

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -160,13 +160,6 @@ functions:
160160
- ${workdir}/bin
161161
binary: scripts/evergreen/setup_kind.sh
162162

163-
setup_docker_datadir: &setup_docker_datadir
164-
command: subprocess.exec
165-
type: setup
166-
params:
167-
working_dir: src/github.com/mongodb/mongodb-kubernetes
168-
binary: scripts/evergreen/configure-docker-datadir.sh
169-
170163
setup_preflight:
171164
- command: subprocess.exec
172165
type: setup
@@ -227,6 +220,15 @@ functions:
227220
- ${workdir}/bin
228221
binary: scripts/dev/configure_docker_auth.sh
229222

223+
setup_evg_host: &setup_evg_host
224+
command: subprocess.exec
225+
type: setup
226+
params:
227+
working_dir: src/github.com/mongodb/mongodb-kubernetes
228+
add_to_path:
229+
- ${workdir}/bin
230+
binary: scripts/dev/setup_evg_host.sh
231+
230232
lint_repo:
231233
- command: subprocess.exec
232234
type: setup
@@ -249,7 +251,7 @@ functions:
249251
- *switch_context
250252
- *setup_aws
251253
- *configure_docker_auth
252-
- *setup_docker_datadir
254+
- *setup_evg_host
253255
- *python_venv
254256

255257
prune_docker_resources:

‎.githooks/pre-commit

Lines changed: 43 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
set -Eeou pipefail
44

55
source scripts/dev/set_env_context.sh
6+
source scripts/funcs/printing
7+
68
if [ -f "${PROJECT_DIR}"/venv/bin/activate ]; then
79
source "${PROJECT_DIR}"/venv/bin/activate
810
fi
@@ -107,55 +109,64 @@ function update_release_json() {
107109
git add release.json
108110
}
109111

110-
function pre_commit() {
111-
# Update release.json first in case there is a newer version
112-
update_release_json
113-
# Generate operator manifests (CRDs, etc)
114-
generate_manifests
115-
# We need to generate the values files first
116-
update_values_yaml_files
117-
# The values files are used for generating the standalone yaml
118-
generate_standalone_yaml
119-
# Run black on python files that have changed
120-
python_formatting
121-
# Generate MCO evergreen tests
122-
update_mco_tests
123-
124-
source scripts/evergreen/lint_code.sh
112+
function regenerate_public_rbac_multi_cluster() {
113+
if echo "$git_last_changed" | grep -q 'public/tools/multicluster'; then
114+
echo 'regenerating multicluster RBAC public example'
115+
pushd public/tools/multicluster
116+
EXPORT_RBAC_SAMPLES="true" go test -run TestPrintingOutRolesServiceAccountsAndRoleBindings
117+
popd
118+
git add public/samples/multi-cluster-cli-gitops
119+
fi
120+
}
125121

126-
echo 'regenerating licenses.csv'
127-
scripts/evergreen/update_licenses.sh
128-
git add LICENSE-THIRD-PARTY
122+
function update_licenses() {
123+
echo 'regenerating licenses'
124+
time scripts/evergreen/update_licenses.sh 2>&1 | prepend "update_licenses"
125+
git add LICENSE-THIRD-PARTY
126+
}
129127

130-
if echo "$git_last_changed" | grep -q 'public/tools/multicluster'; then
131-
echo 'regenerating multicluster RBAC public example'
132-
pushd public/tools/multicluster
133-
EXPORT_RBAC_SAMPLES="true" go test -run TestPrintingOutRolesServiceAccountsAndRoleBindings
134-
popd
135-
git add public/samples/multi-cluster-cli-gitops
128+
function check_erroneous_kubebuilder_annotations() {
129+
# Makes sure there are not erroneous kubebuilder annotations that can
130+
# end up in CRDs as descriptions.
131+
if grep "// kubebuilder" ./* -r --exclude-dir=vendor --include=\*.go; then
132+
echo "Found an erroneous kubebuilder annotation"
133+
exit 1
136134
fi
135+
}
137136

137+
function check_incorrect_makefile_variable_brackets() {
138138
if find . -name "Makefile" | grep -v vendor | xargs grep "\${"; then
139139
echo 'ERROR: Makefiles should NEVER contain curly brackets variables'
140140
exit 1
141141
fi
142+
}
142143

143-
# Makes sure there are not erroneous kubebuilder annotations that can
144-
# end up in CRDs as descriptions.
145-
if grep "// kubebuilder" ./* -r --exclude-dir=vendor --include=\*.go; then
146-
echo "Found an erroneous kubebuilder annotation"
147-
exit 1
148-
fi
144+
function pre_commit() {
145+
# Update release.json first in case there is a newer version
146+
(time update_release_json) 2>&1 | prepend "update_release_json"
147+
# We need to generate the values files first
148+
(time update_values_yaml_files) 2>&1 | prepend "update_values_yaml_files"
149+
# The values files are used for generating the standalone yaml
150+
(time generate_standalone_yaml) 2>&1 | prepend "generate_standalone_yaml"
151+
# Run black on python files that have changed
152+
(time python_formatting) 2>&1 | prepend "python_formatting"
149153

150-
start_shellcheck
154+
(time regenerate_public_rbac_multi_cluster) 2>&1 | prepend "regenerate_public_rbac_multi_cluster"
155+
156+
(time start_shellcheck) 2>&1 | prepend "shellcheck"
157+
158+
(time check_erroneous_kubebuilder_annotations) 2>&1 | prepend "check_erroneous_kubebuilder_annotations"
159+
160+
(time scripts/evergreen/lint_code.sh) 2>&1 | prepend "lint_code.sh"
151161

162+
(time update_licenses) 2>&1 | prepend "update_licenses"
152163
}
153164

154165
# Function to run shellcheck on a single file
155166
run_shellcheck() {
156167
local file="$1"
157168
echo "Running shellcheck on $file"
158-
if ! shellcheck -x "$file" -e SC2154 -e SC1091 -e SC1090 -e SC2148 -o require-variable-braces -P "scripts"; then
169+
if ! shellcheck --color=always -x "$file" -e SC2154 -e SC1091 -e SC1090 -e SC2148 -o require-variable-braces -P "scripts"; then
159170
echo "shellcheck failed on $file"
160171
exit 1
161172
fi

‎scripts/dev/delete_om_projects.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77

88
set -euo pipefail
99

10+
test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
11+
1012
source scripts/dev/set_env_context.sh
1113

1214
delete_project() {

‎scripts/dev/evg_host.sh

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,11 @@
88

99
set -Eeou pipefail
1010

11+
test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
12+
1113
source scripts/dev/set_env_context.sh
14+
source scripts/funcs/printing
15+
1216

1317
if [[ -z "${EVG_HOST_NAME}" ]]; then
1418
echo "EVG_HOST_NAME env var is missing"
@@ -96,16 +100,18 @@ get-kubeconfig() {
96100

97101
recreate-kind-clusters() {
98102
DELETE_KIND_NETWORK=${DELETE_KIND_NETWORK:-"false"}
103+
configure "${1-"amd64"}" 2>&1| prepend "evg_host.sh configure"
99104
echo "Recreating kind clusters on ${EVG_HOST_NAME} (${host_url})..."
100105
# shellcheck disable=SC2088
101106
ssh -T "${host_url}" "cd ~/mongodb-kubernetes; DELETE_KIND_NETWORK=${DELETE_KIND_NETWORK} scripts/dev/recreate_kind_clusters.sh"
102107
echo "Copying kubeconfig to ${kubeconfig_path}"
103-
get-kubeconfig
108+
get-kubeconfig 2>&1| prepend "evg_host.sh configure"
104109
}
105110

106111
recreate-kind-cluster() {
107112
shift 1
108113
cluster_name=$1
114+
configure "${1-"amd64"}" 2>&1| prepend "evg_host.sh configure"
109115
echo "Recreating kind cluster ${cluster_name} on ${EVG_HOST_NAME} (${host_url})..."
110116
# shellcheck disable=SC2088
111117
ssh -T "${host_url}" "cd ~/mongodb-kubernetes; scripts/dev/recreate_kind_cluster.sh ${cluster_name}"
@@ -134,6 +140,19 @@ tunnel() {
134140
set +x
135141
}
136142

143+
retry_with_sleep() {
144+
shift 1
145+
cmd=$1
146+
local sleep_time
147+
sleep_time=5
148+
149+
while true; do
150+
${cmd} || true
151+
echo "Retrying command after ${sleep_time} of sleep: ${cmd}"
152+
sleep 5;
153+
done
154+
}
155+
137156
ssh_to_host() {
138157
shift 1
139158
# shellcheck disable=SC2029
@@ -167,9 +186,9 @@ PREREQUISITES:
167186
- VPN connection
168187
169188
COMMANDS:
189+
recreate-kind-clusters all-you-need to configure host and kind clusters; deletes and recreates all kind clusters (for single and multi runs)
170190
configure <architecture> installs on a host: calls sync, switches context, installs necessary software
171191
sync rsync of project directory
172-
recreate-kind-clusters executes scripts/dev/recreate_kind_clusters.sh and executes get-kubeconfig
173192
recreate-kind-cluster test-cluster executes scripts/dev/recreate_kind_cluster.sh test-cluster and executes get-kubeconfig
174193
remote-prepare-local-e2e-run executes prepare-local-e2e on the remote evg host
175194
get-kubeconfig copies remote kubeconfig locally to ~/.operator-dev/evg-host.kubeconfig
@@ -183,12 +202,12 @@ COMMANDS:
183202

184203
case ${cmd} in
185204
configure) configure "$@" ;;
186-
recreate-kind-clusters) recreate-kind-clusters ;;
205+
recreate-kind-clusters) recreate-kind-clusters "${1-"amd64"}";;
187206
recreate-kind-cluster) recreate-kind-cluster "$@" ;;
188207
get-kubeconfig) get-kubeconfig ;;
189208
remote-prepare-local-e2e-run) remote-prepare-local-e2e-run ;;
190209
ssh) ssh_to_host "$@" ;;
191-
tunnel) tunnel "$@" ;;
210+
tunnel) retry_with_sleep tunnel "$@" ;;
192211
sync) sync ;;
193212
cmd) cmd "$@" ;;
194213
upload-my-ssh-private-key) upload-my-ssh-private-key ;;

‎scripts/dev/install_csi_driver.sh

Lines changed: 21 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,17 @@
22

33
set -eux
44

5+
source scripts/funcs/kubernetes
6+
57
# Path to the deploy script
68
DEPLOY_SCRIPT_PATH="./deploy/kubernetes-latest/deploy.sh"
9+
EXTRACTED_DIR="csi-driver-host-path-1.14.1"
710

8-
# Function to deploy to a single cluster
9-
deploy_to_cluster() {
10-
local context="$1"
11-
echo "Switching to context: ${context}"
12-
13-
# Set the current context to the target cluster
14-
if ! kubectl config use-context "${context}";then
15-
echo "Failed to switch to context: ${context}"
16-
fi
17-
11+
csi_driver_download() {
1812
echo "install resizable csi"
1913
# Define variables
2014
REPO_URL="https://github.com/kubernetes-csi/csi-driver-host-path/archive/refs/tags/v1.14.1.tar.gz"
2115
TAR_FILE="csi-driver-host-path-v1.14.1.tar.gz"
22-
EXTRACTED_DIR="csi-driver-host-path-1.14.1"
2316

2417
# Download the tar.gz file
2518
echo "Downloading ${REPO_URL}..."
@@ -28,56 +21,37 @@ deploy_to_cluster() {
2821
# Extract the tar.gz file
2922
echo "Extracting ${TAR_FILE}..."
3023
tar -xzf "${TAR_FILE}"
24+
}
25+
26+
# Function to deploy to a single cluster
27+
csi_driver_deploy() {
28+
local context="$1"
3129

3230
# Navigate to the extracted directory
3331
cd "${EXTRACTED_DIR}"
34-
3532
# Change to the latest supported snapshotter release branch
3633
SNAPSHOTTER_BRANCH=release-6.3
37-
34+
3835
# Apply VolumeSnapshot CRDs
39-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_BRANCH}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
40-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_BRANCH}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
41-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_BRANCH}/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
42-
36+
kubectl apply --context "${context}" -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_BRANCH}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
37+
kubectl apply --context "${context}" -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_BRANCH}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
38+
kubectl apply --context "${context}" -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_BRANCH}/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
39+
4340
# Change to the latest supported snapshotter version
4441
SNAPSHOTTER_VERSION=v6.3.3
45-
42+
4643
# Create snapshot controller
47-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
48-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
44+
kubectl apply --context "${context}" -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
45+
kubectl apply --context "${context}" -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
4946

50-
# Run the deploy script
51-
echo "Running deploy script on ${context}"
52-
if ! bash "${DEPLOY_SCRIPT_PATH}"; then
53-
echo "Failed to run deploy script on ${context}"
54-
return 1
47+
# Run the deploy script with kubectl wrapped to force it to use specific context rather than rely on current context
48+
if ! run_script_with_wrapped_kubectl "${DEPLOY_SCRIPT_PATH}" "${context}"; then
49+
return 1
5550
fi
5651

5752
echo "Installing csi storageClass"
58-
kubectl apply -f ./examples/csi-storageclass.yaml
53+
kubectl apply --context "${context}" -f ./examples/csi-storageclass.yaml
5954

6055
echo "Deployment successful on ${context}"
6156
return 0
6257
}
63-
64-
CLUSTER_CONTEXTS=()
65-
66-
# Set default values for the context variables if they are not set
67-
CTX_CLUSTER="${CTX_CLUSTER:-}"
68-
CTX_CLUSTER1="${CTX_CLUSTER1:-}"
69-
CTX_CLUSTER2="${CTX_CLUSTER2:-}"
70-
CTX_CLUSTER3="${CTX_CLUSTER3:-}"
71-
72-
# Add to CLUSTER_CONTEXTS only if the environment variable is set and not empty
73-
[[ -n "${CTX_CLUSTER}" ]] && CLUSTER_CONTEXTS+=("${CTX_CLUSTER}")
74-
[[ -n "${CTX_CLUSTER1}" ]] && CLUSTER_CONTEXTS+=("${CTX_CLUSTER1}")
75-
[[ -n "${CTX_CLUSTER2}" ]] && CLUSTER_CONTEXTS+=("${CTX_CLUSTER2}")
76-
[[ -n "${CTX_CLUSTER3}" ]] && CLUSTER_CONTEXTS+=("${CTX_CLUSTER3}")
77-
78-
# Main deployment loop
79-
for context in "${CLUSTER_CONTEXTS[@]}"; do
80-
deploy_to_cluster "${context}"
81-
done
82-
83-
echo "Deployment completed for all clusters."

0 commit comments

Comments
 (0)