Skip to content
This repository was archived by the owner on Oct 8, 2025. It is now read-only.

Commit c33ed4e

Browse files
author
Jason Schmidt
authored
fix: stability and bug fixes (#138)
* fix: remove erroneous dash in shebang * fix: missing ''' in Jenkinsfiles * fix: updated node label * fix: documentation updates * test: do change * fix: update to createpw() function to stop Jenkins hang * fix: update Jenkinsfiles * test: add tracing to kube start * test: more tracing * test: update the createpw() function * test: move the createpw logic * test: move tr to sed * test: still futzing with password generation * test: zeroing in on PW fix * fix: revert tests and apply fix for createpw() * fix: issues with eks and 1.22, revert to 1.21 for eks * chore: update nginx ic version and chart version * fix: typo in configuration * feat: update start script with new additions * fix: add pulumi args to calls to pulumi to get stack * fix: downgrade pulumi-docker to address ecr issue * chore: remove post-run command block from jenkinsfiles * chore: remove version info for ic from aws jenkinsfile * chore: bump nginx ic version for repo deploy * fix: version 5.2.0 moves to new api ver for auth * fix: trying to find a working aws/eks pulumi combo * fix: trying to find a working aws/eks pulumi combo * fix: specify and lock all python modules * fix: add stack identifier to call to application * feat: add in information on versions in use * fix: cleanup random temp file * fix: move version info to AWS script for debugging * test: working on kubectl version issue * feat: add version information to start * fix: address issue with curl command for kubectl * test: trying new K8 version with older kubectl * fix: cleaning up requirements.txt file * fix: revert to 1.21 * feat: add call to pull K8 server information * fix: clean up kubectl update section * fix: adjust to work around #134
1 parent d00fd8b commit c33ed4e

File tree

20 files changed

+194
-146
lines changed

20 files changed

+194
-146
lines changed

‎bin/setup_venv.sh‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -235,16 +235,34 @@ else
235235
exit 1
236236
fi
237237

238-
# Add local kubectl to the virtual environment
238+
#
239+
# This section originally pulled the most recent version of Kubectl down; however it turned out that
240+
# was causing isues with our AWS deploy (see the issues in the repo). Addtionally, this was only
241+
# downloading the kubectl if it did not exist; this could result in versions not being updated if the
242+
# MARA project was run in the same environment w/o a refresh.
243+
#
244+
# The two fixes here are to hardcode (For now) to a known good version (1.23.6) and force the script to
245+
# always download this version.
246+
#
247+
# TODO: Figure out a way to not hardocde the kubectl version
248+
# TODO: Should not always download if the versions match; need a version check
249+
#
250+
#
239251
if [ ! -x "${VIRTUAL_ENV}/bin/kubectl" ]; then
240252
echo "Downloading kubectl into virtual environment"
241-
KUBECTL_VERSION="$(${download_cmd} https://dl.k8s.io/release/stable.txt)"
253+
KUBECTL_VERSION="v1.23.6"
242254
${download_cmd} "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/${OS}/${ARCH}/kubectl" >"${VIRTUAL_ENV}/bin/kubectl"
243255
KUBECTL_CHECKSUM="$(${download_cmd} "https://dl.k8s.io/${KUBECTL_VERSION}/bin/${OS}/${ARCH}/kubectl.sha256")"
244256
echo "${KUBECTL_CHECKSUM} ${VIRTUAL_ENV}/bin/kubectl" | ${sha256sum_cmd}
245257
chmod +x "${VIRTUAL_ENV}/bin/kubectl"
246258
else
247-
echo "kubectl is already installed"
259+
echo "kubectl is already installed, but will overwrite to ensure correct version"
260+
echo "Downloading kubectl into virtual environment"
261+
KUBECTL_VERSION="v1.23.6"
262+
${download_cmd} "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/${OS}/${ARCH}/kubectl" >"${VIRTUAL_ENV}/bin/kubectl"
263+
KUBECTL_CHECKSUM="$(${download_cmd} "https://dl.k8s.io/${KUBECTL_VERSION}/bin/${OS}/${ARCH}/kubectl.sha256")"
264+
echo "${KUBECTL_CHECKSUM} ${VIRTUAL_ENV}/bin/kubectl" | ${sha256sum_cmd}
265+
chmod +x "${VIRTUAL_ENV}/bin/kubectl"
248266
fi
249267

250268
#

‎bin/start.sh‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
1717
# Check to see if the venv has been installed, since this is only going to be used to start pulumi/python based
1818
# projects.
1919
#
20-
if ! command -v "${script_dir}/../pulumi/python/venv/bin/python" > /dev/null ; then
20+
if ! command -v "${script_dir}/../pulumi/python/venv/bin/python" >/dev/null; then
2121
echo "NOTICE! Unable to find the venv directory. This is required for the pulumi/python deployment process."
2222
echo "Please run ./setup_venv.sh from this directory to install the required virtual environment."
2323
echo " "
@@ -31,7 +31,6 @@ if ! command -v pulumi >/dev/null; then
3131
if [ -x "${script_dir}/../pulumi/python/venv/bin/pulumi" ]; then
3232
echo "Adding to [${script_dir}/venv/bin] to PATH"
3333
export PATH="${script_dir}/../pulumi/python/venv/bin:$PATH"
34-
3534
if ! command -v pulumi >/dev/null; then
3635
echo >&2 "Pulumi must be installed to continue"
3736
exit 1
@@ -60,7 +59,7 @@ fi
6059
echo " "
6160
echo "NOTICE! This shell script will call the appropriate helper script depending on your answer to the next question."
6261
echo " "
63-
echo "This script currently supports standing up an AWS environment (including ECR, EKS, and VPC resources), provided "
62+
echo "This script currently supports standing up AWS, Linode, and Digital Ocean kubernetes deployments, provided "
6463
echo "the correct credentials are supplied. It also supports the user of a kubeconfig file with a defined cluster name"
6564
echo "and context, which must be provided by the user."
6665
echo " "
@@ -76,7 +75,7 @@ if [ -s "${script_dir}/../config/pulumi/environment" ] && grep --quiet '^PULUMI_
7675
read -r -e -p "Environment file exists and is not empty. Answer yes to use, no to delete. " yn
7776
case $yn in
7877
[Yy]*) # We have an environment file and they want to keep it....
79-
if pulumi config get kubernetes:infra_type -C ${script_dir}/../pulumi/python/config>/dev/null 2>&1; then
78+
if pulumi config get kubernetes:infra_type -C ${script_dir}/../pulumi/python/config >/dev/null 2>&1; then
8079
INFRA="$(pulumi config get kubernetes:infra_type -C ${script_dir}/../pulumi/python/config)"
8180
if [ $INFRA == 'AWS' ]; then
8281
exec ${script_dir}/start_aws.sh

‎bin/start_aws.sh‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -152,8 +152,8 @@ else
152152
fi
153153

154154
function createpw() {
155-
base64 /dev/random | tr -dc '[:alnum:]' | head -c${1:-16}
156-
return 0
155+
PWORD=$(dd if=/dev/urandom count=1 2>/dev/null | base64 | head -c16)
156+
echo $PWORD
157157
}
158158

159159
# The bank of sirius configuration file is stored in the ./sirius/config
@@ -208,13 +208,13 @@ function header() {
208208
}
209209

210210
function add_kube_config() {
211-
pulumi_region="$(pulumi config get aws:region -C ${script_dir}/../pulumi/python/config)"
211+
pulumi_region="$(pulumi ${pulumi_args} config get aws:region -C ${script_dir}/../pulumi/python/config)"
212212
if [ "${pulumi_region}" != "" ]; then
213213
region_arg="--region ${pulumi_region}"
214214
else
215215
region_arg=""
216216
fi
217-
pulumi_aws_profile="$(pulumi config get aws:profile -C ${script_dir}/../pulumi/python/config)"
217+
pulumi_aws_profile="$(pulumi ${pulumi_args} config get aws:profile -C ${script_dir}/../pulumi/python/config)"
218218
if [ "${pulumi_aws_profile}" != "" ]; then
219219
echo "Using AWS profile [${pulumi_aws_profile}] from Pulumi configuration"
220220
profile_arg="--profile ${pulumi_aws_profile}"
@@ -225,7 +225,7 @@ function add_kube_config() {
225225
profile_arg=""
226226
fi
227227

228-
cluster_name="$(pulumi stack output cluster_name -C ${script_dir}/../pulumi/python/infrastructure/aws/eks)"
228+
cluster_name="$(pulumi ${pulumi_args} stack output cluster_name -C ${script_dir}/../pulumi/python/infrastructure/aws/eks)"
229229

230230
echo "adding ${cluster_name} cluster to local kubeconfig"
231231
"${script_dir}"/../pulumi/python/venv/bin/aws ${profile_arg} ${region_arg} eks update-kubeconfig --name ${cluster_name}
@@ -284,6 +284,19 @@ pulumi config set kubernetes:infra_type -C ${script_dir}/../pulumi/python/config
284284
# configuration because of the encryption needed for the passwords.
285285
pulumi config set kubernetes:infra_type -C ${script_dir}/../pulumi/python/kubernetes/applications/sirius AWS
286286

287+
header "Version Info"
288+
echo "Version and Account Information"
289+
echo "====================================================================="
290+
echo "Pulumi version is: $(pulumi version)"
291+
echo "Pulumi user is: $(pulumi whoami)"
292+
echo "Python version is: $(python --version)"
293+
echo "Kubectl version information: "
294+
echo "$(kubectl version -o json)"
295+
echo "Python module information: "
296+
echo "$(pip list)"
297+
echo "====================================================================="
298+
echo " "
299+
287300
header "AWS VPC"
288301
cd "${script_dir}/../pulumi/python/infrastructure/aws/vpc"
289302
pulumi $pulumi_args up
@@ -300,6 +313,11 @@ if command -v kubectl >/dev/null; then
300313
retry 30 kubectl version >/dev/null
301314
fi
302315

316+
# Display the server information
317+
echo "Kubernetes client/server version information:"
318+
kubectl version -o json
319+
echo " "
320+
303321
#
304322
# This is used to streamline the pieces that follow. Moving forward we can add new logic behind this and this
305323
# should abstract away for us. This way we just call the kubeconfig project to get the needed information and
@@ -357,7 +375,7 @@ header "Bank of Sirius"
357375
cd "${script_dir}/../pulumi/python/kubernetes/applications/sirius"
358376

359377
pulumi $pulumi_args up
360-
app_url="$(pulumi stack output --json | python3 "${script_dir}"/../pulumi/python/kubernetes/applications/sirius/verify.py)"
378+
app_url="$(pulumi ${pulumi_args} stack output --json | python3 "${script_dir}"/../pulumi/python/kubernetes/applications/sirius/verify.py)"
361379

362380
header "Finished!"
363381
echo "The startup process has finished successfully"
@@ -376,4 +394,3 @@ echo "Bank of Sirius (Example Application) Configuration: pulumi config -C ${scr
376394
echo "K8 Loadbalancer IP: kubectl get services --namespace nginx-ingress"
377395
echo " "
378396
echo "Please see the documentation in the github repository for more information"
379-

‎bin/start_do.sh‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -124,8 +124,8 @@ else
124124
fi
125125

126126
function createpw() {
127-
base64 /dev/random | tr -dc '[:alnum:]' | head -c${1:-16}
128-
return 0
127+
PWORD=$(dd if=/dev/urandom count=1 2>/dev/null | base64 | head -c16)
128+
echo $PWORD
129129
}
130130

131131
# The bank of sirius configuration file is stored in the ./sirius/config
@@ -287,6 +287,19 @@ pulumi config set kubernetes:infra_type -C ${script_dir}/../pulumi/python/config
287287
# configuration because of the encryption needed for the passwords.
288288
pulumi config set kubernetes:infra_type -C ${script_dir}/../pulumi/python/kubernetes/applications/sirius DO
289289

290+
header "Version Info"
291+
echo "Version and Account Information"
292+
echo "====================================================================="
293+
echo "Pulumi version is: $(pulumi version)"
294+
echo "Pulumi user is: $(pulumi whoami)"
295+
echo "Python version is: $(python --version)"
296+
echo "Kubectl version information: "
297+
echo "$(kubectl version -o json)"
298+
echo "Python module information: "
299+
echo "$(pip list)"
300+
echo "====================================================================="
301+
echo " "
302+
290303
header "DO Kubernetes"
291304
cd "${script_dir}/../pulumi/python/infrastructure/digitalocean/domk8s"
292305
pulumi $pulumi_args up
@@ -300,6 +313,12 @@ if command -v kubectl >/dev/null; then
300313
retry 30 kubectl version >/dev/null
301314
fi
302315

316+
# Display the server information
317+
echo "Kubernetes client/server version information:"
318+
kubectl version -o json
319+
echo " "
320+
321+
303322
#
304323
# This is used to streamline the pieces that follow. Moving forward we can add new logic behind this and this
305324
# should abstract away for us. This way we just call the kubeconfig project to get the needed information and
@@ -339,7 +358,7 @@ pulumi $pulumi_args up
339358

340359
header "Finished!"
341360
THE_FQDN=$(pulumi config get kic-helm:fqdn -C ${script_dir}/../pulumi/python/config || echo "Cannot Retrieve")
342-
THE_IP=$(kubectl get service kic-nginx-ingress --namespace nginx-ingress --output=jsonpath='{.status.loadBalancer.ingress[*].ip}' || echo "Cannot Retrieve")
361+
THE_IP=$(kubectl get service kic-nginx-ingress --namespace nginx-ingress --output=jsonpath='{.status.loadBalancer.ingress[*].ip}' || echo "Cannot Retrieve")
343362

344363
echo " "
345364
echo "The startup process has finished successfully"
@@ -358,4 +377,4 @@ echo "Main Configuration: pulumi config -C ${script_dir}/../pulumi/python/config
358377
echo "Bank of Sirius (Example Application) Configuration: pulumi config -C ${script_dir}/../pulumi/python/kubernetes/applications/sirius"
359378
echo "K8 Loadbalancer IP: kubectl get services --namespace nginx-ingress"
360379
echo " "
361-
echo "Please see the documentation in the github repository for more information"
380+
echo "Please see the documentation in the github repository for more information"

‎bin/start_kube.sh‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -131,8 +131,8 @@ function retry() {
131131
}
132132

133133
function createpw() {
134-
base64 /dev/random | tr -dc '[:alnum:]' | head -c${1:-16}
135-
return 0
134+
PWORD=$(dd if=/dev/urandom count=1 2>/dev/null | base64 | head -c16)
135+
echo $PWORD
136136
}
137137

138138
#
@@ -288,6 +288,24 @@ else
288288
pulumi_args="--color never --stack ${PULUMI_STACK}"
289289
fi
290290

291+
#
292+
# Note that this is somewhat different than the other startup scripts, because at the point we run this
293+
# here we know that we have a server so we can get the version. The other builds do not have server info
294+
# at this point in time.
295+
#
296+
header "Version Info"
297+
echo "Version and Account Information"
298+
echo "====================================================================="
299+
echo "Pulumi version is: $(pulumi version)"
300+
echo "Pulumi user is: $(pulumi whoami)"
301+
echo "Python version is: $(python --version)"
302+
echo "Kubectl version information: "
303+
echo "$(kubectl version -o json)"
304+
echo "Python module information: "
305+
echo "$(pip list)"
306+
echo "====================================================================="
307+
echo " "
308+
291309
header "Kubeconfig"
292310
cd "${script_dir}/../pulumi/python/infrastructure/kubeconfig"
293311
pulumi $pulumi_args up
@@ -323,7 +341,7 @@ pulumi $pulumi_args up
323341

324342
header "Finished!!"
325343
THE_FQDN=$(pulumi config get kic-helm:fqdn -C ${script_dir}/../pulumi/python/config || echo "Cannot Retrieve")
326-
THE_IP=$(kubectl get service kic-nginx-ingress --namespace nginx-ingress --output=jsonpath='{.status.loadBalancer.ingress[*].ip}' || echo "Cannot Retrieve")
344+
THE_IP=$(kubectl get service kic-nginx-ingress --namespace nginx-ingress --output=jsonpath='{.status.loadBalancer.ingress[*].ip}' || echo "Cannot Retrieve")
327345

328346
echo " "
329347
echo "The startup process has finished successfully"

‎bin/start_lke.sh‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
-#!/usr/bin/env bash
1+
#!/usr/bin/env bash
22

33
set -o errexit # abort on nonzero exit status
44
set -o nounset # abort on unbound variable
@@ -99,8 +99,8 @@ else
9999
fi
100100

101101
function createpw() {
102-
base64 /dev/random | tr -dc '[:alnum:]' | head -c${1:-16}
103-
return 0
102+
PWORD=$(dd if=/dev/urandom count=1 2>/dev/null | base64 | head -c16)
103+
echo $PWORD
104104
}
105105

106106
source "${script_dir}/../config/pulumi/environment"
@@ -296,6 +296,19 @@ pulumi config set kubernetes:infra_type -C ${script_dir}/../pulumi/python/config
296296
# configuration because of the encryption needed for the passwords.
297297
pulumi config set kubernetes:infra_type -C ${script_dir}/../pulumi/python/kubernetes/applications/sirius LKE
298298

299+
header "Version Info"
300+
echo "Version and Account Information"
301+
echo "====================================================================="
302+
echo "Pulumi version is: $(pulumi version)"
303+
echo "Pulumi user is: $(pulumi whoami)"
304+
echo "Python version is: $(python --version)"
305+
echo "Kubectl version information: "
306+
echo "$(kubectl version -o json)"
307+
echo "Python module information: "
308+
echo "$(pip list)"
309+
echo "====================================================================="
310+
echo " "
311+
299312
header "Linode LKE"
300313
cd "${script_dir}/../pulumi/python/infrastructure/linode/lke"
301314
pulumi $pulumi_args up
@@ -313,6 +326,11 @@ pulumi $pulumi_args up
313326
cluster_name=$(pulumi stack output cluster_id -s "${PULUMI_STACK}" -C ${script_dir}/../pulumi/python/infrastructure/linode/lke)
314327
add_kube_config
315328

329+
# Display the server information
330+
echo "Kubernetes client/server version information:"
331+
kubectl version -o json
332+
echo " "
333+
316334
if command -v kubectl >/dev/null; then
317335
echo "Attempting to connect to newly create kubernetes cluster"
318336
retry 30 kubectl version >/dev/null
@@ -348,7 +366,7 @@ pulumi $pulumi_args up
348366

349367
header "Finished!"
350368
THE_FQDN=$(pulumi config get kic-helm:fqdn -C ${script_dir}/../pulumi/python/config || echo "Cannot Retrieve")
351-
THE_IP=$(kubectl get service kic-nginx-ingress --namespace nginx-ingress --output=jsonpath='{.status.loadBalancer.ingress[*].ip}' || echo "Cannot Retrieve")
369+
THE_IP=$(kubectl get service kic-nginx-ingress --namespace nginx-ingress --output=jsonpath='{.status.loadBalancer.ingress[*].ip}' || echo "Cannot Retrieve")
352370

353371
echo " "
354372
echo "The startup process has finished successfully"

‎config/pulumi/Pulumi.stackname.yaml.example‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ config:
6262
############################################################################
6363

6464
# This is the Kubernetes version to install using EKS.
65-
eks:k8s_version: 1.22
65+
eks:k8s_version: 1.21
6666
# This is the default instance type used by EKS.
6767
eks:instance_type: t2.large
6868
# The minimum number of compute instances to provision for the EKS cluster.
@@ -94,7 +94,7 @@ config:
9494
# Chart name for the helm chart for kic
9595
kic-helm:chart_name: nginx-ingress
9696
# Chart version for the helm chart for kic
97-
kic-helm:chart_version: 0.12.0
97+
kic-helm:chart_version: 0.13.1
9898
# Name of the repo to pull the kic chart from
9999
kic-helm:helm_repo_name: nginx-stable
100100
# URL of the chart repo to pull kic from
@@ -118,12 +118,12 @@ config:
118118
# https://docs.nginx.com/nginx-ingress-controller/installation/pulling-ingress-controller-image/
119119
#
120120
# The following are all valid image names:
121-
# kic:image_name: private-registry.nginx.com/nginx-ic/nginx-plus-ingress:2.1.0
122-
# kic:image_name: private-registry.nginx.com/nginx-ic/nginx-plus-ingress:2.1.0-ot
123-
# kic:image_name: docker.io/nginx/nginx-ingress:2.1.0
124-
# kic:image_name: nginx/nginx-ingress:2.1.0
125-
# kic:image_name: nginx/nginx-ingress:2.1.0-alpine
126-
kic:image_name: nginx/nginx-ingress:2.1.0
121+
# kic:image_name: private-registry.nginx.com/nginx-ic/nginx-plus-ingress:2.2.0
122+
# kic:image_name: private-registry.nginx.com/nginx-ic/nginx-plus-ingress:2.2.0-ot
123+
# kic:image_name: docker.io/nginx/nginx-ingress:2.2.0
124+
# kic:image_name: nginx/nginx-ingress:2.2.0
125+
# kic:image_name: nginx/nginx-ingress:2.2.0-alpine
126+
kic:image_name: nginx/nginx-ingress:2.2.0
127127

128128

129129
############################################################################
@@ -174,10 +174,10 @@ config:
174174
# ssh://[email protected]:nginxinc/kubernetes-ingress.git
175175
#
176176
# Github URL with a tag specified:
177-
# https://github.com/nginxinc/kubernetes-ingress.git#v2.1.0
178-
# [email protected]:nginxinc/kubernetes-ingress.git#v2.1.0
179-
# ssh://[email protected]:nginxinc/kubernetes-ingress.git#v2.1.0
180-
kic:src_url: https://github.com/nginxinc/kubernetes-ingress.git#v2.1.0
177+
# https://github.com/nginxinc/kubernetes-ingress.git#v2.2.0
178+
# [email protected]:nginxinc/kubernetes-ingress.git#v2.2.0
179+
# ssh://[email protected]:nginxinc/kubernetes-ingress.git#v2.2.0
180+
kic:src_url: https://github.com/nginxinc/kubernetes-ingress.git#v2.2.0
181181

182182
# When set to true, Pulumi's diff logic is circumvented and the image will always be
183183
# rebuilt regardless of the input variables to Pulumi being the same or not.

0 commit comments

Comments
 (0)