From a2d68a47f0217a70f2709e0572163b2c7dd378d2 Mon Sep 17 00:00:00 2001 From: Techassi Date: Mon, 11 Nov 2024 14:16:08 +0100 Subject: [PATCH 1/4] chore: Add demo upgrade issue template --- .../pre-release-nightly-testing.md | 115 ++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/pre-release-nightly-testing.md diff --git a/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md b/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md new file mode 100644 index 00000000..0615b03c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md @@ -0,0 +1,115 @@ +--- +name: Pre-Release Demo Upgrade Testing from Stable to Nightly +about: | + This template can be used to track the upgrade testing of demos from stable to nightly leading + up to the next Stackable release +title: "chore(tracking): Test demos on nightly versions" +labels: ['epic'] +assignees: '' +--- + + + + + +## Pre-Release Demo Testing on Nightly + +Part of + +This is testing: + +1. The demos documented in nightly (with the updated product versions) still work. +2. That the operators can be upgraded from the current release to the nightly release and do not + negatively impact the products. + +> [!NOTE] +> Record any issues or anomalies during the process in a comment on this issue. +> Eg: +> +> ```plain +> :green_circle: **airflow-scheduled-job** +> +> The CRD had been updated and I needed to change the following in the manifest: +> ... +> ``` + +Replace the items in the task lists below with the applicable Pull Requests (if any). + + + +```[tasklist] +### Testing Demos on Nightly +- [ ] [airflow-scheduled-job](https://docs.stackable.tech/home/nightly/demos/airflow-scheduled-job) +- [ ] [data-lakehouse-iceberg-trino-spark](https://docs.stackable.tech/home/nightly/demos/data-lakehouse-iceberg-trino-spark) +- [ ] [end-to-end-security](https://docs.stackable.tech/home/nightly/demos/end-to-end-security) +- [ ] [hbase-hdfs-load-cycling-data](https://docs.stackable.tech/home/nightly/demos/hbase-hdfs-load-cycling-data) +- [ ] [jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data](https://docs.stackable.tech/home/nightly/demos/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data) +- [ ] [logging](https://docs.stackable.tech/home/nightly/demos/logging) +- [ ] [nifi-kafka-druid-earthquake-data](https://docs.stackable.tech/home/nightly/demos/nifi-kafka-druid-earthquake-data) +- [ ] [nifi-kafka-druid-water-level-data](https://docs.stackable.tech/home/nightly/demos/nifi-kafka-druid-water-level-data) +- [ ] [signal-processing](https://docs.stackable.tech/home/nightly/demos/signal-processing) +- [ ] [spark-k8s-anomaly-detection-taxi-data](https://docs.stackable.tech/home/nightly/demos/spark-k8s-anomaly-detection-taxi-data) +- [ ] [trino-iceberg](https://docs.stackable.tech/home/nightly/demos/trino-iceberg) +- [ ] [trino-taxi-data](https://docs.stackable.tech/home/nightly/demos/trino-taxi-data) +``` + +### Testing Instructions + +These instructions are for deploying the nightly demo, as well as upgrading the operators and CRDS. + +```shell +# Install demo (stable operators) +stackablectl demo install + +# --- IMPORTANT --- +# Run through the nightly demo instructions (refer to the tasklist below). + +# Get a list of installed operators +stackablectl operator installed --output=plain + +# --- OPTIONAL --- +# Sometimes it is necessary to upgrade Helm charts. Look for other Helm Charts +# which might need updating. + +# First, see which charts are installed. You can ignore the stackable-operator +# charts, or anything that might have been installed outside of this demo. +helm list + +# Next, add the applicable Helm Chart repositories. For example: +helm repo add minio https://charts.min.io/ +helm repo add bitnami https://charts.bitnami.com/bitnami + +# Finally, upgrade the Charts. For example: +helm upgrade minio minio/minio --version x.x.x +helm upgrade postgresql-hive bitnami/postgresql --version x.x.x +# --- OPTIONAL END --- + +# Uninstall operators +stackablectl release uninstall + +# Update CRDs to nightly version (on main) +# Repeat this for every operator used by the demo +kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/...-operator/main/deploy/helm/...-operator/crds/crds.yaml + +# Install nightly version of operators (use the list from the earlier step +# before deleting the operators) +stackablectl operator install commons ... + +# Optionally update the product versions in the CRDs, e.g.: +kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed +``` From 51d8119d546bfc262024b679b88573169753263a Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 12 Nov 2024 11:35:19 +0100 Subject: [PATCH 2/4] chore: Adjust demo upgrade instructions --- .../pre-release-nightly-testing.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md b/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md index 0615b03c..2e1f6f4a 100644 --- a/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md +++ b/.github/ISSUE_TEMPLATE/pre-release-nightly-testing.md @@ -3,7 +3,7 @@ name: Pre-Release Demo Upgrade Testing from Stable to Nightly about: | This template can be used to track the upgrade testing of demos from stable to nightly leading up to the next Stackable release -title: "chore(tracking): Test demos on nightly versions" +title: "chore(tracking): Test demos on nightly versions for XX.(X)X" labels: ['epic'] assignees: '' --- @@ -71,9 +71,17 @@ Replace the items in the task lists below with the applicable Pull Requests (if These instructions are for deploying the nightly demo, as well as upgrading the operators and CRDS. + + ```shell -# Install demo (stable operators) -stackablectl demo install +# Install demo (stable operators) for the previous release (24.7) +# For now, we have to deploy from the release branch, otherwise we get new changes. +# Stackablectl doesn't yet support deploying a demo from a branch +git checkout release-24.7 +git pull +stackablectl --stack-file=stacks/stacks-v2.yaml --demo-file=demos/demos-v2.yaml demo install # --- IMPORTANT --- # Run through the nightly demo instructions (refer to the tasklist below). @@ -93,23 +101,23 @@ helm list helm repo add minio https://charts.min.io/ helm repo add bitnami https://charts.bitnami.com/bitnami -# Finally, upgrade the Charts. For example: +# Finally, upgrade the Charts to what is defined in `main`. +# For example: helm upgrade minio minio/minio --version x.x.x helm upgrade postgresql-hive bitnami/postgresql --version x.x.x # --- OPTIONAL END --- # Uninstall operators -stackablectl release uninstall +stackablectl release uninstall 24.7 # Update CRDs to nightly version (on main) -# Repeat this for every operator used by the demo +# Repeat this for every operator used by the demo (use the list from the earlier step before deleting the operators) kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/...-operator/main/deploy/helm/...-operator/crds/crds.yaml -# Install nightly version of operators (use the list from the earlier step -# before deleting the operators) +# Install nightly version of operators (use the list from the earlier step before deleting the operators) stackablectl operator install commons ... -# Optionally update the product versions in the CRDs, e.g.: +# Optionally update the product versions in the CRDs (to the latest non-experimental version for the new release), e.g.: kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed ``` From 4db307de53302e08fba43f828f5a4a1ee2ee4d07 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 12 Nov 2024 11:36:17 +0100 Subject: [PATCH 3/4] chore: Add Chart update issue template --- .../pre-release-chart-updates.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/pre-release-chart-updates.md diff --git a/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md b/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md new file mode 100644 index 00000000..785c486a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md @@ -0,0 +1,57 @@ +--- +name: Pre-Release Demo Chart Updates +about: This template can be used to track the Helm chart updates for all demos. +title: "chore(tracking): Update Helm charts for XX.(X)X" +labels: [] +assignees: '' +--- + +## Pre-Release Demo Chart Updates + +There are a bunch of demos which directly use Helm charts to install some required parts of the +demo / stack. These charts need updating **before** we start testing the stable to nightly upgrade. + +Most of the charts are located in the `stacks/_templated` folder. Additional folders are: +`stacks/observability` and `stacks/signal-processing`. It is recommended to search for `releaseName` +to find all referenced charts. + +### Update Instructions + +These instructions help to update the charts: + +```shell +# List all available repos +helm repo list +``` + +Make sure, you have the following repos added: + +```plain +NAME URL +bitnami https://charts.bitnami.com/bitnami +stackable-dev https://repo.stackable.tech/repository/helm-dev/ +stackable-stable https://repo.stackable.tech/repository/helm-stable/ +timescale https://charts.timescale.com/ +prometheus-community https://prometheus-community.github.io/helm-charts +vector https://helm.vector.dev +grafana https://grafana.github.io/helm-charts +jaeger https://jaegertracing.github.io/helm-charts +opentelemetry https://open-telemetry.github.io/opentelemetry-helm-charts +jupyterhub https://jupyterhub.github.io/helm-chart/ +opensearch https://opensearch-project.github.io/helm-charts +``` + +Next, search for the latest version of the desired chart, eg `grafana/grafana`: + +```shell +# Display the latest versions of a chart +$ helm search repo grafana/grafana --versions | head -5 +NAME CHART VERSION APP VERSION +grafana/grafana 8.6.0 11.3.0 +grafana/grafana 8.5.12 11.3.0 +grafana/grafana 8.5.11 11.3.0 +grafana/grafana 8.5.10 11.3.0 +``` + +Use the latest version displayed to replace the current one in the YAML files. Take a look at +previous PRs for additional guidance, eg. . From 716d3c84200e91128720885083fe9470197d4b9c Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 14 Nov 2024 16:10:55 +0100 Subject: [PATCH 4/4] chore: Apply suggestions Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> --- .../pre-release-chart-updates.md | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md b/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md index 785c486a..1a64b6f6 100644 --- a/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md +++ b/.github/ISSUE_TEMPLATE/pre-release-chart-updates.md @@ -41,6 +41,18 @@ jupyterhub https://jupyterhub.github.io/helm-chart/ opensearch https://opensearch-project.github.io/helm-charts ``` +To add any that are missing, run: + +```shell +helm repo add bitnami https://charts.bitnami.com/bitnami --force-update +``` + +Then make sure you have the latest indexes: + +```shell +helm repo update +``` + Next, search for the latest version of the desired chart, eg `grafana/grafana`: ```shell @@ -53,5 +65,11 @@ grafana/grafana 8.5.11 11.3.0 grafana/grafana 8.5.10 11.3.0 ``` -Use the latest version displayed to replace the current one in the YAML files. Take a look at -previous PRs for additional guidance, eg. . +Use the applicable chart version displayed to replace the current one in the YAML files. Make sure +to add the app version in the comments. Eg: + +```yml +version: 3.2.1 # 1.2.3 +``` + +Take a look at previous PRs for additional guidance, eg. .