From 73d04bfee06a83388ce95f31525b30bf6298a42a Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 15 Nov 2024 10:14:49 +0100 Subject: [PATCH 01/61] Add 24.11 release notes headings --- modules/ROOT/pages/release-notes.adoc | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 35e0f9316..6bab88fba 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -40,6 +40,36 @@ Here are the headings you can use for the next release. Saves time checking inde //// +== Release 24.11 + +=== New / extended platform features + +=== Product versions + +==== New versions + +==== Deprecated versions + +==== Removed versions + +=== stackablectl + +=== Supported Kubernetes versions + +=== Supported OpenShift versions + +=== Breaking changes + +=== Upgrade from 24.7 + +==== Using stackablectl + +==== Using Helm + +==== Known upgrade issues + +===== All operators + == Release 24.7 === New / extended platform features From cf5e87dcb85977b692ee8fd07fc33732ee36abb7 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 16:33:29 +0100 Subject: [PATCH 02/61] airflow versions --- modules/ROOT/pages/release-notes.adoc | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 6bab88fba..1b407aad9 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -48,10 +48,22 @@ Here are the headings you can use for the next release. Saves time checking inde ==== New versions +The following new product versions are now supported: + +* Apache Airflow: https://github.com/stackabletech/airflow-operator/pull/494[2.9.3 (LTS)], https://github.com/stackabletech/airflow-operator/pull/512[2.10.2 (experimental)] + ==== Deprecated versions +The following product versions are deprecated and will be removed in a later release: + +* Apache Airflow: 2.9.2 + ==== Removed versions +The following product versions are no longer supported (although images for released product versions remain available https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable[here]): + +* Apache Airflow: 2.8.4, 2.8.1, 2.6.3 + === stackablectl === Supported Kubernetes versions From 2bef74a01fabb2a45ff79be0949ebf54c23f00ba Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 16:39:15 +0100 Subject: [PATCH 03/61] druid versions --- modules/ROOT/pages/release-notes.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 1b407aad9..23185fbb6 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -51,18 +51,21 @@ Here are the headings you can use for the next release. Saves time checking inde The following new product versions are now supported: * Apache Airflow: https://github.com/stackabletech/airflow-operator/pull/494[2.9.3 (LTS)], https://github.com/stackabletech/airflow-operator/pull/512[2.10.2 (experimental)] +* Apache Druid: https://github.com/stackabletech/druid-operator/pull/631[30.0.0 (LTS)] ==== Deprecated versions The following product versions are deprecated and will be removed in a later release: * Apache Airflow: 2.9.2 +* Apache Druid: 26.0.0 ==== Removed versions The following product versions are no longer supported (although images for released product versions remain available https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable[here]): * Apache Airflow: 2.8.4, 2.8.1, 2.6.3 +* Apache Druid: 28.0.1 === stackablectl From 768c94d7eda77c338492542258e9fc2ca066bdd9 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 16:52:25 +0100 Subject: [PATCH 04/61] link supported versions --- modules/ROOT/pages/release-notes.adoc | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 23185fbb6..8b4013e42 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -46,12 +46,18 @@ Here are the headings you can use for the next release. Saves time checking inde === Product versions +As with previous SDP releases, many product images have been updated to their latest versions. +The LTS version has in many cases also been adjusted in line with our https://docs.stackable.tech/home/stable/policies[support policy]. + +Refer to the https://docs.stackable.tech/home/stable/operators/supported_versions/[supported versions] documentation for a complete overview. + ==== New versions The following new product versions are now supported: * Apache Airflow: https://github.com/stackabletech/airflow-operator/pull/494[2.9.3 (LTS)], https://github.com/stackabletech/airflow-operator/pull/512[2.10.2 (experimental)] * Apache Druid: https://github.com/stackabletech/druid-operator/pull/631[30.0.0 (LTS)] +* ==== Deprecated versions @@ -131,7 +137,7 @@ The status is still xref:concepts:multi-platform-support.adoc[experimental], as Security:: Support for OIDC with/without TLS has been added to Apache Druid in this release. - +* Apache Druid: 26.0.0 NOTE: SDP now provides OIDC-support for Druid, Superset and Trino In this release we provide experimental HBase 2.6.0 support with a new experimental policy based authorizer (with OPA). From 47066541eaf0568205ab9d9801c0f4d1617f68b8 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 16:58:26 +0100 Subject: [PATCH 05/61] kafka versions --- modules/ROOT/pages/release-notes.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 8b4013e42..e2cddb481 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -57,7 +57,8 @@ The following new product versions are now supported: * Apache Airflow: https://github.com/stackabletech/airflow-operator/pull/494[2.9.3 (LTS)], https://github.com/stackabletech/airflow-operator/pull/512[2.10.2 (experimental)] * Apache Druid: https://github.com/stackabletech/druid-operator/pull/631[30.0.0 (LTS)] -* +* Apache Hive: https://github.com/stackabletech/hive-operator/pull/508[4.0.0 (experimental)] +* Apache Kafka: https://github.com/stackabletech/kafka-operator/pull/753/[3.8.0] ==== Deprecated versions @@ -72,6 +73,7 @@ The following product versions are no longer supported (although images for rele * Apache Airflow: 2.8.4, 2.8.1, 2.6.3 * Apache Druid: 28.0.1 +* Apache Kafka: 3.6.2, 3.6.1, 3.4.1 === stackablectl From c5755442ed10afe244748e17c261070a47f15b9c Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 17:01:28 +0100 Subject: [PATCH 06/61] nifi versions --- modules/ROOT/pages/release-notes.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index e2cddb481..5a88f5e53 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -59,6 +59,7 @@ The following new product versions are now supported: * Apache Druid: https://github.com/stackabletech/druid-operator/pull/631[30.0.0 (LTS)] * Apache Hive: https://github.com/stackabletech/hive-operator/pull/508[4.0.0 (experimental)] * Apache Kafka: https://github.com/stackabletech/kafka-operator/pull/753/[3.8.0] +* Apache NiFi: https://github.com/stackabletech/nifi-operator/pull/702[2.0.0 (experimental)] ==== Deprecated versions @@ -74,6 +75,7 @@ The following product versions are no longer supported (although images for rele * Apache Airflow: 2.8.4, 2.8.1, 2.6.3 * Apache Druid: 28.0.1 * Apache Kafka: 3.6.2, 3.6.1, 3.4.1 +* Apache NiFi: 2.0.0-M4, 1.25.0, 1.21.0 === stackablectl From 0791e9f247930bbd2e230ec7b880f808dc8af500 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 17:03:13 +0100 Subject: [PATCH 07/61] opa versions --- modules/ROOT/pages/release-notes.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 5a88f5e53..e3e83e43a 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -60,6 +60,7 @@ The following new product versions are now supported: * Apache Hive: https://github.com/stackabletech/hive-operator/pull/508[4.0.0 (experimental)] * Apache Kafka: https://github.com/stackabletech/kafka-operator/pull/753/[3.8.0] * Apache NiFi: https://github.com/stackabletech/nifi-operator/pull/702[2.0.0 (experimental)] +* Open Policy Agent: https://github.com/stackabletech/opa-operator/pull/616[0.67.1] ==== Deprecated versions @@ -67,6 +68,7 @@ The following product versions are deprecated and will be removed in a later rel * Apache Airflow: 2.9.2 * Apache Druid: 26.0.0 +* Open Policy Agent: 0.66.0 ==== Removed versions @@ -76,6 +78,7 @@ The following product versions are no longer supported (although images for rele * Apache Druid: 28.0.1 * Apache Kafka: 3.6.2, 3.6.1, 3.4.1 * Apache NiFi: 2.0.0-M4, 1.25.0, 1.21.0 +* Open Policy Agent: 0.61.0 === stackablectl From 86042e17fd70c1bc26c714cd0e1a65b7a4f4ee16 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 17:08:11 +0100 Subject: [PATCH 08/61] spark versions --- modules/ROOT/pages/release-notes.adoc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index e3e83e43a..a0dbc150c 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -61,6 +61,8 @@ The following new product versions are now supported: * Apache Kafka: https://github.com/stackabletech/kafka-operator/pull/753/[3.8.0] * Apache NiFi: https://github.com/stackabletech/nifi-operator/pull/702[2.0.0 (experimental)] * Open Policy Agent: https://github.com/stackabletech/opa-operator/pull/616[0.67.1] +* Trino: https://github.com/stackabletech/trino-operator/pull/638[455] +* Apache Spark: https://github.com/stackabletech/spark-k8s-operator/pull/459[3.5.2 (LTS)] ==== Deprecated versions @@ -79,6 +81,8 @@ The following product versions are no longer supported (although images for rele * Apache Kafka: 3.6.2, 3.6.1, 3.4.1 * Apache NiFi: 2.0.0-M4, 1.25.0, 1.21.0 * Open Policy Agent: 0.61.0 +* Trino: 442, 414 +* Apache Spark: 3.4.3, 3.4.2 === stackablectl From 95550fbf37d5ffda501bd99367391184fce24047 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 17:11:08 +0100 Subject: [PATCH 09/61] superset and zookeeper versions --- modules/ROOT/pages/release-notes.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index a0dbc150c..0dfa14615 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -83,6 +83,8 @@ The following product versions are no longer supported (although images for rele * Open Policy Agent: 0.61.0 * Trino: 442, 414 * Apache Spark: 3.4.3, 3.4.2 +* Apache Superset: 3.1.3, 3.1.0, 2.1.3 +* Apache ZooKeeper: 3.8.4 === stackablectl From 2ce7709be039cbdee9918e121acdcc8a24386032 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 17:27:29 +0100 Subject: [PATCH 10/61] airflow(oidc), nifi(oidc), kafka (kerberos) --- modules/ROOT/pages/release-notes.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 0dfa14615..50a00068b 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -44,6 +44,14 @@ Here are the headings you can use for the next release. Saves time checking inde === New / extended platform features +Improved Authentication: + +In this release we introduced several authentication mechanisms in different products: + +* Apache Airflow: https://github.com/stackabletech/airflow-operator/issues/337[OIDC support] +* Apache Kafka: https://github.com/stackabletech/kafka-operator/issues/655[Kerberos support] +* Apache NiFi: https://github.com/stackabletech/nifi-operator/issues/633[OIDC support] + === Product versions As with previous SDP releases, many product images have been updated to their latest versions. From 156c6e3351b10946f2bcfc646600d129519dcffb Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Mon, 18 Nov 2024 17:35:12 +0100 Subject: [PATCH 11/61] improve supported versions text --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 50a00068b..6fa8063a8 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -57,7 +57,7 @@ In this release we introduced several authentication mechanisms in different pro As with previous SDP releases, many product images have been updated to their latest versions. The LTS version has in many cases also been adjusted in line with our https://docs.stackable.tech/home/stable/policies[support policy]. -Refer to the https://docs.stackable.tech/home/stable/operators/supported_versions/[supported versions] documentation for a complete overview. +Refer to the https://docs.stackable.tech/home/stable/operators/supported_versions/[supported versions] documentation for a complete overview including LTS versions or deprecations. ==== New versions From e813f3f0a14652c9128aff7ffab51c2c924c54ed Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Tue, 19 Nov 2024 11:29:20 +0100 Subject: [PATCH 12/61] add supported os / kubernetes versions --- modules/ROOT/pages/release-notes.adoc | 29 +++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 6fa8063a8..6ffe68e1d 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -52,6 +52,14 @@ In this release we introduced several authentication mechanisms in different pro * Apache Kafka: https://github.com/stackabletech/kafka-operator/issues/655[Kerberos support] * Apache NiFi: https://github.com/stackabletech/nifi-operator/issues/633[OIDC support] +Improved Authorization: + +* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] of the hdfs-utils for details + +Monitoring: + +* In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX as Prometheus metrics and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not released yet. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. + === Product versions As with previous SDP releases, many product images have been updated to their latest versions. @@ -98,8 +106,29 @@ The following product versions are no longer supported (although images for rele === Supported Kubernetes versions +This release supports the following Kubernetes versions: + +* `1.31` +* `1.30` +* `1.29` + +These Kubernetes versions are no longer supported: + +* `1.26` +* `1.25` as we removed internal forks required to support Kubernetes `1.25` and below. This includes OpenShift `4.12`, which is using Kubernetes `1.25`. + === Supported OpenShift versions +This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions: + +* `4.15` +* `4.14` + +These OpenShift versions are no longer supported: + +* `4.13` +* `4.12` + === Breaking changes === Upgrade from 24.7 From 14da45be10eaa651b09b24427b793e04d570362f Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Tue, 19 Nov 2024 12:07:28 +0100 Subject: [PATCH 13/61] add more platform features --- modules/ROOT/pages/release-notes.adoc | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 6ffe68e1d..92fea67b7 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -55,11 +55,30 @@ In this release we introduced several authentication mechanisms in different pro Improved Authorization: * The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] of the hdfs-utils for details +* The User Info Fetcher HTTP API has been replaced with a Rego library, please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api for more information Monitoring: * In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX as Prometheus metrics and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not released yet. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. +Security: + +* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. +* Added support for customizing sAMAccountName generation + +Listener: + +* The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized. + +Misc: + +* Apache NiFi: allow users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. Documentation: https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check + +Bug fixes: + +* Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. +* Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. + === Product versions As with previous SDP releases, many product images have been updated to their latest versions. From cb6ba7eda7395783d3a51e99489a8b8c11de8045 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 11:35:50 +0100 Subject: [PATCH 14/61] Apply suggestions from code review Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com> --- modules/ROOT/pages/release-notes.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 92fea67b7..8ddc96eaf 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -54,12 +54,12 @@ In this release we introduced several authentication mechanisms in different pro Improved Authorization: -* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] of the hdfs-utils for details -* The User Info Fetcher HTTP API has been replaced with a Rego library, please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api for more information +* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. +* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api for more information Monitoring: -* In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX as Prometheus metrics and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not released yet. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. +* In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX metrics in Prometheus and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not yet released. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. Security: @@ -72,7 +72,7 @@ Listener: Misc: -* Apache NiFi: allow users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. Documentation: https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check +* Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. Documentation: https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check Bug fixes: From 2e4fc6c249d38f77de5f07eadfc3373983c718ce Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 11:37:57 +0100 Subject: [PATCH 15/61] linter --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 8ddc96eaf..a00cc171d 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -64,7 +64,7 @@ Monitoring: Security: * The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. -* Added support for customizing sAMAccountName generation +* Added support for customizing sAMAccountName generation Listener: From 20104d5611352255649850beb3c9518a6fea0881 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 11:48:02 +0100 Subject: [PATCH 16/61] add missing platform features --- modules/ROOT/pages/release-notes.adoc | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index a00cc171d..ff861f537 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -57,6 +57,10 @@ Improved Authorization: * The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api for more information +Logging: + +* Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. + Monitoring: * In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX metrics in Prometheus and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not yet released. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. @@ -64,12 +68,16 @@ Monitoring: Security: * The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. -* Added support for customizing sAMAccountName generation +* Added support for customizing sAMAccountName generation in secret operator. Listener: * The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized. +Operations: + +* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. This process requires some manual intervention, however. + Misc: * Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. Documentation: https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check @@ -78,6 +86,8 @@ Bug fixes: * Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. * Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. +* Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed +* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described [here](https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain). === Product versions From 42fb12053a942c56b5474b3dd6a9c54c458ec436 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 13:22:28 +0100 Subject: [PATCH 17/61] improve links --- modules/ROOT/pages/release-notes.adoc | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index ff861f537..477f7868f 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -55,7 +55,7 @@ In this release we introduced several authentication mechanisms in different pro Improved Authorization: * The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. -* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api for more information +* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging: @@ -67,7 +67,7 @@ Monitoring: Security: -* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. +* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager (). * Added support for customizing sAMAccountName generation in secret operator. Listener: @@ -80,14 +80,14 @@ Operations: Misc: -* Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. Documentation: https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check +* Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. See documentation https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check[here]. Bug fixes: * Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. * Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed -* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described [here](https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain). +* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[here]. === Product versions @@ -133,6 +133,8 @@ The following product versions are no longer supported (although images for rele === stackablectl +* Bump Rust dependencies to fix critical vulnerability in quinn-proto, see https://github.com/advisories/GHSA-vr26-jcq5-fjj8[CVE-2024-45311] (https://github.com/stackabletech/stackable-cockpit/pull/318). + === Supported Kubernetes versions This release supports the following Kubernetes versions: From f50f6395eb7e093efb6a6e41c461ef57b3ee30f2 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 13:25:48 +0100 Subject: [PATCH 18/61] fix sub headers --- modules/ROOT/pages/release-notes.adoc | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 477f7868f..ef609bd7b 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -44,7 +44,7 @@ Here are the headings you can use for the next release. Saves time checking inde === New / extended platform features -Improved Authentication: +Improved Authentication:: In this release we introduced several authentication mechanisms in different products: @@ -52,37 +52,37 @@ In this release we introduced several authentication mechanisms in different pro * Apache Kafka: https://github.com/stackabletech/kafka-operator/issues/655[Kerberos support] * Apache NiFi: https://github.com/stackabletech/nifi-operator/issues/633[OIDC support] -Improved Authorization: +Improved Authorization:: * The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. -Logging: +Logging:: * Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. -Monitoring: +Monitoring:: * In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX metrics in Prometheus and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not yet released. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. -Security: +Security:: -* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager (). +* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. * Added support for customizing sAMAccountName generation in secret operator. -Listener: +Listener:: * The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized. -Operations: +Operations:: * The Stackable Operator for HDFS now supports upgrading existing HDFS installations. This process requires some manual intervention, however. -Misc: +Misc:: * Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. See documentation https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check[here]. -Bug fixes: +Bug fixes:: * Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. * Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. From d6ab28e893c156cc7c5a1be9330b0951a333ce9d Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 14:23:09 +0100 Subject: [PATCH 19/61] missing pr snippets --- modules/ROOT/pages/release-notes.adoc | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index ef609bd7b..92f71a6bd 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -67,12 +67,23 @@ Monitoring:: Security:: -* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. +* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. * Added support for customizing sAMAccountName generation in secret operator. +* The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with `listeners.stackable.tech/listener-name`. + +Commons:: + +* Pod Enrichment is now deprecated, and will be removed in the next release. Once removed, the SDP will no longer set any `enrichment.stackable.tech/` annotations on Pods. Listener:: * The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized. +* The `ListenerClass.spec.serviceAnnotations` are now correctly propagated to created Service objects. +* Listeners can now be configured to use either IP addresses or DNS hostnames. + +Dependencies:: + +* Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. See the usage guide for detailed information. Operations:: @@ -81,6 +92,13 @@ Operations:: Misc:: * Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. See documentation https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check[here]. +* Apache Airflow: Allow custom arbitrary python code in webserver_config.py. +* Apache Superset: Allow custom arbitrary python code in superset_config.py + +Images:: + +* Our Docker images now exclusively make use of numeric user IDs in `USER` statements allowing the use of `securityContext.runAsNonRoot` +* The group id of all files relevant to our products is now set to `0`. This allows the images to be used with any arbitrary user as every container user will always belong to the root group (`0`). This is especially useful on OpenShift when trying to move to the `restricted-v2` SecurityContextConstraint (SCC), Stackable currently defaults to the `nonroot-v2` SCC but we plan on migrating to `restricted-v2` in the future Bug fixes:: @@ -88,6 +106,7 @@ Bug fixes:: * Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[here]. +* Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. === Product versions @@ -162,6 +181,10 @@ These OpenShift versions are no longer supported: === Breaking changes +==== Listener operator + +* BREAKING: All ListenerClasses now default to using DNS hostnames, previously NodePort ListenerClasses (such as external-unstable) would use IP addresses. Hence, all Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured to set .spec.preferredAddressType: IP. + === Upgrade from 24.7 ==== Using stackablectl From 18790fce673cc649f57c56b4f76a77174eb4a53f Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 15:54:34 +0100 Subject: [PATCH 20/61] add missing issue 211 --- modules/ROOT/pages/release-notes.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 92f71a6bd..98b62c2c4 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -107,6 +107,7 @@ Bug fixes:: * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[here]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. +* Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. === Product versions From 9f709ffc5220eb625dc94b69f6152aef078c0720 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 16:15:38 +0100 Subject: [PATCH 21/61] mention hbase bug fix --- modules/ROOT/pages/release-notes.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 98b62c2c4..102cee201 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -108,6 +108,7 @@ Bug fixes:: * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[here]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. +* Apache HBase: The operator now does not https://github.com/stackabletech/hbase-operator/pull/584[ignore the `hbaseRootdir` config property at role level]. === Product versions From 2906cb076f74359ecf1c4167138d68eeff7b8834 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 16:25:54 +0100 Subject: [PATCH 22/61] mention crd size reduction --- modules/ROOT/pages/release-notes.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 102cee201..70cb71291 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -94,6 +94,7 @@ Misc:: * Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. See documentation https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check[here]. * Apache Airflow: Allow custom arbitrary python code in webserver_config.py. * Apache Superset: Allow custom arbitrary python code in superset_config.py +* The size of the operator deployed CRDs was reduced significantly https://github.com/stackabletech/issues/issues/627[here]. Images:: From 7c5cf1fb2a4b1c707a99be1d4314ceb9925d7bc6 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 16:43:10 +0100 Subject: [PATCH 23/61] mention cve fixes --- modules/ROOT/pages/release-notes.adoc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 70cb71291..355161c46 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -44,6 +44,10 @@ Here are the headings you can use for the next release. Saves time checking inde === New / extended platform features +Vulnerabilities:: + +* More than 142 CVEs were fixed in the Stackable product images. This included 11 CVEs of critical and 55 CVEs of High severity. + Improved Authentication:: In this release we introduced several authentication mechanisms in different products: From f582e4b0a64b6cec40128da61b44ca205eb59ba0 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 16:46:54 +0100 Subject: [PATCH 24/61] mention kafka bugfix --- modules/ROOT/pages/release-notes.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 355161c46..9baa41f20 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -114,6 +114,7 @@ Bug fixes:: * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. * Apache HBase: The operator now does not https://github.com/stackabletech/hbase-operator/pull/584[ignore the `hbaseRootdir` config property at role level]. +* Apache Kakfa: The bootstrap Kafka service is now included in https://github.com/stackabletech/kafka-operator/pull/741[certificate SANs]. === Product versions From 65a8425b25c3927eafe8ccb16427ae9a56ebec33 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Wed, 20 Nov 2024 16:50:45 +0100 Subject: [PATCH 25/61] fix typo --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 9baa41f20..a7fb543bd 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -46,7 +46,7 @@ Here are the headings you can use for the next release. Saves time checking inde Vulnerabilities:: -* More than 142 CVEs were fixed in the Stackable product images. This included 11 CVEs of critical and 55 CVEs of High severity. +* More than 142 CVEs were fixed in the Stackable product images. This includes 11 CVEs of critical and 55 CVEs of high severity. Improved Authentication:: From 881a7cf3a156e7d71512f08e60543a4e8130c12a Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Thu, 21 Nov 2024 11:01:02 +0100 Subject: [PATCH 26/61] fill in the Upgrade from 24.7 section --- modules/ROOT/pages/release-notes.adoc | 126 +++++++++++++++++++++++++- 1 file changed, 125 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index a7fb543bd..d0d638573 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -197,11 +197,135 @@ These OpenShift versions are no longer supported: ==== Using stackablectl +Uninstall the `24.7` release + +[source,console] +---- +$ stackablectl release uninstall 24.7 + +Uninstalled release '24.7' + +Use "stackablectl release list" to list available releases. +# ... +---- + +Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. +The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`. + +[source] +---- +kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.11.0/deploy/helm/airflow-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.11.0/deploy/helm/commons-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.11.0/deploy/helm/druid-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.11.0/deploy/helm/hbase-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.11.0/deploy/helm/hdfs-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.11.0/deploy/helm/hello-world-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.11.0/deploy/helm/hive-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.11.0/deploy/helm/kafka-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.11.0/deploy/helm/listener-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.11.0/deploy/helm/nifi-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.11.0/deploy/helm/opa-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.11.0/deploy/helm/secret-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.11.0/deploy/helm/superset-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.11.0/deploy/helm/trino-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml +---- + +[source,console] +---- +customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced +... +---- + +Install the `24.11` release + +[source,console] +---- +$ stackablectl release install 24.11 + +Installed release '24.11' + +Use "stackablectl operator installed" to list installed operators. +---- + ==== Using Helm +Use `helm list` to list the currently installed operators. + +You can use the following command to uninstall all operators that are part of the `24.3` release: + +[source,console] +---- +$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hello-world-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator +release "airflow-operator" uninstalled +release "commons-operator" uninstalled +... +---- + +Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. +The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`: + +[source] +---- +kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.11.0/deploy/helm/airflow-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.11.0/deploy/helm/commons-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.11.0/deploy/helm/druid-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.11.0/deploy/helm/hbase-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.11.0/deploy/helm/hdfs-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.11.0/deploy/helm/hello-world-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.11.0/deploy/helm/hive-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.11.0/deploy/helm/kafka-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.11.0/deploy/helm/listener-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.11.0/deploy/helm/nifi-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.11.0/deploy/helm/opa-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.11.0/deploy/helm/secret-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.11.0/deploy/helm/superset-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.11.0/deploy/helm/trino-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml +---- + +[source,console] +---- +customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced +... +---- + +Install the `24.11` release + +[source,console] +---- +helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/ +helm repo update stackable-stable +helm install --wait airflow-operator stackable-stable/airflow-operator --version 24.11.0 +helm install --wait commons-operator stackable-stable/commons-operator --version 24.11.0 +helm install --wait druid-operator stackable-stable/druid-operator --version 24.11.0 +helm install --wait hbase-operator stackable-stable/hbase-operator --version 24.11.0 +helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 24.11.0 +helm install --wait hive-operator stackable-stable/hive-operator --version 24.11.0 +helm install --wait kafka-operator stackable-stable/kafka-operator --version 24.11.0 +helm install --wait listener-operator stackable-stable/listener-operator --version 24.11.0 +helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 24.11.0 +helm install --wait nifi-operator stackable-stable/nifi-operator --version 24.11.0 +helm install --wait opa-operator stackable-stable/opa-operator --version 24.11.0 +helm install --wait secret-operator stackable-stable/secret-operator --version 24.11.0 +helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 24.11.0 +helm install --wait superset-operator stackable-stable/superset-operator --version 24.11.0 +helm install --wait trino-operator stackable-stable/trino-operator --version 24.11.0 +helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 24.11.0 +---- + ==== Known upgrade issues -===== All operators +> todo: mention JMX Exporter? +> todo: do we mention NiFi 2.0.0 (experimental)? because NiFi 1.27.0 -> 2.0.0 requires manual intervention (basically same as in the 24.7 note) == Release 24.7 From 50bbb40f3177a6e1ddb0c7fe6926a1db88d9f5c5 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 13:14:05 +0100 Subject: [PATCH 27/61] Apply suggestions from code review Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Co-authored-by: Techassi --- modules/ROOT/pages/release-notes.adoc | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index d0d638573..1932a1317 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -46,7 +46,7 @@ Here are the headings you can use for the next release. Saves time checking inde Vulnerabilities:: -* More than 142 CVEs were fixed in the Stackable product images. This includes 11 CVEs of critical and 55 CVEs of high severity. +* More than 142 CVEs were fixed in the Stackable product images. This includes 11 critical and 55 high-severity CVEs. Improved Authentication:: @@ -67,13 +67,17 @@ Logging:: Monitoring:: -* In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX metrics in Prometheus and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not yet released. This SDP release 24.11 contains a fixed version bringing performance back to normal levels. +* https://github.com/prometheus/jmx_exporter[JMX Exporter] is a tool which allows us to expose JMX metrics as Prometheus metrics. + It is used by the following products: Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. + In the previous SDP release (24.7) we upgraded JMX Exporter from 0.20 to 1.0.1. + Unfortunately version 1.0.1 has a severe performance degradation which has been https://github.com/prometheus/jmx_exporter/pull/1009[fixed upstream] but is not yet released. + This SDP release (24.11) contains a fixed version bringing performance back to normal levels. Security:: -* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager. -* Added support for customizing sAMAccountName generation in secret operator. -* The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with `listeners.stackable.tech/listener-name`. +* The Stackable Data Platform now supports provisioning TLS certificates using https://cert-manager.io/[cert-manager]. +* Added support for customizing `sAMAccountName` generation in secret operator. +* The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with the `listeners.stackable.tech/listener-name` annotation. Commons:: @@ -87,7 +91,7 @@ Listener:: Dependencies:: -* Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. See the usage guide for detailed information. +* Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. Operations:: @@ -95,9 +99,11 @@ Operations:: Misc:: -* Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. See documentation https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check[here]. -* Apache Airflow: Allow custom arbitrary python code in webserver_config.py. -* Apache Superset: Allow custom arbitrary python code in superset_config.py +* Apache NiFi: Permit users to configure allowed hosts when NiFi is running behind a proxy. + The proxy host check is now disabled by default. + See documentation xref:nifi:usage_guide/security#host-header-check[here]. +* Apache Airflow: Allow custom arbitrary python code in `webserver_config.py`. +* Apache Superset: Allow custom arbitrary python code in `superset_config.py`. * The size of the operator deployed CRDs was reduced significantly https://github.com/stackabletech/issues/issues/627[here]. Images:: From 0a3701dcbfe0f89fd3dad06c89a2ff04613197ca Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 13:19:30 +0100 Subject: [PATCH 28/61] mention stackablectl patch release fixes --- modules/ROOT/pages/release-notes.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 1932a1317..974b1761e 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -167,6 +167,7 @@ The following product versions are no longer supported (although images for rele === stackablectl * Bump Rust dependencies to fix critical vulnerability in quinn-proto, see https://github.com/advisories/GHSA-vr26-jcq5-fjj8[CVE-2024-45311] (https://github.com/stackabletech/stackable-cockpit/pull/318). +* We now provide additional completions for Nushell and Elvish, support using SOCK5 and HTTP proxies, and improved the sorting of release versions. === Supported Kubernetes versions From dde6742af142063add32d068e607b85cfc86086a Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 13:21:07 +0100 Subject: [PATCH 29/61] fix link rendering --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 974b1761e..7b3a5da95 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -58,7 +58,7 @@ In this release we introduced several authentication mechanisms in different pro Improved Authorization:: -* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. +* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#\_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging:: From e70d4807a0ea5c40941ab56bce82d40a6138277c Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 13:27:23 +0100 Subject: [PATCH 30/61] add docs link for hdfs upgrade --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 7b3a5da95..2014a1582 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -95,7 +95,7 @@ Dependencies:: Operations:: -* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. This process requires some manual intervention, however. +* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. However, this process requires some manual intervention as described https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/[here]. Misc:: From 3fd9393fcfe2283cfa85e6c12dc18be1b26c28dc Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Thu, 21 Nov 2024 14:21:52 +0100 Subject: [PATCH 31/61] move new sentences to new lines --- modules/ROOT/pages/release-notes.adoc | 36 +++++++++++++++++++-------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 2014a1582..37239e99e 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -58,12 +58,14 @@ In this release we introduced several authentication mechanisms in different pro Improved Authorization:: -* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#\_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. +* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#\_authorization[OPA Authorizer] has been greatly improved. + This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging:: -* Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. +* Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. + Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. Monitoring:: @@ -91,11 +93,14 @@ Listener:: Dependencies:: -* Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. +* Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. + This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. + See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. Operations:: -* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. However, this process requires some manual intervention as described https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/[here]. +* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. + However, this process requires some manual intervention as described https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/[here]. Misc:: @@ -109,15 +114,21 @@ Misc:: Images:: * Our Docker images now exclusively make use of numeric user IDs in `USER` statements allowing the use of `securityContext.runAsNonRoot` -* The group id of all files relevant to our products is now set to `0`. This allows the images to be used with any arbitrary user as every container user will always belong to the root group (`0`). This is especially useful on OpenShift when trying to move to the `restricted-v2` SecurityContextConstraint (SCC), Stackable currently defaults to the `nonroot-v2` SCC but we plan on migrating to `restricted-v2` in the future +* The group id of all files relevant to our products is now set to `0`. + This allows the images to be run with an arbitrary user as every container user will always belong to the root group (`0`). + This is required on OpenShift when migrating to the `restricted-v2` SecurityContextConstraint (SCC). +Stackable currently defaults to the `nonroot-v2` SCC but we plan on migrating to the `restricted-v2` SCC in the future. Bug fixes:: -* Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. +* Apache Spark Operator: Ensure Spark applications are submitted only once. + Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. + This behavior was triggered by different situations, such as when the operator was restarted. * Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed -* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[here]. -* Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. +* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described in https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[Configuring the Kubernetes cluster domain]. +* Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. + This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. * Apache HBase: The operator now does not https://github.com/stackabletech/hbase-operator/pull/584[ignore the `hbaseRootdir` config property at role level]. * Apache Kakfa: The bootstrap Kafka service is now included in https://github.com/stackabletech/kafka-operator/pull/741[certificate SANs]. @@ -180,7 +191,8 @@ This release supports the following Kubernetes versions: These Kubernetes versions are no longer supported: * `1.26` -* `1.25` as we removed internal forks required to support Kubernetes `1.25` and below. This includes OpenShift `4.12`, which is using Kubernetes `1.25`. +* `1.25` as we removed internal forks required to support Kubernetes `1.25` and below. + This includes OpenShift `4.12`, which is using Kubernetes `1.25`. === Supported OpenShift versions @@ -198,7 +210,8 @@ These OpenShift versions are no longer supported: ==== Listener operator -* BREAKING: All ListenerClasses now default to using DNS hostnames, previously NodePort ListenerClasses (such as external-unstable) would use IP addresses. Hence, all Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured to set .spec.preferredAddressType: IP. +* BREAKING: All ListenerClasses now default to using DNS hostnames, previously NodePort ListenerClasses (such as external-unstable) would use IP addresses. + Hence, all Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured to set `.spec.preferredAddressType: IP`. === Upgrade from 24.7 @@ -217,7 +230,8 @@ Use "stackablectl release list" to list available releases. ---- Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. -The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`. +The reason for this is that helm will uninstall the operators but not the CRDs. +This can be done using `kubectl replace`. [source] ---- From 6671d106710753a85231562d6f9289ea8f264e6a Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Thu, 21 Nov 2024 14:34:05 +0100 Subject: [PATCH 32/61] Apply suggestions from code review --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 37239e99e..303258bee 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -89,7 +89,7 @@ Listener:: * The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized. * The `ListenerClass.spec.serviceAnnotations` are now correctly propagated to created Service objects. -* Listeners can now be configured to use either IP addresses or DNS hostnames. +* Listeners can now be configured to use either IP addresses or fully qualified domain names (FQDNs). Dependencies:: From 08aa45a0e553b713fa7b9be08f78331f41e40621 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Thu, 21 Nov 2024 14:52:59 +0100 Subject: [PATCH 33/61] Apply suggestions from code review --- modules/ROOT/pages/release-notes.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 303258bee..662e26322 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -46,7 +46,8 @@ Here are the headings you can use for the next release. Saves time checking inde Vulnerabilities:: -* More than 142 CVEs were fixed in the Stackable product images. This includes 11 critical and 55 high-severity CVEs. +* More than 142 CVEs were fixed in the Stackable product images. + This includes 11 critical and 55 high-severity CVEs. Improved Authentication:: From 2a43d4a741a154ae2ee4f5d799a83c1c80be1f6b Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Thu, 21 Nov 2024 15:11:08 +0100 Subject: [PATCH 34/61] restructure the headings, remove unordered lists with only single items --- modules/ROOT/pages/release-notes.adoc | 103 +++++++++++++++----------- 1 file changed, 59 insertions(+), 44 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 662e26322..75370bb2f 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -10,7 +10,11 @@ Here are the headings you can use for the next release. Saves time checking inde == Release YY.M -=== New / extended platform features +=== New platform features + +=== Platform improvements + +=== Platform deprecations === Product versions @@ -36,20 +40,13 @@ Here are the headings you can use for the next release. Saves time checking inde ==== Known upgrade issues -===== All operators - //// == Release 24.11 -=== New / extended platform features - -Vulnerabilities:: - -* More than 142 CVEs were fixed in the Stackable product images. - This includes 11 critical and 55 high-severity CVEs. +=== New platform features -Improved Authentication:: +Authentication:: In this release we introduced several authentication mechanisms in different products: @@ -57,51 +54,27 @@ In this release we introduced several authentication mechanisms in different pro * Apache Kafka: https://github.com/stackabletech/kafka-operator/issues/655[Kerberos support] * Apache NiFi: https://github.com/stackabletech/nifi-operator/issues/633[OIDC support] -Improved Authorization:: - -* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#\_authorization[OPA Authorizer] has been greatly improved. - This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. -* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. - -Logging:: - -* Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. - Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. - -Monitoring:: - -* https://github.com/prometheus/jmx_exporter[JMX Exporter] is a tool which allows us to expose JMX metrics as Prometheus metrics. - It is used by the following products: Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. - In the previous SDP release (24.7) we upgraded JMX Exporter from 0.20 to 1.0.1. - Unfortunately version 1.0.1 has a severe performance degradation which has been https://github.com/prometheus/jmx_exporter/pull/1009[fixed upstream] but is not yet released. - This SDP release (24.11) contains a fixed version bringing performance back to normal levels. - Security:: * The Stackable Data Platform now supports provisioning TLS certificates using https://cert-manager.io/[cert-manager]. * Added support for customizing `sAMAccountName` generation in secret operator. * The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with the `listeners.stackable.tech/listener-name` annotation. -Commons:: - -* Pod Enrichment is now deprecated, and will be removed in the next release. Once removed, the SDP will no longer set any `enrichment.stackable.tech/` annotations on Pods. - Listener:: * The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized. -* The `ListenerClass.spec.serviceAnnotations` are now correctly propagated to created Service objects. * Listeners can now be configured to use either IP addresses or fully qualified domain names (FQDNs). Dependencies:: -* Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. - This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. - See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. +Apache HBase: The hadoop-azure module was added to the image and is contained in the classpath. +This makes it possible to use the Azure Data Lake Storage Gen2 (ADLS) instead of HDFS. +See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. Operations:: -* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. - However, this process requires some manual intervention as described https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/[here]. +The Stackable Operator for HDFS now supports upgrading existing HDFS installations. +However, this process requires some manual intervention as described https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/[here]. Misc:: @@ -110,15 +83,50 @@ Misc:: See documentation xref:nifi:usage_guide/security#host-header-check[here]. * Apache Airflow: Allow custom arbitrary python code in `webserver_config.py`. * Apache Superset: Allow custom arbitrary python code in `superset_config.py`. -* The size of the operator deployed CRDs was reduced significantly https://github.com/stackabletech/issues/issues/627[here]. Images:: +Support the `restricted-v2` SecurityContextConstraint (SCC) in OpenShift. +Stackable currently defaults to the `nonroot-v2` SCC but we plan on migrating to the `restricted-v2` SCC in the future. + * Our Docker images now exclusively make use of numeric user IDs in `USER` statements allowing the use of `securityContext.runAsNonRoot` * The group id of all files relevant to our products is now set to `0`. This allows the images to be run with an arbitrary user as every container user will always belong to the root group (`0`). - This is required on OpenShift when migrating to the `restricted-v2` SecurityContextConstraint (SCC). -Stackable currently defaults to the `nonroot-v2` SCC but we plan on migrating to the `restricted-v2` SCC in the future. + This is required on OpenShift when migrating to the `restricted-v2` SCC. + +=== Platform improvements + +Vulnerabilities:: + +More than 142 CVEs were fixed in the Stackable product images. +This includes 11 critical and 55 high-severity CVEs. + +Authorization:: + +* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#\_authorization[OPA Authorizer] has been greatly improved. + This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. +* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. + +Logging:: + +Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. +Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. + +Monitoring:: + +https://github.com/prometheus/jmx_exporter[JMX Exporter] is a tool which allows us to expose JMX metrics as Prometheus metrics. +It is used by the following products: Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. +In the previous SDP release (24.7) we upgraded JMX Exporter from 0.20 to 1.0.1. +Unfortunately version 1.0.1 has a severe performance degradation which has been https://github.com/prometheus/jmx_exporter/pull/1009[fixed upstream] but is not yet released. +This SDP release (24.11) contains a fixed version bringing performance back to normal levels. + +Listener:: + +The `ListenerClass.spec.serviceAnnotations` are now correctly propagated to created Service objects. + +Misc:: + +The size of the operator deployed CRDs was reduced significantly (see: https://github.com/stackabletech/issues/issues/627[stackabletech/issues#627]). Bug fixes:: @@ -134,6 +142,13 @@ Bug fixes:: * Apache HBase: The operator now does not https://github.com/stackabletech/hbase-operator/pull/584[ignore the `hbaseRootdir` config property at role level]. * Apache Kakfa: The bootstrap Kafka service is now included in https://github.com/stackabletech/kafka-operator/pull/741[certificate SANs]. +=== Platform deprecations + +Commons:: + +Pod Enrichment is now deprecated, and will be removed in the next release. +Once removed, the SDP will no longer set any `enrichment.stackable.tech/` annotations on Pods. + === Product versions As with previous SDP releases, many product images have been updated to their latest versions. @@ -211,8 +226,8 @@ These OpenShift versions are no longer supported: ==== Listener operator -* BREAKING: All ListenerClasses now default to using DNS hostnames, previously NodePort ListenerClasses (such as external-unstable) would use IP addresses. - Hence, all Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured to set `.spec.preferredAddressType: IP`. +BREAKING: All ListenerClasses now default to using DNS hostnames, previously NodePort ListenerClasses (such as external-unstable) would use IP addresses. +Hence, all Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured to set `.spec.preferredAddressType: IP`. === Upgrade from 24.7 From c8f0104f8d180fc238490ad3074aa2532b333996 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 16:53:24 +0100 Subject: [PATCH 35/61] added last of missing PR/Issues --- modules/ROOT/pages/release-notes.adoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 75370bb2f..4d8d8d0e8 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -59,6 +59,7 @@ Security:: * The Stackable Data Platform now supports provisioning TLS certificates using https://cert-manager.io/[cert-manager]. * Added support for customizing `sAMAccountName` generation in secret operator. * The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with the `listeners.stackable.tech/listener-name` annotation. +* The RSA key length for generated key pairs now can be customized to 2048, 3072 and 4096 bits. The default is 2048 bit. Listener:: @@ -111,6 +112,7 @@ Logging:: Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. +Apache NiFi: Support disabling the create-reporting-task Job as well as podOverrides on that Job. Monitoring:: @@ -133,14 +135,14 @@ Bug fixes:: * Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. -* Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property. -* Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed +* Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed. * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described in https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[Configuring the Kubernetes cluster domain]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. * Apache HBase: The operator now does not https://github.com/stackabletech/hbase-operator/pull/584[ignore the `hbaseRootdir` config property at role level]. * Apache Kakfa: The bootstrap Kafka service is now included in https://github.com/stackabletech/kafka-operator/pull/741[certificate SANs]. +* Trino: Do not print credentials to STDOUT during startup. === Platform deprecations From 8c186246c196e6c84516bb6e56daa9574baf4d90 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 17:01:25 +0100 Subject: [PATCH 36/61] mention nifi reporting task regression --- modules/ROOT/pages/release-notes.adoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 4d8d8d0e8..858cc2d4c 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -363,8 +363,11 @@ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --ver ==== Known upgrade issues -> todo: mention JMX Exporter? -> todo: do we mention NiFi 2.0.0 (experimental)? because NiFi 1.27.0 -> 2.0.0 requires manual intervention (basically same as in the 24.7 note) +> todo: mention JMX Exporter? Malte: Its mentioned in the Monitoring section and not really an upgrade issue? + +* Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask`, use to activate a Prometheus metrics endpoint, was removed. + NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication which is + not supported by the Stackable operator for Apache NiFi in this release. == Release 24.7 From ad0bc5686ac0e57a6c3492f8689a5c7a39d0d8e1 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 18:16:43 +0100 Subject: [PATCH 37/61] mention nifi OIDC problems --- modules/ROOT/pages/release-notes.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 858cc2d4c..f7b69c8a7 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -364,7 +364,9 @@ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --ver ==== Known upgrade issues > todo: mention JMX Exporter? Malte: Its mentioned in the Monitoring section and not really an upgrade issue? - +> todo: Malte: This is not really an upgrade issue? +* The Apache NiFi operator currently cannot share an https://github.com/stackabletech/nifi-operator/issues/716[OIDC AuthenticationClass with other products]. + This is due to an inconsistent implementation in the NiFi operator. * Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask`, use to activate a Prometheus metrics endpoint, was removed. NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication which is not supported by the Stackable operator for Apache NiFi in this release. From 0c65f8b943f23b1733fa530c8e01f6f9bd5c0b30 Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 18:21:58 +0100 Subject: [PATCH 38/61] attempt to fix linter --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index f7b69c8a7..7c3d8d9ba 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -59,7 +59,7 @@ Security:: * The Stackable Data Platform now supports provisioning TLS certificates using https://cert-manager.io/[cert-manager]. * Added support for customizing `sAMAccountName` generation in secret operator. * The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with the `listeners.stackable.tech/listener-name` annotation. -* The RSA key length for generated key pairs now can be customized to 2048, 3072 and 4096 bits. The default is 2048 bit. +* The RSA key length for generated key pairs now can be customized to 2048, 3072 and 4096 bits. The default is 2048 bit. Listener:: From e8763db6672529d7b8583f14fe540b6850c1194e Mon Sep 17 00:00:00 2001 From: Malte Sander Date: Thu, 21 Nov 2024 18:46:16 +0100 Subject: [PATCH 39/61] attempt to fix linter 2 --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 7c3d8d9ba..c9ade8111 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -367,7 +367,7 @@ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --ver > todo: Malte: This is not really an upgrade issue? * The Apache NiFi operator currently cannot share an https://github.com/stackabletech/nifi-operator/issues/716[OIDC AuthenticationClass with other products]. This is due to an inconsistent implementation in the NiFi operator. -* Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask`, use to activate a Prometheus metrics endpoint, was removed. +* Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask`, use to activate a Prometheus metrics endpoint, was removed. NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication which is not supported by the Stackable operator for Apache NiFi in this release. From 62e1943b41ca6515661ff936436c386049a1ca4b Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 10:19:10 +0100 Subject: [PATCH 40/61] Add notes about the service account and OIDC bugs that will appear in the patch release --- modules/ROOT/pages/release-notes.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index c9ade8111..7983ae8f5 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -38,7 +38,7 @@ Here are the headings you can use for the next release. Saves time checking inde ==== Using Helm -==== Known upgrade issues +=== Known issues //// @@ -361,15 +361,19 @@ helm install --wait trino-operator stackable-stable/trino-operator --version 24. helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 24.11.0 ---- -==== Known upgrade issues +=== Known issues -> todo: mention JMX Exporter? Malte: Its mentioned in the Monitoring section and not really an upgrade issue? -> todo: Malte: This is not really an upgrade issue? * The Apache NiFi operator currently cannot share an https://github.com/stackabletech/nifi-operator/issues/716[OIDC AuthenticationClass with other products]. This is due to an inconsistent implementation in the NiFi operator. * Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask`, use to activate a Prometheus metrics endpoint, was removed. NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication which is not supported by the Stackable operator for Apache NiFi in this release. +* https://github.com/stackabletech/issues/issues/675[serviceAccount and roleBinding objects can accidentally get deleted]: + During the release, a bug was reported which affects multiple deployments of the same product sharing a namespace. + This is actively being worked on and will appear in a patch release. +* https://github.com/stackabletech/operator-rs/pull/910[fix!: Correctly construct OIDC endpoints]: + During the release, it was discovered that some OIDC endpoint URLs were constructed in a way that was not compatible with some tooling. + This has been fixed and will appear in a patch release. == Release 24.7 From ef4fadecdad50cb2b43844f0b1c3a72dc63baae4 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 10:27:06 +0100 Subject: [PATCH 41/61] make a sentence shorter --- modules/ROOT/pages/release-notes.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 7983ae8f5..08f4199b1 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -136,7 +136,8 @@ Bug fixes:: Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed. -* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described in https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[Configuring the Kubernetes cluster domain]. +* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators. + Either set the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or the helm value `kubernetesClusterDomain` during installation as described in https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[Configuring the Kubernetes cluster domain]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. From a0eb68f299bcc57bf5bd46e57dface41f75ca3a5 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 10:57:46 +0100 Subject: [PATCH 42/61] Use xrefs instead of direct links to docs --- modules/ROOT/pages/release-notes.adoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 08f4199b1..48035955d 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -75,7 +75,7 @@ See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. Operations:: The Stackable Operator for HDFS now supports upgrading existing HDFS installations. -However, this process requires some manual intervention as described https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/[here]. +However, this process requires some manual intervention as described in xref:hbase:usage-guide/upgrading.adoc[Upgrading HDFS]. Misc:: @@ -104,9 +104,9 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: -* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#\_authorization[OPA Authorizer] has been greatly improved. +* The performance of the xref:hdfs:usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. -* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. +* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging:: @@ -137,7 +137,7 @@ Bug fixes:: This behavior was triggered by different situations, such as when the operator was restarted. * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed. * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators. - Either set the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or the helm value `kubernetesClusterDomain` during installation as described in https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain[Configuring the Kubernetes cluster domain]. + Either set the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or the helm value `kubernetesClusterDomain` during installation as described in xref:guides/kubernetes-cluster-domain.adoc[Configuring the Kubernetes cluster domain]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. @@ -155,9 +155,9 @@ Once removed, the SDP will no longer set any `enrichment.stackable.tech/` annota === Product versions As with previous SDP releases, many product images have been updated to their latest versions. -The LTS version has in many cases also been adjusted in line with our https://docs.stackable.tech/home/stable/policies[support policy]. +The LTS version has in many cases also been adjusted in line with our xref:ROOT:policies.adoc[support policy]. -Refer to the https://docs.stackable.tech/home/stable/operators/supported_versions/[supported versions] documentation for a complete overview including LTS versions or deprecations. +Refer to the xref:operators/supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. ==== New versions From 63bc9c4cac6c8beb5aee5e1ef984c0e1c3d9f062 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 11:10:54 +0100 Subject: [PATCH 43/61] fix xrefs and use instead of _ for emphasis --- modules/ROOT/pages/release-notes.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 48035955d..6f8a683fe 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -75,7 +75,7 @@ See the xref:hbase:usage-guide/adls.adoc[usage guide] for detailed information. Operations:: The Stackable Operator for HDFS now supports upgrading existing HDFS installations. -However, this process requires some manual intervention as described in xref:hbase:usage-guide/upgrading.adoc[Upgrading HDFS]. +However, this process requires some manual intervention as described in xref:hdfs:usage-guide/upgrading.adoc[Upgrading HDFS]. Misc:: @@ -105,7 +105,7 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: * The performance of the xref:hdfs:usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. - This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. + This can be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging:: @@ -137,7 +137,7 @@ Bug fixes:: This behavior was triggered by different situations, such as when the operator was restarted. * Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed. * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators. - Either set the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or the helm value `kubernetesClusterDomain` during installation as described in xref:guides/kubernetes-cluster-domain.adoc[Configuring the Kubernetes cluster domain]. + Either set the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or the helm value `kubernetesClusterDomain` during installation as described in xref:guides:kubernetes-cluster-domain.adoc[Configuring the Kubernetes cluster domain]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. @@ -157,7 +157,7 @@ Once removed, the SDP will no longer set any `enrichment.stackable.tech/` annota As with previous SDP releases, many product images have been updated to their latest versions. The LTS version has in many cases also been adjusted in line with our xref:ROOT:policies.adoc[support policy]. -Refer to the xref:operators/supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. +Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. ==== New versions From eb27ab0a2396d69bd7a28231f7f83f2af9b019be Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 11:16:53 +0100 Subject: [PATCH 44/61] replace html suffix with adoc --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 6f8a683fe..825b9b030 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -104,7 +104,7 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: -* The performance of the xref:hdfs:usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. +* The performance of the xref:hdfs:usage-guide/security.adoc#_authorization[OPA Authorizer] has been greatly improved. This can be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. From 922937232e3af365ee9efb8100302adc1340b381 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 11:42:06 +0100 Subject: [PATCH 45/61] Try to fix the emphasis --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 825b9b030..7bdc5e358 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -105,7 +105,7 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: * The performance of the xref:hdfs:usage-guide/security.adoc#_authorization[OPA Authorizer] has been greatly improved. - This can be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. + This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging:: From 9c06fe977ded0af3649113bad6de0c15e6026d35 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 11:52:33 +0100 Subject: [PATCH 46/61] Escape _ in one URL fragment to preent the emphasis breaking --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 7bdc5e358..550837680 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -104,7 +104,7 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: -* The performance of the xref:hdfs:usage-guide/security.adoc#_authorization[OPA Authorizer] has been greatly improved. +* The performance of the xref:hdfs:usage-guide/security.adoc#\_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. From a7a9439c1fc36e9ec063a5432b0aae00ef538555 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 12:01:09 +0100 Subject: [PATCH 47/61] Reword to remove broken emphasis --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 550837680..7c4f11718 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -105,7 +105,7 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: * The performance of the xref:hdfs:usage-guide/security.adoc#\_authorization[OPA Authorizer] has been greatly improved. - This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. + This can in some cases be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. Logging:: From 99cb96e2b91edb36b4a2d7ef800698ea44180c2a Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 12:41:37 +0100 Subject: [PATCH 48/61] Apply suggestions from code review Thanks @adwk67 Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com> --- modules/ROOT/pages/release-notes.adoc | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 7c4f11718..ad7c59d79 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -59,7 +59,7 @@ Security:: * The Stackable Data Platform now supports provisioning TLS certificates using https://cert-manager.io/[cert-manager]. * Added support for customizing `sAMAccountName` generation in secret operator. * The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with the `listeners.stackable.tech/listener-name` annotation. -* The RSA key length for generated key pairs now can be customized to 2048, 3072 and 4096 bits. The default is 2048 bit. +* The RSA key length for generated key pairs can now be customized to 2048, 3072 or 4096 bits. The default is 2048 bits. Listener:: @@ -110,9 +110,9 @@ Authorization:: Logging:: -Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. -Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds. -Apache NiFi: Support disabling the create-reporting-task Job as well as podOverrides on that Job. +Apache NiFi: The default size of ephemeral EmptyDir Volumes used to store log files before aggregation has been increased from 33 MiB to 500 MiB. +Additionally the interval in which Logback checks if the maximum log file size has been reached has been reduced from 60 seconds to 5 seconds. +Apache NiFi: the create-reporting-task Job (and podOverrides on that Job) can now be disabled. Monitoring:: @@ -135,7 +135,7 @@ Bug fixes:: * Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. -* Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed. +* Trino, Spark, HBase, Airflow: The https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides did not work consistently have now been fixed. * The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators. Either set the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or the helm value `kubernetesClusterDomain` during installation as described in xref:guides:kubernetes-cluster-domain.adoc[Configuring the Kubernetes cluster domain]. * Apache Airflow: In release 24.7 Airflow did not propagate git credentials correctly to the gitsync containers. @@ -197,7 +197,7 @@ The following product versions are no longer supported (although images for rele === stackablectl * Bump Rust dependencies to fix critical vulnerability in quinn-proto, see https://github.com/advisories/GHSA-vr26-jcq5-fjj8[CVE-2024-45311] (https://github.com/stackabletech/stackable-cockpit/pull/318). -* We now provide additional completions for Nushell and Elvish, support using SOCK5 and HTTP proxies, and improved the sorting of release versions. +* We now provide additional completions for Nushell and Elvish, support using SOCK5 and HTTP proxies, and have improved the sorting of release versions. === Supported Kubernetes versions @@ -366,9 +366,9 @@ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --ver * The Apache NiFi operator currently cannot share an https://github.com/stackabletech/nifi-operator/issues/716[OIDC AuthenticationClass with other products]. This is due to an inconsistent implementation in the NiFi operator. -* Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask`, use to activate a Prometheus metrics endpoint, was removed. - NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication which is - not supported by the Stackable operator for Apache NiFi in this release. +* Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask` (which was used to activate a Prometheus metrics endpoint) has been removed. + NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication. + This is not supported by the Stackable operator for Apache NiFi in this release. * https://github.com/stackabletech/issues/issues/675[serviceAccount and roleBinding objects can accidentally get deleted]: During the release, a bug was reported which affects multiple deployments of the same product sharing a namespace. This is actively being worked on and will appear in a patch release. From 0036977c35ed30204a497b313e66055d5ce2034a Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 12:42:10 +0100 Subject: [PATCH 49/61] Apply suggestions from code review Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com> --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index ad7c59d79..afb680024 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -57,7 +57,7 @@ In this release we introduced several authentication mechanisms in different pro Security:: * The Stackable Data Platform now supports provisioning TLS certificates using https://cert-manager.io/[cert-manager]. -* Added support for customizing `sAMAccountName` generation in secret operator. +* Support has been added for customizing `sAMAccountName` generation in secret operator. * The Stackable Secret Operator now requests permission to read Listeners, which is required to provision secrets for listener volumes with the `listeners.stackable.tech/listener-name` annotation. * The RSA key length for generated key pairs can now be customized to 2048, 3072 or 4096 bits. The default is 2048 bits. From 338141162cb9ced129b4da8a4c967f988c502465 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 12:53:53 +0100 Subject: [PATCH 50/61] Remove change from 24.7 release notes --- modules/ROOT/pages/release-notes.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index afb680024..9480ee755 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -422,7 +422,6 @@ The status is still xref:concepts:multi-platform-support.adoc[experimental], as Security:: Support for OIDC with/without TLS has been added to Apache Druid in this release. -* Apache Druid: 26.0.0 NOTE: SDP now provides OIDC-support for Druid, Superset and Trino In this release we provide experimental HBase 2.6.0 support with a new experimental policy based authorizer (with OPA). From 83e56c0234e3ed3233b12052dacf043cb17232d5 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 14:10:17 +0100 Subject: [PATCH 51/61] fix list of changes for logging improvements --- modules/ROOT/pages/release-notes.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 9480ee755..e1d4f6fc6 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -110,9 +110,9 @@ Authorization:: Logging:: -Apache NiFi: The default size of ephemeral EmptyDir Volumes used to store log files before aggregation has been increased from 33 MiB to 500 MiB. -Additionally the interval in which Logback checks if the maximum log file size has been reached has been reduced from 60 seconds to 5 seconds. -Apache NiFi: the create-reporting-task Job (and podOverrides on that Job) can now be disabled. +* Apache NiFi: The default size of ephemeral EmptyDir Volumes used to store log files before aggregation has been increased from 33 MiB to 500 MiB. + Additionally the interval in which Logback checks if the maximum log file size has been reached has been reduced from 60 seconds to 5 seconds. +* Apache NiFi: the create-reporting-task Job (and podOverrides on that Job) can now be disabled. Monitoring:: From 3197e67ddec9050eaf05743741463fedb9094aba Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 14:10:35 +0100 Subject: [PATCH 52/61] update template headings for breaking changes --- modules/ROOT/pages/release-notes.adoc | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index e1d4f6fc6..76fde8dcb 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -32,6 +32,20 @@ Here are the headings you can use for the next release. Saves time checking inde === Breaking changes +Of the changes mentioned above, the following are breaking (or could lead to breaking behaviour), and you will need to adapt your existing CRDs accordingly: + +==== Stackable Operator for Example Product + +* Description of the change 1 +* Description of the change 2 + +.Breaking changes details +[%collapsible] +==== +* `spec.a`: This field has been removed. +* `spec.b`: This field has been changed to a number. +==== + === Upgrade from YY.M ==== Using stackablectl From cd80f378d38a0aa4b91dbf032e7c91092d152acf Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:14:38 +0100 Subject: [PATCH 53/61] Apply suggestions from code review Co-authored-by: Sebastian Bernauer Co-authored-by: Malte Sander --- modules/ROOT/pages/release-notes.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 76fde8dcb..844433310 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -118,7 +118,7 @@ This includes 11 critical and 55 high-severity CVEs. Authorization:: -* The performance of the xref:hdfs:usage-guide/security.adoc#\_authorization[OPA Authorizer] has been greatly improved. +* The performance of the xref:hdfs:usage-guide/security.adoc#\_authorization[HDFS OPA Authorizer] has been greatly improved. This can in some cases be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details. * The User Info Fetcher HTTP API has been replaced with a Rego library. Please see xref:opa:usage-guide/user-info-fetcher#_user_info_fetcher_api[user-info-fetcher API] for more information. @@ -146,7 +146,7 @@ The size of the operator deployed CRDs was reduced significantly (see: https://g Bug fixes:: -* Apache Spark Operator: Ensure Spark applications are submitted only once. +* Apache Spark: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted. * Trino, Spark, HBase, Airflow: The https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides did not work consistently have now been fixed. @@ -156,7 +156,7 @@ Bug fixes:: This has now been corrected and works for both celery- and kubernetes workers. * Operators now do not stop reconciling existing clusters if one of the https://github.com/stackabletech/issues/issues/211[cluster objects cannot be deserialized]. * Apache HBase: The operator now does not https://github.com/stackabletech/hbase-operator/pull/584[ignore the `hbaseRootdir` config property at role level]. -* Apache Kakfa: The bootstrap Kafka service is now included in https://github.com/stackabletech/kafka-operator/pull/741[certificate SANs]. +* Apache Kafka: The bootstrap Kafka service is now included in https://github.com/stackabletech/kafka-operator/pull/741[certificate SANs]. * Trino: Do not print credentials to STDOUT during startup. === Platform deprecations From 01416167eff31e3c3bd11ba26a59f3f681ab373a Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 14:18:32 +0100 Subject: [PATCH 54/61] remove hello-world upgrade instructions --- modules/ROOT/pages/release-notes.adoc | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 844433310..d8a09eabe 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -273,7 +273,6 @@ kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-opera kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.11.0/deploy/helm/druid-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.11.0/deploy/helm/hbase-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.11.0/deploy/helm/hdfs-operator/crds/crds.yaml -kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.11.0/deploy/helm/hello-world-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.11.0/deploy/helm/hive-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.11.0/deploy/helm/kafka-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.11.0/deploy/helm/listener-operator/crds/crds.yaml @@ -314,7 +313,7 @@ You can use the following command to uninstall all operators that are part of th [source,console] ---- -$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hello-world-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator +$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator release "airflow-operator" uninstalled release "commons-operator" uninstalled ... @@ -330,7 +329,6 @@ kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-opera kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.11.0/deploy/helm/druid-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.11.0/deploy/helm/hbase-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.11.0/deploy/helm/hdfs-operator/crds/crds.yaml -kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.11.0/deploy/helm/hello-world-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.11.0/deploy/helm/hive-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.11.0/deploy/helm/kafka-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.11.0/deploy/helm/listener-operator/crds/crds.yaml @@ -366,7 +364,6 @@ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 24.11 helm install --wait hive-operator stackable-stable/hive-operator --version 24.11.0 helm install --wait kafka-operator stackable-stable/kafka-operator --version 24.11.0 helm install --wait listener-operator stackable-stable/listener-operator --version 24.11.0 -helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 24.11.0 helm install --wait nifi-operator stackable-stable/nifi-operator --version 24.11.0 helm install --wait opa-operator stackable-stable/opa-operator --version 24.11.0 helm install --wait secret-operator stackable-stable/secret-operator --version 24.11.0 From 31344b978fbb8038057b8f6442533778302c5a98 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:29:02 +0100 Subject: [PATCH 55/61] Restructure the Breaking changes section --- modules/ROOT/pages/release-notes.adoc | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index d8a09eabe..af4a8d7cf 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -241,10 +241,20 @@ These OpenShift versions are no longer supported: === Breaking changes +Of the changes mentioned above, the following are breaking (or could lead to breaking behaviour), and you will need to adapt your existing CRDs accordingly: + ==== Listener operator -BREAKING: All ListenerClasses now default to using DNS hostnames, previously NodePort ListenerClasses (such as external-unstable) would use IP addresses. -Hence, all Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured to set `.spec.preferredAddressType: IP`. +All ListenerClasses now default to using Fully Qualified Domain Names (FQDNs). +Previously, NodePort ListenerClasses (such as external-unstable) would use the IP addresses. + +All Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured with `spec.preferredAddressType: IP`. + +.Breaking changes details +[%collapsible] +==== +* `spec.preferredAddressType`: Defaults to `HostnameConservative`, but can be set to `Hostname` or `IP`. +==== === Upgrade from 24.7 From 13342b71df4ce5c0646f3adeac09feb6938bfb87 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 14:31:15 +0100 Subject: [PATCH 56/61] add known issue abot nifi 2.0.0 renamed processors --- modules/ROOT/pages/release-notes.adoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index af4a8d7cf..5871eae43 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -245,7 +245,7 @@ Of the changes mentioned above, the following are breaking (or could lead to bre ==== Listener operator -All ListenerClasses now default to using Fully Qualified Domain Names (FQDNs). +All ListenerClasses now default to using Fully Qualified Domain Names (FQDNs). Previously, NodePort ListenerClasses (such as external-unstable) would use the IP addresses. All Nodes must now have resolvable hostnames, or the NodePort ListenerClasses must be configured with `spec.preferredAddressType: IP`. @@ -387,9 +387,12 @@ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --ver * The Apache NiFi operator currently cannot share an https://github.com/stackabletech/nifi-operator/issues/716[OIDC AuthenticationClass with other products]. This is due to an inconsistent implementation in the NiFi operator. -* Apache NiFi: For the experimental NiFi version `2.0.0`, the `PrometheusReportingTask` (which was used to activate a Prometheus metrics endpoint) has been removed. +* Apache NiFi: In the experimental NiFi version `2.0.0`, the `PrometheusReportingTask` (which was used to activate a Prometheus metrics endpoint) has been removed. NiFi now has its own API to directly access metrics. In contrast to previous versions, the metrics endpoints now requires authentication. This is not supported by the Stackable operator for Apache NiFi in this release. +* Apache NiFi: In the experimental NiFi version `2.0.0`, some processors have been renamed or have a different class path since NiFi 1.x.x. + Flows affected by these changes will need manually updating. + See https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[NiFi 2.0.0 Release Notes] for further details. * https://github.com/stackabletech/issues/issues/675[serviceAccount and roleBinding objects can accidentally get deleted]: During the release, a bug was reported which affects multiple deployments of the same product sharing a namespace. This is actively being worked on and will appear in a patch release. From f5c4645ec8786c41b7ceca1270dd65216997104d Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 14:33:58 +0100 Subject: [PATCH 57/61] add accidentally removed newline from previous fixup --- modules/ROOT/pages/release-notes.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 5871eae43..9f68530d2 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -446,6 +446,7 @@ The status is still xref:concepts:multi-platform-support.adoc[experimental], as Security:: Support for OIDC with/without TLS has been added to Apache Druid in this release. + NOTE: SDP now provides OIDC-support for Druid, Superset and Trino In this release we provide experimental HBase 2.6.0 support with a new experimental policy based authorizer (with OPA). From 5055a01b76078c925a395c2e9f61e976852bf262 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:51:34 +0100 Subject: [PATCH 58/61] Apply suggestions from code review add 8 bits Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com> --- modules/ROOT/pages/release-notes.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 9f68530d2..57caf6eb9 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -104,7 +104,7 @@ Images:: Support the `restricted-v2` SecurityContextConstraint (SCC) in OpenShift. Stackable currently defaults to the `nonroot-v2` SCC but we plan on migrating to the `restricted-v2` SCC in the future. -* Our Docker images now exclusively make use of numeric user IDs in `USER` statements allowing the use of `securityContext.runAsNonRoot` +* Our Docker images now exclusively make use of numeric user IDs in `USER` statements allowing the use of `securityContext.runAsNonRoot`. * The group id of all files relevant to our products is now set to `0`. This allows the images to be run with an arbitrary user as every container user will always belong to the root group (`0`). This is required on OpenShift when migrating to the `restricted-v2` SCC. From 3ed9b9f2f1003a5e9a2d0e17730a319bebe1f5d4 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Tue, 26 Nov 2024 15:03:52 +0100 Subject: [PATCH 59/61] partially add a missing breaking change, needs revising. --- modules/ROOT/pages/release-notes.adoc | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 57caf6eb9..40a3c247e 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -243,6 +243,18 @@ These OpenShift versions are no longer supported: Of the changes mentioned above, the following are breaking (or could lead to breaking behaviour), and you will need to adapt your existing CRDs accordingly: +==== Kafka operator + +* Existing services will be migrated to the new format. Clients will need to re-read settings from the discovery configmap. +* Kafka is now only accessible from within the Kubernetes cluster by default. Set listener classes manually to expose it to the outside world (again). +* To complete an upgrade to this kafka-operator, all existing Kafka StatefulSets must be deleted manually. This will cause some downtime. + +.Breaking changes details +[%collapsible] +==== +* `spec.blah`: Describe a change. +==== + ==== Listener operator All ListenerClasses now default to using Fully Qualified Domain Names (FQDNs). From 1411d556c00c76d516375f4cb8ab1f87e0235c0c Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 16:05:47 +0100 Subject: [PATCH 60/61] Apply suggestions from code review Co-authored-by: Malte Sander --- modules/ROOT/pages/release-notes.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 40a3c247e..0f58a61b7 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -91,7 +91,7 @@ Operations:: The Stackable Operator for HDFS now supports upgrading existing HDFS installations. However, this process requires some manual intervention as described in xref:hdfs:usage-guide/upgrading.adoc[Upgrading HDFS]. -Misc:: +Miscellaneous:: * Apache NiFi: Permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now disabled by default. @@ -140,7 +140,7 @@ Listener:: The `ListenerClass.spec.serviceAnnotations` are now correctly propagated to created Service objects. -Misc:: +Miscellaneous:: The size of the operator deployed CRDs was reduced significantly (see: https://github.com/stackabletech/issues/issues/627[stackabletech/issues#627]). From 12b6913d3396443192dddf412fdb6c83cc9dbd23 Mon Sep 17 00:00:00 2001 From: Nick <10092581+NickLarsenNZ@users.noreply.github.com> Date: Tue, 26 Nov 2024 16:19:28 +0100 Subject: [PATCH 61/61] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Natalie Klestrup Röijezon --- modules/ROOT/pages/release-notes.adoc | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 0f58a61b7..981e77830 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -245,14 +245,33 @@ Of the changes mentioned above, the following are breaking (or could lead to bre ==== Kafka operator -* Existing services will be migrated to the new format. Clients will need to re-read settings from the discovery configmap. -* Kafka is now only accessible from within the Kubernetes cluster by default. Set listener classes manually to expose it to the outside world (again). -* To complete an upgrade to this kafka-operator, all existing Kafka StatefulSets must be deleted manually. This will cause some downtime. +* Existing Kafka clusters will need to be migrated to using the Listener Operator. + Kafka clients will need to re-read settings from the discovery configmap (restart required). + Existing Kafka StatefulSets must be deleted manually. This will cause some downtime. +* Kafka is now only accessible from within the Kubernetes cluster by default. .Breaking changes details [%collapsible] ==== -* `spec.blah`: Describe a change. +Migrating Kafka clusters to use the Listener Operator is done by deleting the Kafka StatefulSet after the new Stackable Operator for Kafka has been installed, by running the following: + +[source,console] +.... +kubectl delete --all-namespaces StatefulSet --selector=app.kubernetes.io/managed-by=kafka.stackable.tech_kafkacluster +.... + +The operator will then recreate it. +Please note that the Kafka cluster will be unavailable during the procedure. + +After the upgrade, Kafka clusters will default to only being accessible from inside the Kubernetes cluster. + +To make the cluster accessible from the outside, set the following _before deleting the StatefulSet_: + +* `KafkaCluster.spec.brokers.config.bootstrapListenerClass: external-stable` +* `KafkaCluster.spec.brokers.config.brokerListenerClass: external-unstable` + +Please note that this upgrade will randomize the address that users will have to connect to, so any external clients must re-read it from the discovery configuration after the upgrade has been completed. +This can be done by restarting the client. ==== ==== Listener operator