Add 24.11 Release Notes

modules/ROOT/pages/release-notes.adoc

@@ -40,6 +40,136 @@ Here are the headings you can use for the next release. Saves time checking inde
 
 ////
 
+== Release 24.11
+
+=== New / extended platform features
+
+Improved Authentication:
+
+In this release we introduced several authentication mechanisms in different products:
+
+* Apache Airflow: https://github.com/stackabletech/airflow-operator/issues/337[OIDC support]
+* Apache Kafka: https://github.com/stackabletech/kafka-operator/issues/655[Kerberos support]
+* Apache NiFi: https://github.com/stackabletech/nifi-operator/issues/633[OIDC support]
+
+Improved Authorization:
+
+* The performance of the https://docs.stackable.tech/home/stable/hdfs/usage-guide/security.html#_authorization[OPA Authorizer] has been greatly improved. This _can_ be a breaking change so please make sure to read the hdfs-utils https://github.com/stackabletech/hdfs-utils/releases/tag/v0.4.0[release notes] for details.
+* The User Info Fetcher HTTP API has been replaced with a Rego library. Please see https://docs.stackable.tech/home/nightly/opa/usage-guide/user-info-fetcher#_user_info_fetcher_api for more information
+
+Logging:
+
+* Apache NiFi: The ephemeral EmptyDir Volumes used to store log files before being aggregated have their size increased from a default of 33 MiB to 500 MiB. Additionally the interval in which Logback checks if the maximum log file size has been reached was lowered from 60 seconds to 5 seconds.
+
+Monitoring:
+
+* In SDP 24.7 we upgraded the version of [JMX Exporter](https://github.com/prometheus/jmx_exporter) from 0.20 to 1.0.1. This is the tool which allows us to expose JMX metrics in Prometheus and is in use for Hadoop, HBase, Hive, Kafka, Spark, Trino and ZooKeeper. Unfortunately the version 1.0.1 has a severe performance degradation which has been [fixed upstream](https://github.com/prometheus/jmx_exporter/pull/1009) but is not yet released. This SDP release 24.11 contains a fixed version bringing performance back to normal levels.
+
+Security:
+
+* The Stackable Data Platform now supports provisioning TLS certificates using cert-manager.
+* Added support for customizing sAMAccountName generation in secret operator.
+
+Listener:
+
+* The Stackable Operator for Kafka now uses the Stackable Listener Operator, allowing connectivity to be customized.
+
+Operations:
+
+* The Stackable Operator for HDFS now supports upgrading existing HDFS installations. This process requires some manual intervention, however.
+
+Misc:
+
+* Apache NiFi: permit users to configure allowed hosts when NiFi is running behind a proxy. The proxy host check is now turned off by default. Documentation: https://docs.stackable.tech/home/nightly/nifi/usage_guide/security#host-header-check
+
+Bug fixes:
+
+* Apache Spark Operator: Ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted.
+* Apache Spark Operator: Environment variables can now be overridden with the role group’s envOverrides property.
+* Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed
+* The cluster domain (default `cluster.local`) which caused problems in non-default cluster setups can now be configured in all operators using the ENV variable `KUBERNETES_CLUSTER_DOMAIN` or setting the helm value `kubernetesClusterDomain` during installation as described [here](https://docs.stackable.tech/home/nightly/guides/kubernetes-cluster-domain).
+
+=== Product versions
+
+As with previous SDP releases, many product images have been updated to their latest versions.
+The LTS version has in many cases also been adjusted in line with our https://docs.stackable.tech/home/stable/policies[support policy].
+
+Refer to the https://docs.stackable.tech/home/stable/operators/supported_versions/[supported versions] documentation for a complete overview including LTS versions or deprecations.
+
+==== New versions
+
+The following new product versions are now supported:
+
+* Apache Airflow: https://github.com/stackabletech/airflow-operator/pull/494[2.9.3 (LTS)], https://github.com/stackabletech/airflow-operator/pull/512[2.10.2 (experimental)]
+* Apache Druid: https://github.com/stackabletech/druid-operator/pull/631[30.0.0 (LTS)]
+* Apache Hive: https://github.com/stackabletech/hive-operator/pull/508[4.0.0 (experimental)]
+* Apache Kafka: https://github.com/stackabletech/kafka-operator/pull/753/[3.8.0]
+* Apache NiFi: https://github.com/stackabletech/nifi-operator/pull/702[2.0.0 (experimental)]
+* Open Policy Agent: https://github.com/stackabletech/opa-operator/pull/616[0.67.1]
+* Trino: https://github.com/stackabletech/trino-operator/pull/638[455]
+* Apache Spark: https://github.com/stackabletech/spark-k8s-operator/pull/459[3.5.2 (LTS)]
+
+==== Deprecated versions
+
+The following product versions are deprecated and will be removed in a later release:
+
+* Apache Airflow: 2.9.2
+* Apache Druid: 26.0.0
+* Open Policy Agent: 0.66.0
+
+==== Removed versions
+
+The following product versions are no longer supported (although images for released product versions remain available https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable[here]):
+
+* Apache Airflow: 2.8.4, 2.8.1, 2.6.3
+* Apache Druid: 28.0.1
+* Apache Kafka: 3.6.2, 3.6.1, 3.4.1
+* Apache NiFi: 2.0.0-M4, 1.25.0, 1.21.0
+* Open Policy Agent: 0.61.0
+* Trino: 442, 414
+* Apache Spark: 3.4.3, 3.4.2
+* Apache Superset: 3.1.3, 3.1.0, 2.1.3
+* Apache ZooKeeper: 3.8.4
+
+=== stackablectl
+
+=== Supported Kubernetes versions
+
+This release supports the following Kubernetes versions:
+
+* `1.31`
+* `1.30`
+* `1.29`
+
+These Kubernetes versions are no longer supported:
+
+* `1.26`
+* `1.25` as we removed internal forks required to support Kubernetes `1.25` and below. This includes OpenShift `4.12`, which is using Kubernetes `1.25`.
+
+=== Supported OpenShift versions
+
+This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions:
+
+* `4.15`
+* `4.14`
+
+These OpenShift versions are no longer supported:
+
+* `4.13`
+* `4.12`
+
+=== Breaking changes
+
+=== Upgrade from 24.7
+
+==== Using stackablectl
+
+==== Using Helm
+
+==== Known upgrade issues
+
+===== All operators
+
 == Release 24.7
 
 === New / extended platform features
@@ -86,7 +216,7 @@ The status is still xref:concepts:multi-platform-support.adoc[experimental], as
 Security::
 
 Support for OIDC with/without TLS has been added to Apache Druid in this release.
-
+* Apache Druid: 26.0.0
 NOTE: SDP now provides OIDC-support for Druid, Superset and Trino
 
 In this release we provide experimental HBase 2.6.0 support with a new experimental policy based authorizer (with OPA).