Merge branch 'main' of https://github.com/stackabletech/docker-images into feat/custom-product-versions

Author: dervoeti

.github/ISSUE_TEMPLATE/update-product-trino.md

@@ -30,7 +30,6 @@ Add/Change/Remove anything that isn't applicable anymore
 ### Trino
 
 - [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions).
-- [ ] Upload new version (see `trino/*.sh` scripts).
 - [ ] Update `versions.py` to the latest supported version of JVM (base and devel).
 - [ ] Update other dependencies if applicable (eg: jmx_exporter, opa_authorizer, storage_connector, etc).
 - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below.

.github/workflows/ubi-rust-builder.yml

@@ -7,6 +7,12 @@ on:
   push:
     branches:
       - main
+    paths:
+      - ubi9-rust-builder/**
+      - ubi10-rust-builder/**
+      - .github/actions/**
+      - .github/workflows/ubi-rust-builder.yml
+      - .github/workflows/reusable_build_image.yaml
   schedule:
     - cron: '30 4 * * *'
   workflow_dispatch:

CHANGELOG.md

@@ -37,17 +37,22 @@ All notable changes to this project will be documented in this file.
 - superset: Add version `4.1.2` ([#1102]).
 - trino: check for correct permissions and ownerships in /stackable folder via
   `check-permissions-ownership.sh` provided in stackable-base image ([#1025]).
+- trino: Add `476` ([#1095]).
+- trino-storage-connector: Add `476` ([#1095]).
 - zookeeper: check for correct permissions and ownerships in /stackable folder via
   `check-permissions-ownership.sh` provided in stackable-base image ([#1043]).
 - nifi: Build and add OPA authorizer plugin nar ([#1058]).
 - nifi: Add [nifi-iceberg-bundle](https://github.com/stackabletech/nifi-iceberg-bundle) for NiFi `2.2.0` ([#1060], [#1106]).
 - java: Add JDK 24 ([#1097]).
 - ci: Add golang image to mirror workflow ([#1103]).
-- omid: bump version to 1.1.3 ([#1105])
-- hbase: add 2.6.2 and upgrade dependencies ([#1101])
-- kafka: Add `4.0.0` ([#1117])
-- Include `.tar.gz` snapshots of the product source code in container images ([#1126])
-- airflow: OPA authorizer for Airflow 3.x ([#1127])
+- omid: bump version to 1.1.3 ([#1105]).
+- hbase: add 2.6.2 and upgrade dependencies ([#1101]).
+- kafka: Add `4.0.0` ([#1117]).
+- Include `.tar.gz` snapshots of the product source code in container images ([#1126]).
+- airflow: OPA authorizer for Airflow 3.x ([#1127]).
+- kafka: Add `3.9.1` ([#1149]).
+- spark-k8s: Add `3.5.6` ([#1142]).
+- spark-connect-client: Add `3.5.6` ([#1142]).
 - git-sync: Bump version to 4.4.1 ([#1151]).
 
 ### Changed
@@ -62,17 +67,19 @@ All notable changes to this project will be documented in this file.
 - kubectl: Bump products to use `1.33.0` ([#1090]).
 - yq: Bump products to use `4.45.2` ([#1090]).
 - cyclonedx-bom: Bump airflow and superset to use `6.0.0` ([#1090]).
+- trino-cli: Bump to `476` ([#1095]).
 - vector: Bump to `0.46.1` ([#1098]).
-- spark: update dependencies for 3.5.5 ([#1094])
-- nifi: include NAR SBOMs ([#1119])
+- spark: update dependencies for 3.5.5 ([#1094]).
+- nifi: include NAR SBOMs ([#1119]).
 - nifi: update patch allowing to bypass host header validation starting with NiFi 2.4.0 ([#1125]).
 - BREAKING: kcat: Stop building kcat image ([#1124]).
-- containerdebug updated to 0.2.0 ([#1128])
-- Build Hadoop as `stackable` and configure the Stackable Nexus build-repo for the `root` user ([#1133])
+- containerdebug updated to 0.2.0 ([#1128]).
+- Build Hadoop as `stackable` and configure the Stackable Nexus build-repo for the `root` user ([#1133]).
 - patchable: The base branch is now configured as the git upstream branch ([#1131]).
-- airflow: Updates the entrypoint script and removes the check for GID == 0 ([#1138])
+- airflow: Updates the entrypoint script and removes the check for GID == 0 ([#1138]).
 - druid: Bump druiod-opa-authorizer to `0.7.0` ([#1139]).
 - vector: Bump to `0.47.0` ([#1152]).
+- zookeeper: backport ZOOKEEPER-4846, ZOOKEEPER-4921, ZOOKEEPER-4925 into Zookeeper 3.9.3 ([#1150]).
 
 ### Fixed
 
@@ -100,17 +107,20 @@ All notable changes to this project will be documented in this file.
 
 - ci: Remove Nexus steps from build, mirror and release workflows ([#1056]).
   Also remove the old release workflow.
+- trino: Remove `455` ([#1095]).
+- trino-storage-connector: Remove `455` ([#1095]).
 - zookeeper: Remove 3.9.2 ([#1093]).
 - Remove ubi8-rust-builder image ([#1091]).
-- spark: remove 3.5.2 ([#1094])
+- spark: remove 3.5.2 ([#1094]).
 - hadoop: Remove `3.3.4` and `3.4.0` ([#1099]).
 - opa: Remove `0.67.1` ([#1103]).
 - opa: Remove legacy bundle-builder from container build ([#1103]).
 - omid: Remove 1.1.3-SNAPSHOT ([#1105]).
 - hbase: Remove 2.4.18 ([#1101])
 - druid: Remove `30.0.0` ([#1110]).
 - nifi: Remove `2.2.0` ([#1114]).
-- kafka: Remove `3.7.1` and `3.8.0` ([#1117])
+- kafka: Remove `3.7.1` and `3.8.0` ([#1117]).
+- spark-connect-client: Remove `3.5.5` ([#1142]).
 
 [nifi-iceberg-bundle]: https://github.com/stackabletech/nifi-iceberg-bundle
 [#1025]: https://github.com/stackabletech/docker-images/pull/1025
@@ -137,6 +147,7 @@ All notable changes to this project will be documented in this file.
 [#1091]: https://github.com/stackabletech/docker-images/pull/1091
 [#1093]: https://github.com/stackabletech/docker-images/pull/1093
 [#1094]: https://github.com/stackabletech/docker-images/pull/1094
+[#1095]: https://github.com/stackabletech/docker-images/pull/1095
 [#1097]: https://github.com/stackabletech/docker-images/pull/1097
 [#1098]: https://github.com/stackabletech/docker-images/pull/1098
 [#1099]: https://github.com/stackabletech/docker-images/pull/1099
@@ -165,6 +176,9 @@ All notable changes to this project will be documented in this file.
 [#1137]: https://github.com/stackabletech/docker-images/pull/1137
 [#1138]: https://github.com/stackabletech/docker-images/pull/1138
 [#1139]: https://github.com/stackabletech/docker-images/pull/1139
+[#1142]: https://github.com/stackabletech/docker-images/pull/1142
+[#1149]: https://github.com/stackabletech/docker-images/pull/1149
+[#1150]: https://github.com/stackabletech/docker-images/pull/1150
 [#1151]: https://github.com/stackabletech/docker-images/pull/1151
 [#1152]: https://github.com/stackabletech/docker-images/pull/1152
 

airflow/stackable/constraints/3.0.1/constraints-python3.12.txt

@@ -122,7 +122,14 @@ apache-airflow-providers-asana==2.9.1
 apache-airflow-providers-atlassian-jira==3.0.2
 apache-airflow-providers-celery==3.10.6
 apache-airflow-providers-cloudant==4.1.1
-apache-airflow-providers-cncf-kubernetes==10.4.3
+# Stackable patch:
+# The 10.4.3 Kubernetes provider shipped with Airflow 3.0.1 has a problem.
+# https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/changelog.html#id1
+# It uses a busybox sidecar container to start tasks using the kubernetes executor
+# Two problems with that:
+# 1) The securityContext says "runAsNonRoot" but the busybox image defaults to root so it fails to be scheduled (at least on a standard OpenShift)
+# 2) The busybox image might not be available on air-gapped clusters
+apache-airflow-providers-cncf-kubernetes==10.5.0
 apache-airflow-providers-cohere==1.4.3
 apache-airflow-providers-common-compat==1.6.1
 apache-airflow-providers-common-io==1.5.4

kafka/stackable/patches/3.9.1/0001-Add-CycloneDX-plugin.patch

@@ -0,0 +1,62 @@
+From 9f5ef4ff9f1868244326a56ea7c22c5c7c21253c Mon Sep 17 00:00:00 2001
+From: Lukas Voetmand <[email protected]>
+Date: Thu, 17 Oct 2024 11:01:40 +0200
+Subject: Add CycloneDX plugin
+
+---
+ build.gradle | 42 ++++++++++++++++++++++++++++++++++++++++++
+ 1 file changed, 42 insertions(+)
+
+diff --git a/build.gradle b/build.gradle
+index 6b4725d542..f7894f379e 100644
+--- a/build.gradle
++++ b/build.gradle
+@@ -48,6 +48,48 @@ plugins {
+   //  We are going to drop JDK8 support. Hence, the spotless is upgrade to newest version and be applied only if the build env is compatible with JDK 11.
+   //  spotless 6.15.0+ has issue in runtime with JDK8 even through we define it with `apply:false`. see https://github.com/diffplug/spotless/issues/2156 for more details
+   id 'com.diffplug.spotless' version "6.14.0" apply false
++  id 'org.cyclonedx.bom' version '1.10.0'
++}
++
++cyclonedxBom {
++    // Specified the type of project being built. Defaults to 'library'
++    projectType = "application"
++    // Specified the version of the CycloneDX specification to use. Defaults to '1.5'
++    schemaVersion = "1.5"
++    // Boms destination directory. Defaults to 'build/reports'
++    destination = file("build/reports")
++    // The file name for the generated BOMs (before the file format suffix). Defaults to 'bom'
++    outputName = "bom"
++    // The file format generated, can be xml, json or all for generating both. Defaults to 'all'
++    outputFormat = "json"
++    includeConfigs = ["runtimeClasspath"]
++    // Exclude test components. This list needs to be checked and, if it changed, updated for every new Kafka version.
++    // The list can be obtained by running `gradle projects | grep upgrade-system-tests`
++    skipProjects = [
++      'upgrade-system-tests-0100',
++      'upgrade-system-tests-0101',
++      'upgrade-system-tests-0102',
++      'upgrade-system-tests-0110',
++      'upgrade-system-tests-10',
++      'upgrade-system-tests-11',
++      'upgrade-system-tests-20',
++      'upgrade-system-tests-21',
++      'upgrade-system-tests-22',
++      'upgrade-system-tests-23',
++      'upgrade-system-tests-24',
++      'upgrade-system-tests-25',
++      'upgrade-system-tests-26',
++      'upgrade-system-tests-27',
++      'upgrade-system-tests-28',
++      'upgrade-system-tests-30',
++      'upgrade-system-tests-31',
++      'upgrade-system-tests-32',
++      'upgrade-system-tests-33',
++      'upgrade-system-tests-34',
++      'upgrade-system-tests-35',
++      'upgrade-system-tests-36',
++      'upgrade-system-tests-37'
++    ]
+ }
+ 
+ ext {

kafka/stackable/patches/3.9.1/0002-Change-Gradle-to-use-the-Nexus-Build-Repo.patch

@@ -0,0 +1,35 @@
+From 4de60cd0100bffdd4b92133840ab25247187763d Mon Sep 17 00:00:00 2001
+From: Lars Francke <[email protected]>
+Date: Thu, 12 Dec 2024 10:09:47 +0100
+Subject: Change Gradle to use the Nexus Build Repo
+
+---
+ build.gradle | 8 ++++++--
+ 1 file changed, 6 insertions(+), 2 deletions(-)
+
+diff --git a/build.gradle b/build.gradle
+index f7894f379e..a176efd936 100644
+--- a/build.gradle
++++ b/build.gradle
+@@ -20,7 +20,9 @@ import java.nio.charset.StandardCharsets
+ 
+ buildscript {
+   repositories {
+-    mavenCentral()
++    maven {
++      url 'https://build-repo.stackable.tech/repository/maven-public/'
++    }
+   }
+   apply from: "$rootDir/gradle/dependencies.gradle"
+ 
+@@ -168,7 +170,9 @@ ext {
+ allprojects {
+ 
+   repositories {
+-    mavenCentral()
++    maven {
++      url 'https://build-repo.stackable.tech/repository/maven-public/'
++    }
+   }
+ 
+   dependencyUpdates {

kafka/stackable/patches/3.9.1/patchable.toml

@@ -0,0 +1,2 @@
+base = "f745dfdcee2b9851204ddbbcd423626ab87294bc"
+mirror = "https://github.com/stackabletech/kafka.git"

kafka/versions.py

@@ -17,6 +17,15 @@
         "opa_authorizer": "1.5.1",
         "jmx_exporter": "1.2.0",
     },
+    {
+        "product": "3.9.1",
+        "java-base": "21",
+        "java-devel": "21",
+        "scala": "2.13",
+        "kafka/kcat": "1.7.0",
+        "opa_authorizer": "1.5.1",
+        "jmx_exporter": "1.2.0",
+    },
     {
         "product": "4.0.0",
         "java-base": "23",

spark-connect-client/versions.py

@@ -1,6 +1,6 @@
 versions = [
     {
-        "product": "3.5.5",
+        "product": "3.5.6",
         "spark-k8s": "3.5.5",
         "java-base": "17",
         "python": "3.11",

spark-k8s/stackable/patches/3.5.6/0001-Update-CycloneDX-plugin.patch

@@ -0,0 +1,40 @@
+From 94ccf32b4d0eb7c3191b4e5a646605e7386c39ff Mon Sep 17 00:00:00 2001
+From: Razvan-Daniel Mihai <[email protected]>
+Date: Tue, 11 Mar 2025 17:29:39 +0200
+Subject: Update CycloneDX plugin
+
+---
+ dev/make-distribution.sh | 1 -
+ pom.xml                  | 7 ++++++-
+ 2 files changed, 6 insertions(+), 2 deletions(-)
+
+diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
+index ef7c010e93..0f4c1c74e4 100755
+--- a/dev/make-distribution.sh
++++ b/dev/make-distribution.sh
+@@ -171,7 +171,6 @@ BUILD_COMMAND=("$MVN" clean package \
+     -Dmaven.javadoc.skip=true \
+     -Dmaven.scaladoc.skip=true \
+     -Dmaven.source.skip \
+-    -Dcyclonedx.skip=true \
+     $@)
+ 
+ # Actually build the jar
+diff --git a/pom.xml b/pom.xml
+index 68e2c422a2..6216ebb08e 100644
+--- a/pom.xml
++++ b/pom.xml
+@@ -3534,7 +3534,12 @@
+       <plugin>
+         <groupId>org.cyclonedx</groupId>
+         <artifactId>cyclonedx-maven-plugin</artifactId>
+-        <version>2.7.9</version>
++        <version>2.8.0</version>
++        <configuration>
++          <projectType>application</projectType>
++          <schemaVersion>1.5</schemaVersion>
++          <skipNotDeployed>false</skipNotDeployed>
++        </configuration>
+         <executions>
+           <execution>
+             <phase>package</phase>