Merge branch 'main' into feat/spark-4.0.0

Author: razvan

.github/ISSUE_TEMPLATE/early-pre-release.md

@@ -45,6 +45,7 @@ Part of stackabletech/issues#xxx.
 - [ ] [Create issue from template: update-product-kafka.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-kafka.md)
 - [ ] [Create issue from template: update-product-nifi.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-nifi.md)
 - [ ] [Create issue from template: update-product-opa.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-opa.md)
+- [ ] [Create issue from template: update-product-opensearch.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-opensearch.md)
 - [ ] [Create issue from template: update-product-spark.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-spark.md)
 - [ ] [Create issue from template: update-product-superset.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-superset.md)
 - [ ] [Create issue from template: update-product-trino.md](https://github.com/stackabletech/docker-images/issues/new?template=update-product-trino.md)

.github/ISSUE_TEMPLATE/update-product-opensearch.md

@@ -0,0 +1,79 @@
+---
+name: Update OpenSearch
+about: >-
+  This template contains instructions specific to updating this product and/or
+  container image(s).
+title: >-
+  chore(opensearch): Update container images ahead of Stackable Release YY.M.X
+labels: []
+# Currently, projects cannot be assigned via front-matter.
+projects: ['stackabletech/10']
+assignees: ''
+---
+
+Part of #xxx.
+
+<!--
+This gives hints to the person doing the work.
+Add/Change/Remove anything that isn't applicable anymore
+-->
+- Add: `x.x.x`
+- Remove: `y.y.y`
+
+> [!TIP]
+> Please add the `scheduled-for/20XX-XX` label, and add to the [Stackable Engineering][1] project.
+>
+> [1]: https://github.com/orgs/stackabletech/projects/10
+
+## Update tasks
+
+- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions).
+- [ ] Update `versions.py` to the latest supported version of JVM (base and devel).
+- [ ] Update other dependencies if applicable (eg: security-plugin, etc).
+- [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below.
+- [ ] Update the version in demos. Add the PR(s) to the list below.
+
+## Related Pull Requests
+
+> [!TIP]
+> Delete any items that do not apply so that all applicable items can be checked.
+> For example, if you add release notes to the documentation repository, you do not need the latter two criteria.
+
+- _Link to the docker-images PR (product update)_
+- _Link to the operator PR (getting_started / kuttl / supported-versions)_
+- _Link to any other operator PRs (getting_started / kuttl)_
+- _Link to demo PR (raise against the `main` branch)_
+- _Link to the Release Notes PR in the documentation repo (if not a comment below)_
+
+## Acceptance
+
+> [!TIP]
+> This list should be completed by the assignee(s), once respective PRs have been merged. Once all items have been
+> checked, the issue can be moved into _Development: Done_.
+
+- [ ] Can build image (either locally, or in CI)
+- [ ] Kuttl smoke tests passes (either locally, or in CI)
+- [ ] Release notes added to documentation and linked as a PR above
+- [ ] Release notes written in a comment below
+- [ ] Applicable `release-note` label added to this issue
+
+<details>
+<summary>Testing instructions</summary>
+
+```shell
+# See the latest version at https://pypi.org/project/image-tools-stackabletech/
+pip install image-tools-stackabletech==0.0.16
+
+bake --product opensearch=x.y.z # where x.y.z is the new version added in this PR
+
+kind load docker-image oci.stackable.tech/sdp/opensearch:x.y.z-stackable0.0.0-dev
+
+# Change directory into the opensearch-operator repository and update the
+# product version in tests/test-definition.yaml
+./scripts/run-tests --test-suite smoke-latest # or similar
+```
+
+</details>
+
+_Please consider updating this template if these instructions are wrong, or
+could be made clearer._

.github/workflows/build_opensearch.yaml

@@ -0,0 +1,36 @@
+---
+name: Build OpenSearch
+run-name: |
+  Build OpenSearch (attempt #${{ github.run_attempt }})
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 0 2/2 * *' # https://crontab.guru/#0_0_2/2_*_*
+  push:
+    branches: [main]
+    tags: ['*']
+    paths:
+      # To check dependencies, run this ( you will need to consider transitive dependencies)
+      # bake --product PRODUCT -d | grep -v 'docker buildx bake' | jq '.target | keys[]'
+      - opensearch/**
+      - vector/**
+      - stackable-base/**
+      - stackable-devel/**
+      - jdk-base/**
+      - java-devel/**
+      - .github/actions/**
+      - .github/workflows/build_opensearch.yaml
+      - .github/workflows/reusable_build_image.yaml
+
+jobs:
+  build_image:
+    name: Reusable Workflow
+    uses: ./.github/workflows/reusable_build_image.yaml
+    secrets:
+      harbor-robot-secret: ${{ secrets.HARBOR_ROBOT_SDP_GITHUB_ACTION_BUILD_SECRET }}
+      slack-token: ${{ secrets.SLACK_CONTAINER_IMAGE_TOKEN }}
+    with:
+      product-name: opensearch
+      sdp-version: ${{ github.ref_type == 'tag' && github.ref_name || '0.0.0-dev' }}
+      registry-namespace: sdp

.github/workflows/preflight.yaml

@@ -66,6 +66,7 @@ jobs:
           - nifi
           - omid
           - opa
+          - opensearch
           - spark-k8s
           - superset
           - trino

.scripts/enumerate-product-versions.py

@@ -18,6 +18,7 @@
     "krb5",
     "nifi",
     "opa",
+    "opensearch",
     "omid",
     "spark-k8s",
     "superset",

.scripts/update_feature_tracker_db.sh

@@ -33,6 +33,7 @@ PRODUCT_CODE_NAMES=(
   hive
   kafka
   nifi
+  opensearch
   spark-k8s
   superset
   trino

CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Added
+
+- ubi9-rust-builder: Include `.tar.gz` snapshots of the operator source code in container images ([#1207])
+- opensearch: Add Opensearch as new product with version `3.1.0` ([#1215]).
+- opensearch: Use build-repo.stackable.tech instead of Maven Central ([#1222]).
+
+### Changed
+
+- all: Use our build-repo to cache NPM dependencies ([#1219])
+- java: Use a more recent Maven version for all Java based products ([#1220])
+
+[#1207]: https://github.com/stackabletech/docker-images/pull/1207
+[#1215]: https://github.com/stackabletech/docker-images/pull/1215
+[#1219]: https://github.com/stackabletech/docker-images/pull/1219
+[#1220]: https://github.com/stackabletech/docker-images/pull/1220
+[#1222]: https://github.com/stackabletech/docker-images/pull/1222
+
 ## [25.7.0] - 2025-07-23
 
 ## [25.7.0-rc1] - 2025-07-18

README.md

@@ -8,9 +8,9 @@ This repository contains Dockerfiles and scripts to build base images for use wi
 | [![Build Airflow]][build_airflow.yaml] | [![Build Druid]][build_druid.yaml] | [![Build Hadoop]][build_hadoop.yaml] | [![Build HBase]][build_hbase.yaml] |
 | [![Build Hive]][build_hive.yaml] | [![Build Java Base]][build_java-base.yaml] | [![Build Java Development]][build_java-devel.yaml] | [![Build Kafka Testing Tools]][build_kafka-testing-tools.yaml] |
 | [![Build Kafka]][build_kafka.yaml] | [![Build Krb5]][build_krb5.yaml] | [![Build NiFi]][build_nifi.yaml] | [![Build Omid]][build_omid.yaml] |
-| [![Build OPA]][build_opa.yaml] | [![Build Spark Connect Client]][build_spark-connect-client.yaml] | [![Build Spark K8s]][build_spark-k8s.yaml] | [![Build Stackable Base]][build_stackable-base.yaml] |
-| [![Build Superset]][build_superset.yaml] | [![Build Testing Tools]][build_testing-tools.yaml] | [![Build Tools]][build_tools.yaml] | [![Build Trino CLI]][build_trino-cli.yaml] |
-| [![Build Trino]][build_trino.yaml] | [![Build Vector]][build_vector.yaml] | [![Build ZooKeeper]][build_zookeeper.yaml] | |
+| [![Build OPA]][build_opa.yaml] | [![Build OpenSearch]][build_opensearch.yaml] | [![Build Spark Connect Client]][build_spark-connect-client.yaml] | [![Build Spark K8s]][build_spark-k8s.yaml] |
+| [![Build Stackable Base]][build_stackable-base.yaml] | [![Build Superset]][build_superset.yaml] | [![Build Testing Tools]][build_testing-tools.yaml] | [![Build Tools]][build_tools.yaml] |
+| [![Build Trino CLI]][build_trino-cli.yaml] | [![Build Trino]][build_trino.yaml] | [![Build Vector]][build_vector.yaml] | [![Build ZooKeeper]][build_zookeeper.yaml] |
 <!-- end:badges -->
 
 ## Prerequisites
@@ -239,6 +239,8 @@ ENTRYPOINT ["/stackable-zookeeper-operator"]
 [build_omid.yaml]: https://github.com/stackabletech/docker-images/actions/workflows/build_omid.yaml
 [Build OPA]: https://github.com/stackabletech/docker-images/actions/workflows/build_opa.yaml/badge.svg
 [build_opa.yaml]: https://github.com/stackabletech/docker-images/actions/workflows/build_opa.yaml
+[Build OpenSearch]: https://github.com/stackabletech/docker-images/actions/workflows/build_opensearch.yaml/badge.svg
+[build_opensearch.yaml]: https://github.com/stackabletech/docker-images/actions/workflows/build_opensearch.yaml
 [Build Spark Connect Client]: https://github.com/stackabletech/docker-images/actions/workflows/build_spark-connect-client.yaml/badge.svg
 [build_spark-connect-client.yaml]: https://github.com/stackabletech/docker-images/actions/workflows/build_spark-connect-client.yaml
 [Build Spark K8s]: https://github.com/stackabletech/docker-images/actions/workflows/build_spark-k8s.yaml/badge.svg

conf.py

@@ -22,12 +22,17 @@
 hive = importlib.import_module("hive.versions")
 java_base = importlib.import_module("java-base.versions")
 java_devel = importlib.import_module("java-devel.versions")
+jdk_base = importlib.import_module("jdk-base.versions")
 kafka = importlib.import_module("kafka.versions")
 krb5 = importlib.import_module("krb5.versions")
 vector = importlib.import_module("vector.versions")
 nifi = importlib.import_module("nifi.versions")
 omid = importlib.import_module("omid.versions")
 opa = importlib.import_module("opa.versions")
+opensearch = importlib.import_module("opensearch.versions")
+opensearch_security_plugin = importlib.import_module(
+    "opensearch.security-plugin.versions"
+)
 spark_k8s = importlib.import_module("spark-k8s.versions")
 stackable_base = importlib.import_module("stackable-base.versions")
 stackable_devel = importlib.import_module("stackable-devel.versions")
@@ -58,12 +63,18 @@
     {"name": "hive", "versions": hive.versions},
     {"name": "java-base", "versions": java_base.versions},
     {"name": "java-devel", "versions": java_devel.versions},
+    {"name": "jdk-base", "versions": jdk_base.versions},
     {"name": "kafka", "versions": kafka.versions},
     {"name": "krb5", "versions": krb5.versions},
     {"name": "vector", "versions": vector.versions},
     {"name": "nifi", "versions": nifi.versions},
     {"name": "omid", "versions": omid.versions},
     {"name": "opa", "versions": opa.versions},
+    {"name": "opensearch", "versions": opensearch.versions},
+    {
+        "name": "opensearch/security-plugin",
+        "versions": opensearch_security_plugin.versions,
+    },
     {"name": "spark-k8s", "versions": spark_k8s.versions},
     {"name": "stackable-base", "versions": stackable_base.versions},
     {"name": "stackable-devel", "versions": stackable_devel.versions},
@@ -91,6 +102,7 @@
     "kafka": {"id": "625ff25b91bdcd4b49c823a4"},
     "nifi": {"id": "625586a32e9e14bc8118e203"},
     "opa": {"id": "6255838bea1feb8bec4aaaa3"},
+    "opensearch": {"id": "6880fe690db664aa303d3a28"},
     "spark-k8s": {"id": "62613e81f8ce82a2f247dda5"},
     "superset": {"id": "62557e5fea1feb8bec4aaaa0"},
     "tools": {"id": "62557cd575ab7e30884aaaa0"},

druid/Dockerfile

@@ -73,8 +73,6 @@ cp -r /stackable/patched-libs/maven/* /stackable/.m2/repository
 tar -czf /stackable/druid-${NEW_VERSION}-src.tar.gz .
 
 mvn \
-  --batch-mode \
-  --no-transfer-progress \
   clean install \
   -Pdist,stackable-bundle-contrib-exts \
   -Dhadoop.compile.version=${HADOOP_VERSION}-stackable${RELEASE} \