diff --git a/.github/ISSUE_TEMPLATE/add-product.md b/.github/ISSUE_TEMPLATE/add-product.md index 7828dd229..a4ef78f1e 100644 --- a/.github/ISSUE_TEMPLATE/add-product.md +++ b/.github/ISSUE_TEMPLATE/add-product.md @@ -17,7 +17,8 @@ assignees: '' - [ ] Create a README.md file outlining special considerations required to update or run the product. See existing README files as a guide of reference. -- [ ] Add a `versions.py` file to the folder. Add all required key-value pairs. +- [ ] Add a `boil-config.toml` file to the folder. Add all required local image + dependencies and build arguments. - [ ] Add a new "Update Product" issue template in `.github/ISSUE_TEMPLATE/` folder. See existing ones as a guide of reference. - [ ] Add a new `build_.yml` GitHub Action workflow in the diff --git a/.github/ISSUE_TEMPLATE/update-base-java.md b/.github/ISSUE_TEMPLATE/update-base-java.md index cceead28f..2a164a7cc 100644 --- a/.github/ISSUE_TEMPLATE/update-base-java.md +++ b/.github/ISSUE_TEMPLATE/update-base-java.md @@ -41,8 +41,8 @@ we should also make new versions of Java available for use. ## Update tasks -- [ ] Add any new versions of java to both `java-base/versions.py` and `java-devel/versions.py` -- [ ] Remove versions when there are no long any references (eg: `grep java- **/versions.py | grep "1.8.0"`) +- [ ] Add any new versions of java to both `java-base/boil-config.toml` and `java-devel/boil-config.toml` +- [ ] Remove versions when there are no long any references (eg: `grep java- **/boil-config.toml | grep "1.8.0"`) ## Related Pull Requests @@ -55,18 +55,16 @@ we should also make new versions of Java available for use. > checked, the issue can be moved into _Development: Done_. - [ ] Can build a product image that uses the new version(s) -- [ ] Both `java-base` and `java-devel` have the same Java versions in `versions.py` +- [ ] Both `java-base` and `java-devel` have the same Java versions in `boil-config.toml` - [ ] Kuttl smoke test passes locally for a product using the new Java version
Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -# Test a product image can build, eg: ZooKeeper -bake --product zookeeper=x.y.z # where x.y.z is a valid product version using the newly added Java version +# Test a product image can build, eg: ZooKeeper where x.y.z is a valid product +# version using the newly added Java version +boil build zookeeper=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/zookeeper:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-base-stackable.md b/.github/ISSUE_TEMPLATE/update-base-stackable.md index 5fcf5412f..25a18689e 100644 --- a/.github/ISSUE_TEMPLATE/update-base-stackable.md +++ b/.github/ISSUE_TEMPLATE/update-base-stackable.md @@ -55,10 +55,8 @@ Part of . Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product vector=x.y.z # where x.y.z is a valid version +# Where x.y.z is a valid version +boil build vector=x.y.z --strip-architecture --load ```
diff --git a/.github/ISSUE_TEMPLATE/update-base-vector.md b/.github/ISSUE_TEMPLATE/update-base-vector.md index 1b1b27c22..80ec582de 100644 --- a/.github/ISSUE_TEMPLATE/update-base-vector.md +++ b/.github/ISSUE_TEMPLATE/update-base-vector.md @@ -33,8 +33,8 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). -- [ ] Update all `versions.py` files which reference vector. +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update all `boil-config.toml` files which reference vector. - [ ] Upload new version (see `vector/upload_new_vector_version.sh`). - [ ] Update other dependencies if applicable (eg: inotify_tools, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. @@ -68,10 +68,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product vector=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build vector=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/vector:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-airflow.md b/.github/ISSUE_TEMPLATE/update-product-airflow.md index 6d8c335ff..bcdb7d2fc 100644 --- a/.github/ISSUE_TEMPLATE/update-product-airflow.md +++ b/.github/ISSUE_TEMPLATE/update-product-airflow.md @@ -27,7 +27,7 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Download new constraints file (see `airflow/download_constraints.sh`). - [ ] Update other dependencies if applicable (eg: python, statsd_exporter, cyclonedx-bom, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. @@ -62,10 +62,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product airflow=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build airflow=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/airflow:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-druid.md b/.github/ISSUE_TEMPLATE/update-product-druid.md index 995e47e70..f2f699738 100644 --- a/.github/ISSUE_TEMPLATE/update-product-druid.md +++ b/.github/ISSUE_TEMPLATE/update-product-druid.md @@ -28,12 +28,12 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks - [ ] Release a new version of [druid-opa-authorizer](https://github.com/stackabletech/druid-opa-authorizer) -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Update the [druid-opa-authorizer](https://github.com/stackabletech/druid-opa-authorizer/) with the new set of versions. - [ ] Upload new version (see `druid/upload_new_druid_version.sh`). - [ ] Create a file: `druid/stackable/patches/x.y.z/.gitkeep`, add patches if applicable. - [ ] Delete old patch directories. -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Ensure prerequisites are up to date (required-external-components.adoc). - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -67,10 +67,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product druid=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build druid=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/druid:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-hbase-phoenix-omid.md b/.github/ISSUE_TEMPLATE/update-product-hbase-phoenix-omid.md index f23681437..e06c9614e 100644 --- a/.github/ISSUE_TEMPLATE/update-product-hbase-phoenix-omid.md +++ b/.github/ISSUE_TEMPLATE/update-product-hbase-phoenix-omid.md @@ -29,17 +29,17 @@ Add/Change/Remove anything that isn't applicable anymore ### HBase and Phoenix -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new versions (see the `hbase/*.sh` scripts). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: phoenix, opa_authorizer, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR to the list below. ### Omid -- [ ] Update `omid/versions.py`to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `omid/boil-config.toml`to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `omid/upload_new_omid_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: jmx_exporter, etc). ## Related Pull Requests @@ -70,11 +70,9 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product hbase=x.y.z # where x.y.z is the new version added in this PR -bake --product omid=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build hbase=x.y.z --strip-architecture --load +boil build omid=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/hbase:x.y.z-stackable0.0.0-dev kind load docker-image oci.stackable.tech/sdp/omid:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-hdfs.md b/.github/ISSUE_TEMPLATE/update-product-hdfs.md index 433ab44d0..846a7e1ae 100644 --- a/.github/ISSUE_TEMPLATE/update-product-hdfs.md +++ b/.github/ISSUE_TEMPLATE/update-product-hdfs.md @@ -27,9 +27,9 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `hadoop/upload_new_hadoop_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: hdfs_utils, jmx_exporter, protobuf, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -63,10 +63,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product hadoop=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build hadoop=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/hadoop:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-hive.md b/.github/ISSUE_TEMPLATE/update-product-hive.md index 6e8557281..0dfa635aa 100644 --- a/.github/ISSUE_TEMPLATE/update-product-hive.md +++ b/.github/ISSUE_TEMPLATE/update-product-hive.md @@ -27,9 +27,9 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `hive/upload_new_hive_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: jmx_exporter, aws_java_sdk_bundle, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Ensure prerequisites are up to date (required-external-components.adoc). @@ -63,10 +63,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product hive=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build hive=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/hive:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-kafka.md b/.github/ISSUE_TEMPLATE/update-product-kafka.md index be04908ff..45308bafa 100644 --- a/.github/ISSUE_TEMPLATE/update-product-kafka.md +++ b/.github/ISSUE_TEMPLATE/update-product-kafka.md @@ -29,9 +29,9 @@ Add/Change/Remove anything that isn't applicable anymore ### Kafka -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `kafka/upload_new_kafka_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: jmx_exporter, kcat, scala, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -40,10 +40,10 @@ Add/Change/Remove anything that isn't applicable anymore -- [ ] Update `kcat/versions.py`. -- [ ] Update `kafka-testing-tools/versions.py`. +- [ ] Update `kcat/boil-config.toml`. +- [ ] Update `kafka-testing-tools/boil-config.toml`. - [ ] Upload new version (see `.scripts/upload_new_kcat_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -75,11 +75,11 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 +# Where x.y.z is the new version added in this PR +boil build kafka=x.y.z --strip-architecture --load -bake --product kafka=x.y.z # where x.y.z is the new version added in this PR -bake --product kafka-testing-tools=1.0.0 # This version doesn't change +# This version doesn't change +boil build kafka-testing-tools=1.0.0 --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/kafka:x.y.z-stackable0.0.0-dev kind load docker-image oci.stackable.tech/sdp/kafka-testing-tools:1.0.0-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-nifi.md b/.github/ISSUE_TEMPLATE/update-product-nifi.md index 50b504a18..5e3a1a4ce 100644 --- a/.github/ISSUE_TEMPLATE/update-product-nifi.md +++ b/.github/ISSUE_TEMPLATE/update-product-nifi.md @@ -27,9 +27,9 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `nifi/upload_new_nifi_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: jmx_exporter, kcat, scala, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -62,10 +62,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product nifi=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build nifi=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/nifi:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-opa.md b/.github/ISSUE_TEMPLATE/update-product-opa.md index dbbd31dd6..ac80ef91f 100644 --- a/.github/ISSUE_TEMPLATE/update-product-opa.md +++ b/.github/ISSUE_TEMPLATE/update-product-opa.md @@ -27,7 +27,7 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `opa/upload_new_opa_version.sh`). - [ ] Update other dependencies if applicable (eg: opa_bundle_builder, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. @@ -61,10 +61,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product opa=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build opa=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/opa:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-opensearch.md b/.github/ISSUE_TEMPLATE/update-product-opensearch.md index 74dfa3c35..8f0761d68 100644 --- a/.github/ISSUE_TEMPLATE/update-product-opensearch.md +++ b/.github/ISSUE_TEMPLATE/update-product-opensearch.md @@ -27,8 +27,8 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: security-plugin, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -61,10 +61,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product opensearch=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build opensearch=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/opensearch:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-spark.md b/.github/ISSUE_TEMPLATE/update-product-spark.md index ef5c5c24d..3080a3688 100644 --- a/.github/ISSUE_TEMPLATE/update-product-spark.md +++ b/.github/ISSUE_TEMPLATE/update-product-spark.md @@ -27,10 +27,10 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `spark/upload_new_spark_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). -- [ ] Ensure that the transitive dependencies in `versions.py` are compatible with the Spark version(s). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). +- [ ] Ensure that the transitive dependencies in `boil-config.toml` are compatible with the Spark version(s). - [ ] Update other dependencies if applicable (eg: python, jmx_exporter, aws_java_sdk_bundle, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -63,10 +63,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product spark-k8s=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build spark-k8s=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/spark-k8s:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-superset.md b/.github/ISSUE_TEMPLATE/update-product-superset.md index 67f8b8f35..d98e63bc5 100644 --- a/.github/ISSUE_TEMPLATE/update-product-superset.md +++ b/.github/ISSUE_TEMPLATE/update-product-superset.md @@ -27,7 +27,7 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Create a new constraints file (see `superset/README.md`). - [ ] Create a file: `superset/stackable/patches/x.y.z/.gitkeep`, add patches if applicable. - [ ] Delete old constraint files and patch directories. @@ -64,10 +64,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product superset=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build superset=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/superset:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-trino.md b/.github/ISSUE_TEMPLATE/update-product-trino.md index 1168043c9..c3fadbf46 100644 --- a/.github/ISSUE_TEMPLATE/update-product-trino.md +++ b/.github/ISSUE_TEMPLATE/update-product-trino.md @@ -29,8 +29,8 @@ Add/Change/Remove anything that isn't applicable anymore ### Trino -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: jmx_exporter, opa_authorizer, storage_connector, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Maybe update versions in `tests/templates/kuttl/opa-authorization/check-opa.py.j2` (if tests fail) @@ -39,9 +39,9 @@ Add/Change/Remove anything that isn't applicable anymore ### trino-cli -- [ ] Update `trino-cli/versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `trino-cli/boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `trino-cli/upload_new_trino_version.sh` scripts). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. - [ ] Update versions used in the documentation repository @@ -74,11 +74,9 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product trino=x.y.z # where x.y.z is the new version added in this PR -bake --product trino-cli=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build trino=x.y.z --strip-architecture --load +boil build trino-cli=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/trino:x.y.z-stackable0.0.0-dev kind load docker-image oci.stackable.tech/sdp/trino-cli:x.y.z-stackable0.0.0-dev diff --git a/.github/ISSUE_TEMPLATE/update-product-zookeeper.md b/.github/ISSUE_TEMPLATE/update-product-zookeeper.md index 08212d9d8..89427ef62 100644 --- a/.github/ISSUE_TEMPLATE/update-product-zookeeper.md +++ b/.github/ISSUE_TEMPLATE/update-product-zookeeper.md @@ -27,9 +27,9 @@ Add/Change/Remove anything that isn't applicable anymore ## Update tasks -- [ ] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). +- [ ] Update `boil-config.toml` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions). - [ ] Upload new version (see `zookeeper/upload_new_zookeeper_version.sh`). -- [ ] Update `versions.py` to the latest supported version of JVM (base and devel). +- [ ] Update `boil-config.toml` to the latest supported version of JVM (base and devel). - [ ] Update other dependencies if applicable (eg: jmx_exporter, etc). - [ ] Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below. - [ ] Update the version in demos. Add the PR(s) to the list below. @@ -62,10 +62,8 @@ Add/Change/Remove anything that isn't applicable anymore Testing instructions ```shell -# See the latest version at https://pypi.org/project/image-tools-stackabletech/ -pip install image-tools-stackabletech==0.0.16 - -bake --product zookeeper=x.y.z # where x.y.z is the new version added in this PR +# Where x.y.z is the new version added in this PR +boil build zookeeper=x.y.z --strip-architecture --load kind load docker-image oci.stackable.tech/sdp/zookeeper:x.y.z-stackable0.0.0-dev diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 8e88a3faf..9cae5bc38 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -22,9 +22,9 @@ Please make sure all these things are done and tick the boxes The image can be built and uploaded to the kind cluster with the following commands: ```shell -bake --product --image-version -kind load docker-image --name= +boil build --image-version --strip-architecture --load +kind load docker-image --name= ``` -See the output of `bake` to retrieve the image tag for ``. +See the output of `boil` to retrieve the image manifest URI for ``. diff --git a/.scripts/enumerate-product-versions.py b/.scripts/enumerate-product-versions.py deleted file mode 100644 index 6beaa085c..000000000 --- a/.scripts/enumerate-product-versions.py +++ /dev/null @@ -1,38 +0,0 @@ -import sys -import os - -# NOTE: This script (used in the release workflow as of 2024-07-23) relies on conf.py being in its parent folder. Should either file be moved or the structure changed in any way remember to update this script as well. -# Add parent directory to the sys.path -sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -import conf - -PRODUCTS = [ - "airflow", - "druid", - "hadoop", - "hbase", - "hive", - "kafka", - "kafka-testing-tools", - "krb5", - "nifi", - "opa", - "opensearch", - "omid", - "spark-k8s", - "superset", - "testing-tools", - "trino", - "trino-cli", - "tools", - "zookeeper", -] - -for product in conf.products: - product_name = product["name"] - if product_name not in PRODUCTS: - continue - - for version in product["versions"]: - print(f"{product_name}#{version['product']}") diff --git a/README.md b/README.md index efd84522c..dc8bc3857 100644 --- a/README.md +++ b/README.md @@ -38,161 +38,16 @@ both `amd64` and `arm64`. The workflow is triggered * by tag pushes to produce (release candidate) images for a particular SDP release, * and by manual workflow dispatches. -## Patches +## Patch Images -Many products apply Stackable-specific patches, managed by [Patchable](rust/patchable). +Images are patched using `patchable`. Consult the [patchable README](./rust/patchable/README.md) which contains a detailed usage guide. -Patchable doesn't _edit_ anything by itself. Instead, it's a uniform way to apply a set of patches -to an upstream Git repository, and then export your local changes back into patch files. +## Verify Images -It doesn't care about how you make your local changes - you can edit the branch created by -patchable using any Git frontend, such as the git CLI or [jj](https://jj-vcs.github.io/jj/latest/). - -This way, the patch files are the global source of truth and track the history of our patch series, -while you can still use the same familiar Git tools to manipulate them. - -### Check out patched sources locally - -> [!NOTE] -> This is not required for building images, but is used for when hacking on or debugging patch series. - -```sh -# Fetches the upstream repository (if required), and creates a git worktree to work with it -# It also creates two branches: -# - patchable/{version} (HEAD, has all patches applied) -# - patchable/base/{version} (the upstream) -pushd $(cargo patchable checkout druid 26.0.0) - -# Commit to add new patches -# NOTE: the commit message will be used to construct the patch filename. Spaces -# will be converted to hyphens automatically. -git commit - -# Rebase against the base commit to edit or remove patches -git rebase --interactive patchable/base/26.0.0 -# jj edit also works, but make sure to go back to the tip before exporting - -# When done, export your patches and commit them (to docker-images) -popd -cargo patchable export druid 26.0.0 -git status -``` - -> ![CAUTION] -> `cargo patchable export` exports whatever is currently checked out (`HEAD`) in the worktree. -> If you use `jj edit` (or `git switch`) then you _must_ go back to the tip before exporting, or -> any patches after that point will be omitted from the export. - -### Initialize a new patch series - -Patchable stores metadata about each patch series in its `patchable.toml`, and will not be able to check out -a patch series that lacks one. It can be generated using `cargo patchable init`: +To verify if a container image is compatible with OpenShift, run the following `preflight` command: ```sh -cargo patchable init druid 28.0.0 --upstream https://github.com/apache/druid.git --base druid-28.0.0 -cargo patchable checkout druid 28.0.0 -``` - -### Importing patch series into Patchable - -Patchable is stricter about applying invalid patches (both metadata and patches themselves) than Git is. - -If an initial `cargo patchable checkout` fails then `git am` can be useful for the initial migration: - -```sh -# Create Patchable configuration for the new version, if it doesn't already exist -cargo patchable init druid 30.0.0 --upstream https://github.com/apache/druid.git --base druid-30.0.0 -# Check out the upstream base commit, without trying to apply the patches -pushd $(cargo patchable checkout druid 30.0.0 --base-only) - -# Apply the patch series -git am ../../../stackable/patches/30.0.0/*.patch -# Resolve any conflicts that arise, and `git am --continue` until done - -# Leave and export the new patch series! -popd -cargo patchable export druid 30.0.0 -``` - -### Porting patch series to a new version - -Patchable doesn't support restoring a patch series that doesn't apply cleanly. Instead, use `git cherry-pick` to rebase the patch series. - -For example, let's try rebasing our patch series from Druid 26.0.0 to Druid 28.0.0 (which is not packaged by SDP): - -```sh -# Restore the old version -# In addition to creating the version worktree, this also creates the branches patchable/26.0.0 (26.0.0 with our patches applied) and -# patchable/base/26.0.0 (upstream 26.0.0 with no patches). -cargo patchable checkout druid 26.0.0 -# Tell Patchable about the new version 28.0.0, which can be fetched from https://github.com/apache/druid.git, and has the tag druid-28.0.0 -cargo patchable init druid 28.0.0 --upstream https://github.com/apache/druid.git --base druid-28.0.0 -# Create and go to the worktree for the new version -pushd $(cargo patchable checkout druid 28.0.0) - -# Cherry pick the old patch series -git cherry-pick patchable/base/26.0.0..patchable/26.0.0 -# Solve conflicts and `git cherry-pick --continue` until done -# You can also use `git cherry-pick --skip` to skip resolving conflicts for patches that are no longer required - -# If some patches are no longer required, use an interactive rebase to remove them (or do other cleanup) -git rebase --interactive patchable/base/28.0.0 - -# Leave and export the new patch series! -popd -cargo patchable export druid 28.0.0 -git status -``` - -### Porting patches between versions - -Individual patches can also be cherry-picked across versions. - -For example, assuming we are in the Druid 28.0.0 workspace and want to port the last patch of the Druid 26.0.0 series: - -```sh -# git cherry-pick is also fine for grabbing arbitrary patches -git cherry-pick patchable/26.0.0 -``` - -## Verify Product Images - -To verify if Apache Zookeeper validate against OpenShift preflight, run: - -```sh -check-container --product zookeeper --image 0.0.0-dev -``` - -## ubi8-rust-builder / ubi9-rust-builder - -These images are meant to be used in multi-stage builds as a base image for projects building Rust projects. -They are automatically rebuilt and pushed every night and also on every push to the main branch, in addition a build can be triggered using GitHub Actions. - -The image will run `cargo build --release` in the current context and copy all binaries to an `/app` directory. - -This will bake in the current stable Rust version at the time this image was built, which means it should be rebuilt (and tagged) for every release of Rust. - -## Example usage - -```dockerfile -FROM oci.stackable.tech/ubi9-rust-builder AS builder - -FROM registry.access.redhat.com/ubi9/ubi-minimal AS operator -LABEL maintainer="Stackable GmbH" - -# Update image -RUN microdnf update \ - && microdnf install \ - shadow-utils \ - && rm -rf /var/cache/yum - -COPY --from=builder /app/stackable-zookeeper-operator / - -RUN groupadd -g 1000 stackable && adduser -u 1000 -g stackable -c 'Stackable Operator' stackable - -USER 1000:1000 - -ENTRYPOINT ["/stackable-zookeeper-operator"] +preflight check container oci.stackable.tech/sdp/:-stackable --platform amd64 ``` diff --git a/hadoop/Dockerfile b/hadoop/Dockerfile index 5a8892fc3..2757b4e8c 100644 --- a/hadoop/Dockerfile +++ b/hadoop/Dockerfile @@ -11,7 +11,7 @@ ARG RELEASE_VERSION ARG STACKABLE_USER_UID ARG HADOOP_HADOOP_VERSION # Reassign the arg to `HADOOP_VERSION` for better readability. -# It is passed as `HADOOP_HADOOP_VERSION`, because versions.py has to contain `hadoop/hadoop` to establish a dependency on the Hadoop builder. +# It is passed as `HADOOP_HADOOP_VERSION`, because boil-config.toml has to contain `hadoop/hadoop` to establish a dependency on the Hadoop builder. # The value of `hadoop/hadoop` is transformed by `bake` and automatically passed as `HADOOP_HADOOP_VERSION` arg. ENV HADOOP_VERSION=${HADOOP_HADOOP_VERSION} diff --git a/hbase/Dockerfile b/hbase/Dockerfile index ab41ba14a..f954a7304 100644 --- a/hbase/Dockerfile +++ b/hbase/Dockerfile @@ -22,7 +22,7 @@ ARG HADOOP_HADOOP_VERSION ENV HADOOP_VERSION=${HADOOP_HADOOP_VERSION} ARG HBASE_HBASE_VERSION # Reassign the arg to `HBASE_VERSION` for better readability. -# It is passed as `HBASE_HBASE_VERSION`, because versions.py has to contain `hbase/hbase` to establish a dependency on the HBase builder. +# It is passed as `HBASE_HBASE_VERSION`, because boil-config.toml has to contain `hbase/hbase` to establish a dependency on the HBase builder. # The value of `hbase/hbase` is transformed by `bake` and automatically passed as `HBASE_HBASE_VERSION` arg. ENV HBASE_VERSION=${HBASE_HBASE_VERSION} ARG STACKABLE_USER_UID diff --git a/rust/patchable/README.md b/rust/patchable/README.md index 80bbe59d5..b83816a08 100644 --- a/rust/patchable/README.md +++ b/rust/patchable/README.md @@ -6,10 +6,9 @@ Patchable works by keeping a series of .patch files (in `docker-images/ as its source of truth, but using temporary Git repositories as an interface to modify those patches. This lets us track the history of patches over time, but reuse the existing familiarity and tooling around Git. -Patchable designates a commit as the upstream _base_ for each version, and considers each commit made on top of that -to be an individual patch. +Patchable designates a commit as the upstream _base_ for each version, and considers each commit made on top of that to be an individual patch. -## Usage +## Basic Usage ```sh pushd $(cargo patchable checkout druid 26.0.0) @@ -22,48 +21,160 @@ git status For more details, run `cargo patchable --help`. -## Notes +## Patches -- patchable only supports linear patch series (no merges beyond the base commit) -- patchable doesn't support merging "materialized" trees, merge the .patch files instead, and `checkout`/`export` to update the hashes -- `patchable checkout` doesn't support resolving patch conflicts, use `git am` instead (and then `patchable export` the resolved patches) -- Always run patchable via `cargo patchable` (rather than `cargo install`ing it), to ensure that you use the correct version for a given checkout of docker-images +Patchable doesn't _edit_ anything by itself. Instead, it's a uniform way to apply a set of patches +to an upstream Git repository, and then export your local changes back into patch files. + +It doesn't care about how you make your local changes - you can edit the branch created by +patchable using any Git frontend, such as the git CLI or [jj](https://jj-vcs.github.io/jj/latest/). -## Configuration +This way, the patch files are the global source of truth and track the history of our patch series, +while you can still use the same familiar Git tools to manipulate them. -Patchable uses a two-level configuration system: +### Check out patched sources locally -1. A product-level config file at `docker-images//stackable/patches/patchable.toml` -2. A version-level config file at `docker-images//stackable/patches//patchable.toml` +> [!NOTE] +> This is not required for building images, but is used for when hacking on or debugging patch series. -The product-level config contains: +```sh +# Fetches the upstream repository (if required), and creates a git worktree to work with it +# It also creates two branches: +# - patchable/{version} (HEAD, has all patches applied) +# - patchable/base/{version} (the upstream) +pushd $(cargo patchable checkout druid 26.0.0) + +# Commit to add new patches +# NOTE: the commit message will be used to construct the patch filename. Spaces +# will be converted to hyphens automatically. +git commit + +# Rebase against the base commit to edit or remove patches +git rebase --interactive patchable/base/26.0.0 +# jj edit also works, but make sure to go back to the tip before exporting + +# When done, export your patches and commit them (to docker-images) +popd +cargo patchable export druid 26.0.0 +git status +``` + +> ![CAUTION] +> `cargo patchable export` exports whatever is currently checked out (`HEAD`) in the worktree. +> If you use `jj edit` (or `git switch`) then you _must_ go back to the tip before exporting, or +> any patches after that point will be omitted from the export. + +### Initialize a new product + +If you're adding a completely new product, you need to initialize the product-level config once +using patchable: + +```sh +cargo patchable init product druid \ + --upstream https://github.com/apache/druid.git \ + --default-mirror https://github.com/stackabletech/druid.git +``` + +This will create the product-level configuration in `docker-images/druid/stackable/patches/patchable.toml` +containing the following fields: - `upstream` - the URL of the upstream repository (such as `https://github.com/apache/druid.git`) - `default_mirror` - optional: default URL of a mirror repository (such as `https://github.com/stackabletech/druid.git`) -The version-level config contains: +### Initialize a new patch series + +Patchable stores metadata about each patch series in its version-level config, and will not be able +to check out a patch series that lacks one. It can be generated using the following command: + +```sh +cargo patchable init version druid 28.0.0 --base druid-28.0.0 --mirror +cargo patchable checkout druid 28.0.0 +``` + +This will initialize the version-level config in `docker-images/druid/stackable/patches/28.0.0/patchable.toml` +with the base commit hash and the default mirror URL from the product-level config: - `base` - the commit hash of the upstream base commit - `mirror` - optional: URL of the mirror repository for this version, if mirroring is enabled -### Template +You can optionally provide the `--ssh` flag to use SSH instead of HTTPS for Git operations. + +### Importing patch series into Patchable -If you're adding a completely new product, you need to initialize the product-level config once using patchable: +Patchable is stricter about applying invalid patches (both metadata and patches themselves) than Git is. + +If an initial `cargo patchable checkout` fails then `git am` can be useful for the initial migration: ```sh -cargo patchable init product druid --upstream https://github.com/apache/druid.git --default-mirror https://github.com/stackabletech/druid.git +# Create Patchable configuration for the new version, if it doesn't already exist +cargo patchable init druid 30.0.0 --upstream https://github.com/apache/druid.git --base druid-30.0.0 +# Check out the upstream base commit, without trying to apply the patches +pushd $(cargo patchable checkout druid 30.0.0 --base-only) + +# Apply the patch series +git am ../../../stackable/patches/30.0.0/*.patch +# Resolve any conflicts that arise, and `git am --continue` until done + +# Leave and export the new patch series! +popd +cargo patchable export druid 30.0.0 ``` -This will create the product-level configuration in `docker-images/druid/stackable/patches/patchable.toml`. +### Porting patch series to a new version -If you just want to add a new version, initialize the version-level config with patchable: +Patchable doesn't support restoring a patch series that doesn't apply cleanly. Instead, use `git cherry-pick` to rebase the patch series. + +For example, let's try rebasing our patch series from Druid 26.0.0 to Druid 28.0.0 (which is not packaged by SDP): ```sh -cargo patchable init version druid 28.0.0 --base druid-28.0.0 --mirror +# Restore the old version +# In addition to creating the version worktree, this also creates the branches +# patchable/26.0.0 (26.0.0 with our patches applied) and +# patchable/base/26.0.0 (upstream 26.0.0 with no patches). +cargo patchable checkout druid 26.0.0 + +# Tell Patchable about the new version 28.0.0, which can be fetched from +# https://github.com/apache/druid.git, and has the tag druid-28.0.0 +cargo patchable init druid 28.0.0 --upstream https://github.com/apache/druid.git --base druid-28.0.0 + +# Create and go to the worktree for the new version +pushd $(cargo patchable checkout druid 28.0.0) + +# Cherry pick the old patch series +git cherry-pick patchable/base/26.0.0..patchable/26.0.0 + +# Solve conflicts and `git cherry-pick --continue` until done +# You can also use `git cherry-pick --skip` to skip resolving conflicts for +# patches that are no longer required + +# If some patches are no longer required, use an interactive rebase to remove +# them (or do other cleanup) +git rebase --interactive patchable/base/28.0.0 + +# Leave and export the new patch series! +popd +cargo patchable export druid 28.0.0 +git status ``` -This will initialize the version-level config in `docker-images/druid/stackable/patches/28.0.0/patchable.toml` with the base commit hash and the default mirror URL from the product-level config. -You can optionally provide the `--ssh` flag to use SSH instead of HTTPS for Git operations. +### Porting patches between versions + +Individual patches can also be cherry-picked across versions. + +For example, assuming we are in the Druid 28.0.0 workspace and want to port the last patch of the +Druid 26.0.0 series: + +```sh +# git cherry-pick is also fine for grabbing arbitrary patches +git cherry-pick patchable/26.0.0 +``` + +## Notes + +- patchable only supports linear patch series (no merges beyond the base commit) +- patchable doesn't support merging "materialized" trees, merge the .patch files instead, and `checkout`/`export` to update the hashes +- `patchable checkout` doesn't support resolving patch conflicts, use `git am` instead (and then `patchable export` the resolved patches) +- Always run patchable via `cargo patchable` (rather than `cargo install`ing it), to ensure that you use the correct version for a given checkout of docker-images ## Glossary diff --git a/shared/statsd-exporter/README.md b/shared/statsd-exporter/README.md index ee66a158b..ae4a0786b 100644 --- a/shared/statsd-exporter/README.md +++ b/shared/statsd-exporter/README.md @@ -3,5 +3,5 @@ This is the Docker image that builds statsd_exporter. This image does _not_ need to be built manually. -It will be built automatically if specified as a dependency in our `conf.py` file. +It will be built automatically if specified as a dependency in our `boil-config.toml` file. Which will then be transformed to a dependency in a Docker `buildx bake` file. diff --git a/ubi9-rust-builder/README.md b/ubi9-rust-builder/README.md new file mode 100644 index 000000000..946c1d407 --- /dev/null +++ b/ubi9-rust-builder/README.md @@ -0,0 +1,31 @@ +# ubi9-rust-builder + +These images are meant to be used in multi-stage builds as a base image for projects building Rust projects. +They are automatically rebuilt and pushed every night and also on every push to the main branch, in addition a build can be triggered using GitHub Actions. + +The image will run `cargo build --release` in the current context and copy all binaries to an `/app` directory. + +This will bake in the current stable Rust version at the time this image was built, which means it should be rebuilt (and tagged) for every release of Rust. + +## Example usage + +```dockerfile +FROM oci.stackable.tech/ubi9-rust-builder AS builder + +FROM registry.access.redhat.com/ubi9/ubi-minimal AS operator +LABEL maintainer="Stackable GmbH" + +# Update image +RUN microdnf update \ + && microdnf install \ + shadow-utils \ + && rm -rf /var/cache/yum + +COPY --from=builder /app/stackable-zookeeper-operator / + +RUN groupadd -g 1000 stackable && adduser -u 1000 -g stackable -c 'Stackable Operator' stackable + +USER 1000:1000 + +ENTRYPOINT ["/stackable-zookeeper-operator"] +``` diff --git a/vector/README.md b/vector/README.md index 6fe8b9469..a662dfff5 100644 --- a/vector/README.md +++ b/vector/README.md @@ -3,5 +3,5 @@ This is the Docker image for our Vector image. This image does _not_ need to be built manually. -It will be built automatically if specified as a dependency in our `conf.py` file. +It will be built automatically if specified as a dependency in our `boil-config.toml` file. Which will then be transformed to a dependency in a Docker `buildx bake` file.