Skip to content

Commit 92327a9

Browse files
committed
Merge branch 'main' of https://github.com/stackabletech/docker-images into feat/move-patch-apply-logic-to-patchable
2 parents 237d275 + c80ae9a commit 92327a9

File tree

8 files changed

+216
-142
lines changed

8 files changed

+216
-142
lines changed

CHANGELOG.md

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,21 +8,29 @@ All notable changes to this project will be documented in this file.
88

99
- airflow: check for correct permissions and ownerships in /stackable folder via
1010
`check-permissions-ownership.sh` provided in stackable-base image ([#1054]).
11+
- druid: check for correct permissions and ownerships in /stackable folder via
12+
`check-permissions-ownership.sh` provided in stackable-base image ([#1039]).
1113
- hadoop: check for correct permissions and ownerships in /stackable folder via
1214
`check-permissions-ownership.sh` provided in stackable-base image ([#1029]).
1315
- hbase: check for correct permissions and ownerships in /stackable folder via
1416
`check-permissions-ownership.sh` provided in stackable-base image ([#1028]).
15-
- druid: check for correct permissions and ownerships in /stackable folder via
16-
`check-permissions-ownership.sh` provided in stackable-base image ([#1039]).
17+
- hive: check for correct permissions and ownerships in /stackable folder via
18+
`check-permissions-ownership.sh` provided in stackable-base image ([#1040]).
1719
- spark-connect-client: A new image for Spark connect tests and demos ([#1034])
20+
- kafka: check for correct permissions and ownerships in /stackable folder via
21+
`check-permissions-ownership.sh` provided in stackable-base image ([#1041]).
1822
- nifi: check for correct permissions and ownerships in /stackable folder via
1923
`check-permissions-ownership.sh` provided in stackable-base image ([#1027]).
2024
- opa: check for correct permissions and ownerships in /stackable folder via
2125
`check-permissions-ownership.sh` provided in stackable-base image ([#1038]).
26+
- spark-k8s: check for correct permissions and ownerships in /stackable folder via
27+
`check-permissions-ownership.sh` provided in stackable-base image ([#1055]).
2228
- superset: check for correct permissions and ownerships in /stackable folder via
2329
`check-permissions-ownership.sh` provided in stackable-base image ([#1053]).
2430
- trino: check for correct permissions and ownerships in /stackable folder via
2531
`check-permissions-ownership.sh` provided in stackable-base image ([#1025]).
32+
- zookeeper: check for correct permissions and ownerships in /stackable folder via
33+
`check-permissions-ownership.sh` provided in stackable-base image ([#1043]).
2634

2735
### Changed
2836

@@ -31,14 +39,17 @@ All notable changes to this project will be documented in this file.
3139

3240
### Fixed
3341

42+
- druid: reduce docker image size by removing the recursive chown/chmods in the final image ([#1039]).
3443
- hadoop: reduce docker image size by removing the recursive chown/chmods in the final image ([#1029]).
3544
- hbase: reduce docker image size by removing the recursive chown/chmods in the final image ([#1028]).
36-
- druid: reduce docker image size by removing the recursive chown/chmods in the final image ([#1039]).
45+
- hive: reduce docker image size by removing the recursive chown/chmods in the final image ([#1040]).
46+
- kafka: reduce docker image size by removing the recursive chown/chmods in the final image ([#1041]).
47+
- Add `--locked` flag to `cargo install` commands for reproducible builds ([#1044]).
3748
- nifi: reduce docker image size by removing the recursive chown/chmods in the final image ([#1027]).
3849
- opa: reduce docker image size by removing the recursive chown/chmods in the final image ([#1038]).
3950
- spark-k8s: reduce docker image size by removing the recursive chown/chmods in the final image ([#1042]).
4051
- trino: reduce docker image size by removing the recursive chown/chmods in the final image ([#1025]).
41-
- Add `--locked` flag to `cargo install` commands for reproducible builds ([#1044]).
52+
- zookeeper: reduce docker image size by removing the recursive chown/chmods in the final image ([#1043]).
4253

4354
[#1025]: https://github.com/stackabletech/docker-images/pull/1025
4455
[#1027]: https://github.com/stackabletech/docker-images/pull/1027
@@ -47,11 +58,15 @@ All notable changes to this project will be documented in this file.
4758
[#1034]: https://github.com/stackabletech/docker-images/pull/1034
4859
[#1038]: https://github.com/stackabletech/docker-images/pull/1038
4960
[#1039]: https://github.com/stackabletech/docker-images/pull/1039
61+
[#1040]: https://github.com/stackabletech/docker-images/pull/1040
62+
[#1041]: https://github.com/stackabletech/docker-images/pull/1041
5063
[#1042]: https://github.com/stackabletech/docker-images/pull/1042
64+
[#1043]: https://github.com/stackabletech/docker-images/pull/1043
5165
[#1044]: https://github.com/stackabletech/docker-images/pull/1044
5266
[#1050]: https://github.com/stackabletech/docker-images/pull/1050
5367
[#1053]: https://github.com/stackabletech/docker-images/pull/1053
5468
[#1054]: https://github.com/stackabletech/docker-images/pull/1054
69+
[#1055]: https://github.com/stackabletech/docker-images/pull/1055
5570

5671
## [25.3.0] - 2025-03-21
5772

hive/Dockerfile

Lines changed: 45 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ FROM stackable/image/java-devel AS hive-builder
1313
ARG PRODUCT
1414
ARG HADOOP
1515
ARG JMX_EXPORTER
16+
ARG AWS_JAVA_SDK_BUNDLE
17+
ARG AZURE_STORAGE
18+
ARG AZURE_KEYVAULT_CORE
1619
ARG STACKABLE_USER_UID
1720

1821
# Setting this to anything other than "true" will keep the cache folders around (e.g. for Maven, NPM etc.)
@@ -25,6 +28,10 @@ COPY --chown=${STACKABLE_USER_UID}:0 hive/stackable/patches/${PRODUCT} /stackabl
2528
COPY --chown=${STACKABLE_USER_UID}:0 hive/stackable/jmx /stackable/jmx
2629
# Copy the start script into the builder
2730
COPY --chown=${STACKABLE_USER_UID}:0 hive/stackable/bin /stackable/bin
31+
# It is useful to see which version of Hadoop is used at a glance
32+
# Therefore the use of the full name here
33+
# TODO: Do we really need all of Hadoop in here?
34+
COPY --chown=${STACKABLE_USER_UID}:0 --from=hadoop-builder /stackable/hadoop /stackable/hadoop-${HADOOP}
2835

2936
USER ${STACKABLE_USER_UID}
3037
WORKDIR /stackable
@@ -64,6 +71,18 @@ cp bin/start-metastore apache-hive-metastore-${PRODUCT}-bin/bin
6471
curl "https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" -o "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar"
6572
ln -s "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" /stackable/jmx/jmx_prometheus_javaagent.jar
6673

74+
# The next two sections for S3 and Azure use hardcoded version numbers on purpose instead of wildcards
75+
# This way the build will fail should one of the files not be available anymore in a later Hadoop version!
76+
77+
# Add S3 Support for Hive (support for s3a://)
78+
cp /stackable/hadoop-${HADOOP}/share/hadoop/tools/lib/hadoop-aws-${HADOOP}.jar /stackable/apache-hive-metastore-${PRODUCT}-bin/lib/
79+
cp /stackable/hadoop-${HADOOP}/share/hadoop/tools/lib/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE}.jar /stackable/apache-hive-metastore-${PRODUCT}-bin/lib/
80+
81+
# Add Azure ABFS support (support for abfs://)
82+
cp /stackable/hadoop-${HADOOP}/share/hadoop/tools/lib/hadoop-azure-${HADOOP}.jar /stackable/apache-hive-metastore-${PRODUCT}-bin/lib/
83+
cp /stackable/hadoop-${HADOOP}/share/hadoop/tools/lib/azure-storage-${AZURE_STORAGE}.jar /stackable/apache-hive-metastore-${PRODUCT}-bin/lib/
84+
cp /stackable/hadoop-${HADOOP}/share/hadoop/tools/lib/azure-keyvault-core-${AZURE_KEYVAULT_CORE}.jar /stackable/apache-hive-metastore-${PRODUCT}-bin/lib/
85+
6786
# We're removing these to make the intermediate layer smaller
6887
# This can be necessary even though it's only a builder image because the GitHub Action Runners only have very limited space available
6988
# and we are sometimes running into errors because we're out of space.
@@ -73,6 +92,9 @@ if [ "${DELETE_CACHES}" = "true" ] ; then
7392
rm -rf /stackable/.npm/*
7493
rm -rf /stackable/.cache/*
7594
fi
95+
96+
# change groups
97+
chmod --recursive g=u /stackable
7698
EOF
7799

78100

@@ -81,9 +103,6 @@ FROM stackable/image/java-base AS final
81103
ARG PRODUCT
82104
ARG HADOOP
83105
ARG RELEASE
84-
ARG AWS_JAVA_SDK_BUNDLE
85-
ARG AZURE_STORAGE
86-
ARG AZURE_KEYVAULT_CORE
87106
ARG STACKABLE_USER_UID
88107

89108

@@ -112,47 +131,45 @@ LABEL io.k8s.display-name="${NAME}"
112131
WORKDIR /stackable
113132

114133
COPY --chown=${STACKABLE_USER_UID}:0 --from=hive-builder /stackable/apache-hive-metastore-${PRODUCT}-bin /stackable/apache-hive-metastore-${PRODUCT}-bin
134+
COPY --chown=${STACKABLE_USER_UID}:0 --from=hive-builder /stackable/hadoop-${HADOOP} /stackable/hadoop-${HADOOP}
135+
COPY --chown=${STACKABLE_USER_UID}:0 --from=hive-builder /stackable/jmx /stackable/jmx
115136

116-
# It is useful to see which version of Hadoop is used at a glance
117-
# Therefore the use of the full name here
118-
# TODO: Do we really need all of Hadoop in here?
119-
COPY --chown=${STACKABLE_USER_UID}:0 --from=hadoop-builder /stackable/hadoop /stackable/hadoop-${HADOOP}
137+
COPY hive/licenses /licenses
120138

121139
RUN <<EOF
122140
microdnf update
123141
microdnf clean all
124142
rpm -qa --qf "%{NAME}-%{VERSION}-%{RELEASE}\n" | sort > /stackable/package_manifest.txt
143+
chown ${STACKABLE_USER_UID}:0 /stackable/package_manifest.txt
144+
chmod g=u /stackable/package_manifest.txt
125145
rm -rf /var/cache/yum
126146

127147
ln -s /stackable/apache-hive-metastore-${PRODUCT}-bin /stackable/hive-metastore
148+
chown -h ${STACKABLE_USER_UID}:0 /stackable/hive-metastore
149+
chmod g=u /stackable/hive-metastore
128150
ln -s /stackable/hadoop-${HADOOP} /stackable/hadoop
151+
chown -h ${STACKABLE_USER_UID}:0 /stackable/hadoop
152+
chmod g=u /stackable/hadoop
129153

130-
# The next two sections for S3 and Azure use hardcoded version numbers on purpose instead of wildcards
131-
# This way the build will fail should one of the files not be available anymore in a later Hadoop version!
154+
# fix missing permissions
155+
chmod --recursive g=u /stackable/jmx
156+
EOF
132157

133-
# Add S3 Support for Hive (support for s3a://)
134-
cp /stackable/hadoop/share/hadoop/tools/lib/hadoop-aws-${HADOOP}.jar /stackable/hive-metastore/lib/
135-
cp /stackable/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE}.jar /stackable/hive-metastore/lib/
158+
# ----------------------------------------
159+
# Checks
160+
# This section is to run final checks to ensure the created final images
161+
# adhere to several minimal requirements like:
162+
# - check file permissions and ownerships
163+
# ----------------------------------------
136164

137-
# Add Azure ABFS support (support for abfs://)
138-
cp /stackable/hadoop/share/hadoop/tools/lib/hadoop-azure-${HADOOP}.jar /stackable/hive-metastore/lib/
139-
cp /stackable/hadoop/share/hadoop/tools/lib/azure-storage-${AZURE_STORAGE}.jar /stackable/hive-metastore/lib/
140-
cp /stackable/hadoop/share/hadoop/tools/lib/azure-keyvault-core-${AZURE_KEYVAULT_CORE}.jar /stackable/hive-metastore/lib/
141-
142-
# All files and folders owned by root group to support running as arbitrary users.
143-
# This is best practice as all container users will belong to the root group (0).
144-
chown -R ${STACKABLE_USER_UID}:0 /stackable
145-
chmod -R g=u /stackable
165+
# Check that permissions and ownership in /stackable are set correctly
166+
# This will fail and stop the build if any mismatches are found.
167+
RUN <<EOF
168+
/bin/check-permissions-ownership.sh /stackable ${STACKABLE_USER_UID} 0
146169
EOF
147170

148-
COPY --chown=${STACKABLE_USER_UID}:0 --from=hive-builder /stackable/jmx /stackable/jmx
149-
COPY hive/licenses /licenses
150-
151171
# ----------------------------------------
152-
# Attention: We are changing the group of all files in /stackable directly above
153-
# If you do any file based actions (copying / creating etc.) below this comment you
154-
# absolutely need to make sure that the correct permissions are applied!
155-
# chown ${STACKABLE_USER_UID}:0
172+
# Attention: Do not perform any file based actions (copying/creating etc.) below this comment because the permissions would not be checked.
156173
# ----------------------------------------
157174

158175
USER ${STACKABLE_USER_UID}

kafka/Dockerfile

Lines changed: 56 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -14,29 +14,35 @@ ARG STACKABLE_USER_UID
1414
USER ${STACKABLE_USER_UID}
1515
WORKDIR /stackable
1616

17+
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/jmx/ /stackable/jmx/
1718
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/patches/${PRODUCT} /stackable/src/kafka/stackable/patches/${PRODUCT}
1819

19-
RUN cd "$(/stackable/patchable --images-repo-root=src checkout kafka ${PRODUCT})" && \
20-
# TODO: Try to install gradle via package manager (if possible) instead of fetching it from the internet
21-
# We don't specify "-x test" to skip the tests, as we might bump some Kafka internal dependencies in the future and
22-
# it's a good idea to run the tests in this case.
23-
./gradlew clean releaseTarGz && \
24-
./gradlew cyclonedxBom && \
25-
tar -xf core/build/distributions/kafka_${SCALA}-${PRODUCT}.tgz -C /stackable && \
26-
cp build/reports/bom.json /stackable/kafka_${SCALA}-${PRODUCT}.cdx.json && \
27-
rm -rf /stackable/kafka_${SCALA}-${PRODUCT}/site-docs/ && \
28-
rm -rf /stackable/kafka-${PRODUCT}-src
20+
RUN <<EOF
21+
cd "$(/stackable/patchable --images-repo-root=src checkout kafka ${PRODUCT})"
22+
23+
# TODO: Try to install gradle via package manager (if possible) instead of fetching it from the internet
24+
# We don't specify "-x test" to skip the tests, as we might bump some Kafka internal dependencies in the future and
25+
# it's a good idea to run the tests in this case.
26+
./gradlew clean releaseTarGz
27+
./gradlew cyclonedxBom
28+
tar -xf core/build/distributions/kafka_${SCALA}-${PRODUCT}.tgz -C /stackable
29+
cp build/reports/bom.json /stackable/kafka_${SCALA}-${PRODUCT}.cdx.json
30+
rm -rf /stackable/kafka_${SCALA}-${PRODUCT}/site-docs/
31+
(cd .. && rm -rf ${PRODUCT})
2932

3033
# TODO (@NickLarsenNZ): Compile from source: https://github.com/StyraInc/opa-kafka-plugin
31-
RUN curl https://repo.stackable.tech/repository/packages/kafka-opa-authorizer/opa-authorizer-${OPA_AUTHORIZER}-all.jar \
32-
-o /stackable/kafka_${SCALA}-${PRODUCT}/libs/opa-authorizer-${OPA_AUTHORIZER}-all.jar
34+
curl https://repo.stackable.tech/repository/packages/kafka-opa-authorizer/opa-authorizer-${OPA_AUTHORIZER}-all.jar \
35+
-o /stackable/kafka_${SCALA}-${PRODUCT}/libs/opa-authorizer-${OPA_AUTHORIZER}-all.jar
3336

34-
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/jmx/ /stackable/jmx/
35-
RUN curl https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar \
36-
-o /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar && \
37-
chmod +x /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar && \
38-
ln -s /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar /stackable/jmx/jmx_prometheus_javaagent.jar
37+
# JMX exporter
38+
curl https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar \
39+
-o /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar
40+
chmod +x /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar
41+
ln -s /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar /stackable/jmx/jmx_prometheus_javaagent.jar
3942

43+
# change groups
44+
chmod -R g=u /stackable
45+
EOF
4046

4147
FROM stackable/image/java-base AS final
4248

@@ -46,21 +52,23 @@ ARG SCALA
4652
ARG KCAT
4753
ARG STACKABLE_USER_UID
4854

49-
LABEL name="Apache Kafka" \
50-
maintainer="[email protected]" \
51-
vendor="Stackable GmbH" \
52-
version="${PRODUCT}" \
53-
release="${RELEASE}" \
54-
summary="The Stackable image for Apache Kafka." \
55-
description="This image is deployed by the Stackable Operator for Apache Kafka."
55+
LABEL \
56+
name="Apache Kafka" \
57+
maintainer="[email protected]" \
58+
vendor="Stackable GmbH" \
59+
version="${PRODUCT}" \
60+
release="${RELEASE}" \
61+
summary="The Stackable image for Apache Kafka." \
62+
description="This image is deployed by the Stackable Operator for Apache Kafka."
5663

57-
COPY --chown=${STACKABLE_USER_UID}:0 kafka/licenses /licenses
5864
COPY --chown=${STACKABLE_USER_UID}:0 --from=kafka-builder /stackable/kafka_${SCALA}-${PRODUCT} /stackable/kafka_${SCALA}-${PRODUCT}
5965
COPY --chown=${STACKABLE_USER_UID}:0 --from=kafka-builder /stackable/kafka_${SCALA}-${PRODUCT}.cdx.json /stackable/kafka_${SCALA}-${PRODUCT}/kafka_${SCALA}-${PRODUCT}.cdx.json
6066
COPY --chown=${STACKABLE_USER_UID}:0 --from=kafka-builder /stackable/jmx/ /stackable/jmx/
6167
COPY --chown=${STACKABLE_USER_UID}:0 --from=kcat /stackable/kcat-${KCAT}/kcat /stackable/bin/kcat-${KCAT}
6268
COPY --chown=${STACKABLE_USER_UID}:0 --from=kcat /licenses /licenses
6369

70+
COPY --chown=${STACKABLE_USER_UID}:0 kafka/licenses /licenses
71+
6472
WORKDIR /stackable
6573

6674
RUN <<EOF
@@ -71,24 +79,39 @@ microdnf install \
7179

7280
microdnf clean all
7381
rpm -qa --qf "%{NAME}-%{VERSION}-%{RELEASE}\n" | sort > /stackable/package_manifest.txt
82+
chown ${STACKABLE_USER_UID}:0 /stackable/package_manifest.txt
83+
chmod g=u /stackable/package_manifest.txt
7484
rm -rf /var/cache/yum
7585

7686
ln -s /stackable/bin/kcat-${KCAT} /stackable/bin/kcat
87+
chown -h ${STACKABLE_USER_UID}:0 /stackable/bin/kcat
7788
# kcat was located in /stackable/kcat - legacy
7889
ln -s /stackable/bin/kcat /stackable/kcat
90+
chown -h ${STACKABLE_USER_UID}:0 /stackable/kcat
7991
ln -s /stackable/kafka_${SCALA}-${PRODUCT} /stackable/kafka
92+
chown -h ${STACKABLE_USER_UID}:0 /stackable/kafka
8093

81-
# All files and folders owned by root group to support running as arbitrary users.
82-
# This is best practice as all container users will belong to the root group (0).
83-
chown -R ${STACKABLE_USER_UID}:0 /stackable
84-
chmod -R g=u /stackable
94+
# fix missing permissions
95+
chmod g=u /stackable/bin
96+
chmod g=u /stackable/jmx
97+
chmod g=u /stackable/kafka_${SCALA}-${PRODUCT}
98+
EOF
99+
100+
# ----------------------------------------
101+
# Checks
102+
# This section is to run final checks to ensure the created final images
103+
# adhere to several minimal requirements like:
104+
# - check file permissions and ownerships
105+
# ----------------------------------------
106+
107+
# Check that permissions and ownership in /stackable are set correctly
108+
# This will fail and stop the build if any mismatches are found.
109+
RUN <<EOF
110+
/bin/check-permissions-ownership.sh /stackable ${STACKABLE_USER_UID} 0
85111
EOF
86112

87113
# ----------------------------------------
88-
# Attention: We are changing the group of all files in /stackable directly above
89-
# If you do any file based actions (copying / creating etc.) below this comment you
90-
# absolutely need to make sure that the correct permissions are applied!
91-
# chown ${STACKABLE_USER_UID}:0
114+
# Attention: Do not perform any file based actions (copying/creating etc.) below this comment because the permissions would not be checked.
92115
# ----------------------------------------
93116

94117
USER ${STACKABLE_USER_UID}

kcat/Dockerfile

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,9 @@ FROM stackable/image/java-base AS builder
99
ARG PRODUCT
1010
ARG STACKABLE_USER_UID
1111

12-
RUN microdnf update \
13-
&& microdnf install \
12+
RUN <<EOF
13+
microdnf update
14+
microdnf install \
1415
cmake \
1516
cyrus-sasl-devel \
1617
gcc-c++ \
@@ -22,16 +23,21 @@ RUN microdnf update \
2223
wget \
2324
which \
2425
zlib \
25-
zlib-devel && \
26-
microdnf clean all && \
27-
rm -rf /var/cache/yum
26+
zlib-devel
27+
microdnf clean all
28+
rm -rf /var/cache/yum
29+
EOF
2830

2931
WORKDIR /stackable
3032

31-
RUN curl -O https://repo.stackable.tech/repository/packages/kcat/kcat-${PRODUCT}.tar.gz \
32-
&& tar xvfz kcat-${PRODUCT}.tar.gz \
33-
&& cd kcat-${PRODUCT} \
34-
&& ./bootstrap.sh
33+
RUN <<EOF
34+
curl -O https://repo.stackable.tech/repository/packages/kcat/kcat-${PRODUCT}.tar.gz
35+
tar xvfz kcat-${PRODUCT}.tar.gz
36+
cd kcat-${PRODUCT}
37+
./bootstrap.sh
38+
# set correct permissions
39+
chmod --recursive g=u /stackable/kcat-${PRODUCT}
40+
EOF
3541

3642
COPY --chown=${STACKABLE_USER_UID}:0 kcat/licenses /licenses
3743

0 commit comments

Comments
 (0)