Skip to content

Commit 6bc9b97

Browse files
maltesandermaltesander
authored
Fix: kafka reduce image size (#1041)
* reduce image size * adapted changelog * fix linter * consolidation
1 parent e35bb73 commit 6bc9b97

File tree

3 files changed

+76
-44
lines changed

3 files changed

+76
-44
lines changed

‎CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ All notable changes to this project will be documented in this file.
1717
- hive: check for correct permissions and ownerships in /stackable folder via
1818
`check-permissions-ownership.sh` provided in stackable-base image ([#1040]).
1919
- spark-connect-client: A new image for Spark connect tests and demos ([#1034])
20+
- kafka: check for correct permissions and ownerships in /stackable folder via
21+
`check-permissions-ownership.sh` provided in stackable-base image ([#1041]).
2022
- nifi: check for correct permissions and ownerships in /stackable folder via
2123
`check-permissions-ownership.sh` provided in stackable-base image ([#1027]).
2224
- opa: check for correct permissions and ownerships in /stackable folder via
@@ -37,6 +39,7 @@ All notable changes to this project will be documented in this file.
3739
- hadoop: reduce docker image size by removing the recursive chown/chmods in the final image ([#1029]).
3840
- hbase: reduce docker image size by removing the recursive chown/chmods in the final image ([#1028]).
3941
- hive: reduce docker image size by removing the recursive chown/chmods in the final image ([#1040]).
42+
- kafka: reduce docker image size by removing the recursive chown/chmods in the final image ([#1041]).
4043
- nifi: reduce docker image size by removing the recursive chown/chmods in the final image ([#1027]).
4144
- opa: reduce docker image size by removing the recursive chown/chmods in the final image ([#1038]).
4245
- spark-k8s: reduce docker image size by removing the recursive chown/chmods in the final image ([#1042]).
@@ -51,6 +54,7 @@ All notable changes to this project will be documented in this file.
5154
[#1038]: https://github.com/stackabletech/docker-images/pull/1038
5255
[#1039]: https://github.com/stackabletech/docker-images/pull/1039
5356
[#1040]: https://github.com/stackabletech/docker-images/pull/1040
57+
[#1041]: https://github.com/stackabletech/docker-images/pull/1041
5458
[#1042]: https://github.com/stackabletech/docker-images/pull/1042
5559
[#1044]: https://github.com/stackabletech/docker-images/pull/1044
5660
[#1050]: https://github.com/stackabletech/docker-images/pull/1050

‎kafka/Dockerfile

Lines changed: 57 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -27,30 +27,35 @@ WORKDIR /stackable
2727

2828
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/patches/apply_patches.sh /stackable/kafka-${PRODUCT}-src/patches/apply_patches.sh
2929
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/patches/${PRODUCT} /stackable/kafka-${PRODUCT}-src/patches/${PRODUCT}
30+
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/jmx/ /stackable/jmx/
3031

31-
RUN curl "https://repo.stackable.tech/repository/packages/kafka/kafka-${PRODUCT}-src.tgz" | tar -xzC . && \
32-
cd kafka-${PRODUCT}-src && \
33-
./patches/apply_patches.sh ${PRODUCT} && \
34-
# TODO: Try to install gradle via package manager (if possible) instead of fetching it from the internet
35-
# We don't specify "-x test" to skip the tests, as we might bump some Kafka internal dependencies in the future and
36-
# it's a good idea to run the tests in this case.
37-
./gradlew clean releaseTarGz && \
38-
./gradlew cyclonedxBom && \
39-
tar -xf core/build/distributions/kafka_${SCALA}-${PRODUCT}.tgz -C /stackable && \
40-
cp build/reports/bom.json /stackable/kafka_${SCALA}-${PRODUCT}.cdx.json && \
41-
rm -rf /stackable/kafka_${SCALA}-${PRODUCT}/site-docs/ && \
42-
rm -rf /stackable/kafka-${PRODUCT}-src
32+
RUN <<EOF
33+
curl "https://repo.stackable.tech/repository/packages/kafka/kafka-${PRODUCT}-src.tgz" | tar -xzC .
34+
cd kafka-${PRODUCT}-src
35+
./patches/apply_patches.sh ${PRODUCT}
36+
# TODO: Try to install gradle via package manager (if possible) instead of fetching it from the internet
37+
# We don't specify "-x test" to skip the tests, as we might bump some Kafka internal dependencies in the future and
38+
# it's a good idea to run the tests in this case.
39+
./gradlew clean releaseTarGz
40+
./gradlew cyclonedxBom
41+
tar -xf core/build/distributions/kafka_${SCALA}-${PRODUCT}.tgz -C /stackable
42+
cp build/reports/bom.json /stackable/kafka_${SCALA}-${PRODUCT}.cdx.json
43+
rm -rf /stackable/kafka_${SCALA}-${PRODUCT}/site-docs/
44+
rm -rf /stackable/kafka-${PRODUCT}-src
4345

4446
# TODO (@NickLarsenNZ): Compile from source: https://github.com/StyraInc/opa-kafka-plugin
45-
RUN curl https://repo.stackable.tech/repository/packages/kafka-opa-authorizer/opa-authorizer-${OPA_AUTHORIZER}-all.jar \
46-
-o /stackable/kafka_${SCALA}-${PRODUCT}/libs/opa-authorizer-${OPA_AUTHORIZER}-all.jar
47+
curl https://repo.stackable.tech/repository/packages/kafka-opa-authorizer/opa-authorizer-${OPA_AUTHORIZER}-all.jar \
48+
-o /stackable/kafka_${SCALA}-${PRODUCT}/libs/opa-authorizer-${OPA_AUTHORIZER}-all.jar
4749

48-
COPY --chown=${STACKABLE_USER_UID}:0 kafka/stackable/jmx/ /stackable/jmx/
49-
RUN curl https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar \
50-
-o /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar && \
51-
chmod +x /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar && \
52-
ln -s /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar /stackable/jmx/jmx_prometheus_javaagent.jar
50+
# JMX exporter
51+
curl https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar \
52+
-o /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar
53+
chmod +x /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar
54+
ln -s /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar /stackable/jmx/jmx_prometheus_javaagent.jar
5355

56+
# change groups
57+
chmod -R g=u /stackable
58+
EOF
5459

5560
FROM stackable/image/java-base AS final
5661

@@ -60,21 +65,23 @@ ARG SCALA
6065
ARG KCAT
6166
ARG STACKABLE_USER_UID
6267

63-
LABEL name="Apache Kafka" \
64-
maintainer="[email protected]" \
65-
vendor="Stackable GmbH" \
66-
version="${PRODUCT}" \
67-
release="${RELEASE}" \
68-
summary="The Stackable image for Apache Kafka." \
69-
description="This image is deployed by the Stackable Operator for Apache Kafka."
68+
LABEL \
69+
name="Apache Kafka" \
70+
maintainer="[email protected]" \
71+
vendor="Stackable GmbH" \
72+
version="${PRODUCT}" \
73+
release="${RELEASE}" \
74+
summary="The Stackable image for Apache Kafka." \
75+
description="This image is deployed by the Stackable Operator for Apache Kafka."
7076

71-
COPY --chown=${STACKABLE_USER_UID}:0 kafka/licenses /licenses
7277
COPY --chown=${STACKABLE_USER_UID}:0 --from=kafka-builder /stackable/kafka_${SCALA}-${PRODUCT} /stackable/kafka_${SCALA}-${PRODUCT}
7378
COPY --chown=${STACKABLE_USER_UID}:0 --from=kafka-builder /stackable/kafka_${SCALA}-${PRODUCT}.cdx.json /stackable/kafka_${SCALA}-${PRODUCT}/kafka_${SCALA}-${PRODUCT}.cdx.json
7479
COPY --chown=${STACKABLE_USER_UID}:0 --from=kafka-builder /stackable/jmx/ /stackable/jmx/
7580
COPY --chown=${STACKABLE_USER_UID}:0 --from=kcat /stackable/kcat-${KCAT}/kcat /stackable/bin/kcat-${KCAT}
7681
COPY --chown=${STACKABLE_USER_UID}:0 --from=kcat /licenses /licenses
7782

83+
COPY --chown=${STACKABLE_USER_UID}:0 kafka/licenses /licenses
84+
7885
WORKDIR /stackable
7986

8087
RUN <<EOF
@@ -85,24 +92,39 @@ microdnf install \
8592

8693
microdnf clean all
8794
rpm -qa --qf "%{NAME}-%{VERSION}-%{RELEASE}\n" | sort > /stackable/package_manifest.txt
95+
chown ${STACKABLE_USER_UID}:0 /stackable/package_manifest.txt
96+
chmod g=u /stackable/package_manifest.txt
8897
rm -rf /var/cache/yum
8998

9099
ln -s /stackable/bin/kcat-${KCAT} /stackable/bin/kcat
100+
chown -h ${STACKABLE_USER_UID}:0 /stackable/bin/kcat
91101
# kcat was located in /stackable/kcat - legacy
92102
ln -s /stackable/bin/kcat /stackable/kcat
103+
chown -h ${STACKABLE_USER_UID}:0 /stackable/kcat
93104
ln -s /stackable/kafka_${SCALA}-${PRODUCT} /stackable/kafka
105+
chown -h ${STACKABLE_USER_UID}:0 /stackable/kafka
94106

95-
# All files and folders owned by root group to support running as arbitrary users.
96-
# This is best practice as all container users will belong to the root group (0).
97-
chown -R ${STACKABLE_USER_UID}:0 /stackable
98-
chmod -R g=u /stackable
107+
# fix missing permissions
108+
chmod g=u /stackable/bin
109+
chmod g=u /stackable/jmx
110+
chmod g=u /stackable/kafka_${SCALA}-${PRODUCT}
111+
EOF
112+
113+
# ----------------------------------------
114+
# Checks
115+
# This section is to run final checks to ensure the created final images
116+
# adhere to several minimal requirements like:
117+
# - check file permissions and ownerships
118+
# ----------------------------------------
119+
120+
# Check that permissions and ownership in /stackable are set correctly
121+
# This will fail and stop the build if any mismatches are found.
122+
RUN <<EOF
123+
/bin/check-permissions-ownership.sh /stackable ${STACKABLE_USER_UID} 0
99124
EOF
100125

101126
# ----------------------------------------
102-
# Attention: We are changing the group of all files in /stackable directly above
103-
# If you do any file based actions (copying / creating etc.) below this comment you
104-
# absolutely need to make sure that the correct permissions are applied!
105-
# chown ${STACKABLE_USER_UID}:0
127+
# Attention: Do not perform any file based actions (copying/creating etc.) below this comment because the permissions would not be checked.
106128
# ----------------------------------------
107129

108130
USER ${STACKABLE_USER_UID}

‎kcat/Dockerfile

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,9 @@ FROM stackable/image/java-base AS builder
99
ARG PRODUCT
1010
ARG STACKABLE_USER_UID
1111

12-
RUN microdnf update \
13-
&& microdnf install \
12+
RUN <<EOF
13+
microdnf update
14+
microdnf install \
1415
cmake \
1516
cyrus-sasl-devel \
1617
gcc-c++ \
@@ -22,16 +23,21 @@ RUN microdnf update \
2223
wget \
2324
which \
2425
zlib \
25-
zlib-devel && \
26-
microdnf clean all && \
27-
rm -rf /var/cache/yum
26+
zlib-devel
27+
microdnf clean all
28+
rm -rf /var/cache/yum
29+
EOF
2830

2931
WORKDIR /stackable
3032

31-
RUN curl -O https://repo.stackable.tech/repository/packages/kcat/kcat-${PRODUCT}.tar.gz \
32-
&& tar xvfz kcat-${PRODUCT}.tar.gz \
33-
&& cd kcat-${PRODUCT} \
34-
&& ./bootstrap.sh
33+
RUN <<EOF
34+
curl -O https://repo.stackable.tech/repository/packages/kcat/kcat-${PRODUCT}.tar.gz
35+
tar xvfz kcat-${PRODUCT}.tar.gz
36+
cd kcat-${PRODUCT}
37+
./bootstrap.sh
38+
# set correct permissions
39+
chmod --recursive g=u /stackable/kcat-${PRODUCT}
40+
EOF
3541

3642
COPY --chown=${STACKABLE_USER_UID}:0 kcat/licenses /licenses
3743

0 commit comments

Comments
 (0)