-
-
Notifications
You must be signed in to change notification settings - Fork 6
Fix: trino reduce image size #1025
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 14 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
9f51159
improve chmod/chown to reduce image size
maltesander d61b9e9
fix auto lint
maltesander 47cacf7
summarize chmod
maltesander 4c86f09
remove log4shell check
maltesander 6bf1a21
fixes
maltesander 463954f
fix linter
maltesander da10bdc
cleanup
maltesander 7072c1a
Merge remote-tracking branch 'origin/main' into fix/trino-reduce-imag…
maltesander 86163ae
adapted changelog
maltesander 28dd434
linter
maltesander 28f0039
Update CHANGELOG.md
maltesander 81fc95c
check permissions
maltesander cd201ad
Merge remote-tracking branch 'origin/main' into fix/trino-reduce-imag…
maltesander 007041a
consolidation
maltesander 5d382db
Update trino/Dockerfile
maltesander f7452ef
Update trino/Dockerfile
maltesander 53b53eb
missing doc comment
maltesander f7a06e7
Merge remote-tracking branch 'origin/main' into fix/trino-reduce-imag…
maltesander 16cdf3a
copy storage connector directly
maltesander File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,6 +7,7 @@ FROM stackable/image/java-devel AS trino-builder | |
|
|
||
| ARG PRODUCT | ||
| ARG STACKABLE_USER_UID | ||
| ARG JMX_EXPORTER | ||
|
|
||
| RUN <<EOF | ||
| microdnf update | ||
|
|
@@ -25,6 +26,8 @@ RUN curl "https://repo.stackable.tech/repository/packages/trino-server/trino-ser | |
|
|
||
| COPY --chown=${STACKABLE_USER_UID}:0 trino/stackable/patches/apply_patches.sh /stackable/trino-server-${PRODUCT}-src/patches/apply_patches.sh | ||
| COPY --chown=${STACKABLE_USER_UID}:0 trino/stackable/patches/${PRODUCT} /stackable/trino-server-${PRODUCT}-src/patches/${PRODUCT} | ||
| COPY --chown=${STACKABLE_USER_UID}:0 --from=trino-storage-connector-image /stackable/trino-storage-${PRODUCT}-src/target/trino-storage-${PRODUCT} /trino-storage-${PRODUCT} | ||
| COPY --chown=${STACKABLE_USER_UID}:0 trino/stackable/jmx /stackable/jmx | ||
|
|
||
| # adding a hadolint ignore for SC2215, due to https://github.com/hadolint/hadolint/issues/980 | ||
| # hadolint ignore=SC2215 | ||
|
|
@@ -47,90 +50,83 @@ git tag ${PRODUCT} | |
| # We need to use ./mvnw instead of mvn to get a recent maven version (which is required to build Trino) | ||
| ./mvnw --batch-mode --no-transfer-progress package -DskipTests --projects="!docs,!core/trino-server-rpm" | ||
|
|
||
| # Delete the worst intermediate build products to free some space | ||
| # Delete intermediate build products to free some space and keep runners happy | ||
| rm -r /stackable/trino-server-${PRODUCT}-src/plugin/*/target /stackable/trino-server-${PRODUCT}-src/core/trino-server/target/trino-server-${PRODUCT} | ||
|
|
||
| # Extract from tarball to save space; the tarball deduplicates jars (replacing them with symlinks), | ||
| # while the raw output folder does not | ||
| tar -xzf /stackable/trino-server-${PRODUCT}-src/core/trino-server/target/trino-server-${PRODUCT}.tar.gz -C /stackable | ||
| mv /stackable/trino-server-${PRODUCT}-src/core/trino-server/target/bom.json /stackable/trino-server-${PRODUCT}/trino-server-${PRODUCT}.cdx.json | ||
| chown --recursive ${STACKABLE_USER_UID}:0 /stackable/trino-server-${PRODUCT} | ||
|
|
||
| # Delete all intermediate build products to free some more space | ||
| # Delete intermediate build products to free some space and keep runners happy | ||
| rm -r /stackable/trino-server-${PRODUCT}-src | ||
| rm -r /stackable/.m2 | ||
|
|
||
| # JMX Exporter | ||
| curl --fail https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar -o /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar | ||
| chmod +x /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar | ||
|
|
||
| # Storage connector | ||
| mv /trino-storage-${PRODUCT}/ /stackable/trino-server-${PRODUCT}/plugin/trino-storage-${PRODUCT}/ | ||
siegfriedweber marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| # Softlinks | ||
| ln -s /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar /stackable/jmx/jmx_prometheus_javaagent.jar | ||
| ln -s /stackable/trino-server-${PRODUCT} /stackable/trino-server | ||
|
|
||
| # Set correct permissions | ||
| chmod --recursive g=u /stackable | ||
| EOF | ||
|
|
||
| COPY --from=trino-storage-connector-image /stackable/trino-storage-${PRODUCT}-src/target/trino-storage-${PRODUCT} /stackable/trino-server-${PRODUCT}/plugin/trino-storage-${PRODUCT} | ||
| # For earlier versions this script removes the .class file that contains the | ||
| # vulnerable code. | ||
| # TODO: This can be restricted to target only versions which do not honor the environment | ||
| # varible that has been set above but this has not currently been implemented | ||
| COPY shared/log4shell.sh /bin | ||
| RUN /bin/log4shell.sh /stackable/trino-server-${PRODUCT} | ||
|
|
||
| # Ensure no vulnerable files are left over | ||
| # This will currently report vulnerable files being present, as it also alerts on | ||
| # SocketNode.class, which we do not remove with our scripts. | ||
| # Further investigation will be needed whether this should also be removed. | ||
| COPY shared/log4shell_1.6.1-log4shell_Linux_x86_64 /bin/log4shell_scanner_x86_64 | ||
| COPY shared/log4shell_1.6.1-log4shell_Linux_aarch64 /bin/log4shell_scanner_aarch64 | ||
| COPY shared/log4shell_scanner /bin/log4shell_scanner | ||
| RUN /bin/log4shell_scanner s /stackable/trino-server-${PRODUCT} | ||
| # === | ||
|
|
||
| FROM stackable/image/java-base AS final | ||
| FROM stackable/image/java-base | ||
|
|
||
| ARG PRODUCT | ||
| ARG JMX_EXPORTER | ||
| ARG RELEASE | ||
| ARG STACKABLE_USER_UID | ||
|
|
||
| LABEL name="Trino" \ | ||
| maintainer="[email protected]" \ | ||
| vendor="Stackable GmbH" \ | ||
| version="${PRODUCT}" \ | ||
| release="${RELEASE}" \ | ||
| summary="The Stackable image for Trino." \ | ||
| description="This image is deployed by the Stackable Operator for Trino." | ||
| LABEL \ | ||
| name="Trino" \ | ||
| maintainer="[email protected]" \ | ||
| vendor="Stackable GmbH" \ | ||
| version="${PRODUCT}" \ | ||
| release="${RELEASE}" \ | ||
| summary="The Stackable image for Trino." \ | ||
| description="This image is deployed by the Stackable Operator for Trino." | ||
|
|
||
| RUN microdnf update && \ | ||
| microdnf install \ | ||
| RUN <<EOF | ||
| microdnf update | ||
| # libstdc++ required by snappy and duckdb, see https://github.com/trinodb/trino/pull/25143 | ||
| microdnf install \ | ||
| gzip \ | ||
| httpd-tools \ | ||
| python \ | ||
| tar \ | ||
| # Required by snappy and duckdb, see https://github.com/trinodb/trino/pull/25143 | ||
| libstdc++ \ | ||
| zip && \ | ||
| microdnf clean all && \ | ||
| rm -rf /var/cache/yum | ||
|
|
||
| WORKDIR /stackable | ||
| zip | ||
| microdnf clean all | ||
| rm -rf /var/cache/yum | ||
| EOF | ||
|
|
||
| COPY --chown=${STACKABLE_USER_UID}:0 trino/stackable /stackable | ||
| # If /stackable has any build artifacts / leftovers make sure its removed properly | ||
| # or only copy what is actually required in the final image like: | ||
| # COPY --from=trino-builder --chown=${STACKABLE_USER_UID}:0 /stackable/foo /stackable/foo | ||
maltesander marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| COPY --from=trino-builder --chown=${STACKABLE_USER_UID}:0 /stackable /stackable | ||
| COPY --chown=${STACKABLE_USER_UID}:0 trino/licenses /licenses | ||
|
|
||
| COPY --from=trino-builder /stackable/trino-server-${PRODUCT} /stackable/trino-server-${PRODUCT} | ||
| # ---------------------------------------- | ||
| # Checks | ||
| # This section is to run final checks to ensure the created final images | ||
| # adhere to several minimal requirements like: | ||
| # - check file permissions and ownerships | ||
| # ---------------------------------------- | ||
|
|
||
| # Check that permissions and ownership in /stackable are set correctly | ||
| # This will fail and stop the build if any mismatches are found. | ||
| RUN <<EOF | ||
| ln -s /stackable/trino-server-${PRODUCT} /stackable/trino-server | ||
|
|
||
| curl --fail https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar -o /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar | ||
| chmod +x /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar | ||
| ln -s /stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar /stackable/jmx/jmx_prometheus_javaagent.jar | ||
|
|
||
| # All files and folders owned by root group to support running as arbitrary users. | ||
| # This is best practice as all container users will belong to the root group (0). | ||
| chown -R ${STACKABLE_USER_UID}:0 /stackable | ||
| chmod -R g=u /stackable | ||
| /bin/check-permissions-ownership.sh /stackable ${STACKABLE_USER_UID} 0 | ||
| EOF | ||
|
|
||
| # ---------------------------------------- | ||
| # Attention: We are changing the group of all files in /stackable directly above | ||
| # If you do any file based actions (copying / creating etc.) below this comment you | ||
| # absolutely need to make sure that the correct permissions are applied! | ||
| # chown ${STACKABLE_USER_UID}:0 | ||
| # ---------------------------------------- | ||
| # Attention: Do not perform any file based actions (copying/creating etc.) below this comment because the permissions would not be checked. | ||
|
|
||
| USER ${STACKABLE_USER_UID} | ||
| WORKDIR /stackable/trino-server | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.