-
Notifications
You must be signed in to change notification settings - Fork 3
ci: Add GitHub workflows to build and publish coordinator runtime, worker runtime, and worker development environment container images. #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 21 commits
4f23c7b
48dff13
9bd9207
4997a02
d74b885
4815351
e7fe20c
fc2cc18
cc2c510
60dd19d
a534bec
9e17bc0
aeb4e48
dbb28fa
1e69681
4ca014c
fec665a
20a7054
283b751
64983f0
a6ebbaa
5ca0420
fa06d23
d120672
4c90aad
3b76c12
b55aeda
74787c7
313a648
47b9c10
ceaa809
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,5 +46,69 @@ jobs: | |
| run: | | ||
| export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}" | ||
| ./mvnw install -B -V -T 1C -DskipTests -Dmaven.javadoc.skip=true --no-transfer-progress -P ci -pl '!presto-test-coverage,!:presto-docs' | ||
| - name: Upload presto-server tarball | ||
| if: matrix.java == '8.0.442' | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: presto-server | ||
| path: presto-server/target/presto-server-0.293.tar.gz | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - name: Upload presto-cli executable | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| if: matrix.java == '8.0.442' | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: presto-cli | ||
| path: presto-cli/target/presto-cli-0.293-executable.jar | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - name: Clean Maven Output | ||
|
||
| run: ./mvnw clean -pl '!:presto-server,!:presto-cli,!presto-test-coverage' | ||
|
|
||
coderabbitai[bot] marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| presto-coordinator-with-clp-connector-runtime-image: | ||
|
||
| name: presto-coordinator-runtime-image | ||
| needs: maven-checks | ||
| runs-on: ubuntu-22.04 | ||
| if: ${{ always() && success() }} | ||
|
||
| steps: | ||
coderabbitai[bot] marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - uses: "actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| with: | ||
| submodules: "recursive" | ||
|
|
||
| - name: Download presto-server | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: presto-server | ||
| path: ./docker | ||
|
|
||
| - name: Download presto-cli | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: presto-cli | ||
| path: ./docker | ||
|
|
||
| - name: "Login to image registry" | ||
| uses: "docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| with: | ||
| registry: ghcr.io | ||
| username: ${{github.actor}} | ||
| password: ${{secrets.GITHUB_TOKEN}} | ||
|
|
||
| - name: "Update Metadata" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| id: "meta" | ||
| uses: "docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| with: | ||
| images: >- | ||
| ghcr.io/${{github.repository}}/coordinator-with-clp-connector-runtime | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| tags: | | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| type=raw,value=centos9 | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| - name: "Build and push" | ||
| uses: "docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| with: | ||
| build-args: |- | ||
| PRESTO_VERSION=0.293 | ||
| JMX_PROMETHEUS_JAVA_AGENT_VERSION=0.20.0 | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| context: "./docker/" | ||
| file: "./docker/Dockerfile" | ||
| push: >- | ||
| ${{github.event_name != 'pull_request' | ||
| && github.ref == 'refs/heads/release-0.293-clp-connector'}} | ||
| tags: "${{steps.meta.outputs.tags}}" | ||
| labels: "${{steps.meta.outputs.labels}}" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| name: prestocpp-worker-with-clp-connector-runtime-image-build | ||
|
||
|
|
||
| on: | ||
| # TODO: specifiy the branch to the release-0.293 when finalize the PR | ||
| pull_request: | ||
| push: | ||
|
|
||
|
||
| jobs: | ||
| prestocpp-worker-with-clp-connector-runtime-image: | ||
|
||
| name: prestocpp-worker-with-clp-connector-runtime-image | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - uses: "actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683" | ||
| with: | ||
| submodules: "recursive" | ||
|
|
||
| - name: "Login to Image Registry" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| uses: "docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772" | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{github.actor}} | ||
| password: ${{secrets.GITHUB_TOKEN}} | ||
|
|
||
| - name: "Update Metadata for Dependency Image" | ||
|
||
| id: "meta-dependency" | ||
|
||
| uses: "docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804" | ||
| with: | ||
| images: >- | ||
| ghcr.io/${{github.repository}}/prestissimo-with-clp-connector-dependency | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| tags: | | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| type=raw,value=ubuntu-22.04 | ||
|
||
| - name: "Update Metadata for Runtime Image" | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| id: "meta-runtime" | ||
|
||
| uses: "docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804" | ||
| with: | ||
| images: >- | ||
| ghcr.io/${{github.repository}}/prestissimo-with-clp-connector-runtime | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| tags: | | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| type=raw,value=ubuntu-22.04 | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
| - name: Check for dependency changes or missing dependency image | ||
| id: dependency-changes | ||
| run: | | ||
| git fetch origin ${{ github.event.before }} --depth=1 || true | ||
| dep_changed=false | ||
| if git diff --name-only ${{ github.event.before }} ${{ github.sha }} | grep -q '^presto-native-execution/scripts'; then | ||
| dep_changed=true | ||
| fi | ||
| velox_old_sha=$(git ls-tree ${{ github.event.before }} presto-native-execution/velox | awk '{print $3}') | ||
| velox_new_sha=$(git ls-tree ${{ github.sha }} presto-native-execution/velox | awk '{print $3}') | ||
| cd presto-native-execution/velox | ||
| if git diff --name-only ${velox_old_sha} ${velox_new_sha} | grep -q '^scripts'; then | ||
| dep_changed=true | ||
| fi | ||
| echo "dep-changed=${dep_changed}" >> $GITHUB_OUTPUT | ||
| image_exists=true | ||
| if ! docker manifest inspect "${{ steps.meta-dependency.outputs.tags }}" > /dev/null 2>&1; then | ||
| image_exists=false | ||
| fi | ||
| echo "image-exists=${image_exists}" >> $GITHUB_OUTPUT | ||
|
||
| - name: "Build and push dependency image" | ||
| if: steps.dependency-changes.outputs.dep-changed == 'true' || steps.dependency-changes.outputs.image-exists == 'false' | ||
|
||
| uses: "docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4" | ||
| with: | ||
| context: "./presto-native-execution/" | ||
| file: "./presto-native-execution/scripts/dockerfiles/ubuntu-22.04-dependency.dockerfile" | ||
| push: >- | ||
| ${{github.event_name != 'pull_request' | ||
| && github.ref == 'refs/heads/release-0.293-clp-connector'}} | ||
| tags: "${{steps.meta-dependency.outputs.tags}}" | ||
| labels: "${{steps.meta-dependency.outputs.labels}}" | ||
|
|
||
|
||
| - name: Get number of CPU | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| id: get-cores | ||
| run: | | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| echo "num-threads=$(nproc)" >> $GITHUB_OUTPUT | ||
|
||
| - name: "Build and push runtime image" | ||
| uses: "docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4" | ||
| with: | ||
| build-args: |- | ||
| NUM_THREADS=${{ steps.get-cores.outputs.num-threads }} | ||
| DEPENDENCY_IMAGE=${{ steps.meta-dependency.outputs.tags }} | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| BASE_IMAGE=ubuntu:22.04 | ||
| OSNAME=ubuntu | ||
| EXTRA_CMAKE_FLAGS=-DPRESTO_ENABLE_TESTING=OFF -DPRESTO_ENABLE_PARQUET=ON -DPRESTO_ENABLE_S3=ON | ||
|
||
| context: "./presto-native-execution/" | ||
| file: "./presto-native-execution/scripts/dockerfiles/prestissimo-runtime.dockerfile" | ||
| push: >- | ||
| ${{github.event_name != 'pull_request' | ||
| && github.ref == 'refs/heads/release-0.293-clp-connector'}} | ||
| tags: "${{steps.meta-runtime.outputs.tags}}" | ||
| labels: "${{steps.meta-runtime.outputs.labels}}" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,7 +12,7 @@ | |
|
|
||
| ARG DEPENDENCY_IMAGE=presto/prestissimo-dependency:centos9 | ||
| ARG BASE_IMAGE=quay.io/centos/centos:stream9 | ||
| FROM ${DEPENDENCY_IMAGE} as prestissimo-image | ||
|
Check warning on line 15 in presto-native-execution/scripts/dockerfiles/prestissimo-runtime.dockerfile
|
||
|
|
||
| ARG OSNAME=centos | ||
| ARG BUILD_TYPE=Release | ||
|
|
@@ -39,6 +39,15 @@ | |
| ENV BUILD_BASE_DIR=_build | ||
| ENV BUILD_DIR="" | ||
|
|
||
| # TODO: This is a temporary fix for issue: https://github.com/prestodb/presto/issues/25531 | ||
| # When there is a proper fix, update this piece of code. | ||
anlowee marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| RUN apt-get update && \ | ||
| DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata && \ | ||
| ln -snf /usr/share/zoneinfo/America/Toronto /etc/localtime && \ | ||
| echo "America/New_York" > /etc/timezone && \ | ||
|
||
| apt-get clean && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
|
||
|
|
||
| COPY --chmod=0775 --from=prestissimo-image /prestissimo/${BUILD_BASE_DIR}/${BUILD_DIR}/presto_cpp/main/presto_server /usr/bin/ | ||
| COPY --chmod=0775 --from=prestissimo-image /runtime-libraries/* /usr/lib64/prestissimo-libs/ | ||
| COPY --chmod=0755 ./etc /opt/presto-server/etc | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.