Skip to content

Adjust release process after quickstart changes #4104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from
Open

Adjust release process after quickstart changes #4104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e30a3fe
Adjust release process after quickstart changes
schustmi Oct 27, 2025
831a33a
Remove more quickstart CI code
schustmi Oct 31, 2025
6fdc3f3
Merge branch 'develop' into misc/release-changes-after-new-quickstart
schustmi Oct 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 1 addition & 10 deletions .github/workflows/release_prepare.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,6 @@ jobs:
fi
done
git add examples/quickstart
- name: Regenerate quickstart cloud requirements
Copy link
Contributor

@bcdurak bcdurak Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see this used to generate examples/quickstart/requirements_{aws,azure,gcp}.txt files for the quickstart testing script. Since this is now disabled (which was the right call I think), wouldn't the qs_run_release_flow fail?

It has a check:

...
REQS="requirements_${CLOUD}.txt"
...
if [[ ! -f "${REQS}" ]]; then
  echo "Error: Requirements file not found: ${REQS}" >&2
  exit 1
fi
...
``

Copy link
Contributor

@bcdurak bcdurak Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same script also looks for config files that do not exist.

Copy link
Contributor

@bcdurak bcdurak Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The installation seems a bit weird as well. The runner starts, checks out to the branch, already install ZenML once, passes the branch name to the test script, the test script manually replaces the hard-coded zenml requirement (which is the only requirement), pip installs zenml throught the requirements.txt file again and only then runs the example. This seems like a very complicated and redundant process.

Copy link
Contributor Author

@schustmi schustmi Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this whole thing has been a mess. I removed as much of it as I could now.

run: |
scripts/generate_cloud_requirements.sh --zenml-version "${{ needs.fetch-versions.outputs.new_version }}"
git add examples/quickstart/requirements_*.txt
# Generate and append release notes
- name: Generate release notes
# Only run this part once when the release branch is created
Expand Down Expand Up @@ -231,11 +227,8 @@ jobs:
matrix:
include:
- cloud: aws
parent_image: 715803424590.dkr.ecr.eu-central-1.amazonaws.com/prepare-release:quickstart-aws
- cloud: azure
parent_image: zenmldocker/prepare-release:quickstart-azure
- cloud: gcp
parent_image: zenmldocker/prepare-release:quickstart-gcp
steps:
# Check out the code
- name: Checkout code
Expand All @@ -253,6 +246,4 @@ jobs:
run: |
scripts/qs_run_release_flow.sh \
--cloud "${{ matrix.cloud }}" \
--parent-image "${{ matrix.parent_image }}" \
--new-version "${{ needs.fetch-versions.outputs.new_version }}" \
--branch-ref "${{ github.ref }}"
--new-version "${{ needs.fetch-versions.outputs.new_version }}"
26 changes: 0 additions & 26 deletions docker/zenml-quickstart-dev.Dockerfile
View file
Open in desktop

This file was deleted.

29 changes: 0 additions & 29 deletions docker/zenml-quickstart.Dockerfile
View file
Open in desktop

This file was deleted.

7 changes: 1 addition & 6 deletions examples/quickstart/pipelines/simple_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,7 @@
from steps.simple_step import simple_step

from zenml import pipeline
from zenml.config import CORSConfig, DeploymentSettings, DockerSettings

docker_settings = DockerSettings(
requirements="requirements.txt",
)
from zenml.config import CORSConfig, DeploymentSettings

deployment_settings = DeploymentSettings(
app_title="Simple Pipeline",
Expand All @@ -19,7 +15,6 @@

@pipeline(
settings={
"docker": docker_settings,
"deployment": deployment_settings,
},
enable_cache=False,
Expand Down
105 changes: 0 additions & 105 deletions release-cloudbuild-preparation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,35 +10,6 @@ steps:
- USERNAME
- PASSWORD

# Build base image
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker build . \
--platform linux/amd64 \
-f docker/zenml-dev.Dockerfile \
-t $$USERNAME/prepare-release:base-${_ZENML_NEW_VERSION}

id: build-base
waitFor: ['-']
entrypoint: bash
secretEnv:
- USERNAME

# Push base image
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- docker push $$USERNAME/prepare-release:base-${_ZENML_NEW_VERSION}
id: push-base
waitFor:
- docker-login
- build-base
entrypoint: bash
secretEnv:
- USERNAME

# Build server image
- name: gcr.io/cloud-builders/docker
args:
Expand Down Expand Up @@ -68,82 +39,6 @@ steps:
secretEnv:
- USERNAME

# Build Quickstart GCP Image
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker build . \
--platform linux/amd64 \
--build-arg BASE_IMAGE=$$USERNAME/prepare-release:base-${_ZENML_NEW_VERSION} \
--build-arg CLOUD_PROVIDER=gcp \
--build-arg ZENML_BRANCH=${_ZENML_BRANCH} \
Copy link
Contributor

@bcdurak bcdurak Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused about all this. Maybe you can clear it for me. I see now that we don't even build any quickstart images for the release (which I like). But if that's the case, why do we even need to build them now for testing the release? Isn't ok for us to use the base image to test?

Previously, different quickstarts required a different "pre-specified" list of requirements to run the quickstart. As far as I can see, this is no longer a concern of the new one.

Copy link
Contributor Author

@schustmi schustmi Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These image still installed the stack requirements in there, but I don't even understand why that was necessary back then. I removed all of that now though.

-f docker/zenml-quickstart-dev.Dockerfile \
-t $$USERNAME/prepare-release:quickstart-gcp-${_ZENML_NEW_VERSION}

id: build-quickstart-gcp
waitFor:
- push-base
entrypoint: bash
secretEnv:
- USERNAME

# Build Quickstart AWS image
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker build . \
--platform linux/amd64 \
--build-arg BASE_IMAGE=$$USERNAME/prepare-release:base-${_ZENML_NEW_VERSION} \
--build-arg CLOUD_PROVIDER=aws \
--build-arg ZENML_BRANCH=${_ZENML_BRANCH} \
-f docker/zenml-quickstart-dev.Dockerfile \
-t $$USERNAME/prepare-release:quickstart-aws-${_ZENML_NEW_VERSION}
id: build-quickstart-aws
waitFor:
- push-base
entrypoint: bash
secretEnv:
- USERNAME

# Build Quickstart Azure image
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker build . \
--platform linux/amd64 \
--build-arg BASE_IMAGE=$$USERNAME/prepare-release:base-${_ZENML_NEW_VERSION} \
--build-arg CLOUD_PROVIDER=azure \
--build-arg ZENML_BRANCH=${_ZENML_BRANCH} \
-f docker/zenml-quickstart-dev.Dockerfile \
-t $$USERNAME/prepare-release:quickstart-azure-${_ZENML_NEW_VERSION}
id: build-quickstart-azure
waitFor:
- push-base
entrypoint: bash
secretEnv:
- USERNAME

# Push Quickstart images
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker push $$USERNAME/prepare-release:quickstart-aws-${_ZENML_NEW_VERSION}
docker push $$USERNAME/prepare-release:quickstart-azure-${_ZENML_NEW_VERSION}
docker push $$USERNAME/prepare-release:quickstart-gcp-${_ZENML_NEW_VERSION}
id: push-quickstart
waitFor:
- docker-login
- build-quickstart-gcp
- build-quickstart-aws
- build-quickstart-azure
entrypoint: bash
secretEnv:
- USERNAME

timeout: 3600s
availableSecrets:
secretManager:
Expand Down
69 changes: 0 additions & 69 deletions release-cloudbuild.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,75 +156,6 @@ steps:
secretEnv:
- USERNAME

# build client quickstart gcp image - python 3.11
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker buildx build \
--platform linux/amd64 \
--build-arg ZENML_VERSION=$TAG_NAME \
--build-arg PYTHON_VERSION=3.11 \
--build-arg CLOUD_PROVIDER=gcp \
-f docker/zenml-quickstart.Dockerfile . \
-t $$USERNAME/zenml-public-pipelines:quickstart-$TAG_NAME-py3.11-gcp
id: build-quickstart-3.11-gcp
waitFor: [ 'push-base' ]
entrypoint: bash
secretEnv:
- USERNAME

# build client quickstart aws image - python 3.11
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker buildx build \
--platform linux/amd64 \
--build-arg ZENML_VERSION=$TAG_NAME \
--build-arg PYTHON_VERSION=3.11 \
--build-arg CLOUD_PROVIDER=aws \
-f docker/zenml-quickstart.Dockerfile . \
-t $$USERNAME/zenml-public-pipelines:quickstart-$TAG_NAME-py3.11-aws
id: build-quickstart-3.11-aws
waitFor: [ 'push-base' ]
entrypoint: bash
secretEnv:
- USERNAME

# build client quickstart azure image - python 3.11
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- |
docker buildx build \
--platform linux/amd64 \
--build-arg ZENML_VERSION=$TAG_NAME \
--build-arg PYTHON_VERSION=3.11 \
--build-arg CLOUD_PROVIDER=azure \
-f docker/zenml-quickstart.Dockerfile . \
-t $$USERNAME/zenml-public-pipelines:quickstart-$TAG_NAME-py3.11-azure
id: build-quickstart-3.11-azure
waitFor: [ 'push-base' ]
entrypoint: bash
secretEnv:
- USERNAME

# push quickstart images
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- docker push --all-tags $$USERNAME/zenml-public-pipelines
id: push-quickstart
waitFor:
- build-quickstart-3.11-gcp
- build-quickstart-3.11-aws
- build-quickstart-3.11-azure
- docker-login
entrypoint: bash
secretEnv:
- USERNAME

timeout: 3600s
availableSecrets:
secretManager:
Expand Down
Loading
Loading