Skip to content

Build one aws, azure, and gcp-flavored image per release #12367

@kevingrismore

Description

@kevingrismore

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar request and didn't find it.
  • I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Users must currently either pip install a prefect-* integration package at worker startup time, or build their own worker images, for the following worker types:

AWS Elastic Container Service: prefect-aws
Azure Container Instances: prefect-azure
Google Cloud Run: prefect-gcp
Google Cloud Run V2: prefect-gcp
Google Vertex AI: prefect-gcp

Installing packages at startup time can lead to a few undesirable outcomes, depending on the startup command they choose for their worker.

  1. The integration package requires a newer version of prefect than is on the chosen image and the install is attempted by prefect itself, leading to errors.
  2. The integration package requires a newer version of prefect than is on the chosen image and is installed prior to worker startup, making the running prefect version no longer in sync with the chosen image

The alternative of asking users to build their own images to maintain prefect and integration package compatibility feels like a bumpy experience for maintaining workers.

Describe the proposed behavior

Build a version.number-python3.x-aws, version.number-python3.x-azure, and version.number-python3.x-gcp image per release, where 3.x is strictly one python version. Whether it's 3.11 or 3.12 I'll leave to the experts, but our Helm chart points to 2-python3.11-kubernetes.

This should significantly reduce potential for worker startup related issues, and make onboarding workers significantly easier, especially as we approach agent deprecation. It has the added benefit of pairing Prefect versions with compatible integration packages, as our Kubernetes images already do.

If this is taken up, we should also update any documentation or guides for hosting workers that reference startup commands and images.

Example Use

image: 2-python3.11-gcp
command: prefect worker start -p my-cloud-run-v2-pool

Additional context

No response

Metadata

Metadata

Assignees

Labels

docsenhancementAn improvement of an existing featureintegrationsRelated to integrations with other services

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions