-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Description
First check
- I added a descriptive title to this issue.
- I used the GitHub search to find a similar request and didn't find it.
- I searched the Prefect documentation for this feature.
Prefect Version
2.x
Describe the current behavior
Users must currently either pip install a prefect-* integration package at worker startup time, or build their own worker images, for the following worker types:
AWS Elastic Container Service: prefect-aws
Azure Container Instances: prefect-azure
Google Cloud Run: prefect-gcp
Google Cloud Run V2: prefect-gcp
Google Vertex AI: prefect-gcp
Installing packages at startup time can lead to a few undesirable outcomes, depending on the startup command they choose for their worker.
- The integration package requires a newer version of prefect than is on the chosen image and the install is attempted by prefect itself, leading to errors.
- The integration package requires a newer version of prefect than is on the chosen image and is installed prior to worker startup, making the running prefect version no longer in sync with the chosen image
The alternative of asking users to build their own images to maintain prefect and integration package compatibility feels like a bumpy experience for maintaining workers.
Describe the proposed behavior
Build a version.number-python3.x-aws, version.number-python3.x-azure, and version.number-python3.x-gcp image per release, where 3.x is strictly one python version. Whether it's 3.11 or 3.12 I'll leave to the experts, but our Helm chart points to 2-python3.11-kubernetes.
This should significantly reduce potential for worker startup related issues, and make onboarding workers significantly easier, especially as we approach agent deprecation. It has the added benefit of pairing Prefect versions with compatible integration packages, as our Kubernetes images already do.
If this is taken up, we should also update any documentation or guides for hosting workers that reference startup commands and images.
Example Use
image: 2-python3.11-gcp
command: prefect worker start -p my-cloud-run-v2-pool
Additional context
No response