Support all work pool types

[chrisguidry]

In the current state, work pools may be of type process or kubernetes (due to a quick shortcut/hack to use the -kubernetes prefect image). Any other type of work pool would require a custom image or the PIP_EXTRA_PACKAGES setting to load the additional integration packages with the correct worker implementations.

We should be able to support any kind of Prefect-maintained work pool declaratively by arranging for the installation of the appropriate integration package. For example, when using ecs, we'd arrange for prefect-aws to be installed in the worker, etc. The general-purpose way to accomplish this would be by reading the global collection catalog and installing (via PIP_EXTRA_PACKAGES?) the correct Prefect integration.

Hi @chrisguidry - Would you mind expanding on this feature request? How would this feature request look from the customer perspective of instantiating a PrefectWorkpool with the CRD interface?

[chrisguidry]

Hi @josephmcasey 👋, thanks for reaching out!

So this was just kind of aspirational when I wrote it, because I'm not a huge fan of having container images install extra software at startup. But since then, we've had some discussions about publishing more flavors of the prefecthq/prefect image for the top cloud providers where the prefect-* integration packages would be pre-installed.

Let's assume that we start publishing a prefecthq/prefect:3-latest-aws image that has the prefect-aws Python package with the ECSWorker and friends installed. If that were the case, this would be a lot cleaner.

From the CRD perspective, I think it would be pretty simple:

apiVersion: prefect.io/v1
kind: PrefectWorkPool
metadata:
  name: my-ecs-pool  # corresponds to the name of the pool on the Prefect server
spec:
  type: ecs
  queue: high-priority
  server:
    name: prefect-ephemeral
  workers: 3

This would be the case with any of the hybrid work pool types, like the GCP cloud-run-v2 worker. In essence, the CRD is saying how many workers to run in Kubernetes for the given (pre-existing) work pool on the Prefect server. These workers would be configured to run prefect worker start --pool <metadata.name> --type <spec.type> --queue <spec.queue>.

It reads to me like what you're describing is a two-part feature in which the list below is published as images on public registries similar to how the *-.kubernetes tagged images are currently being released. Is this a correct interpretation?

prefect-aws
prefect-azure
prefect-bitbucket
prefect-dask
prefect-databricks
prefect-dbt
prefect-docker
prefect-email
prefect-gcp
prefect-github
prefect-gitlab
prefect-kubernetes
prefect-ray
prefect-redis
prefect-shell
prefect-slack
prefect-snowflake
prefect-sqlalchemy

If so, following the releases of those images, the feature would then support taking the value specified in the property .spec.type and placing that into the long-option --type.

Feature One:
https://github.com/PrefectHQ/prefect/blob/378fc4f6b72aace32343264ee6cea198d949ba6a/.github/workflows/docker-images.yaml#L24-L37

Feature Two:

prefect-operator/deploy/charts/prefect-operator/crds/prefect.io_prefectworkpools.yaml

Lines 62 to 64 in 3ecf633

	spec:
	description: PrefectWorkPoolSpec defines the desired state of PrefectWorkPool
	properties:

prefect-operator/internal/controller/prefectworkpool_controller.go

Line 117 in 3ecf633

Args: workPool.EntrypointArguments(),

So if my interpretation is correct. I have some questions:

• What happens if I provide an incorrect type?
• What happens if i don't provide a type at all?
• Can I provide multiple types?
• What if I provide a valid type that doesn't have a corresponding special image published?

[chrisguidry]

Yes, exactly, @josephmcasey! We have some open conversations about publishing images for the "big three" (AWS, GCP, and Azure), which would be a good start.

All of the prefecthq/prefect images include a "steam valve" called EXTRA_PIP_PACKAGES, which allows a user to install additional Python pacakges at container start time. It's definitely not as ideal as having dedicated images, but it gives us a way to install those integration packages at runtime at least.

Taking those together, I think we could add support for the other types by using the stock prefect image and adding the right integration package (prefect-aws, prefect-gcp, etc) via EXTRA_PIP_PACKAGES.

Referenced Issue: PrefectHQ/prefect#12367

Released Contribution: PrefectHQ/prefect#18020