How to deploy connect pipelines declaritively?

[BryanDollery]

How am I meant to get a pipeline into my cluster, exactly? Is rpk the only way? All I can find in the rp-operator is a stanza to configure connectors on startup. I don't want to restart my cluster just to deploy a single pipeline. rpk means it's possible to apply new pipelines dynamically, so what does everyone else do? I'd really like to apply these things as crds -- is that possible?

[mihaitodor]

Hey @BryanDollery, would you mind joining the Redpanda Community Slack? This repo is focused on Connect and the Kubernetes stuff is maintained separately.

I'm not familiar with your setup, but Connect is not coupled with Kubernetes in any way. You can, for example, grab the static binary from the latest release or the prebuilt Docker container, stick it in a Kubernetes Deployment manifest along with a yaml config (which you can maintain in a ConfigMap and deploy it. There's also Streams Mode which allows you to expose a REST API over a HTTP endpoint and then you can POST pipeline configs to it that way. The only gotcha is that the state isn't maintained during pod restarts. You'll have to come up with a mechanism to recreate pipelines if a restart occurs. In practice, the simplest thing to do is to have a single pipeline / pod because that allows you to define precise resource constraints.

[rockwotj]

@BryanDollery we have a helm chart here: https://docs.redpanda.com/redpanda-connect/get-started/quickstarts/helm-chart/

[emaxerrno]

ah i must have had a cached version of this. just replied w/ the same.

[emaxerrno]

@BryanDollery - https://docs.redpanda.com/redpanda-connect/get-started/quickstarts/helm-chart/ - official chart repo.

[BryanDollery]

Ok, so we use helm, not crds. Got it, thanks.

[david-yu]

@BryanDollery There is a future state where we will want to expose Pipelines Declaratively using CRDs, as we have an API on our Cloud Products that allows us to manage Pipelines for Connect. We don't yet expose this through Repdanda K8s and it will likely take some time before we expose this.

[BryanDollery]

Thanks for this. I've actually just written a cheap and cheerful operator that'll apply pipeline 'crds' using rpk. I'll post it to github eventually and I'll come post a link here when I do for other's to maybe benefit from.