Support for running Multi-Cluster Argo Workflows

[dmildh-absci]

Hello Karmada Gurus! I am in the process of seeing if I can get Karmada working to support a Multi-Cluster setup similar to how one of the Kubestellar folks documented this here. So far I have the kind cluster setup from your repo that stood up the control cluster with the Karmada api components and the 3 member servers. I have also installed Argo workflows on the member clusters and the Karama-host and current am able to submit the below sample workflow and see it get created on the member servers but what I am trying to do is similar to the Kubesteller article in which you would preferably have a single workflow UI that can show you the status of all of the member cluster's workflows and not quite sure if this is supported. I think this could be a really useful use case for those that are trying to set this up along with some supporting docs and some example code.

I have diagrammed out the design below to help clarify what I think it would looks like but not sure yet about the Argo Workflows server component and where that sits.

Here is the example workflow and propagation policy I am using.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  labels:
    app: sample0
  name: sample0
  namespace: argo
spec:
  parallelism: 1
  serviceAccountName: default
  entrypoint: volumes-existing-example
  tolerations:
  - key: 'key'
    operation: 'Equal'
    value: 'value'
    effect: 'NoSchedule'

  templates:
  - name: volumes-existing-example
    steps:
    - - name: hello
        template: whalesay

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 1Mi

---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: sample0
  namespace: argo
  labels:
    app: sample0
spec:
  propagateDeps: true
  resourceSelectors:
    - apiVersion: argoproj.io/v1alpha1
      kind: Workflow
      name: sample0
  placement:
    clusterAffinity:
      clusterNames:
        - member1
        - member3
    replicaScheduling:
      replicaSchedulingType: Duplicated

If anyone can provide any guidance on this or can help please let me know.

[likku123]

I am early looking to implement the same approach as well . Installing Argoworkflow in Karmada control plane and submitting the jobs to members clusters which does not have the Argoworkflow installed.

@RainbowMango Any pointers on this ask ?

[chaosi-zju]

I am early looking to implement the same approach as well . Installing Argoworkflow in Karmada control plane and submitting the jobs to members clusters which does not have the Argoworkflow installed.

I used to achieved this purpose

but we can only achieve simple work through examples like DAG

as for complex example like Artifacts, we can't, since we can't resolve such scene that the output of one step is exactly the input of another step.

[likku123]

@chaosi-zju

Is it possible to share a sample workflow yaml ? I just want to make sure what changes in have to make to run the workflow in multi cluster.

Thank you !

[likku123]

I am early looking to implement the same approach as well . Installing Argoworkflow in Karmada control plane and submitting the jobs to members clusters which does not have the Argoworkflow installed.

I used to achieved this purpose

but we can only achieve simple work through examples like DAG

as for complex example like Artifacts, we can't, since we can't resolve such scene that the output of one step is exactly the input of another step.

@chaosi-zju I believe one can achieve output of one job can be input of others by maintaining a centralised artifacts bucket like s3 where the jobs can communicate to this bucket for respective input and outputs.

[RainbowMango]

I have also installed Argo workflows on the member clusters and the Karama-host and current am able to submit the below sample workflow and see it get created on the member servers but what I am trying to do is similar to the Kubesteller article in which you would preferably have a single workflow UI that can show you the status of all of the member cluster's workflows and not quite sure if this is supported.

Hi @dmildh-absci Interesting use case, and thanks @likku123 for bringing up.

Just went through the docs and I don't see any gap for doing the same thing with Karmada.
Regarding the status reflection, we achieve that by declaring a ResourceInterpreter .

In addition for the Workflow, Karmada maintains a built-in ResourceInterpreters here , seems all we need to add two more operations InterpretStatus and AggregateStatus.

Let me know if you are still interested in this. Glad to talk more.

[RainbowMango]

Hi @likku123, how are you doing? Are you running Karmda now?

Just a short update for the ArgoWorkflow status aggregation thing. I've talked to the maintainer of argo-workflow, they are looking at the argoproj/argo-workflows#11082 now.

[likku123]

Hi @RainbowMango

I am doing good. How are you ? Long time no see.

we just pause for a moment now because running argoworkflow jobs are our major criteria. Glad ,they started working on it now.

Appreciate your help on following up on this.

Thank you !

[RainbowMango]

I'm doing fine, thanks.

we just pause for a moment now because running argoworkflow jobs are our major criteria.

I don't quite understand, do you mean running jobs on multi-cluster is not your first priority?

[likku123]

What I mean to say is . Running argoworkflow jobs on multi cluster is our first and high priority from a centralised multi cluster Orchestrator.

Right now instead of multi cluster orchestrator we are individually logging to cluster and submitting the argoworkflow jobs.

[RainbowMango]

Get it, thanks. Will let you know once there is any update.