GraphScope Operator Design Proposal

[siyuan0322]

Discussion and implementation details of GraphScope Operator.

[siyuan0322]

First phase of design and TODO:

• GS Operator 设计：
  • Coordinator without resource/process launching.
  • gsctl 提交算法 （算法作为 CRD） 提交算法是绑定crd到 cluster/图
  • Operator 拉起 coordinator/engine
  • engine fail 的容错
  • coordinator 考虑和 flex 发展路线的 align
  • gsctl
    • create resource
    • load graph
    • create gar
    • run application
  • operator reconcile 考虑多个crd同时提交时候的先后关系
  • expose details metrics, such like the progress of app algorithms.

[wuyueandrew]

设计了大概5个k8s crd
GaeCluster: 一组gae资源，coordinator + worker
Gar：gae使用的算法，cython/cpp
Graph：图数据
GaeApps：一组gae app，包含依赖关系构成dag，目前只考虑按顺序运行
GaeApp：gae app，包括输入，gar，输出

---
apiVersion: graphscope.io/v1beta1
kind: GaeCluster
metadata:
  annotations: 
  name: gae-cluster-1
  namespace: graphscope
spec:
  show_log: true
  log_level: "DEBUG"
  timeout_seconds: 600
  k8s_waiting_for_delete: false
  service:
    type: "NodePort"
  podGroup:
    coordinator:
      preemptive: true
      k8s_image_registry: "registry.cn-hongkong.aliyuncs.com"
      k8s_image_repository: "graphscope"
      k8s_image_tag: "0.21.0"
      k8s_image_pull_policy: "IfNotPresent"
      k8s_image_pull_secrets: 
      k8s_coordinator_cpu: 0.5
      k8s_coordinator_mem: "512Mi"
      k8s_coordinator_pod_node_selector: 
    etcd:
      etcd_addrs:
      etcd_listening_client_port: 2379
      etcd_listening_peer_port: 2380
    worker:
      num_workers: 2
      k8s_engine_cpu: 0.2
      k8s_engine_mem: "1Gi"
      k8s_engine_pod_node_selector:
      with_dataset: false
      dataset_download_retries: 3
    vineyard:
      k8s_vineyard_image: "vineyardcloudnative/vineyardd:latest"
      k8s_vineyard_deployment: None
      k8s_vineyard_cpu: 0.5
      k8s_vineyard_mem: "512Mi"
      vineyard_shared_mem: "4Gi"
  volumes:
    my-data:
      type: persistentVolumeClaim
      field:
        claimName: your-pvc-name
      mounts:
      - mountPath: <path1>

---
apiVersion: graphscope.io/v1beta1
kind: Gar
metadata:
  annotations: 
  name: gar-with-sssp-pregel
  namespace: graphscope
spec:
  app:
		- algo: "SSSP_Pregel_1690991373"
  	  class_name: gs::PregelPropertyAppBase
      compatible_graph:
      - vineyard::ArrowFragment
      context_type: labeled_vertex_data
      md_type: double
      pregel_combine: true
      type: cython_pregel
      vd_type: double
  builtin: false
  version: xxx
  desc: xxx
  binary:

---
apiVersion: graphscope.io/v1beta1
kind: Graph
metadata:
  annotations: 
  name: graph
  namespace: graphscope
spec:
  vertices:
  - label: student
    loader:
      source: "s3://test/student.csv"
      delimiter: ","
      header_row: true
      filetype: "CSV"
      args:
        key: ""
        secret: ""
        endpoint: ""
  edges:
  - label: teacher_student
    loader:
      source: "s3://test/teacher_student.csv"
      delimiter: ","
      header_row: true
      filetype: "CSV"
      args:
        key: ""
        secret: ""
        endpoint: ""
  directed: true
  oid_type: "int64_t"
  generate_eid: true
  retain_oid: true
  vformat:
  eformat:
  vertex_map: "global"

---
apiVersion: graphscope.io/v1beta1
kind: GaeApp
metadata:
  annotations: 
  name: gae-app
  namespace: graphscope
spec:
  graph:
    selector:
      metadata:
        name: graph
  gar:
    selector:
      metadata:
        name: gar-with-sssp-pregel
  config:
  output:
    fd: "s3://test/student.csv"
    selector:
      - name: id
        value: "v:index.id"
    vertex_range:
    args:
      key: ""
      secret: ""
      endpoint: ""

---
apiVersion: graphscope.io/v1beta1
kind: GaeApps
metadata:
  annotations: 
  name: gae-apps
  namespace: graphscope
spec:
  - name: gae-app
    
      

[siyuan0322]

1. GaeCluster 看着不好看，改成 GAECluster 或者 AnalyticalCluster 如何
2. Gar：gae使用的算法 既然是算法，包括builtin 和 gar，Gar有点不全面反映它的内容，不如就称 App, 或 Application 如何
3. GaeApp 包括了输入和输出，其实就是绑定了图的app，叫 BoundedApp 吧。
4. GaeApps 包括一组 GaeApp，这个有必要单独抽出一个CRD么，上次我记得可以提供一个包括多个子yaml的文件来着。 如果是CRD的话就叫 DAG 吧。
   其他都挺好的 👍

[wuyueandrew]

感谢回复

1. GaeCluster调整为AnalyticalCluster，Gar调整为App，GaeApp调整为BoundedApp(这里是不是笔误，BundledApp绑定的app)，GaeApps调整为DAG。
2. GaeApps除了包括一组BoundedApp，还指定了运行的AnalyticalCluster，我的考虑是AnalyticalCluster和DAG绑定后可以在多个BoundedApp之间复用AnalyticalCluster，这样可以节省多次载图。

---
apiVersion: graphscope.io/v1beta1
kind: AnalyticalCluster
metadata:
  annotations: 
  name: analyticalcluster-1
  namespace: graphscope
spec:
  show_log: true
  log_level: "DEBUG"
  timeout_seconds: 600
  k8s_waiting_for_delete: false
  service:
    type: "NodePort"
  podGroup:
    coordinator:
      preemptive: true
      k8s_image_registry: "registry.cn-hongkong.aliyuncs.com"
      k8s_image_repository: "graphscope"
      k8s_image_tag: "0.21.0"
      k8s_image_pull_policy: "IfNotPresent"
      k8s_image_pull_secrets: 
      k8s_coordinator_cpu: 0.5
      k8s_coordinator_mem: "512Mi"
      k8s_coordinator_pod_node_selector: 
    etcd:
      etcd_addrs:
      etcd_listening_client_port: 2379
      etcd_listening_peer_port: 2380
    worker:
      num_workers: 2
      k8s_engine_cpu: 0.2
      k8s_engine_mem: "1Gi"
      k8s_engine_pod_node_selector:
      with_dataset: false
      dataset_download_retries: 3
    vineyard:
      k8s_vineyard_image: "vineyardcloudnative/vineyardd:latest"
      k8s_vineyard_deployment: None
      k8s_vineyard_cpu: 0.5
      k8s_vineyard_mem: "512Mi"
      vineyard_shared_mem: "4Gi"
  volumes:
    my-data:
      type: persistentVolumeClaim
      field:
        claimName: your-pvc-name
      mounts:
      - mountPath: <path1>

---
apiVersion: graphscope.io/v1beta1
kind: App
metadata:
  annotations: 
  name: app-sssp-pregel
  namespace: graphscope
spec:
  app:
		- algo: "SSSP_Pregel_1690991373"
  	  class_name: gs::PregelPropertyAppBase
      compatible_graph:
      - vineyard::ArrowFragment
      context_type: labeled_vertex_data
      md_type: double
      pregel_combine: true
      type: cython_pregel
      vd_type: double
  builtin: false
  version: xxx
  desc: xxx
  binary:

---
apiVersion: graphscope.io/v1beta1
kind: Graph
metadata:
  annotations: 
  name: graph
  namespace: graphscope
spec:
  vertices:
  - label: student
    loader:
      source: "s3://test/student.csv"
      delimiter: ","
      header_row: true
      filetype: "CSV"
      args:
        key: ""
        secret: ""
        endpoint: ""
  edges:
  - label: teacher_student
    loader:
      source: "s3://test/teacher_student.csv"
      delimiter: ","
      header_row: true
      filetype: "CSV"
      args:
        key: ""
        secret: ""
        endpoint: ""
  directed: true
  oid_type: "int64_t"
  generate_eid: true
  retain_oid: true
  vformat:
  eformat:
  vertex_map: "global"

---
apiVersion: graphscope.io/v1beta1
kind: BoundedApp
metadata:
  annotations: 
  name: bounded-app
  namespace: graphscope
spec:
  graph:
    selector:
      metadata:
        name: graph
  app:
    selector:
      metadata:
        name: app-sssp-pregel
  config:
  output:
    fd: "s3://test/student.csv"
    selector:
      - name: id
        value: "v:index.id"
    vertex_range:
    args:
      key: ""
      secret: ""
      endpoint: ""

---
apiVersion: graphscope.io/v1beta1
kind: DAG
metadata:
  annotations: 
  name: gae-apps
  namespace: graphscope
spec:
  cluster:
    selector:
      metadata:
        name: analyticalcluster-1
  boundedApps:
    - name: bounded-app
    
      

[siyuan0322]

1. BoundedApp 不是笔误，就是 app bind 到 graph， 被 bind 的 bounded 这个语义
2. 无异议

[wuyueandrew]

ok，确认BoundedApp