maestro.md

Overview

Maestro leverages CloudEvents to transport Kubernetes resources to target clusters and relay the resource status back. These resources and their statuses are stored and updated in a database. The system is composed of two main parts: the Maestro server and the Maestro agent.

The Maestro Server includes various components to fulfill its functions, as illustrated in the diagram below.

gRPC server

Authentication and Authorization

To enable gRPC server-side TLS, specify your server cert and key with --grpc-tls-cert-file and --grpc-tls-key-file.

For authorization, the gRPC server uses a mock authorizer by default. To enable real authorization, set --grpc-authn-type to either mtls or token. Depending on the authorizer type, you will need to create authorization rule resources, which are standard Kubernetes RBAC resources.

1. mTLS-Based Authorization

For mTLS-based authorization, specify the client CA file using --grpc-client-ca-file. The server will validate the client certificate against this CA.

Then create authorization rules based on the CN (Common Name) or O (Organization) in the client certificate, representing the user or group. For example, to allow the user "Alice" to publish and subscribe to the policy source, use the following Kubernetes RBAC configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: policy-pub-sub
rules:
- nonResourceURLs:
  - /sources/policy
  verbs:
  - pub
  - sub
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: policy-pub-sub
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: policy-pub-sub
subjects:
- kind: User
  name: Alice
  apiGroup: rbac.authorization.k8s.io

On the gRPC client side, configure the gRPC options with the client certificate and key files, as follows:

grpcOptions = grpcoptions.NewGRPCOptions()
grpcOptions.URL = grpcServerAddr
grpcOptions.CAFile = grpcServerCAFile
grpcOptions.ClientCertFile = grpcClientCertFile
grpcOptions.ClientKeyFile = grpcClientKeyFile

The grpcClientCertFile and grpcClientKeyFile should contain the certificate signed by the client CA. For the example above, the CN must be "Alice".

2. Token-Based Authorization

For token-based authorization, the gRPC server authenticates the client using a Kubernetes service account token. The service account is expected to be created by the gRPC client.

Create authorization rules based on the service account associated with the token. For example, to allow the service account open-cluster-management/policy-controller to publish and subscribe to the policy source, use the following Kubernetes RBAC configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: policy-pub-sub
rules:
- nonResourceURLs:
  - /sources/policy
  verbs:
  - pub
  - sub
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: policy-pub-sub
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: policy-pub-sub
subjects:
- kind: ServiceAccount
  name: policy-controller
  namespace: open-cluster-management

On the gRPC client side, configure the gRPC options with the token file, as follows:

grpcOptions = grpcoptions.NewGRPCOptions()
grpcOptions.URL = grpcServerAddr
grpcOptions.CAFile = grpcServerCAFile
grpcOptions.TokenFile = grpcClientTokenFile

The grpcClientTokenFile stores the token for the corresponding service account. In the example above, it holds the token for the open-cluster-management/policy-controller service account.

How to Use gRPC Client

• See this example for how to use the gRPC client to publish and subscribe to CloudEvents.
• See this example for how to use the MaestroGRPCSourceWorkClient client to publish and subscribe to ManifestWorks.

Maestro Resource Flow

1. Resource create flow with gRPC

   

2. Resource patch flow with gRPC

   

3. Resource delete flow with gRPC

   

Maestro Resource Status Flow

maestro-resource-status-flow

Maestro Resource Data Flow

Maestro Publish Resource with MQTT

1. The consumer (e.g., ClustersService) uses the MaestroGRPCSourceWorkClient to create a ManifestWork.

   {
     "apiVersion": "work.open-cluster-management.io/v1",
     "kind": "ManifestWork",
     "metadata": {
         "name": "e44ec579-9646-549a-b679-db8d19d6da37",
         ...
     },
     "spec": {
       "workload": {
         "manifests": [
             {
                 "kind": "Deployment",
                 "apiVersion": "apps/v1",
                 "metadata": {
                     "name": "maestro-e2e-upgrade-test",
                     "namespace": "default"
                 },
                 ...
             }
         ]
       }
     }
   }

   The CS uses the uuid.NewSHA1(uuid.NameSpaceOID, manifests[0].Name + manifests[0].Namespace + manifests[0].GVK) to generate the ManifestWork name.

2. The MaestroGRPCSourceWorkClient sends this ManifestWork as a CloudEvent to the Maestro server via gRPC.

   {
     "source": "mw-client-example",
     "type": "io.open-cluster-management.works.v1alpha1.manifestbundles.spec.create_request",
     "datacontenttype": "application/json",
     "data": {...},
     "metadata": "{\"name\":\"e44ec579-9646-549a-b679-db8d19d6da37\",\"uid\":\"55c61e54-a3f6-563d-9fec-b1fe297bdfdb\",...}",
     "resourceid": "55c61e54-a3f6-563d-9fec-b1fe297bdfdb",
     ...
   }

   • The MaestroGRPCSourceWorkClient generates a UID (uuid.NewSHA1(uuid.NameSpaceOID, sourceID + manifestwork.GR + manifestwork.Namespace + manifestwork.Name)) for this ManifestWork and set the CloudEvent resourceid extension attribute to this UID.

3. The Maestro server receives this CloudEvent, creates a corresponding Resource record in its database.

   • Maestro server sets the resourceid as the Resource record ID
   • Maestro server sets the CloudEvent as the Resource record payload

4. After persisting the record, the Maestro server publishes a CloudEvent representing the stored Resource to Maestro agent.

   • Maestro server uses the Resource record ID as the ManifestWork name.

5. The Maestro agent receives this CloudEvent, converts it back into a ManifestWork, and applies it to the target Kubernetes cluster. After applied, there is a corresponding appliedmanifestwork created on the agent side.

Maestro Subscribe Resource Status with MQTT

1. The Maestro agent watches applied manifest status changes.

2. The Maestro agent updates the corresponding ManifestWork status and publishes a CloudEvent representing it.

3. The Maestro server receives the ManifestWork status and updates the corresponding Resource record status in its database (Maestro server finds the corresponding Resource via CloudEvent resourceid).

4. The Maestro server sends this update using a CloudEvent.

5. The MaestroGRPCSourceWorkClient receives this CloudEvent and converts it back to a ManifestWork.

6. The Consumer watches this updated ManifestWork.