How to join hub and spoke with aws-auth using helmcharts

[softcane]

https://github.com/open-cluster-management-io/ocm/tree/main/solutions/joining-hub-and-spoke-with-aws-auth

I'm trying to find documentation on how to do this process using Helm charts. We don't plan to use clusteradm. Even though the public charts for cluster-manager and klusterlet are available, we want to use them to handle the cluster registration process.

[dosubot]

There isn’t a single end-to-end guide for joining hub and spoke clusters with aws-auth using only Helm charts, but you can achieve this by combining the manual AWS IAM/IRSA setup from the solution guide with the Helm chart values for cluster-manager and klusterlet.

First, follow the solution guide to set up the required AWS IAM roles, policies, and tags for both hub and spoke clusters. These steps are mandatory regardless of the tool you use.

For the hub, install the cluster-manager Helm chart. You don’t need to set AWS-specific values here, but ensure the hub is ready to accept AWS IAM-based registration.

For the spoke, use the klusterlet Helm chart. You’ll need to provide the bootstrapHubKubeConfig (the hub’s kubeconfig) and can customize registration settings via the registrationConfiguration block in your values file or with --set flags. The klusterlet chart supports IRSA (aws-auth) integration—see its README and values.yaml for details on serviceAccount annotations and enabling IRSA. You can also create the bootstrap-hub-kubeconfig secret manually after installation if you prefer.

There’s no explicit aws_auth_config block in the Helm charts, but the registration process works by combining the manual IAM/IRSA setup with the Helm chart’s registration parameters. Recent OCM updates are improving AWS IAM integration, but the workflow above is the current supported path source.

If you need more concrete examples, the solution guide and klusterlet chart README are the best references. Let me know if you want a sample values.yaml or Helm command for your use case.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[softcane]

@dosu, what if I don't want to use bootstrapHubKubeConfig files? Is there any other alternative?

[dosubot]

There isn't currently an alternative to using a bootstrapHubKubeConfig file (or the corresponding secret) for the initial registration handshake—even when using AWS IAM/IRSA integration with the klusterlet Helm chart. The code and CRD require this kubeconfig for the first connection to the hub, regardless of whether you later use IRSA for ongoing authentication. You can install the klusterlet Helm chart without specifying bootstrapHubKubeConfig, but you'll still need to manually create the bootstrap-hub-kubeconfig secret in the agent namespace before registration will succeed. There is no tokenless or kubeconfig-free bootstrap path as of now, even with awsirsa mode enabled. See the klusterlet CRD and Helm chart for details:
klusterlet CRD
klusterlet Helm README
registration logic

If you want, I can share an example workflow for creating the secret manually after Helm install. Let me know!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[mikeshng]

CC @ramekris3163 @suvaanshkumar @jaswalkiranavtar

[ramekris3163]

@softcane the bootstraphubkubeconfig is mandatory for the manged cluster to create the managedcluster resource on the hub. If you don't want to pass the bootstraphubkubeconfig file to the helm chart, you can create a secret called "bootstrap-hub-kubeconfig" in "open-cluster-management-agent" namespace , then you don't have to pass the file to helm install.

Note: Currently we are using this approach and creating the secret directly from our custom helm chart which also installs the klusterlet helm chart.

[softcane]

Thank You!
@ramekris3163 Can you clarify this last statement? What's it referring to, the clusteradm or something else.

Note: Currently we are using this approach and creating the secret directly from our custom helm chart which also installs the klusterlet helm chart.

[ramekris3163]

@softcane

No this is not referring to clusteradm.

We have the following:

custom-ocm-hub-helm(custom helm chart)
->templates
-->cluster-manager-helm-release.yaml(This yaml has the HelmRelease which installs cluster-manager helmchart)
-->rotate-token-cron-job.yaml(This yaml creates a cronjob which runs every 40 minutes generates the required token by running the command:"kubectl create token agent-registration-bootstrap --namespace open-cluster-management" and then generate a kubeconfig and store it somewhere)
-->cluster-manager-post-install.yaml(This yaml has a post install job which creates a initial job from the cronjob since the cronjob only runs every 40 minutes)
->Chart.yaml

custom-ocm-managed-helm(custom helm chart)
->templates
-->klusterlet-helm-release.yaml(This yaml has the HelmRelease which installs klusterlet helmchart)
-->klusterlet-post-install.yaml(This yaml has a post install job which creates the required secret(bootstrap-hub-kubeconfig) after retreiving the kubeconfig generated by hub from storage)
->chart.yaml

When installing the custom-ocm-hub-helm chart, it will generate the kubeconfig and store it in the storage.
While installing the custom-ocm-managed-helm, it will retrieve the secret from storage and create the required secret(bootstrap-hub-kubeconfig).

Please let me know if you have and further questions.

[softcane]

@ramekris3163 @ramekris3163 @suvaanshkumar @jaswalkiranavtar
Where can I find these custom charts? If it's not possible to share the link, then at least can you paste the spec.

[ramekris3163]


@softcane
I'm attaching snippets of the helm charts FYR. Please let me know if you have any questions on the helm chart i can answer them.

In cluster-manager-helm-release.yaml, we create a flux helm repository object pointing to ocm helm chart and a flux helm release which installs the helm chart in the cluster.

ex:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: ocm-hub
  namespace: {{ .Values.namespace.shared }}
spec:
  interval: {{ .Values.helmRepositoryInterval }}
  url: {{ .Values.hubChartUrl | quote }}
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ocm-hub
  namespace: {{ .Values.namespace.shared }}
  annotations:
    meta.helm.sh/release-name: ocm-hub-release
spec:
  # flux reconciles the deleted k8s resources only if enabled
  driftDetection:
    mode: enabled
  releaseName: ocm-hub
  targetNamespace: {{ .Values.namespace.ocm }}
  storageNamespace: {{ .Values.namespace.ocm }}
  interval: {{ .Values.helmReleaseInterval }}
  chart:
    spec:
      chart: {{ .Values.hubChartName | quote }}
      version: {{ .Values.hubChartVersion | quote }}
      sourceRef:
        kind: HelmRepository
        name: ocm-hub
        namespace: {{ .Values.namespace.shared }}
     values:
    replicaCount: {{ .Values.replicaCount }}
    resources:
      requests:
        cpu: {{ .Values.resourcesOverrides.requests.cpu }}
        memory: {{ .Values.resourcesOverrides.requests.memory }}
    enableSyncLabels: true
    clusterManager:
      registrationConfiguration:
          featureGates:
          - feature: ManagedClusterAutoApproval
            mode: {{ .Values.managedClusterAutoApproval }}
          registrationDrivers:
          - authType: {{ .Values.authType }}
            awsirsa:
              {{- if and (eq .Values.managedClusterAutoApproval "Enable") (not (empty .Values.autoApprovedIdentities)) }}
              autoApprovedIdentities:
              {{- range .Values.autoApprovedIdentities }}
                - {{ . }}
              {{- end }}
              {{- end }}
              hubClusterArn: {{ .Values.hubClusterArn }}
              {{- if .Values.tags }}
              tags:
              {{- range .Values.tags }}
                - {{ . }}
              {{- end }}
            {{- end }}
             createBootstrapSA: true

Then we have another template called rotate-token-cron-job.yaml. In this we rotate the bootstrap token.

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Chart.Name }}-rotate-token-script
  namespace: {{ .Values.namespace.helm }}
data:
  rotate-token.sh: |
{{ .Files.Get "scripts/rotate-token.sh" | indent 4 }}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Chart.Name }}-rotate-hub-kubeconfig-sa
  namespace: {{ .Values.namespace.helm }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: {{ .Values.namespace.helm }}
  name: {{ .Chart.Name }}-rotate-hub-kubeconfig-role
rules:
  - apiGroups: [""] # "" indicates the core API group
    resources: ["configmaps"]
    verbs: ["get"] 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: {{ .Chart.Name }}-rotate-hub-kubeconfig-rolebinding
  namespace: {{ .Values.namespace.helm }}  # same as Role
subjects:
- kind: ServiceAccount
  name: {{ .Chart.Name }}-rotate-hub-kubeconfig-sa       # CALLER SA
  namespace: {{ .Values.namespace.helm }}
roleRef:
  kind: Role
  name: {{ .Chart.Name }}-rotate-hub-kubeconfig-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Chart.Name }}-allow-sa-token-create
  namespace: {{ .Values.namespace.ocm }}  # TARGET SA's namespace
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: {{ .Chart.Name }}-bind-token-create
  namespace: {{ .Values.namespace.ocm }}
subjects:
  - kind: ServiceAccount
    name: {{ .Chart.Name }}-rotate-hub-kubeconfig-sa
    namespace: {{ .Values.namespace.helm }}
roleRef:
  kind: Role
  name: {{ .Chart.Name }}-allow-sa-token-create
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: {{ .Chart.Name }}-rotate-hub-kubeconfig
  namespace: {{ .Values.namespace.helm }}
spec:
  schedule: "*/50 * * * *"
  jobTemplate:
    spec:
      template:
        metadata:
        spec:
          serviceAccountName: {{ .Chart.Name }}-rotate-hub-kubeconfig-sa
          restartPolicy: OnFailure
          containers:
            - name: rotate-hub-kubeconfig-container
              image: "alpine/k8s:{{ .Values.alpineImageVersion}}"
              imagePullPolicy: IfNotPresent
              command: ["/bin/bash", "-ce", "/scripts/rotate-token.sh"]
              volumeMounts:
                - name: rotate-token-vol
                  mountPath: /scripts
              env:
              - name: OCM_NAMESPACE
                value: "{{ .Values.namespace.ocm }}"
          volumes:
            - name: rotate-token-vol
              configMap:
                name: {{ .Chart.Name }}-rotate-token-script
                defaultMode: 493

rotate-token-script.sh

#!/bin/bash

set -e

hubBootstrapToken=$(
  kubectl create token agent-registration-bootstrap --namespace ${OCM_NAMESPACE}
)

also get the hub_cluster_arn, hub_server_url and certificate authority data.

##store this generated token and other values in an external system

ocm-post-install.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: rotate-hub-kubeconfig-init
  namespace: {{ .Values.namespace.helm }}
  annotations:
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook": post-install, post-upgrade
spec:
  backoffLimit: 5
  ttlSecondsAfterFinished: 1200
  template:
    metadata:
      name: rotate-hub-kubeconfig-init
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: {{ .Chart.Name }}-rotate-hub-kubeconfig-sa
      restartPolicy: OnFailure
      containers:
        - name: rotate-hub-kubeconfig-container
          image: "alpine/k8s:{{ .Values.alpineImageVersion }}"
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "-ce", "sleep 30 && /scripts/rotate-token.sh"]
          volumeMounts:
            - name: rotate-token-vol
              mountPath: /scripts
          env:
            - name: OCM_NAMESPACE
              value: "{{ .Values.namespace.ocm }}"
      volumes:
        - name: rotate-token-vol
          configMap:
            name: {{ .Chart.Name }}-rotate-token-script
            defaultMode: 493

When we apply this above helm chart, it will ensure the required token is generated on post install and stored in an external system and rotated every 50 mins.

Similarly for managed, we have a custom helm chart and klusterlet-helm-release.yaml installs the helm repository and helm release of ocm-managed helm chart and we have a post install job which talks to the external system and creates the required secret in the managed cluster.

post-install-script.sh

kubeconfigSecretJson=#retrieve from the external system wherever stored(can be ssm or systems manager)
# Extract CERTIFICATE_AUTHORITY_DATA from the kubeconfigSecretJson
CERTIFICATE_AUTHORITY_DATA=$(echo "$kubeconfigSecretJson" | jq -r '.certificateAuthorityData')
HUB_SERVER_URL=$(echo "$kubeconfigSecretJson" | jq -r '.hubServerUrl')
HUB_CLUSTER_ARN=$(echo "$kubeconfigSecretJson" | jq -r '.hubClusterArn')
hubBootstrapToken=$(echo "$kubeconfigSecretJson" | jq -r '.hubBootstrapToken')

bootstrap_hub_kubeconfig="apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: ${CERTIFICATE_AUTHORITY_DATA}\n    server: ${HUB_SERVER_URL}\n  name: ${HUB_CLUSTER_ARN}\ncontexts:\n- context:\n    cluster: ${HUB_CLUSTER_ARN}\n    namespace: default\n    user: bootstrap\n  name: bootstrap\ncurrent-context: bootstrap\nkind: Config\npreferences: {}\nusers:\n- name: bootstrap\n  user:\n    token: ${hubBootstrapToken}"

yaml_tmp=$(cat <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-hub-kubeconfig
  namespace: ${AGENT_NAMESPACE}
type: Opaque
data:
  kubeconfig: $(echo -e "${bootstrap_hub_kubeconfig}" | base64 -w 0)
EOF
)

echo "$yaml_tmp" | kubectl apply -f -

In the post-install.yaml you should create the required roles and bindings and jobs. With this managed cluster when joining will be able to pull the secrets for kubeconfig from the external system whenever required and it is always up to date. We have a CICD pipleine which passes the required values and installs these custom helm charts on our custers.