Skip to content

Commit 629a4bb

Browse files
dkoshkindkoshkin
authored
feat: deploy registry syncer for workload clusters (#1189)
**What problem does this PR solve?**: This adds a new feature gate `SynchronizeWorkloadClusterRegistry` that will deploy another [HCP](mesosphere/charts#1591) against the management cluster that will setup a Job and a Deployment to sync images from the management cluster to the workload cluster being depoyed. HCPs use a local clusterRef so we will always to create the HCP in the management cluster's namespace but deploy the Pods into the workload cluster's namespace. Stacked on #1175 **Which issue(s) this PR fixes**: Fixes # **How Has This Been Tested?**: <!-- Please describe the tests that you ran to verify your changes. Provide output from the tests and any manual steps needed to replicate the tests. --> Added unit and integration tests. Tested manually: <details> <summary>1. Create a management cluster and make it self-managed</summary> ```bash # Create the bootstrap cluster make dev.run-on-kind CAREN_FEATURE_GATES="SynchronizeWorkloadClusterRegistry=true" eval $(make kind.kubeconfig) # Create the management cluster export CLUSTER_NAME=dk-agf-syncer export CLUSTER_FILE=examples/capi-quick-start/docker-cluster-cilium-helm-addon.yaml export KUBERNETES_VERSION=v1.33.1 clusterctl generate cluster ${CLUSTER_NAME} \ --from ${CLUSTER_FILE} \ --kubernetes-version ${KUBERNETES_VERSION} \ --worker-machine-count 1 | \ kubectl apply --server-side -f - # Get the management cluster kubeconfig kubectl wait clusters/${CLUSTER_NAME} \ --for=condition=ControlPlaneInitialized \ --timeout=1m clusterctl get kubeconfig ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf kubectl config set-cluster ${CLUSTER_NAME} \ --kubeconfig ${CLUSTER_NAME}.conf \ --server=https://$(docker container port ${CLUSTER_NAME}-lb 6443/tcp) kubectl --kubeconfig ${CLUSTER_NAME}.conf wait nodes \ --all \ --for=condition=Ready \ --timeout=5m # Deploy CAREN on the management cluster make dev.run-on-kind \ CAREN_FEATURE_GATES="SynchronizeWorkloadClusterRegistry=true" \ SKIP_BUILD=true KIND_KUBECONFIG=$CLUSTER_NAME.conf \ KIND_CLUSTER_NAME=$CLUSTER_NAME # Move the management cluster clusterctl move --to-kubeconfig=$CLUSTER_NAME.conf # Delete the bootstrap cluster kind delete cluster -n cluster-api-runtime-extensions-nutanix-dev ``` </details> <details> <summary>2. Push a test image into the management cluster's registry</summary> ```bash export KUBECONFIG=$CLUSTER_NAME.conf kubectl port-forward --address=127.0.0.1 --namespace registry-system service/cncf-distribution-registry-docker-registry-headless 5000:5000 & crane copy alpine:latest 127.0.0.1:5000/library/alpine:agf-test --insecure ``` </details> <details> <summary>3. Create a workload cluster in a new namespace</summary> ```bash export KUBECONFIG=$CLUSTER_NAME.conf # Create the workload cluster kubectl create ns wrkld kubectl label ns wrkld caren.nutanix.com/namespace-sync=true export CLUSTER_NAME=dk-agf-syncer-wrkld clusterctl generate cluster ${CLUSTER_NAME} \ -n wrkld \ --from ${CLUSTER_FILE} \ --kubernetes-version ${KUBERNETES_VERSION} \ --worker-machine-count 1 | \ kubectl apply --server-side -f - ``` </details> <details> <summary>4. Deploy a test workload in the workload cluster</summary> ```bash clusterctl get kubeconfig ${CLUSTER_NAME} -n wrkld > ${CLUSTER_NAME}.conf kubectl config set-cluster ${CLUSTER_NAME} \ --kubeconfig ${CLUSTER_NAME}.conf \ --server=https://$(docker container port ${CLUSTER_NAME}-lb 6443/tcp) kubectl run alpine \ --kubeconfig ${CLUSTER_NAME}.conf \ --image=alpine:agf-test \ --restart=Never \ -it \ --command -- sh -c "echo hello" ``` </details> <details> <summary>5. Clean everything up</summary> ```bash kind delete clusters -A ``` </details> **Special notes for your reviewer**: <!-- Use this to provide any additional information to the reviewers. This may include: - Best way to review the PR. - Where the author wants the most review attention on. - etc. -->
1 parent ebe2c00 commit 629a4bb

File tree

25 files changed

+1358
-78
lines changed

25 files changed

+1358
-78
lines changed

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ repos:
153153
name: License headers - YAML and Makefiles
154154
stages: [pre-commit]
155155
files: (^Makefile|\.(ya?ml|mk))$
156-
exclude: ^(internal/test|pkg/handlers/.+/embedded|examples|charts/cluster-api-runtime-extensions-nutanix/(defaultclusterclasses|addons))/.+\.ya?ml|docs/static/helm/index\.yaml|charts/cluster-api-runtime-extensions-nutanix/templates/helm-config.yaml|hack/examples/files/kube-vip.yaml$
156+
exclude: ^(internal/test|pkg/handlers/.+/embedded|pkg/handlers/.+/testdata|examples|charts/cluster-api-runtime-extensions-nutanix/(defaultclusterclasses|addons))/.+\.ya?ml|docs/static/helm/index\.yaml|charts/cluster-api-runtime-extensions-nutanix/templates/helm-config.yaml|hack/examples/files/kube-vip.yaml|$
157157
args:
158158
- --license-filepath
159159
- hack/license-header.txt

‎api/v1alpha1/constants.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,15 @@ const (
4141

4242
ClusterUUIDAnnotationKey = APIGroup + "/cluster-uuid"
4343

44+
// SkipAutoEnablingWorkloadClusterRegistry is the key of the annotation on the Cluster
45+
// used to skip enabling the registry addon on workload cluster.
4446
SkipAutoEnablingWorkloadClusterRegistry = APIGroup + "/skip-auto-enabling-workload-cluster-registry"
4547

48+
// SkipSynchronizingWorkloadClusterRegistry is the key of the annotation on the Cluster
49+
// used to skip deploying the components that will sync OCI artifacts from the registry
50+
// running on the management cluster to registry running on the workload cluster.
51+
SkipSynchronizingWorkloadClusterRegistry = APIGroup + "/skip-synchronizing-workload-cluster-registry"
52+
4653
// PreflightChecksSkipAnnotationKey is the key of the annotation on the Cluster used to skip preflight checks.
4754
PreflightChecksSkipAnnotationKey = "preflight.cluster.caren.nutanix.com/skip"
4855

‎api/variables/getters.go‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
// Copyright 2025 Nutanix. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
package variables
5+
6+
import (
7+
"fmt"
8+
9+
clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
10+
11+
carenv1 "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1"
12+
)
13+
14+
// RegistryAddon retrieves the RegistryAddon from the cluster's topology variables.
15+
// Returns nil if the addon is not defined.
16+
func RegistryAddon(cluster *clusterv1.Cluster) (*carenv1.RegistryAddon, error) {
17+
spec, err := UnmarshalClusterConfigVariable(cluster.Spec.Topology.Variables)
18+
if err != nil {
19+
return nil, fmt.Errorf("failed to unmarshal cluster variable: %w", err)
20+
}
21+
if spec == nil {
22+
return nil, nil
23+
}
24+
if spec.Addons == nil {
25+
return nil, nil
26+
}
27+
28+
return spec.Addons.Registry, nil
29+
}

‎charts/cluster-api-runtime-extensions-nutanix/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@ A Helm chart for cluster-api-runtime-extensions-nutanix
8989
| hooks.nfd.helmAddonStrategy.defaultValueTemplateConfigMap.name | string | `"default-nfd-helm-values-template"` | |
9090
| hooks.registry.cncfDistribution.defaultValueTemplateConfigMap.create | bool | `true` | |
9191
| hooks.registry.cncfDistribution.defaultValueTemplateConfigMap.name | string | `"default-cncf-distribution-registry-helm-values-template"` | |
92+
| hooks.registrySyncer.defaultValueTemplateConfigMap.create | bool | `true` | |
93+
| hooks.registrySyncer.defaultValueTemplateConfigMap.name | string | `"default-registry-syncer-helm-values-template"` | |
9294
| hooks.serviceLoadBalancer.metalLB.defaultValueTemplateConfigMap.create | bool | `true` | |
9395
| hooks.serviceLoadBalancer.metalLB.defaultValueTemplateConfigMap.name | string | `"default-metallb-helm-values-template"` | |
9496
| hooks.virtualIP.kubeVip.defaultTemplateConfigMap.create | bool | `true` | |

‎charts/cluster-api-runtime-extensions-nutanix/addons/registry-syncer/values-template.yaml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
initContainers:
2+
# The regsync container does not fail when it cannot connect to the destination registry.
3+
# In the case when it runs as a Job, it will prematurely exit.
4+
# This init container will wait for the destination registry to be ready.
5+
- name: wait-for-registry
6+
image: ghcr.io/d2iq-labs/kubectl-betterwait:{{ .KubernetesVersion }}
7+
args:
8+
- --for=condition=Ready
9+
- --timeout=-1s # a negative number here means wait forever
10+
- --interval=5s # poll every 5 seconds to the resources to be created
11+
- --namespace={{ .DestinationRegistryHeadlessServiceNamespace }}
12+
- --kubeconfig=/kubeconfig/admin.conf
13+
# Ideally we would wait for the Service to be ready, but Kubernetes does not have a condition for that.
14+
- pod/{{ .DestinationRegistryAnyPodName }}
15+
volumeMounts:
16+
- mountPath: /kubeconfig
17+
name: kubeconfig
18+
readOnly: true
19+
- name: port-forward-registry
20+
image: ghcr.io/d2iq-labs/kubectl-betterwait:{{ .KubernetesVersion }}
21+
command:
22+
- /bin/kubectl
23+
args:
24+
- port-forward
25+
- --address=127.0.0.1
26+
- --namespace={{ .DestinationRegistryHeadlessServiceNamespace }}
27+
- --kubeconfig=/kubeconfig/admin.conf
28+
# This will port-forward to a single Pod in the Service.
29+
- service/{{ .DestinationRegistryHeadlessServiceName }}
30+
- 5000:{{ .DestinationRegistryHeadlessServicePort }}
31+
resources:
32+
requests:
33+
cpu: 25m
34+
memory: 32Mi
35+
limits:
36+
cpu: 100m
37+
memory: 50Mi
38+
volumeMounts:
39+
- mountPath: /kubeconfig
40+
name: kubeconfig
41+
readOnly: true
42+
# Kubernetes will treat this as a Sidecar container
43+
# https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
44+
restartPolicy: Always
45+
46+
extraVolumes:
47+
- name: kubeconfig
48+
secret:
49+
items:
50+
- key: value
51+
path: admin.conf
52+
secretName: {{ .CusterName }}-kubeconfig
53+
- name: ca-cert
54+
secret:
55+
secretName: {{ .RegistryCASecretName }}
56+
57+
extraVolumeMounts:
58+
# Assume both the source and the target registries have the same CA.
59+
# Source registry running in the cluster.
60+
- mountPath: /etc/docker/certs.d/{{ .SourceRegistryAddress }}/
61+
name: ca-cert
62+
readOnly: true
63+
# Destination registry running in the remote cluster being port-forwarded.
64+
- mountPath: /etc/docker/certs.d/127.0.0.1:5000/
65+
name: ca-cert
66+
readOnly: true
67+
68+
deployment:
69+
config:
70+
creds:
71+
- registry: {{ .SourceRegistryAddress }}
72+
reqPerSec: 1
73+
sync:
74+
- source: {{ .SourceRegistryAddress }}
75+
target: 127.0.0.1:5000
76+
type: registry
77+
interval: 1m
78+
79+
job:
80+
enabled: true
81+
config:
82+
sync:
83+
- source: {{ .SourceRegistryAddress }}
84+
target: 127.0.0.1:5000
85+
type: registry
86+
interval: 1m

‎charts/cluster-api-runtime-extensions-nutanix/templates/helm-config.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,10 @@ data:
5151
ChartName: nutanix-csi-storage
5252
ChartVersion: 3.3.4
5353
RepositoryURL: '{{ if .Values.helmRepository.enabled }}oci://helm-repository.{{ .Release.Namespace }}.svc/charts{{ else }}https://nutanix.github.io/helm-releases/{{ end }}'
54+
registry-syncer: |
55+
ChartName: registry-syncer
56+
ChartVersion: 0.1.0
57+
RepositoryURL: '{{ if .Values.helmRepository.enabled }}oci://helm-repository.{{ .Release.Namespace }}.svc/charts{{ else }}https://mesosphere.github.io/charts/staging/{{ end }}'
5458
snapshot-controller: |
5559
ChartName: snapshot-controller
5660
ChartVersion: 4.0.2

‎charts/cluster-api-runtime-extensions-nutanix/templates/registry-syncer/helm-addon-installation.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Copyright 2025 Nutanix. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
{{- if .Values.hooks.registrySyncer.defaultValueTemplateConfigMap.name }}
5+
apiVersion: v1
6+
kind: ConfigMap
7+
metadata:
8+
name: '{{ .Values.hooks.registrySyncer.defaultValueTemplateConfigMap.name }}'
9+
data:
10+
values.yaml: |-
11+
{{- .Files.Get "addons/registry-syncer/values-template.yaml" | nindent 4 }}
12+
{{- end -}}

‎charts/cluster-api-runtime-extensions-nutanix/values.schema.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -536,6 +536,22 @@
536536
},
537537
"type": "object"
538538
},
539+
"registrySyncer": {
540+
"properties": {
541+
"defaultValueTemplateConfigMap": {
542+
"properties": {
543+
"create": {
544+
"type": "boolean"
545+
},
546+
"name": {
547+
"type": "string"
548+
}
549+
},
550+
"type": "object"
551+
}
552+
},
553+
"type": "object"
554+
},
539555
"serviceLoadBalancer": {
540556
"properties": {
541557
"metalLB": {

‎charts/cluster-api-runtime-extensions-nutanix/values.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,10 @@ hooks:
117117
defaultValueTemplateConfigMap:
118118
create: true
119119
name: default-cncf-distribution-registry-helm-values-template
120+
registrySyncer:
121+
defaultValueTemplateConfigMap:
122+
create: true
123+
name: default-registry-syncer-helm-values-template
120124

121125
helmAddonsConfigMap: default-helm-addons-config
122126

‎docs/content/addons/registry.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,20 @@ annotations:
4747
caren.nutanix.com/skip-auto-enabling-workload-cluster-registry: "true"
4848
```
4949
50+
All images pushed to the management cluster's registry can be automatically synced to the workload cluster's registry.
51+
To enable this behavior, set the following feature gate on the controller:
52+
53+
```text
54+
--feature-gates=SynchronizeWorkloadClusterRegistry=true
55+
```
56+
57+
It is also possible to disable this behavior by setting the following annotation on the Cluster resource:
58+
59+
```yaml
60+
annotations:
61+
caren.nutanix.com/skip-synchronizing-workload-cluster-registry: "true"
62+
```
63+
5064
## Registry Certificate
5165
5266
1. A root CA Certificate is deployed in the provider's namespace.

0 commit comments

Comments
 (0)