Skip to content

Commit 381e56e

Browse files
britaniarbritaniar
authored
feat: add cloudConfig support in member-agent (#939)
1 parent 67c349a commit 381e56e

File tree

9 files changed

+158
-54
lines changed

9 files changed

+158
-54
lines changed

‎charts/member-agent/README.md‎

Lines changed: 56 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -29,18 +29,61 @@ helm upgrade member-agent member-agent/ --namespace fleet-system
2929

3030
## Parameters
3131

32-
| Parameter | Description | Default |
33-
|:-------------------------|:------------------------------------------------------|:------------------------------------------------|
34-
| replicaCount | The number of member-agent replicas to deploy | `1` |
35-
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/member-agent` |
36-
| image.pullPolicy | Image pullPolicy | `IfNotPresent` |
37-
| image.tag | The image tag to use | `v0.1.0` |
38-
| affinity | The node affinity to use for pod scheduling | `{}` |
39-
| tolerations | The toleration to use for pod scheduling | `[]` |
40-
| resources | The resource request/limits for the container image | limits: "2" CPU, 4Gi, requests: 100m CPU, 128Mi |
41-
| namespace | Namespace that this Helm chart is installed on. | `fleet-system` |
42-
| logVerbosity | Log level. Uses V logs (klog) | `3` |
43-
| propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` |
44-
| region | The region where the member cluster resides | `` |
32+
| Parameter | Description | Default |
33+
|:------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------|
34+
| replicaCount | The number of member-agent replicas to deploy | `1` |
35+
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/member-agent` |
36+
| image.pullPolicy | Image pullPolicy | `IfNotPresent` |
37+
| image.tag | The image tag to use | `v0.1.0` |
38+
| affinity | The node affinity to use for pod scheduling | `{}` |
39+
| tolerations | The toleration to use for pod scheduling | `[]` |
40+
| resources | The resource request/limits for the container image | limits: "2" CPU, 4Gi, requests: 100m CPU, 128Mi |
41+
| namespace | Namespace that this Helm chart is installed on. | `fleet-system` |
42+
| logVerbosity | Log level. Uses V logs (klog) | `3` |
43+
| propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` |
44+
| region | The region where the member cluster resides | `` |
45+
| config.azureCloudConfig | The cloud provider configuration | **required if property provider is set to azure** |
46+
47+
## Override Azure cloud config
48+
49+
**If PropertyProvider feature is set to azure, then a cloud configuration is required.**
50+
Cloud configuration provides resource metadata and credentials for `fleet-member-agent` to manipulate Azure resources.
51+
It's embedded into a Kubernetes secret and mounted to the pods.
52+
The values can be modified under `config.azureCloudConfig` section in values.yaml or can be provided as a separate file.
53+
54+
55+
| configuration value | description | Remark |
56+
|-------------------------------------------------------| --- |---------------------------------------------------------------------------|
57+
| `cloud` | The cloud where resources belong. | Required. |
58+
| `tenantId` | The AAD Tenant ID for the subscription where the Azure resources are deployed. | |
59+
| `subscriptionId` | The ID of the subscription where resources are deployed. | |
60+
| `useManagedIdentityExtension` | Boolean indicating whether or not to use a managed identity. | `true` or `false` |
61+
| `userAssignedIdentityID` | ClientID of the user-assigned managed identity with RBAC access to resources. | Required for UserAssignedIdentity and omitted for SystemAssignedIdentity. |
62+
| `aadClientId` | The ClientID for an AAD application with RBAC access to resources. | Required if `useManagedIdentityExtension` is set to `false`. |
63+
| `aadClientSecret` | The ClientSecret for an AAD application with RBAC access to resources. | Required if `useManagedIdentityExtension` is set to `false`. |
64+
| `resourceGroup` | The name of the resource group where cluster resources are deployed. | |
65+
| `userAgent` | The userAgent provided when accessing resources. | |
66+
| `location` | The region where resource group and its resources is deployed. | |
67+
| `vnetName` | The name of the virtual network where the cluster is deployed. | |
68+
| `vnetResourceGroup` | The resource group where the virtual network is deployed. | |
69+
70+
You can create a file `azure.yaml` with the following content, and pass it to `helm install` command: `helm install <release-name> <chart-name> --set propertyProvider=azure -f azure.yaml`
71+
72+
```yaml
73+
config:
74+
azureCloudConfig:
75+
cloud: "AzurePublicCloud"
76+
tenantId: "00000000-0000-0000-0000-000000000000"
77+
subscriptionId: "00000000-0000-0000-0000-000000000000"
78+
useManagedIdentityExtension: false
79+
userAssignedIdentityID: "00000000-0000-0000-0000-000000000000"
80+
aadClientId: "00000000-0000-0000-0000-000000000000"
81+
aadClientSecret: "<your secret>"
82+
userAgent: "fleet-member-agent"
83+
resourceGroup: "<resource group name>"
84+
location: "<resource group location>"
85+
vnetName: "<vnet name>"
86+
vnetResourceGroup: "<vnet resource group>"
87+
```
4588
4689
## Contributing Changes

‎charts/member-agent/templates/cloudconfig.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{{- if eq .Values.propertyProvider "azure" }}
2+
apiVersion: v1
3+
kind: Secret
4+
metadata:
5+
name: cloud-config
6+
namespace: {{ .Values.namespace }}
7+
type: Opaque
8+
data:
9+
config.json: {{ .Values.config.azureCloudConfig | toJson | indent 4 | b64enc | quote }}
10+
{{- end }}

‎charts/member-agent/templates/deployment.yaml‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,9 @@ spec:
3737
{{- if .Values.propertyProvider }}
3838
- --property-provider={{ .Values.propertyProvider }}
3939
{{- end }}
40+
{{- if eq .Values.propertyProvider "azure" }}
41+
- --cloud-config=/etc/kubernetes/provider/config.json
42+
{{- end }}
4043
{{- if .Values.region }}
4144
- --region={{ .Values.region }}
4245
{{- end }}
@@ -80,10 +83,19 @@ spec:
8083
httpGet:
8184
path: /readyz
8285
port: hubhealthz
83-
{{- if not .Values.useCAAuth }}
86+
{{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }}
8487
volumeMounts:
88+
{{- if not .Values.useCAAuth }}
8589
- name: provider-token
8690
mountPath: /config
91+
{{- end }}
92+
{{- if eq .Values.propertyProvider "azure" }}
93+
- name: cloud-provider-config
94+
mountPath: /etc/kubernetes/provider
95+
readOnly: true
96+
{{- end }}
97+
{{- end }}
98+
{{- if not .Values.useCAAuth }}
8799
- name: refresh-token
88100
image: "{{ .Values.refreshtoken.repository }}:{{ .Values.refreshtoken.tag }}"
89101
imagePullPolicy: {{ .Values.refreshtoken.pullPolicy }}
@@ -102,10 +114,19 @@ spec:
102114
volumeMounts:
103115
- name: provider-token
104116
mountPath: /config
117+
{{- end }}
118+
{{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }}
105119
volumes:
120+
{{- if not .Values.useCAAuth }}
106121
- name: provider-token
107122
emptyDir: {}
108123
{{- end }}
124+
{{- if eq .Values.propertyProvider "azure" }}
125+
- name: cloud-provider-config
126+
secret:
127+
secretName: cloud-config
128+
{{- end }}
129+
{{- end }}
109130
{{- with .Values.nodeSelector }}
110131
nodeSelector:
111132
{{- toYaml . | nindent 8 }}

‎charts/member-agent/values.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,19 @@ config:
3535
identityKey: "identity-key-path"
3636
identityCert: "identity-cert-path"
3737
CABundle: "ca-bundle-path"
38+
azureCloudConfig:
39+
cloud: ""
40+
tenantId: ""
41+
subscriptionId: ""
42+
useManagedIdentityExtension: false
43+
userAssignedIdentityID: ""
44+
aadClientId: ""
45+
aadClientSecret: ""
46+
resourceGroup: ""
47+
userAgent: ""
48+
location: ""
49+
vnetName: ""
50+
vnetResourceGroup: ""
3851

3952
secret:
4053
name: "hub-kubeconfig-secret"

‎cmd/memberagent/main.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ var (
7373
enableV1Beta1APIs = flag.Bool("enable-v1beta1-apis", false, "If set, the agents will watch for the v1beta1 APIs.")
7474
propertyProvider = flag.String("property-provider", "none", "The property provider to use for the agent.")
7575
region = flag.String("region", "", "The region where the member cluster resides.")
76+
cloudConfigFile = flag.String("cloud-config", "/etc/kubernetes/provider/config.json", "The path to the cloud cloudconfig file.")
7677
)
7778

7879
func init() {
@@ -367,6 +368,8 @@ func Start(ctx context.Context, hubCfg, memberConfig *rest.Config, hubOpts, memb
367368
klog.V(2).Info("setting up the Azure property provider")
368369
// Note that the property provider, though initialized here, is not started until
369370
// the specific instance wins the leader election.
371+
klog.V(1).InfoS("Property Provider is azure, loading cloud config", "cloudConfigFile", *cloudConfigFile)
372+
// TODO (britaniar): load cloud config for Azure property provider.
370373
pp = azure.New(region)
371374
default:
372375
// Fall back to not using any property provider if the provided type is none or

‎go.mod‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,26 +14,26 @@ require (
1414
github.com/onsi/gomega v1.35.1
1515
github.com/prometheus/client_golang v1.19.1
1616
github.com/prometheus/client_model v0.6.1
17-
github.com/spf13/cobra v1.8.0
17+
github.com/spf13/cobra v1.8.1
1818
github.com/spf13/pflag v1.0.5
1919
github.com/stretchr/testify v1.9.0
2020
go.goms.io/fleet-networking v0.2.7
2121
go.uber.org/atomic v1.11.0
2222
go.uber.org/zap v1.27.0
23-
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8
23+
golang.org/x/exp v0.0.0-20241004190924-225e2abe05e6
2424
golang.org/x/sync v0.8.0
2525
golang.org/x/time v0.7.0
2626
k8s.io/api v0.30.2
2727
k8s.io/apiextensions-apiserver v0.30.2
2828
k8s.io/apimachinery v0.30.2
2929
k8s.io/client-go v0.30.2
3030
k8s.io/component-base v0.30.2
31-
k8s.io/klog/v2 v2.120.1
31+
k8s.io/klog/v2 v2.130.1
3232
k8s.io/metrics v0.25.2
33-
k8s.io/utils v0.0.0-20240502163921-fe8a2dddb1d0
33+
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8
3434
sigs.k8s.io/cloud-provider-azure v1.28.2
3535
sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50
36-
sigs.k8s.io/controller-runtime v0.18.4
36+
sigs.k8s.io/controller-runtime v0.18.5
3737
sigs.k8s.io/work-api v0.0.0-20220407021756-586d707fdb2c
3838
)
3939

@@ -50,7 +50,7 @@ require (
5050
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/privatedns/armprivatedns v1.3.0 // indirect
5151
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0 // indirect
5252
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.6.0 // indirect
53-
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.2.0 // indirect
53+
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.3.0 // indirect
5454
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.1.0 // indirect
5555
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
5656
github.com/Azure/go-autorest/autorest v0.11.29 // indirect
@@ -60,13 +60,13 @@ require (
6060
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
6161
github.com/Azure/go-autorest/logger v0.2.1 // indirect
6262
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
63-
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
63+
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.1 // indirect
6464
github.com/aws/karpenter-core v0.32.2-0.20231109191441-e32aafc81fb5 // indirect
6565
github.com/beorn7/perks v1.0.1 // indirect
6666
github.com/blendle/zapdriver v1.3.1 // indirect
6767
github.com/cespare/xxhash/v2 v2.3.0 // indirect
68-
github.com/davecgh/go-spew v1.1.1 // indirect
69-
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
68+
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
69+
github.com/emicklei/go-restful/v3 v3.12.1 // indirect
7070
github.com/evanphx/json-patch v5.9.0+incompatible // indirect
7171
github.com/fsnotify/fsnotify v1.7.0 // indirect
7272
github.com/go-logr/zapr v1.3.0 // indirect
@@ -96,8 +96,8 @@ require (
9696
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
9797
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
9898
github.com/pkg/errors v0.9.1 // indirect
99-
github.com/pmezard/go-difflib v1.0.0 // indirect
100-
github.com/prometheus/common v0.54.0 // indirect
99+
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
100+
github.com/prometheus/common v0.55.0 // indirect
101101
github.com/prometheus/procfs v0.15.1 // indirect
102102
github.com/samber/lo v1.38.1 // indirect
103103
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.52.0 // indirect
@@ -106,7 +106,7 @@ require (
106106
go.uber.org/multierr v1.11.0 // indirect
107107
golang.org/x/crypto v0.28.0 // indirect
108108
golang.org/x/net v0.30.0 // indirect
109-
golang.org/x/oauth2 v0.21.0 // indirect
109+
golang.org/x/oauth2 v0.23.0 // indirect
110110
golang.org/x/sys v0.26.0 // indirect
111111
golang.org/x/term v0.25.0 // indirect
112112
golang.org/x/text v0.19.0 // indirect
@@ -116,7 +116,7 @@ require (
116116
gopkg.in/inf.v0 v0.9.1 // indirect
117117
gopkg.in/yaml.v2 v2.4.0 // indirect
118118
gopkg.in/yaml.v3 v3.0.1 // indirect
119-
k8s.io/kube-openapi v0.0.0-20240521193020-835d969ad83a // indirect
119+
k8s.io/kube-openapi v0.0.0-20240903163716-9e1beecbcb38 // indirect
120120
knative.dev/pkg v0.0.0-20231010144348-ca8c009405dd // indirect
121121
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
122122
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect

0 commit comments

Comments
 (0)