Skip to content

Commit 5ad42da

Browse files
author
Arvind Thirumurugan
committed
add standalone-metric-collector dir
Signed-off-by: Arvind Thirumurugan <[email protected]>
1 parent bc98d0c commit 5ad42da

24 files changed

+3183
-0
lines changed

‎standalone-metric-collector/HUB_SETUP.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# Hub Cluster RBAC Setup
2+
3+
This guide explains how to set up RBAC permissions on the hub cluster for the MetricCollector controller running on member clusters.
4+
5+
## Overview
6+
7+
The MetricCollector controller needs permissions on the **hub cluster** to:
8+
- Create/update `MetricCollectorReport` resources in `fleet-{cluster}` namespaces
9+
- List namespaces
10+
11+
## Option 1: Using the Helm Chart Template (Recommended)
12+
13+
Generate and apply the RBAC resources from the helm chart:
14+
15+
```bash
16+
# Generate hub RBAC manifest
17+
helm template metric-collector ./charts/metric-collector \
18+
--set hubCluster.createRBAC=true \
19+
--set memberCluster.name=cluster-1 \
20+
--show-only templates/hub-rbac.yaml > hub-rbac.yaml
21+
22+
# Apply on the hub cluster
23+
kubectl apply -f hub-rbac.yaml --context=hub-cluster
24+
```
25+
26+
## Option 2: Manual RBAC Setup
27+
28+
Apply this manifest directly on the hub cluster:
29+
30+
```yaml
31+
---
32+
apiVersion: rbac.authorization.k8s.io/v1
33+
kind: ClusterRole
34+
metadata:
35+
name: metric-collector-hub-access
36+
labels:
37+
app: metric-collector
38+
rules:
39+
# MetricCollectorReport access
40+
- apiGroups: ["placement.kubernetes-fleet.io"]
41+
resources: ["metriccollectorreports"]
42+
verbs: ["get", "list", "create", "update", "patch", "delete"]
43+
# Namespace access
44+
- apiGroups: [""]
45+
resources: ["namespaces"]
46+
verbs: ["get", "list"]
47+
---
48+
apiVersion: rbac.authorization.k8s.io/v1
49+
kind: ClusterRoleBinding
50+
metadata:
51+
name: metric-collector-cluster-1
52+
labels:
53+
app: metric-collector
54+
fleet.kubernetes.io/member-cluster: cluster-1
55+
roleRef:
56+
apiGroup: rbac.authorization.k8s.io
57+
kind: ClusterRole
58+
name: metric-collector-hub-access
59+
subjects:
60+
# Option A: Use ServiceAccount from hub cluster
61+
- kind: ServiceAccount
62+
name: metric-collector-sa
63+
namespace: fleet-member-cluster-1
64+
65+
# Option B: Use token directly (for testing)
66+
# Create token secret on member cluster and reference here
67+
```
68+
69+
## Creating ServiceAccount Token for Member Cluster
70+
71+
On the **hub cluster**:
72+
73+
```bash
74+
# 1. Create namespace for the member cluster
75+
kubectl create namespace fleet-member-cluster-1
76+
77+
# 2. Create ServiceAccount
78+
kubectl create serviceaccount metric-collector-sa -n fleet-member-cluster-1
79+
80+
# 3. Bind to ClusterRole
81+
kubectl create clusterrolebinding metric-collector-cluster-1 \
82+
--clusterrole=metric-collector-hub-access \
83+
--serviceaccount=fleet-member-cluster-1:metric-collector-sa
84+
85+
# 4. Create token secret
86+
kubectl apply -f - <<EOF
87+
apiVersion: v1
88+
kind: Secret
89+
metadata:
90+
name: metric-collector-token
91+
namespace: fleet-member-cluster-1
92+
annotations:
93+
kubernetes.io/service-account.name: metric-collector-sa
94+
type: kubernetes.io/service-account-token
95+
EOF
96+
97+
# 5. Get the token
98+
kubectl get secret metric-collector-token -n fleet-member-cluster-1 \
99+
-o jsonpath='{.data.token}' | base64 -d > hub-token.txt
100+
101+
# 6. Get CA certificate
102+
kubectl get secret metric-collector-token -n fleet-member-cluster-1 \
103+
-o jsonpath='{.data.ca\.crt}' | base64 -d > hub-ca.crt
104+
```
105+
106+
On the **member cluster**:
107+
108+
```bash
109+
# 1. Create namespace
110+
kubectl create namespace fleet-system
111+
112+
# 2. Create token secret
113+
kubectl create secret generic hub-token \
114+
--from-file=token=hub-token.txt \
115+
-n fleet-system
116+
117+
# 3. (Optional) Create CA secret
118+
kubectl create secret generic hub-ca \
119+
--from-file=ca.crt=hub-ca.crt \
120+
-n fleet-system
121+
122+
# 4. Install the chart
123+
helm install metric-collector ./charts/metric-collector \
124+
--namespace fleet-system \
125+
--set memberCluster.name=cluster-1 \
126+
--set hubCluster.url=https://hub-cluster:6443 \
127+
--set hubCluster.tls.caSecretName=hub-ca \
128+
--set prometheus.url=http://prometheus:9090
129+
```
130+
131+
## Verification
132+
133+
On the **hub cluster**:
134+
135+
```bash
136+
# Check if RBAC is created
137+
kubectl get clusterrole metric-collector-hub-access
138+
kubectl get clusterrolebinding metric-collector-cluster-1
139+
140+
# Check if namespace exists for reports
141+
kubectl get namespace fleet-cluster-1
142+
143+
# Watch for reports
144+
kubectl get metriccollectorreports -n fleet-cluster-1 --watch
145+
```
146+
147+
On the **member cluster**:
148+
149+
```bash
150+
# Check controller logs
151+
kubectl logs -n fleet-system deployment/metric-collector -f
152+
153+
# Check for errors
154+
kubectl logs -n fleet-system deployment/metric-collector | grep -i error
155+
```
156+
157+
## Troubleshooting
158+
159+
### Permission Denied Errors
160+
161+
If you see errors like:
162+
```
163+
Failed to sync report to hub: ... forbidden: User "system:serviceaccount:fleet-member-cluster-1:metric-collector-sa" cannot create resource "metriccollectorreports"
164+
```
165+
166+
**Solution**: Verify RBAC is correctly configured on hub cluster:
167+
```bash
168+
kubectl auth can-i create metriccollectorreports \
169+
--as=system:serviceaccount:fleet-member-cluster-1:metric-collector-sa \
170+
-n fleet-cluster-1 \
171+
--context=hub-cluster
172+
```
173+
174+
### Token Expired
175+
176+
If authentication fails:
177+
```
178+
Failed to connect to hub: Unauthorized
179+
```
180+
181+
**Solution**: Regenerate the token secret on the hub cluster and update the secret on the member cluster.
182+
183+
### Namespace Not Found
184+
185+
If reports fail to be created:
186+
```
187+
Failed to sync report: namespace "fleet-cluster-1" not found
188+
```
189+
190+
**Solution**: Ensure the `fleet-{cluster}` namespace exists on the hub cluster. The hub agent typically creates these.
191+
192+
## Multi-Cluster Setup
193+
194+
For multiple member clusters, repeat the RBAC setup for each cluster:
195+
196+
```bash
197+
# For cluster-1
198+
helm template metric-collector ./charts/metric-collector \
199+
--set hubCluster.createRBAC=true \
200+
--set memberCluster.name=cluster-1 \
201+
--show-only templates/hub-rbac.yaml | \
202+
kubectl apply -f - --context=hub-cluster
203+
204+
# For cluster-2
205+
helm template metric-collector ./charts/metric-collector \
206+
--set hubCluster.createRBAC=true \
207+
--set memberCluster.name=cluster-2 \
208+
--show-only templates/hub-rbac.yaml | \
209+
kubectl apply -f - --context=hub-cluster
210+
```
211+
212+
Each member cluster will have its own ClusterRoleBinding with a unique ServiceAccount.

‎standalone-metric-collector/Makefile‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Image URL to use for building/pushing image targets
2+
REGISTRY ?= ghcr.io/kubefleet-dev
3+
IMAGE_NAME ?= metric-collector
4+
TAG ?= latest
5+
IMG ?= $(REGISTRY)/$(IMAGE_NAME):$(TAG)
6+
7+
# Go parameters
8+
GOOS ?= $(shell go env GOOS)
9+
GOARCH ?= $(shell go env GOARCH)
10+
11+
# Directories
12+
ROOT_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
13+
TOOLS_DIR := hack/tools
14+
TOOLS_BIN_DIR := $(abspath $(TOOLS_DIR)/bin)
15+
16+
# Binaries
17+
CONTROLLER_GEN_VER := v0.16.0
18+
CONTROLLER_GEN_BIN := controller-gen
19+
CONTROLLER_GEN := $(abspath $(TOOLS_BIN_DIR)/$(CONTROLLER_GEN_BIN)-$(CONTROLLER_GEN_VER))
20+
21+
GOIMPORTS_VER := latest
22+
GOIMPORTS_BIN := goimports
23+
GOIMPORTS := $(abspath $(TOOLS_BIN_DIR)/$(GOIMPORTS_BIN)-$(GOIMPORTS_VER))
24+
25+
# Scripts
26+
GO_INSTALL := ../hack/go-install.sh
27+
28+
# CRD Options
29+
CRD_OPTIONS ?= "crd"
30+
31+
##@ General
32+
33+
.PHONY: help
34+
help: ## Display this help.
35+
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
36+
37+
##@ Tooling
38+
39+
$(CONTROLLER_GEN):
40+
GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) sigs.k8s.io/controller-tools/cmd/controller-gen $(CONTROLLER_GEN_BIN) $(CONTROLLER_GEN_VER)
41+
42+
$(GOIMPORTS):
43+
GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) golang.org/x/tools/cmd/goimports $(GOIMPORTS_BIN) $(GOIMPORTS_VER)
44+
45+
##@ Development
46+
47+
.PHONY: manifests
48+
manifests: $(CONTROLLER_GEN) ## Generate CRD manifests.
49+
$(CONTROLLER_GEN) \
50+
$(CRD_OPTIONS) paths="./apis/..." output:crd:artifacts:config=config/crd/bases
51+
@echo "Copying CRDs to helm chart..."
52+
@cp config/crd/bases/*.yaml charts/metric-collector/templates/crds/
53+
54+
.PHONY: generate
55+
generate: $(CONTROLLER_GEN) ## Generate code (DeepCopy, etc).
56+
$(CONTROLLER_GEN) \
57+
object:headerFile="../hack/boilerplate.go.txt" paths="./..."
58+
59+
.PHONY: fmt
60+
fmt: ## Run go fmt against code.
61+
go fmt ./...
62+
63+
.PHONY: vet
64+
vet: ## Run go vet against code.
65+
go vet ./...
66+
67+
.PHONY: imports
68+
imports: $(GOIMPORTS) ## Organize imports.
69+
$(GOIMPORTS) -local github.com/kubefleet-dev/standalone-metric-collector -w .
70+
71+
.PHONY: test
72+
test: fmt vet ## Run tests.
73+
go test ./... -coverprofile cover.out
74+
75+
##@ Build
76+
77+
.PHONY: build
78+
build: fmt vet ## Build metric-collector binary.
79+
CGO_ENABLED=0 GOOS=$(GOOS) GOARCH=$(GOARCH) go build -o bin/metric-collector ./cmd/metriccollector
80+
81+
.PHONY: run
82+
run: fmt vet ## Run controller from your host.
83+
go run ./cmd/metriccollector/main.go
84+
85+
.PHONY: docker-build
86+
docker-build: ## Build docker image.
87+
docker build -t ${IMG} -f docker/Dockerfile .
88+
89+
.PHONY: docker-push
90+
docker-push: ## Push docker image.
91+
docker push ${IMG}
92+
93+
.PHONY: docker-build-push
94+
docker-build-push: docker-build docker-push ## Build and push docker image.
95+
96+
##@ Deployment
97+
98+
.PHONY: helm-lint
99+
helm-lint: ## Lint helm chart.
100+
helm lint charts/metric-collector
101+
102+
.PHONY: helm-template
103+
helm-template: ## Template helm chart.
104+
helm template metric-collector charts/metric-collector \
105+
--set memberCluster.name=cluster-1 \
106+
--set hubCluster.url=https://hub-cluster:6443 \
107+
--set prometheus.url=http://prometheus.test-ns:9090
108+
109+
.PHONY: helm-package
110+
helm-package: helm-lint ## Package helm chart.
111+
helm package charts/metric-collector -d dist/
112+
113+
.PHONY: helm-install
114+
helm-install: ## Install helm chart.
115+
helm upgrade --install metric-collector charts/metric-collector \
116+
--namespace fleet-system --create-namespace \
117+
--set memberCluster.name=$(CLUSTER_NAME) \
118+
--set hubCluster.url=$(HUB_URL) \
119+
--set prometheus.url=$(PROMETHEUS_URL) \
120+
--set image.repository=$(REGISTRY)/$(IMAGE_NAME) \
121+
--set image.tag=$(TAG)
122+
123+
.PHONY: helm-uninstall
124+
helm-uninstall: ## Uninstall helm chart.
125+
helm uninstall metric-collector --namespace fleet-system
126+
127+
##@ CRD
128+
129+
.PHONY: install-crds
130+
install-crds: ## Install CRDs into the K8s cluster.
131+
kubectl apply -f config/crd/bases/
132+
133+
.PHONY: uninstall-crds
134+
uninstall-crds: ## Uninstall CRDs from the K8s cluster.
135+
kubectl delete -f config/crd/bases/
136+
137+
##@ Cleanup
138+
139+
.PHONY: clean
140+
clean: ## Clean build artifacts.
141+
rm -rf bin/ dist/ cover.out

0 commit comments

Comments
 (0)