Skip to content

Commit f909a9e

Browse files
nickytdn-boshnakov
andauthored
[blog] Getting started with OpenTelemetry on a Gardener shoot cluster (gardener#674)
* blog: Getting started with OpenTelemetry on a Gardener shoot cluster * blog: add victoria-logs UI for k8s-events logs * blog: fix prometheus workload deployment manifest * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md Co-authored-by: Nikolay Boshnakov <[email protected]> * Update website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md --------- Co-authored-by: Nikolay Boshnakov <[email protected]>
1 parent 74a4625 commit f909a9e

20 files changed

+1426
-1
lines changed

.gitignore

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,9 @@ node_modules
44
hugo/resources/_gen
55
hugo/.forestry
66
.DS_Store
7-
*.yaml
7+
*.yaml
8+
9+
# Allow getting-started-with-opentelemetry-on-gardener-shoot-cluster yaml resources
10+
!website/blog/2025/06/manifests/otel-collectors/*.yaml
11+
!website/blog/2025/06/manifests/otel-prometheus/*.yaml
12+
!website/blog/2025/06/manifests/otel-victorialogs/*.yaml

website/blog/2025/06/06-30-getting-started-with-opentelemetry-on-gardener-shoot-cluster.md

Lines changed: 312 additions & 0 deletions
Large diffs are not rendered by default.
65.4 KB
Loading
50.1 KB
Loading
128 KB
Loading
411 KB
Loading
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
apiVersion: opentelemetry.io/v1beta1
3+
kind: OpenTelemetryCollector
4+
metadata:
5+
name: k8s-events
6+
namespace: kube-system
7+
spec:
8+
managementState: "managed"
9+
mode: statefulset
10+
serviceAccount: k8s-events
11+
resources:
12+
requests:
13+
memory: 32Mi
14+
cpu: 5m
15+
limits:
16+
memory: 200Mi
17+
cpu: 200m
18+
env:
19+
- name: PROMETHEUS_URL
20+
value: "https://prometheus.prometheus.svc:8443"
21+
- name: VICTORIA_LOGS_URL
22+
value: "https://victorialogs-victoria-logs-single-server.victoria-logs.svc:8443"
23+
volumes:
24+
- name: certs
25+
secret:
26+
secretName: otel-collector-tls
27+
volumeMounts:
28+
- name: certs
29+
mountPath: /etc/cert
30+
readOnly: true
31+
observability:
32+
metrics:
33+
disablePrometheusAnnotations: true
34+
config:
35+
receivers:
36+
# Fetch Kubernetes events from the API server
37+
k8s_events:
38+
namespaces: []
39+
40+
# Collects cluster-level metrics and entity events from the Kubernetes API server
41+
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver
42+
k8s_cluster:
43+
node_conditions_to_report: [Ready, MemoryPressure]
44+
allocatable_types_to_report: [cpu, memory, storage]
45+
resource_attributes:
46+
container.id:
47+
enabled: false
48+
49+
processors:
50+
memory_limiter:
51+
check_interval: 1s
52+
limit_percentage: 80
53+
spike_limit_percentage: 20
54+
55+
batch:
56+
timeout: 5s
57+
send_batch_size: 1000
58+
59+
resource:
60+
attributes:
61+
- key: collector.name
62+
value: k8s-events
63+
action: upsert
64+
65+
exporters:
66+
otlphttp/metrics:
67+
metrics_endpoint: "${env:PROMETHEUS_URL}/api/v1/otlp/v1/metrics"
68+
tls:
69+
insecure: false # Ensure server certificate is validated against the CA
70+
ca_file: /etc/cert/ca.crt
71+
cert_file: /etc/cert/tls.crt
72+
key_file: /etc/cert/tls.key
73+
74+
otlphttp/logs:
75+
logs_endpoint: "${env:VICTORIA_LOGS_URL}/insert/opentelemetry/v1/logs"
76+
headers:
77+
VL-Stream-Fields: "k8s.event.reason,k8s.node.name,k8s.namespace.name,k8s.object.name,k8s.pods.name,k8s.container.name,severity"
78+
tls:
79+
insecure: false # Ensure server certificate is validated against the CA
80+
ca_file: /etc/cert/ca.crt
81+
cert_file: /etc/cert/tls.crt
82+
key_file: /etc/cert/tls.key
83+
84+
service:
85+
pipelines:
86+
logs:
87+
receivers: [k8s_events]
88+
processors: [memory_limiter, resource, batch]
89+
exporters: [otlphttp/logs]
90+
metrics:
91+
receivers: [k8s_cluster]
92+
processors: [memory_limiter, resource, batch]
93+
exporters: [otlphttp/metrics]
94+
95+
# Configure the collector own telemetry
96+
telemetry:
97+
# Emit collector logs to stdout
98+
logs:
99+
level: info
100+
encoding: console
101+
output_paths: [stdout]
102+
error_output_paths: [stderr]
103+
# Push collector internal metrics to Prometheus
104+
metrics:
105+
level: detailed
106+
readers:
107+
- # push metrics to Prometheus backend
108+
periodic:
109+
interval: 30000
110+
timeout: 10000
111+
exporter:
112+
otlp:
113+
protocol: http/protobuf
114+
endpoint: "${env:PROMETHEUS_URL}/api/v1/otlp/v1/metrics"
115+
insecure: false # Ensure server certificate is validated against the CA
116+
certificate: /etc/cert/ca.crt
117+
client_certificate: /etc/cert/tls.crt
118+
client_key: /etc/cert/tls.key
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
name: k8s-events
6+
namespace: kube-system
7+
---
8+
apiVersion: rbac.authorization.k8s.io/v1
9+
kind: ClusterRole
10+
metadata:
11+
name: k8s-events-otel
12+
rules:
13+
- nonResourceURLs: ["/metrics"]
14+
verbs: ["get"]
15+
- apiGroups:
16+
- ""
17+
resources:
18+
- events
19+
- namespaces
20+
- namespaces/status
21+
- nodes
22+
- nodes/spec
23+
- nodes/stats
24+
- nodes/metrics
25+
- configmaps
26+
- nodes/proxy
27+
- pods
28+
- pods/status
29+
- replicationcontrollers
30+
- replicationcontrollers/status
31+
- resourcequotas
32+
- services
33+
- endpoints
34+
verbs:
35+
- get
36+
- list
37+
- watch
38+
- apiGroups:
39+
- apps
40+
resources:
41+
- daemonsets
42+
- deployments
43+
- replicasets
44+
- statefulsets
45+
verbs:
46+
- get
47+
- list
48+
- watch
49+
- apiGroups:
50+
- batch
51+
resources:
52+
- jobs
53+
- cronjobs
54+
verbs:
55+
- get
56+
- list
57+
- watch
58+
- apiGroups:
59+
- autoscaling
60+
resources:
61+
- horizontalpodautoscalers
62+
verbs:
63+
- get
64+
- list
65+
- watch
66+
---
67+
apiVersion: rbac.authorization.k8s.io/v1
68+
kind: ClusterRoleBinding
69+
metadata:
70+
name: k8s-events-otel
71+
roleRef:
72+
apiGroup: rbac.authorization.k8s.io
73+
kind: ClusterRole
74+
name: k8s-events-otel
75+
subjects:
76+
- kind: ServiceAccount
77+
name: k8s-events
78+
namespace: kube-system
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
apiVersion: cert.gardener.cloud/v1alpha1
2+
kind: Certificate
3+
metadata:
4+
name: otel-collector
5+
namespace: kube-system
6+
labels:
7+
app: otel-collector
8+
spec:
9+
commonName: client
10+
dnsNames:
11+
- otel-collector
12+
secretName: otel-collector-tls
13+
issuerRef:
14+
name: issuer-selfsigned
15+
namespace: certs
16+
renew: true

0 commit comments

Comments
 (0)