feat(k8s-observability-monitoring): add kubelet scraping to customAlloy

Add native kubelet and cAdvisor metrics scraping to customAlloy.
This enables PVC volume stats (kubelet_volume_stats_*) without relying
on upstream alloy-metrics which has the service.namespace promotion issue.

Enable with customAlloy.kubelet.enabled: true

Author: loafoe

charts/k8s-observability-monitoring/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v2
 name: k8s-observability-monitoring
-version: 0.35.2
+version: 0.36.0
 description: Helm chart for k8s-observability-monitoring
 
 # renovate: datasource=helm depName=k8s-monitoring registryUrl=https://grafana.github.io/helm-charts

charts/k8s-observability-monitoring/README.md

@@ -1,6 +1,6 @@
 # k8s-observability-monitoring
 
-![Version: 0.35.2](https://img.shields.io/badge/Version-0.35.2-informational?style=flat-square) ![AppVersion: 3.8.3](https://img.shields.io/badge/AppVersion-3.8.3-informational?style=flat-square)
+![Version: 0.36.0](https://img.shields.io/badge/Version-0.36.0-informational?style=flat-square) ![AppVersion: 3.8.3](https://img.shields.io/badge/AppVersion-3.8.3-informational?style=flat-square)
 
 Helm chart for k8s-observability-monitoring
 
@@ -105,7 +105,7 @@ This creates a `PolicyException` resource that allows `k8s-monitoring-alloy-*` p
 | clusterMetrics.nodeExporter.deploy | bool | `false` | Deploy node-exporter (set to false if using existing deployment) |
 | clusterMetrics.nodeExporter.enabled | bool | `true` | Enable scraping node-exporter |
 | clusterName | string | `""` | Cluster name for telemetry labeling. Must be set to a non-empty value at install time. |
-| customAlloy | object | `{"attributeCleanup":{"enabled":true},"attributePromotion":{"enabled":false},"clustering":{"enabled":false},"enabled":false,"kubeStateMetrics":{"extraMetricProcessingRules":""},"liveDebugging":{"enabled":true},"replaceUpstreamCollector":false,"replicas":1,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"sendingQueue":{"enabled":true}}` | Custom Alloy deployment for metrics scraping This deploys a separate Alloy instance that can scrape kube-state-metrics and optionally replace the upstream alloy-metrics collector entirely. |
+| customAlloy | object | `{"attributeCleanup":{"enabled":true},"attributePromotion":{"enabled":false},"clustering":{"enabled":false},"enabled":false,"kubeStateMetrics":{"extraMetricProcessingRules":""},"kubelet":{"enabled":false},"liveDebugging":{"enabled":true},"replaceUpstreamCollector":false,"replicas":1,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"sendingQueue":{"enabled":true}}` | Custom Alloy deployment for metrics scraping This deploys a separate Alloy instance that can scrape kube-state-metrics and optionally replace the upstream alloy-metrics collector entirely. |
 | customAlloy.attributeCleanup | object | `{"enabled":true}` | Remove high-cardinality attributes to reduce storage costs Matches k8s-monitoring attribute cleanup |
 | customAlloy.attributeCleanup.enabled | bool | `true` | Enable attribute cleanup |
 | customAlloy.attributePromotion | object | `{"enabled":false}` | Promote useful attributes from datapoint to resource level |
@@ -115,6 +115,8 @@ This creates a `PolicyException` resource that allows `k8s-monitoring-alloy-*` p
 | customAlloy.enabled | bool | `false` | Enable custom Alloy deployment |
 | customAlloy.kubeStateMetrics | object | `{"extraMetricProcessingRules":""}` | kube-state-metrics scraping configuration |
 | customAlloy.kubeStateMetrics.extraMetricProcessingRules | string | `""` | Extra metric processing rules (Alloy relabel config syntax) |
+| customAlloy.kubelet | object | `{"enabled":false}` | Kubelet metrics scraping configuration (includes PVC volume stats) |
+| customAlloy.kubelet.enabled | bool | `false` | Enable kubelet and cAdvisor metrics scraping. Provides kubelet_volume_stats_* metrics for PVC capacity monitoring. |
 | customAlloy.liveDebugging | object | `{"enabled":true}` | Live debugging via Alloy UI (port 12345) |
 | customAlloy.liveDebugging.enabled | bool | `true` | Enable live debugging |
 | customAlloy.replaceUpstreamCollector | bool | `false` | Replace upstream alloy-metrics collector entirely. When true, disables alloy-metrics and customAlloy handles all metrics collection including ServiceMonitors, PodMonitors, and Probes (if prometheusOperatorObjects is enabled). |

charts/k8s-observability-monitoring/templates/custom-alloy-configmap.yaml

@@ -123,6 +123,159 @@ data:
       forward_to = [otelcol.receiver.prometheus.default.receiver]
     }
 
+    {{- if .Values.customAlloy.kubelet.enabled }}
+    // Kubelet Metrics Discovery
+    discovery.kubernetes "kubelet" {
+      role = "node"
+    }
+
+    discovery.relabel "kubelet" {
+      targets = discovery.kubernetes.kubelet.targets
+
+      rule {
+        target_label = "__address__"
+        replacement  = "kubernetes.default.svc.cluster.local:443"
+      }
+
+      rule {
+        source_labels = ["__meta_kubernetes_node_name"]
+        regex         = "(.+)"
+        target_label  = "__metrics_path__"
+        replacement   = "/api/v1/nodes/$1/proxy/metrics"
+      }
+
+      rule {
+        source_labels = ["__meta_kubernetes_node_name"]
+        target_label  = "node"
+      }
+
+      rule {
+        source_labels = ["__meta_kubernetes_node_name"]
+        target_label  = "instance"
+      }
+    }
+
+    prometheus.scrape "kubelet" {
+      targets = discovery.relabel.kubelet.output
+      job_name = "integrations/kubernetes/kubelet"
+      scrape_interval = "60s"
+      scrape_timeout = "10s"
+      scheme = "https"
+
+      authorization {
+        type             = "Bearer"
+        credentials_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
+      }
+
+      tls_config {
+        ca_file              = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+        insecure_skip_verify = true
+      }
+
+      {{- if .Values.customAlloy.clustering.enabled }}
+      clustering {
+        enabled = true
+      }
+      {{- end }}
+
+      forward_to = [prometheus.relabel.kubelet.receiver]
+    }
+
+    // Kubelet cAdvisor Metrics
+    discovery.relabel "cadvisor" {
+      targets = discovery.kubernetes.kubelet.targets
+
+      rule {
+        target_label = "__address__"
+        replacement  = "kubernetes.default.svc.cluster.local:443"
+      }
+
+      rule {
+        source_labels = ["__meta_kubernetes_node_name"]
+        regex         = "(.+)"
+        target_label  = "__metrics_path__"
+        replacement   = "/api/v1/nodes/$1/proxy/metrics/cadvisor"
+      }
+
+      rule {
+        source_labels = ["__meta_kubernetes_node_name"]
+        target_label  = "node"
+      }
+
+      rule {
+        source_labels = ["__meta_kubernetes_node_name"]
+        target_label  = "instance"
+      }
+    }
+
+    prometheus.scrape "cadvisor" {
+      targets = discovery.relabel.cadvisor.output
+      job_name = "integrations/kubernetes/cadvisor"
+      scrape_interval = "60s"
+      scrape_timeout = "10s"
+      scheme = "https"
+
+      authorization {
+        type             = "Bearer"
+        credentials_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
+      }
+
+      tls_config {
+        ca_file              = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+        insecure_skip_verify = true
+      }
+
+      {{- if .Values.customAlloy.clustering.enabled }}
+      clustering {
+        enabled = true
+      }
+      {{- end }}
+
+      forward_to = [prometheus.relabel.cadvisor.receiver]
+    }
+
+    prometheus.relabel "kubelet" {
+      max_cache_size = 100000
+      forward_to = [otelcol.receiver.prometheus.default.receiver]
+    }
+
+    prometheus.relabel "cadvisor" {
+      max_cache_size = 100000
+      // Drop high-cardinality container metrics
+      rule {
+        source_labels = ["__name__"]
+        regex = "container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)"
+        action = "drop"
+      }
+      rule {
+        source_labels = ["__name__"]
+        regex = "container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)"
+        action = "drop"
+      }
+      rule {
+        source_labels = ["__name__"]
+        regex = "container_memory_(mapped_file|swap)"
+        action = "drop"
+      }
+      rule {
+        source_labels = ["__name__"]
+        regex = "container_(file_descriptors|tasks_state|threads_max)"
+        action = "drop"
+      }
+      rule {
+        source_labels = ["__name__", "interface"]
+        regex = "container_network_.*;(cali|cilium|cni|lxc|nodelocaldns|tunl).*"
+        action = "drop"
+      }
+      rule {
+        source_labels = ["__name__"]
+        regex = "container_spec.*"
+        action = "drop"
+      }
+      forward_to = [otelcol.receiver.prometheus.default.receiver]
+    }
+    {{- end }}
+
     // OTEL Pipeline
     otelcol.receiver.prometheus "default" {
       output {

charts/k8s-observability-monitoring/values.yaml

@@ -171,6 +171,11 @@ customAlloy:
   kubeStateMetrics:
     # -- Extra metric processing rules (Alloy relabel config syntax)
     extraMetricProcessingRules: ""
+  # -- Kubelet metrics scraping configuration (includes PVC volume stats)
+  kubelet:
+    # -- Enable kubelet and cAdvisor metrics scraping.
+    # Provides kubelet_volume_stats_* metrics for PVC capacity monitoring.
+    enabled: false
   # -- Resource requests and limits
   resources:
     requests: