Merge pull request #5340 from natasha41575/metrics

KEP 1287: Instrumentation for in-place pod resize

Author: k8s-ci-robot

keps/sig-node/1287-in-place-update-pod-resources/README.md

@@ -38,6 +38,12 @@
   - [QOS Class](#qos-class)
   - [Resource Quota](#resource-quota)
   - [Affected Components](#affected-components)
+  - [Instrumentation](#instrumentation)
+    - [<code>kubelet_container_requested_resizes_total</code>](#kubelet_container_requested_resizes_total)
+    - [<code>kubelet_pod_resize_duration_seconds</code>](#kubelet_pod_resize_duration_seconds)
+    - [<code>kubelet_pod_pending_resizes</code>](#kubelet_pod_pending_resizes)
+    - [<code>kubelet_pod_in_progress_resizes</code>](#kubelet_pod_in_progress_resizes)
+    - [<code>kubelet_pod_deferred_resize_accepted_total</code>](#kubelet_pod_deferred_resize_accepted_total)
   - [Static CPU &amp; Memory Policy](#static-cpu--memory-policy)
   - [Future Enhancements](#future-enhancements)
     - [Mutable QOS Class &quot;Shape&quot;](#mutable-qos-class-shape)
@@ -912,6 +918,74 @@ Other components:
 * check how the change of meaning of resource requests influence other
   Kubernetes components.
 
+### Instrumentation
+
+The kubelet will record the following metrics:
+
+#### `kubelet_container_requested_resizes_total`
+
+This metric tracks the total number of resize attempts observed by the Kubelet, counted at the container level.
+A single pod update changing multiple containers will be considered separate resize attempts.
+
+Labels: 
+- `resource` - what resource. Possible values: `cpu`, or `memory`. If more than one of these is changing in the resize request, we increment the counter multiple times, once for each.
+- `requirement` - Possible values: `limits`, or `requests`. If more than one of these is changing in the resize request, we increment the counter multiple times, once for each.
+- `operation` -  whether the resize is an increase or a decrease. Possible values: `increase`, `decrease`, `add`, or `remove`.
+- `namespace` - the namespace of the pod.
+
+This metric is recorded as a counter.
+
+#### `kubelet_pod_resize_duration_seconds`
+This metric tracks the duration of [doPodResizeAction](https://github.com/kubernetes/kubernetes/blob/92de70895830ea1a9c2c6554bdab4cbee7ce867d/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L699), which 
+is responsible for actuating the resize.
+
+Labels: 
+- `namespace` - the namespace of the pod.
+
+This metric is recorded as a histogram.
+
+#### `kubelet_pod_pending_resizes`
+
+This metric tracks the current count of pods that the kubelet marks as pending. This will make it
+easier for us to see which of the current limitations users are running into the most.
+
+Labels:
+- `reason` - why the resize is pending. Possible values: `infeasible` or `deferred`.
+- `reason_detail` - more details about why the resize is pending. Although a more detailed "message" will be provided in the `PodResizePending`
+condition in the pod, we limit this label to only the following possible values to keep cardinality low:
+  - `guaranteed_pod_cpu_manager_static_policy` - In-place resize is not supported for Guaranteed Pods alongside CPU Manager static policy.
+  - `guaranteed_pod_memory_manager_static_policy` - In-place resize is not supported for Guaranteed Pods alongside Memory Manager static policy.
+  - `static_pod` - In-place resize is not supported for static pods.
+  - `swap_limitation` - In-place resize is not supported for containers with swap.
+  - `insufficient_node_allocatable` - The node doesn't have enough capacity for this resize request.
+- `namespace` - the namespace of the pod.
+
+This list of possible reasons may shrink or grow depending on limitations that are added or removed in the future.
+
+This metric is recorded as a gauge.
+
+#### `kubelet_pod_in_progress_resizes`
+
+This metric tracks the total count of resize requests that the kubelet marks as in progress, meaning that
+the resources have been allocated but not yet actuated.
+
+Labels: 
+- `namespace` - the namespace of the pod.
+
+This metric is recorded as a gauge.
+
+#### `kubelet_pod_deferred_resize_accepted_total`
+
+This metric tracks the total number of resize requests that the Kubelet originally marked as deferred but 
+later accepted. This metric primarily exists because if a deferred resize is accepted through the timed retry (as
+opposed to being triggered by an event such as another pod being deleted or sized down), it indicates an issue in the Kubelet's logic for handling deferred resizes that we should fix.
+
+Labels:
+  - `accepted_reason` - whether the resize was accepted through the timed retry or due to another pod event. Possible values: `periodic_retry`, `event_based`.
+  - `namespace` - the namespace of the pod.
+
+This metric is recorded as a counter.
+
 ### Static CPU & Memory Policy
 
 Resizing pods with static CPU & memory policy configured is out-of-scope for the beta release of

keps/sig-node/1287-in-place-update-pod-resources/kep.yaml

@@ -34,7 +34,7 @@ replaces:
 
 stage: "beta"
 
-latest-milestone: "v1.33"
+latest-milestone: "v1.34"
 
 milestone:
   alpha: "v1.27"