|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Kubernetes 1.27: HorizontalPodAutoscaler ContainerResource type metric moves to beta" |
| 4 | +date: 2023-05-02T12:00:00+0800 |
| 5 | +slug: hpa-container-resource-metric |
| 6 | +--- |
| 7 | + |
| 8 | +**Author:** [Kensei Nakada](https://github.com/sanposhiho) (Mercari) |
| 9 | + |
| 10 | +Kubernetes 1.20 introduced the [`ContainerResource` type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics) |
| 11 | +in HorizontalPodAutoscaler (HPA). |
| 12 | + |
| 13 | +In Kubernetes 1.27, this feature moves to beta and the corresponding feature gate (`HPAContainerMetrics`) gets enabled by default. |
| 14 | + |
| 15 | +## What is the ContainerResource type metric |
| 16 | + |
| 17 | +The ContainerResource type metric allows us to configure the autoscaling based on resource usage of individual containers. |
| 18 | + |
| 19 | +In the following example, the HPA controller scales the target |
| 20 | +so that the average utilization of the cpu in the application container of all the pods is around 60%. |
| 21 | +(See [the algorithm details](/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details) |
| 22 | +to know how the desired replica number is calculated exactly) |
| 23 | + |
| 24 | +```yaml |
| 25 | +type: ContainerResource |
| 26 | +containerResource: |
| 27 | + name: cpu |
| 28 | + container: application |
| 29 | + target: |
| 30 | + type: Utilization |
| 31 | + averageUtilization: 60 |
| 32 | +``` |
| 33 | +
|
| 34 | +## The difference from the Resource type metric |
| 35 | +
|
| 36 | +HPA already had a [Resource type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-resource-metrics). |
| 37 | +
|
| 38 | +You can define the target resource utilization like the following, |
| 39 | +and then HPA will scale up/down the replicas based on the current utilization. |
| 40 | +
|
| 41 | +```yaml |
| 42 | +type: Resource |
| 43 | +resource: |
| 44 | + name: cpu |
| 45 | + target: |
| 46 | + type: Utilization |
| 47 | + averageUtilization: 60 |
| 48 | +``` |
| 49 | +
|
| 50 | +But, this Resource type metric refers to the average utilization of the **Pods**. |
| 51 | +
|
| 52 | +In case a Pod has multiple containers, the utilization calculation would be: |
| 53 | +
|
| 54 | +``` |
| 55 | +sum{the resource usage of each container} / sum{the resource request of each container} |
| 56 | +``` |
| 57 | + |
| 58 | +The resource utilization of each container may not have a direct correlation or may grow at different rates as the load changes. |
| 59 | + |
| 60 | +For example: |
| 61 | +- A sidecar container is only providing an auxiliary service such as log shipping. |
| 62 | + If the application does not log very frequently or does not produce logs in its hotpath |
| 63 | + then the usage of the log shipper will not grow. |
| 64 | +- A sidecar container which provides authentication. Due to heavy caching |
| 65 | + the usage will only increase slightly when the load on the main container increases. |
| 66 | + In the current blended usage calculation approach this usually results in |
| 67 | + the HPA not scaling up the deployment because the blended usage is still low. |
| 68 | +- A sidecar may be injected without resources set which prevents scaling |
| 69 | + based on utilization. In the current logic the HPA controller can only scale |
| 70 | + on absolute resource usage of the pod when the resource requests are not set. |
| 71 | + |
| 72 | +And, in such case, if only one container's resource utilization goes high, |
| 73 | +the Resource type metric may not suggest scaling up. |
| 74 | + |
| 75 | +So, for the accurate autoscaling, you may want to use the ContainerResource type metric for such Pods instead. |
| 76 | + |
| 77 | +## What's new for the beta? |
| 78 | + |
| 79 | +For Kubernetes v1.27, the ContainerResource type metric is available by default as described at the beginning |
| 80 | +of this article. |
| 81 | +(You can still disable it by the `HPAContainerMetrics` feature gate.) |
| 82 | + |
| 83 | +Also, we've improved the observability of HPA controller by exposing some metrics from the kube-controller-manager: |
| 84 | +- `metric_computation_total`: Number of metric computations. |
| 85 | +- `metric_computation_duration_seconds`: The time that the HPA controller takes to calculate one metric. |
| 86 | +- `reconciliations_total`: Number of reconciliation of HPA controller. |
| 87 | +- `reconciliation_duration_seconds`: The time that the HPA controller takes to reconcile a HPA object once. |
| 88 | + |
| 89 | +These metrics have labels `action` (`scale_up`, `scale_down`, `none`) and `error` (`spec`, `internal`, `none`). |
| 90 | +And, in addition to them, the first two metrics have the `metric_type` label |
| 91 | +which corresponds to `.spec.metrics[*].type` for a HorizontalPodAutoscaler. |
| 92 | + |
| 93 | +All metrics are useful for general monitoring of HPA controller, |
| 94 | +you can get deeper insight into which part has a problem, where it takes time, how much scaling tends to happen at which time on your cluster etc. |
| 95 | + |
| 96 | +Another minor stuff, we've changed the `SuccessfulRescale` event's messages |
| 97 | +so that everyone can check whether the events came from the resource metric or |
| 98 | +the container resource metric (See [the related PR](https://github.com/kubernetes/kubernetes/pull/116045)). |
| 99 | + |
| 100 | +## Getting involved |
| 101 | + |
| 102 | +This feature is managed by [SIG Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling). |
| 103 | +Please join us and share your feedback. We look forward to hearing from you! |
| 104 | + |
| 105 | +## How can I learn more? |
| 106 | + |
| 107 | +- [The official document of the ContainerResource type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics) |
| 108 | +- [KEP-1610: Container Resource based Autoscaling](https://github.com/kubernetes/enhancements/tree/master/keps/sig-autoscaling/1610-container-resource-autoscaling) |
0 commit comments