Merge pull request #31558 from chrismetz09/metzResourceMetrics

update resource metrics pipeline section

Author: k8s-ci-robot

content/en/docs/concepts/configuration/manage-resources-containers.md

@@ -231,7 +231,7 @@ The kubelet reports the resource usage of a Pod as part of the Pod
 
 If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
 are available in your cluster, then Pod resource usage can be retrieved either
-from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api)
+from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-api)
 directly or from your monitoring tools.
 
 ## Local ephemeral storage

content/en/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md

@@ -2,75 +2,208 @@
 reviewers:
 - fgrzadkowski
 - piosz
-title: Resource metrics pipeline
+title: Resource metrics pipeline 
 content_type: concept
 ---
 
 <!-- overview -->
 
-Resource usage metrics, such as container CPU and memory usage,
-are available in Kubernetes through the Metrics API. These metrics can be accessed either directly
-by the user with the `kubectl top` command, or by a controller in the cluster, for example
-Horizontal Pod Autoscaler, to make decisions.
+For Kubernetes, the _Metrics API_ offers a basic set of metrics to support automatic scaling and similar use cases.
+This API makes information available about resource usage for node and pod, including metrics for CPU and memory.
+If you deploy the Metrics API into your cluster, clients of the Kubernetes API can then query for this information, and
+you can use Kubernetes' access control mechanisms to manage permissions to do so.
 
-<!-- body -->
+The [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/)  (HPA) and [VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme) (VPA) use data from the metrics API to adjust workload replicas and resources to meet customer demand.
 
-## The Metrics API
+You can also view the resource metrics using the [`kubectl top`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#top) command.
 
-Through the Metrics API, you can get the amount of resource currently used
-by a given node or a given pod. This API doesn't store the metric values,
-so it's not possible, for example, to get the amount of resources used by a
-given node 10 minutes ago.
+{{< note >}}
+The Metrics API, and the metrics pipeline that it enables, only offers the minimum
+CPU and memory metrics to enable automatic scaling using HPA and / or VPA.
+If you would like to provide a more complete set of metrics, you can complement
+the simpler Metrics API by deploying a second
+[metrics pipeline](/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline)
+that uses the _Custom Metrics API_.
+{{< /note >}}
 
-The API is no different from any other API:
 
-- it is discoverable through the same endpoint as the other Kubernetes APIs under the path: `/apis/metrics.k8s.io/`
-- it offers the same security, scalability, and reliability guarantees
+Figure 1 illustrates the architecture of the resource metrics pipeline.
+
+{{< mermaid >}}
+flowchart RL
+subgraph cluster[Cluster]
+direction RL
+S[ <br><br> ]
+A[Metrics-<br>Server]
+subgraph B[Nodes]
+direction TB
+D[cAdvisor] --> C[kubelet]
+E[Container<br>runtime] --> D
+E1[Container<br>runtime] --> D
+P[pod data] -.- C
+end
+L[API<br>server]
+W[HPA]
+C ---->|Summary<br>API| A -->|metrics<br>API| L --> W
+end
+L ---> K[kubectl<br>top]
+classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000;
+class W,B,P,K,cluster,D,E,E1 box
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class S spacewhite
+classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff;
+class A,L,C k8s
+{{< /mermaid >}}
+
+Figure 1. Resource Metrics Pipeline
+
+The architecture components, from right to left in the figure, consist of the following:
+
+* [cAdvisor](https://github.com/google/cadvisor): Daemon for collecting, aggregating and exposing container metrics included in Kubelet.
+* [kubelet](/docs/concepts/overview/components/#kubelet): Node agent for managing container resources. Resource metrics are accessible using the `/metrics/resource` and `/stats` kubelet API endpoints.
+* [Summary API](#summary-api-source): API provided by the kubelet for discovering and retrieving per-node summarized stats available through the `/stats` endpoint.
+* [metrics-server](#metrics-server): Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by the `kubectl top` command. Metrics Server is a reference implementation of the Metrics API.
+* [Metrics API](#metrics-api): Kubernetes API supporting access to CPU and memory used for workload autoscaling. To make this work in your cluster, you need an API extension server that provides the Metrics API.
+  
+  {{< note >}}
+  cAdvisor supports reading metrics from cgroups, which works with typical container runtimes on Linux.
+  If you use a container runtime that uses another resource isolation mechanism, for example virtualization, then that container runtime must support [CRI Container Metrics](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/cri-container-stats.md) in order for metrics to be available to the kubelet.
+  {{< /note >}}
+
+  
+<!-- body -->
 
-The API is defined in [k8s.io/metrics](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1beta1/types.go)
-repository. You can find more information about the API there.
+## Metrics API
+
+The metrics-server implements the Metrics API. This API allows you to access CPU and memory usage for the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8s autoscaler components. 
+
+Here is an example of the Metrics API request for a `minikube` node piped through `jq` for easier reading:
+```shell
+kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq '.'
+```
+
+Here is the same API call using `curl`:
+```shell
+curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube
+```
+Sample reply:
+```json
+{
+  "kind": "NodeMetrics",
+  "apiVersion": "metrics.k8s.io/v1beta1",
+  "metadata": {
+    "name": "minikube",
+    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/minikube",
+    "creationTimestamp": "2022-01-27T18:48:43Z"
+  },
+  "timestamp": "2022-01-27T18:48:33Z",
+  "window": "30s",
+  "usage": {
+    "cpu": "487558164n",
+    "memory": "732212Ki"
+  }
+}
+```
+Here is an example of the Metrics API request for a `kube-scheduler-minikube` pod contained in the `kube-system` namespace and piped through `jq` for easier reading:
+
+```shell
+kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.'
+```
+Here is the same API call using `curl`:
+```shell
+curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube
+```
+Sample reply:
+```json
+{
+  "kind": "PodMetrics",
+  "apiVersion": "metrics.k8s.io/v1beta1",
+  "metadata": {
+    "name": "kube-scheduler-minikube",
+    "namespace": "kube-system",
+    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube",
+    "creationTimestamp": "2022-01-27T19:25:00Z"
+  },
+  "timestamp": "2022-01-27T19:24:31Z",
+  "window": "30s",
+  "containers": [
+    {
+      "name": "kube-scheduler",
+      "usage": {
+        "cpu": "9559630n",
+        "memory": "22244Ki"
+      }
+    }
+  ]
+}
+```
+
+The Metrics API is defined in the [k8s.io/metrics](https://github.com/kubernetes/metrics) repository. You must enable the [API aggregation layer](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) and register an [APIService](/docs/reference/kubernetes-api/cluster-resources/api-service-v1/) for the `metrics.k8s.io` API.
+
+To learn more about the Metrics API, see [resource metrics API design](https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md), the [metrics-server repository](https://github.com/kubernetes-sigs/metrics-server) and the [resource metrics API](https://github.com/kubernetes/metrics#resource-metrics-api).
+
+
+{{< note >}} You must deploy the metrics-server or alternative adapter that serves the Metrics API to be able to access it. {{< /note >}}
+
+## Measuring resource usage
 
-{{< note >}}
-The API requires the metrics server to be deployed in the cluster. Otherwise it will be not available.
-{{< /note >}}
+### CPU
 
-## Measuring Resource Usage
+CPU is reported as the average core usage measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers, and 1 hyper-thread on bare-metal Intel processors.
 
-### CPU
+This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (in both Linux and Windows kernels). The time window used to calculate CPU is shown under window field in Metrics API.
 
-CPU is reported as the average usage, in
-[CPU cores](/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu),
-over a period of time. This value is derived by taking a rate over a cumulative CPU counter
-provided by the kernel (in both Linux and Windows kernels).
-The kubelet chooses the window for the rate calculation.
+To learn more about how Kubernetes allocates and measures CPU resources, see [meaning of CPU](/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu).
 
 ### Memory
 
-Memory is reported as the working set, in bytes, at the instant the metric was collected.
-In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure.
-However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate.
-It includes all anonymous (non-file-backed) memory since Kubernetes does not support swap.
-The metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim such pages.
+Memory is reported as the working set, measured in bytes, at the instant the metric was collected.
+
+In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate. 
+
+The Kubernetes model for a container's working set expects that the container runtime counts anonymous memory associated with the container in question. The working set metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim pages.
+
+To learn more about how Kubernetes allocates and measures memory resources, see [meaning of memory](/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory).
 
 ## Metrics Server
 
-[Metrics Server](https://github.com/kubernetes-sigs/metrics-server) is a cluster-wide aggregator of resource usage data.
-By default, it is deployed in clusters created by `kube-up.sh` script
-as a Deployment object. If you use a different Kubernetes setup mechanism, you can deploy it using the provided
-[deployment components.yaml](https://github.com/kubernetes-sigs/metrics-server/releases) file.
+The metrics-server fetches resource metrics from the kubelets and exposes them in the Kubernetes API server through the Metrics API for use by the HPA and VPA. You can also view these metrics using the `kubectl top` command.
+
+The metrics-server uses the Kubernetes API to track nodes and pods in your cluster. The metrics-server queries each node over HTTP to fetch metrics. The metrics-server also builds an internal view of pod metadata, and keeps a cache of pod health. That cached pod health information is available via the extension API that the metrics-server makes available.
+
+For example with an HPA query, the metrics-server needs to identify which pods fulfill the label selectors in the deployment.
+
+The metrics-server calls the [kubelet](/docs/reference/command-line-tools-reference/kubelet/) API to collect metrics from each node. Depending on the metrics-server version it uses:
+* Metrics resource endpoint `/metrics/resource` in version v0.6.0+ or
+* Summary API endpoint `/stats/summary` in older versions
+
 
-Metrics Server collects metrics from the Summary API, exposed by
-[Kubelet](/docs/reference/command-line-tools-reference/kubelet/) on each node, and is registered with the main API server via
-[Kubernetes aggregator](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/).
+To learn more about the metrics-server, see the [metrics-server repository](https://github.com/kubernetes-sigs/metrics-server).
 
-Learn more about the metrics server in
-[the design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md).
+You can also check out the following:
 
-### Summary API Source
-The [Kubelet](/docs/reference/command-line-tools-reference/kubelet/) gathers stats at node, volume, pod and container level, and emits their statistics in
+* [metrics-server design](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md)
+* [metrics-server FAQ](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md)
+* [metrics-server known issues](https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md)
+* [metrics-server releases](https://github.com/kubernetes-sigs/metrics-server/releases)
+* [Horizontal Pod Autoscaling](/docs/tasks/run-application/horizontal-pod-autoscale/)
+
+### Summary API source
+
+The [kubelet](/docs/reference/command-line-tools-reference/kubelet/) gathers stats at the node, volume, pod and container level, and emits this information in
 the [Summary API](https://github.com/kubernetes/kubernetes/blob/7d309e0104fedb57280b261e5677d919cb2a0e2d/staging/src/k8s.io/kubelet/pkg/apis/stats/v1alpha1/types.go)
 for consumers to read.
 
-Pre-1.23, these resources have been primarily gathered from [cAdvisor](https://github.com/google/cadvisor). However, in 1.23 with the
-introduction of the `PodAndContainerStatsFromCRI` FeatureGate, container and pod level stats can be gathered by the CRI implementation.
-Note: this also requires support from the CRI implementations (containerd >= 1.6.0, CRI-O >= 1.23.0).
+Here is an example of a Summary API request for a `minikube` node:
+
+
+```shell
+kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary"
+```
+Here is the same API call using `curl`:
+```shell
+curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary
+```
+{{< note >}}
+The summary API `/stats/summary` endpoint will be replaced by the `/metrics/resource` endpoint beginning with metrics-server 0.6.x.
+{{< /note >}}