Merge pull request #39904 from saschagrunert/dev-1.27-otel

[KEP-2731] Add docs for Kubelet OpenTelemetry tracing graduation

Author: k8s-ci-robot

content/en/docs/concepts/cluster-administration/system-traces.md

@@ -76,7 +76,7 @@ For more information about the `TracingConfiguration` struct, see
 
 ### kubelet traces
 
-{{< feature-state for_k8s_version="v1.25" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.27" state="beta" >}}
 
 The kubelet CRI interface and authenticated http servers are instrumented to generate
 trace spans. As with the apiserver, the endpoint and sampling rate are configurable.
@@ -86,10 +86,7 @@ Enabled without a configured endpoint, the default OpenTelemetry Collector recei
 
 #### Enabling tracing in the kubelet
 
-To enable tracing, enable the `KubeletTracing`
-[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
-on the kubelet. Also, provide the kubelet with a
-[tracing configuration](https://github.com/kubernetes/component-base/blob/release-1.25/tracing/api/v1/types.go).
+To enable tracing, apply the [tracing configuration](https://github.com/kubernetes/component-base/blob/release-1.27/tracing/api/v1/types.go).
 This is an example snippet of a kubelet config that records spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
 
 ```yaml
@@ -103,6 +100,21 @@ tracing:
   samplingRatePerMillion: 100
 ```
 
+If the `samplingRatePerMillion` is set to one million (`1000000`), then every
+span will be sent to the exporter.
+
+The kubelet in Kubernetes v{{< skew currentVersion >}} collects spans from
+the garbage collection, pod synchronization routine as well as every gRPC
+method. Connected container runtimes like CRI-O and containerd can link the
+traces to their exported spans to provide additional context of information.
+
+Please note that exporting spans always comes with a small performance overhead
+on the networking and CPU side, depending on the overall configuration of the
+system. If there is any issue like that in a cluster which is running with
+tracing enabled, then mitigate the problem by either reducing the
+`samplingRatePerMillion` or disabling tracing completely by removing the
+configuration.
+
 ## Stability
 
 Tracing instrumentation is still under active development, and may change

content/en/docs/reference/command-line-tools-reference/feature-gates.md

@@ -131,7 +131,8 @@ For a reference to old feature gates that are removed, please refer to
 | `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | 1.22 |
 | `KubeletPodResourcesGetAllocatable` | `true` | Beta | 1.23 | |
 | `KubeletPodResourcesDynamicResources` | `false` | Alpha | 1.27 | |
-| `KubeletTracing` | `false` | Alpha | 1.25 | |
+| `KubeletTracing` | `false` | Alpha | 1.25 | 1.26 |
+| `KubeletTracing` | `true` | Beta | 1.27 | |
 | `LegacyServiceAccountTokenTracking` | `false` | Alpha | 1.26 | 1.26 |
 | `LegacyServiceAccountTokenTracking` | `true` | Beta | 1.27 | |
 | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | - |