Skip to content

Commit 1b942ca

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #42075 from mengqiy/apf
Add explanation for APF debug output
2 parents 0209b39 + 0469dc9 commit 1b942ca

File tree

3 files changed

+188
-123
lines changed

3 files changed

+188
-123
lines changed

‎content/en/docs/concepts/cluster-administration/flow-control.md

Lines changed: 3 additions & 123 deletions
Original file line numberDiff line numberDiff line change
@@ -472,23 +472,6 @@ that originate from outside your cluster.
472472

473473
{{% code_sample file="priority-and-fairness/health-for-strangers.yaml" %}}
474474

475-
## Diagnostics
476-
477-
Every HTTP response from an API server with the priority and fairness feature
478-
enabled has two extra headers: `X-Kubernetes-PF-FlowSchema-UID` and
479-
`X-Kubernetes-PF-PriorityLevel-UID`, noting the flow schema that matched the request
480-
and the priority level to which it was assigned, respectively. The API objects'
481-
names are not included in these headers in case the requesting user does not
482-
have permission to view them, so when debugging you can use a command like
483-
484-
```shell
485-
kubectl get flowschemas -o custom-columns="uid:{metadata.uid},name:{metadata.name}"
486-
kubectl get prioritylevelconfigurations -o custom-columns="uid:{metadata.uid},name:{metadata.name}"
487-
```
488-
489-
to get a mapping of UIDs to names for both FlowSchemas and
490-
PriorityLevelConfigurations.
491-
492475
## Observability
493476

494477
### Metrics
@@ -678,110 +661,6 @@ poorly-behaved workloads that may be harming system health.
678661
to a request being dispatched but did not, due to lack of available
679662
concurrency, broken down by `flow_schema` and `priority_level`.
680663

681-
### Debug endpoints
682-
683-
When you enable the API Priority and Fairness feature, the `kube-apiserver`
684-
serves the following additional paths at its HTTP(S) ports.
685-
686-
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of
687-
all the priority levels and the current state of each. You can fetch like this:
688-
689-
```shell
690-
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
691-
```
692-
693-
The output is similar to this:
694-
695-
```none
696-
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests, DispatchedRequests, RejectedRequests, TimedoutRequests, CancelledRequests
697-
catch-all, 0, true, false, 0, 0, 1, 0, 0, 0
698-
exempt, <none>, <none>, <none>, <none>, <none>, <none>, <none>, <none>, <none>
699-
global-default, 0, true, false, 0, 0, 46, 0, 0, 0
700-
leader-election, 0, true, false, 0, 0, 4, 0, 0, 0
701-
node-high, 0, true, false, 0, 0, 34, 0, 0, 0
702-
system, 0, true, false, 0, 0, 48, 0, 0, 0
703-
workload-high, 0, true, false, 0, 0, 500, 0, 0, 0
704-
workload-low, 0, true, false, 0, 0, 0, 0, 0, 0
705-
```
706-
707-
- `/debug/api_priority_and_fairness/dump_queues` - a listing of all the
708-
queues and their current state. You can fetch like this:
709-
710-
```shell
711-
kubectl get --raw /debug/api_priority_and_fairness/dump_queues
712-
```
713-
714-
The output is similar to this:
715-
716-
```none
717-
PriorityLevelName, Index, PendingRequests, ExecutingRequests, VirtualStart,
718-
workload-high, 0, 0, 0, 0.0000,
719-
workload-high, 1, 0, 0, 0.0000,
720-
workload-high, 2, 0, 0, 0.0000,
721-
...
722-
leader-election, 14, 0, 0, 0.0000,
723-
leader-election, 15, 0, 0, 0.0000,
724-
```
725-
726-
- `/debug/api_priority_and_fairness/dump_requests` - a listing of all the requests
727-
that are currently waiting in a queue. You can fetch like this:
728-
729-
```shell
730-
kubectl get --raw /debug/api_priority_and_fairness/dump_requests
731-
```
732-
733-
The output is similar to this:
734-
735-
```none
736-
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime,
737-
exempt, <none>, <none>, <none>, <none>, <none>,
738-
system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:26:57.179170694Z,
739-
```
740-
741-
In addition to the queued requests, the output includes one phantom line
742-
for each priority level that is exempt from limitation.
743-
744-
You can get a more detailed listing with a command like this:
745-
746-
```shell
747-
kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
748-
```
749-
750-
The output is similar to this:
751-
752-
```none
753-
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, UserName, Verb, APIPath, Namespace, Name, APIVersion, Resource, SubResource,
754-
system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:31:03.583823404Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
755-
system, system-nodes, 12, 1, system:node:127.0.0.1, 2020-07-23T15:31:03.594555947Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
756-
```
757-
758-
### Debug logging
759-
760-
At `-v=3` or more verbose the server outputs an httplog line for every
761-
request, and it includes the following attributes.
762-
763-
- `apf_fs`: the name of the flow schema to which the request was classified.
764-
- `apf_pl`: the name of the priority level for that flow schema.
765-
- `apf_iseats`: the number of seats determined for the initial
766-
(normal) stage of execution of the request.
767-
- `apf_fseats`: the number of seats determined for the final stage of
768-
execution (accounting for the associated WATCH notifications) of the
769-
request.
770-
- `apf_additionalLatency`: the duration of the final stage of
771-
execution of the request.
772-
773-
At higher levels of verbosity there will be log lines exposing details
774-
of how APF handled the request, primarily for debugging purposes.
775-
776-
### Response headers
777-
778-
APF adds the following two headers to each HTTP response message.
779-
780-
- `X-Kubernetes-PF-FlowSchema-UID` holds the UID of the FlowSchema
781-
object to which the corresponding request was classified.
782-
- `X-Kubernetes-PF-PriorityLevel-UID` holds the UID of the
783-
PriorityLevelConfiguration object associated with that FlowSchema.
784-
785664
## Good practices for using API Priority and Fairness
786665

787666
When a given priority level exceeds its permitted concurrency, requests can
@@ -898,7 +777,8 @@ Example FlowSchema object to isolate list event requests:
898777

899778
## {{% heading "whatsnext" %}}
900779

901-
For background information on design details for API priority and fairness, see
780+
- You can visit flow control [reference doc](/docs/reference/flow-control/) to learn more about troubleshooting.
781+
- For background information on design details for API priority and fairness, see
902782
the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness).
903-
You can make suggestions and feature requests via [SIG API Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
783+
- You can make suggestions and feature requests via [SIG API Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
904784
or the feature's [slack channel](https://kubernetes.slack.com/messages/api-priority-and-fairness).

‎content/en/docs/reference/debug-cluster/_index.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
title: Debug cluster
3+
weight: 120
4+
no_list: false
5+
---

‎content/en/docs/reference/debug-cluster/flow-control.md

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
---
2+
title: Flow control
3+
weight: 130
4+
---
5+
6+
<!-- overview -->
7+
8+
API Priority and Fairness controls the behavior of the Kubernetes API server in
9+
an overload situation. You can find more information about it in the
10+
[API Priority and Fairness](/docs/concepts/cluster-administration/flow-control/)
11+
documentation.
12+
13+
<!-- body -->
14+
15+
## Diagnostics
16+
17+
Every HTTP response from an API server with the priority and fairness feature
18+
enabled has two extra headers: `X-Kubernetes-PF-FlowSchema-UID` and
19+
`X-Kubernetes-PF-PriorityLevel-UID`, noting the flow schema that matched the request
20+
and the priority level to which it was assigned, respectively. The API objects'
21+
names are not included in these headers (to avoid revealing details in case the
22+
requesting user does not have permission to view them). When debugging, you
23+
can use a command such as:
24+
25+
```shell
26+
kubectl get flowschemas -o custom-columns="uid:{metadata.uid},name:{metadata.name}"
27+
kubectl get prioritylevelconfigurations -o custom-columns="uid:{metadata.uid},name:{metadata.name}"
28+
```
29+
30+
to get a mapping of UIDs to names for both FlowSchemas and
31+
PriorityLevelConfigurations.
32+
33+
## Debug endpoints
34+
35+
With the `APIPriorityAndFairness` feature enabled, the `kube-apiserver`
36+
serves the following additional paths at its HTTP(S) ports.
37+
38+
You need to ensure you have permissions to access these endpoints.
39+
You don't have to do anything if you are using admin.
40+
Permissions can be granted if needed following the [RBAC](/docs/reference/access-authn-authz/rbac/) doc
41+
to access `/debug/api_priority_and_fairness/` by specifying `nonResourceURLs`.
42+
43+
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of
44+
all the priority levels and the current state of each. You can fetch like this:
45+
46+
```shell
47+
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
48+
```
49+
50+
The output will be in CSV and similar to this:
51+
52+
```none
53+
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests, DispatchedRequests, RejectedRequests, TimedoutRequests, CancelledRequests
54+
catch-all, 0, true, false, 0, 0, 1, 0, 0, 0
55+
exempt, 0, true, false, 0, 0, 0, 0, 0, 0
56+
global-default, 0, true, false, 0, 0, 46, 0, 0, 0
57+
leader-election, 0, true, false, 0, 0, 4, 0, 0, 0
58+
node-high, 0, true, false, 0, 0, 34, 0, 0, 0
59+
system, 0, true, false, 0, 0, 48, 0, 0, 0
60+
workload-high, 0, true, false, 0, 0, 500, 0, 0, 0
61+
workload-low, 0, true, false, 0, 0, 0, 0, 0, 0
62+
```
63+
64+
Explanation for selected column names:
65+
- `IsQuiescing` indicates if this priority level will be removed when its queues have been drained.
66+
67+
- `/debug/api_priority_and_fairness/dump_queues` - a listing of all the
68+
queues and their current state. You can fetch like this:
69+
70+
```shell
71+
kubectl get --raw /debug/api_priority_and_fairness/dump_queues
72+
```
73+
74+
The output will be in CSV and similar to this:
75+
76+
```none
77+
PriorityLevelName, Index, PendingRequests, ExecutingRequests, SeatsInUse, NextDispatchR, InitialSeatsSum, MaxSeatsSum, TotalWorkSum
78+
workload-low, 14, 27, 0, 0, 77.64342019ss, 270, 270, 0.81000000ss
79+
workload-low, 74, 26, 0, 0, 76.95387841ss, 260, 260, 0.78000000ss
80+
...
81+
leader-election, 0, 0, 0, 0, 5088.87053833ss, 0, 0, 0.00000000ss
82+
leader-election, 1, 0, 0, 0, 0.00000000ss, 0, 0, 0.00000000ss
83+
...
84+
workload-high, 0, 0, 0, 0, 0.00000000ss, 0, 0, 0.00000000ss
85+
workload-high, 1, 0, 0, 0, 1119.44936475ss, 0, 0, 0.00000000ss
86+
```
87+
88+
Explanation for selected column names:
89+
- `NextDispatchR`: The R progress meter reading, in units of seat-seconds, at
90+
which the next request will be dispatched.
91+
- `InitialSeatsSum`: The sum of InitialSeats associated with all requests in
92+
a given queue.
93+
- `MaxSeatsSum`: The sum of MaxSeats associated with all requests in a given
94+
queue.
95+
- `TotalWorkSum`: The sum of total work, in units of seat-seconds, of all
96+
waiting requests in a given queue.
97+
98+
Note: `seat-second` (abbreviate as `ss`) is a measure of work, in units of
99+
seat-seconds, in the APF world.
100+
101+
- `/debug/api_priority_and_fairness/dump_requests` - a listing of all the requests
102+
including requests waiting in a queue and requests being executing.
103+
You can fetch like this:
104+
105+
```shell
106+
kubectl get --raw /debug/api_priority_and_fairness/dump_requests
107+
```
108+
109+
The output will be in CSV and similar to this:
110+
111+
```none
112+
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, InitialSeats, FinalSeats, AdditionalLatency, StartTime
113+
exempt, exempt, -1, -1, , 2023-07-15T04:51:25.596404345Z, 1, 0, 0s, 2023-07-15T04:51:25.596404345Z
114+
workload-low, service-accounts, 14, 0, system:serviceaccount:default:loadtest, 2023-07-18T00:12:51.386556253Z, 10, 0, 0s, 0001-01-01T00:00:00Z
115+
workload-low, service-accounts, 14, 1, system:serviceaccount:default:loadtest, 2023-07-18T00:12:51.487092539Z, 10, 0, 0s, 0001-01-01T00:00:00Z
116+
```
117+
118+
You can get a more detailed listing with a command like this:
119+
120+
```shell
121+
kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
122+
```
123+
124+
The output will be in CSV and similar to this:
125+
126+
```none
127+
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, InitialSeats, FinalSeats, AdditionalLatency, StartTime, UserName, Verb, APIPath, Namespace, Name, APIVersion, Resource, SubResource
128+
exempt, exempt, -1, -1, , 2023-07-15T04:51:25.596404345Z, 1, 0, 0s, 2023-07-15T04:51:25.596404345Z, system:serviceaccount:system:admin, list, /api/v1/namespaces/kube-stress/configmaps, kube-stress, , v1, configmaps,
129+
workload-low, service-accounts, 14, 0, system:serviceaccount:default:loadtest, 2023-07-18T00:13:08.986534842Z, 10, 0, 0s, 0001-01-01T00:00:00Z, system:serviceaccount:default:loadtest, list, /api/v1/namespaces/kube-stress/configmaps, kube-stress, , v1, configmaps,
130+
workload-low, service-accounts, 14, 1, system:serviceaccount:default:loadtest, 2023-07-18T00:13:09.086476021Z, 10, 0, 0s, 0001-01-01T00:00:00Z, system:serviceaccount:default:loadtest, list, /api/v1/namespaces/kube-stress/configmaps, kube-stress, , v1, configmaps,
131+
```
132+
133+
Explanation for selected column names:
134+
- `QueueIndex`: The index of the queue. It will be -1 for priority levels
135+
without queues.
136+
- `RequestIndexInQueue`: The index in the queue for a given request. It will
137+
be -1 for executing requests.
138+
- `InitialSeats`: The number of seats will be occupied during the initial
139+
(normal) stage of execution of the request.
140+
- `FinalSeats`: The number of seats will be occupied during the final stage
141+
of request execution, accounting for the associated WATCH notifications.
142+
- `AdditionalLatency`: The extra time taken during the final stage of request
143+
execution. FinalSeats will be occupied during this time period. It does not
144+
mean any latency that a user will observe.
145+
- `StartTime`: The time a request starts to execute. It will be
146+
0001-01-01T00:00:00Z for queued requests.
147+
148+
## Debug logging
149+
150+
At `-v=3` or more verbosity, the API server outputs an httplog line for every
151+
request in the API server log, and it includes the following attributes.
152+
153+
- `apf_fs`: the name of the flow schema to which the request was classified.
154+
- `apf_pl`: the name of the priority level for that flow schema.
155+
- `apf_iseats`: the number of seats determined for the initial
156+
(normal) stage of execution of the request.
157+
- `apf_fseats`: the number of seats determined for the final stage of
158+
execution (accounting for the associated `watch` notifications) of the
159+
request.
160+
- `apf_additionalLatency`: the duration of the final stage of
161+
execution of the request.
162+
163+
At higher levels of verbosity there will be log lines exposing details
164+
of how APF handled the request, primarily for debugging purposes.
165+
166+
## Response headers
167+
168+
APF adds the following two headers to each HTTP response message.
169+
They won't appear in the audit log. They can be viewed from the client side.
170+
For client using `klog`, use verbosity `-v=8` or higher to view these headers.
171+
172+
- `X-Kubernetes-PF-FlowSchema-UID` holds the UID of the FlowSchema
173+
object to which the corresponding request was classified.
174+
- `X-Kubernetes-PF-PriorityLevel-UID` holds the UID of the
175+
PriorityLevelConfiguration object associated with that FlowSchema.
176+
177+
## {{% heading "whatsnext" %}}
178+
179+
For background information on design details for API priority and fairness, see
180+
the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness).

0 commit comments

Comments
 (0)