Skip to content

Commit ae7ae36

Browse files
committed
move to reference doc and address comments
1 parent 54e8463 commit ae7ae36

File tree

2 files changed

+23
-124
lines changed

2 files changed

+23
-124
lines changed

content/en/docs/concepts/cluster-administration/flow-control.md

Lines changed: 1 addition & 105 deletions
Original file line numberDiff line numberDiff line change
@@ -661,110 +661,6 @@ poorly-behaved workloads that may be harming system health.
661661
to a request being dispatched but did not, due to lack of available
662662
concurrency, broken down by `flow_schema` and `priority_level`.
663663

664-
### Debug endpoints
665-
666-
When you enable the API Priority and Fairness feature, the `kube-apiserver`
667-
serves the following additional paths at its HTTP(S) ports.
668-
669-
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of
670-
all the priority levels and the current state of each. You can fetch like this:
671-
672-
```shell
673-
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
674-
```
675-
676-
The output is similar to this:
677-
678-
```none
679-
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests, DispatchedRequests, RejectedRequests, TimedoutRequests, CancelledRequests
680-
catch-all, 0, true, false, 0, 0, 1, 0, 0, 0
681-
exempt, <none>, <none>, <none>, <none>, <none>, <none>, <none>, <none>, <none>
682-
global-default, 0, true, false, 0, 0, 46, 0, 0, 0
683-
leader-election, 0, true, false, 0, 0, 4, 0, 0, 0
684-
node-high, 0, true, false, 0, 0, 34, 0, 0, 0
685-
system, 0, true, false, 0, 0, 48, 0, 0, 0
686-
workload-high, 0, true, false, 0, 0, 500, 0, 0, 0
687-
workload-low, 0, true, false, 0, 0, 0, 0, 0, 0
688-
```
689-
690-
- `/debug/api_priority_and_fairness/dump_queues` - a listing of all the
691-
queues and their current state. You can fetch like this:
692-
693-
```shell
694-
kubectl get --raw /debug/api_priority_and_fairness/dump_queues
695-
```
696-
697-
The output is similar to this:
698-
699-
```none
700-
PriorityLevelName, Index, PendingRequests, ExecutingRequests, VirtualStart,
701-
workload-high, 0, 0, 0, 0.0000,
702-
workload-high, 1, 0, 0, 0.0000,
703-
workload-high, 2, 0, 0, 0.0000,
704-
...
705-
leader-election, 14, 0, 0, 0.0000,
706-
leader-election, 15, 0, 0, 0.0000,
707-
```
708-
709-
- `/debug/api_priority_and_fairness/dump_requests` - a listing of all the requests
710-
that are currently waiting in a queue. You can fetch like this:
711-
712-
```shell
713-
kubectl get --raw /debug/api_priority_and_fairness/dump_requests
714-
```
715-
716-
The output is similar to this:
717-
718-
```none
719-
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime,
720-
exempt, <none>, <none>, <none>, <none>, <none>,
721-
system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:26:57.179170694Z,
722-
```
723-
724-
In addition to the queued requests, the output includes one phantom line
725-
for each priority level that is exempt from limitation.
726-
727-
You can get a more detailed listing with a command like this:
728-
729-
```shell
730-
kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
731-
```
732-
733-
The output is similar to this:
734-
735-
```none
736-
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, UserName, Verb, APIPath, Namespace, Name, APIVersion, Resource, SubResource,
737-
system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:31:03.583823404Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
738-
system, system-nodes, 12, 1, system:node:127.0.0.1, 2020-07-23T15:31:03.594555947Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
739-
```
740-
741-
### Debug logging
742-
743-
At `-v=3` or more verbose the server outputs an httplog line for every
744-
request, and it includes the following attributes.
745-
746-
- `apf_fs`: the name of the flow schema to which the request was classified.
747-
- `apf_pl`: the name of the priority level for that flow schema.
748-
- `apf_iseats`: the number of seats determined for the initial
749-
(normal) stage of execution of the request.
750-
- `apf_fseats`: the number of seats determined for the final stage of
751-
execution (accounting for the associated WATCH notifications) of the
752-
request.
753-
- `apf_additionalLatency`: the duration of the final stage of
754-
execution of the request.
755-
756-
At higher levels of verbosity there will be log lines exposing details
757-
of how APF handled the request, primarily for debugging purposes.
758-
759-
### Response headers
760-
761-
APF adds the following two headers to each HTTP response message.
762-
763-
- `X-Kubernetes-PF-FlowSchema-UID` holds the UID of the FlowSchema
764-
object to which the corresponding request was classified.
765-
- `X-Kubernetes-PF-PriorityLevel-UID` holds the UID of the
766-
PriorityLevelConfiguration object associated with that FlowSchema.
767-
768664
## Good practices for using API Priority and Fairness
769665

770666
When a given priority level exceeds its permitted concurrency, requests can
@@ -881,7 +777,7 @@ Example FlowSchema object to isolate list event requests:
881777

882778
## {{% heading "whatsnext" %}}
883779

884-
You can visit flow control [troubleshooting page](/docs/tasks/debug/flow-control.md) to learn how to debug.
780+
You can visit flow control [reference doc](/docs/reference/flow-control/) to learn more about troubleshooting.
885781
For background information on design details for API priority and fairness, see
886782
the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness).
887783
You can make suggestions and feature requests via [SIG API Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)

content/en/docs/tasks/debug/debug-cluster/flow-control.md renamed to content/en/docs/reference/flow-control/_index.md

Lines changed: 22 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,27 @@
11
---
2-
title: Troubleshooting APIServer flow control issues
3-
content_type: task
2+
title: Flow control
3+
weight: 120
4+
no_list: true
45
---
56

67
<!-- overview -->
78

89
API Priority and Fairness controls the behavior of the Kubernetes API server in
910
an overload situation. You can find more information about it in the
10-
[concept doc](/docs/concepts/cluster-administration/flow-control.md)
11+
[API Priority and Fairness](/docs/concepts/cluster-administration/flow-control/)
12+
documentation.
1113

12-
## {{% heading "prerequisites" %}}
13-
14-
* Kubernetes cluster is installed
15-
* `kubectl` is configured to communicate with the cluster
16-
17-
<!-- steps -->
14+
<!-- body -->
1815

1916
## Diagnostics
2017

2118
Every HTTP response from an API server with the priority and fairness feature
2219
enabled has two extra headers: `X-Kubernetes-PF-FlowSchema-UID` and
2320
`X-Kubernetes-PF-PriorityLevel-UID`, noting the flow schema that matched the request
2421
and the priority level to which it was assigned, respectively. The API objects'
25-
names are not included in these headers in case the requesting user does not
26-
have permission to view them, so when debugging you can use a command like
22+
names are not included in these headers (to avoid revealing details in case the
23+
requesting user does not have permission to view them). When debugging, you
24+
can use a command such as:
2725

2826
```shell
2927
kubectl get flowschemas -o custom-columns="uid:{metadata.uid},name:{metadata.name}"
@@ -38,14 +36,19 @@ PriorityLevelConfigurations.
3836
When you enable the API Priority and Fairness feature, the `kube-apiserver`
3937
serves the following additional paths at its HTTP(S) ports.
4038

39+
You need to ensure you have permissions to access these endpoints.
40+
You don't have to do anything if you are using admin.
41+
Permissions can be granted if needed following [this doc](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
42+
to access `/debug/api_priority_and_fairness/` by specifying `nonResourceURLs`.
43+
4144
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of
4245
all the priority levels and the current state of each. You can fetch like this:
4346

4447
```shell
4548
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
4649
```
4750

48-
The output is similar to this:
51+
The output will be in CSV and similar to this:
4952

5053
```none
5154
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests, DispatchedRequests, RejectedRequests, TimedoutRequests, CancelledRequests
@@ -69,7 +72,7 @@ serves the following additional paths at its HTTP(S) ports.
6972
kubectl get --raw /debug/api_priority_and_fairness/dump_queues
7073
```
7174

72-
The output is similar to this:
75+
The output will be in CSV and similar to this:
7376

7477
```none
7578
PriorityLevelName, Index, PendingRequests, ExecutingRequests, SeatsInUse, NextDispatchR, InitialSeatsSum, MaxSeatsSum, TotalWorkSum
@@ -104,7 +107,7 @@ serves the following additional paths at its HTTP(S) ports.
104107
kubectl get --raw /debug/api_priority_and_fairness/dump_requests
105108
```
106109

107-
The output is similar to this:
110+
The output will be in CSV and similar to this:
108111

109112
```none
110113
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, InitialSeats, FinalSeats, AdditionalLatency, StartTime
@@ -119,7 +122,7 @@ serves the following additional paths at its HTTP(S) ports.
119122
kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
120123
```
121124

122-
The output is similar to this:
125+
The output will be in CSV and similar to this:
123126

124127
```none
125128
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, InitialSeats, FinalSeats, AdditionalLatency, StartTime, UserName, Verb, APIPath, Namespace, Name, APIVersion, Resource, SubResource
@@ -145,8 +148,8 @@ serves the following additional paths at its HTTP(S) ports.
145148

146149
## Debug logging
147150

148-
At `-v=3` or more verbose the server outputs an httplog line for every
149-
request, and it includes the following attributes.
151+
At `-v=3` or more verbosity, the API server outputs an httplog line for every
152+
request in the APIServer log, and it includes the following attributes.
150153

151154
- `apf_fs`: the name of the flow schema to which the request was classified.
152155
- `apf_pl`: the name of the priority level for that flow schema.
@@ -164,6 +167,8 @@ of how APF handled the request, primarily for debugging purposes.
164167
## Response headers
165168

166169
APF adds the following two headers to each HTTP response message.
170+
They won't appear in the audit log. They can be viewed from the client side.
171+
For client using klog, use verbosity `-v=8` or higher to view these headers.
167172

168173
- `X-Kubernetes-PF-FlowSchema-UID` holds the UID of the FlowSchema
169174
object to which the corresponding request was classified.
@@ -174,5 +179,3 @@ APF adds the following two headers to each HTTP response message.
174179

175180
For background information on design details for API priority and fairness, see
176181
the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness).
177-
You can make suggestions and feature requests via [SIG API Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
178-
or the feature's [slack channel](https://kubernetes.slack.com/messages/api-priority-and-fairness).

0 commit comments

Comments
 (0)