kubectl apply is slow due to client side throttling

[Rick-van-Dam]

I constantly see these kind of messages when doing kubectl apply -f - --server-side --force-conflicts --prune --applyset=<applyset>:

Waited for 14.1888454s due to client-side throttling, not priority and fairness, request: GET:https:<url>

Likely this is due to having a decent amount of custom CRD's which causes me to hit the client side limit.

I tried to find a way to increase the client limits but these seem to be not configurable. My server can easily handle more (and if not I could simply add more powerful hardware). How to disable or increase this limit when using the cli? Having to wait is annoying and costly. I think its fine to have somewhat conservative limits by default but they should be configurable in a way.

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ardaguclu]

Thanks for filing an issue. Limits are hardcoded in

kubectl/pkg/cmd/cmd.go

Line 97 in 1fe7110

return genericclioptions.NewConfigFlags(true).WithDeprecatedPasswordFlag().WithDiscoveryBurst(300).WithDiscoveryQPS(50.0)

. Incorrect configurations may overload the API server. So that these values are not configurable and I don't think, we'd want to change this behavior.

[Rick-van-Dam]

Everyone's situation is different so having this configurable would better fit everyones situation.

I don't think preventing ppl to overload the api server is a good reason to hardcode this. If someone really wants to he could just make a custom client to do so. A warning in the documentation with some recommended values should probably be enough.

Even better would be to apply some server side control of this actually by sending back a 429.

[mpuckett159]

I'll bring it up that the next sig-cli meeting, but because it is really easy to brick your cluster due to over loading the API server I don't think that we will move forward with making these options configurable.

[Rick-van-Dam]

Could the limits atleast be raised as to not hamper normal use? Iam getting this pretty much everytime now. Surely the api server can handle way more than this without crashing?

[mpuckett159]

Sorry I completely forgot to bring this up I've added it to the next agenda so I don't forget.

[soltysh]

@Rick-van-Dam can you provide more logging output from your invocation. Before we decide what to do forward we'd like to better understand what problems you're actually running into. Also, please make sure to include kubectl and k8s versions in the output. Ideally please also provide us with the following information:

• how many CRDs you have in your cluster?
• how many files you're trying to apply?