HTTP/2 GRPC proxy dispatch timed out

[RagingPuppies]

Hi,
quick question,
i'm aware that by default the maximum requests amount of http2 connection is 100,
when my Kubernetes deployment reaches 300 RPS with 3 replica pod i get "proxy dispatch timed out"
i suspect that linkerd -> pod awaiting for the 100 requests to finish and meanwhile hits the 10s default timeout,
can i tell linkerd to open multiple connections?
this is an example of error message:
[559392.895207265s] WARN inbound:accept{peer.addr=10.42.81.251:36920}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.42.81.208:5000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion

extra info i use this flow:
Client(grpc)->haproxy(h2)->ingress(grpc)->linkerd_ingress_side_car->linkerd_pod_side_car->pod(grpc)
linkerd_pod_side_car is the place i hits the error message, there is a very high load on the environment at that time..
if someone ever seen this and know how to debbug\solve it it will be great!
Thanks

[olix0r]

Hi @RagingPuppies,

i'm aware that by default the maximum requests amount of http2 connection is 100,

Where is that limit being enforced? Linkerd shouldn't be applying any default stream concurrency limits.

There currently is no connection pooling for HTTP/2 clients because this would bypass any server-configured stream concurrency limit.

We've run gRPC load tests at much higher than 300RPS, so it's a bit surprising to hit these problems at such a low volume. What sort of response latency does your service impose?

[RagingPuppies]

Hi @olix0r !
thanks for the quick reply,
That's good to hear that the limits are not on linkerd,
yes, I've seen the response rate going up in the corresponding time in my application container to 3.5s and i'm afraid some calls reached the timeout, so just to be sure, it is probably on my application side? can you maybe point me in the direction i should investigate? (application requests limit? hanging calls?)