How to avoid Linkerd reporting success rate failures when handling Graceful shutdown

[2snEM6]

I have setup Linkerd injection with the --wait-before-exit-seconds 180 flag.

All my containers perform graceful shutdowns by waiting 120 seconds after the SIGTERM signal is sent by Kubernetes with a preStop lifecycle hook.

I understand that the rule is as follows: application life cycle hook sleep < linkerd proxy life cycle hook sleep < terminationGracePeriodSeconds.

Using fortio I am checking whether requests are being missed and there are no missed requests during rolling update, which is great and then the graceful shutdown problem is solved.

Now, since the application shutsdowns before the Linkerd proxy does, the Linkerd proxy tries to connect to the application, and reports the following message:

linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused graceful shutdown

This obviously makes sense, since the application has gracefully shutdown already.

However, what it bugs me is that within Grafana, the success rate lowers during rolling updates despite of the actual traffic being correctly handled with 0 downtime.

Is there a way to avoid this? Since in reality no requests are being missed.

Thanks!

[olix0r]

Is that failure on the admin port? Is the solution simply to break out success rate by port in the grafana dashboard?

[2snEM6]

I checked our logs and it returns this:

[ 10351.337113s] INFO ThreadId(01) inbound:server{port=8912}:rescue{client.addr=10.244.4.62:41706}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 10361.337430s] INFO ThreadId(01) inbound:server{port=8912}:rescue{client.addr=10.244.4.62:41850}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)

Port 8912 is used by our application readiness and liveness probes, which is different than the main application port. Since the app is shutdown already, obviously readiness probes fail and Linkerd picks them up.

Reading your comment, maybe filtering failures on this port within Grafana is enough I guess?

Thanks

[olix0r]

Yeah, Linkerd's UIs don't currently do a great job of breaking out these details per port. I suspect that it's pretty easy to update the queries in grafana, at least. That would be a great PR, if you're up for it ;)

[2snEM6]

I just had to add target_port!=<value> to every query in Grafana in order to achieve this, thanks for suggesting it. I wonder how we could present this in a nice way within Grafana. Maybe displaying a Grafana variable on top, where you can exclude ports that are autodiscovered.

[2snEM6]

Something like this

[olix0r]

or perhaps we should update the queries to aggregate by target_port. e.g. sum(...) by (..., target_port)? We could control the displayed target ports with a grafana variable so that you have the option to show all of the per-port queries or restrict it to a single port.