Pods unable to connect to API Server after enabling linkerd

[listenerme]

Bug Report

What is the issue?

Pods like fluentd, filebeat when meshed, unable to connect to kubernetes api server to grab the metadata and it say connection refused on 10.0.0.1:443

How can it be reproduced?

Mesh filebeat daemonset with linkerd. Prometheus of the linkerd itself fail sometimes with similar error

Logs, error output, etc

level=error ts=2020-06-16T07:46:28.936Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:333: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.0.0.1:443: connect: connection refused"

level=error ts=2020-06-16T07:46:30.939Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:281: Failed to list *v1.Pod: Get https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0: EOF"

(If the output is long, please create a gist and
paste the link here.)

linkerd check output

your output here ...

Environment

• Kubernetes Version:
• Cluster Environment: (GKE, AKS, kops, ...)
• Host OS:
• Linkerd version:

Possible solution

Additional context

[grampelberg]

When are these messages printed?

[listenerme]

Above logs, that I got it from the output of Prometheus container

But I had the same issue(connection refused), while filebeat is configured to collect kubernetes metadata by contacting api server inside the pod. Tried setting the " skip outbound port" parameter to 443 while meshing filebeat, that too didnt work.

[grampelberg]

There's no benefit you get from meshing that, TBH. You'll need to share the filebeat resource definition as skipping the outbound port should have worked.

Normally when we see these issues, it is printed at startup. This is because the container starts up before the proxy and is unable to connect. You either need to make the container wait or skip the outbound port.

[listenerme]

I thought of adding linkerd annotation to the namespace so all app will be meshed there. It was not working by skipping outbound port. I guess linkerd proxy always starts after the original container. Anyways , I decided to inject necessary app instead of meshing namespace

[listenerme]

But any idea why api server access have issues through linkerd proxy? or is it expected behavior?

[grampelberg]

We've not seen any issues, assuming your app retries the connection on startup.

[listenerme]

Above logs, that I got it from the output of Prometheus container

But I had the same issue(connection refused), while filebeat is configured to collect kubernetes metadata by contacting api server inside the pod. Tried setting the " skip outbound port" parameter to 443 while meshing filebeat, that too didnt work.

Prometheus 443 error is not often, but for filebeat , its permanent.

[grampelberg]

Skipping outbound port means that it doesn't go through the proxy (at the iptables level), so it is confusing that filebeat still can't connect.

[listenerme]

ok, thanks for your replies