Incomplete install #10067

fredczj · 2023-01-02T11:35:01Z

fredczj
Jan 2, 2023

Hello !

Now that I have validated linkerd usage using command line interface install, which is terribly efficient, I test the deployment into a traditional cluster which has an already installed prometheus/grafana stack, and which uses a private PKI (cert-manager).

So I move the install to helm chart deployment, trying to change the minimum of things, but I try to re-use cert-manager and prometheus things.

The mesh result is working fine. The viz is incomplete, and I cannot understand why and how to debug this.

Basically, all the network metrics are just absent. But when I navigate to the metrics of a specific deployment, I see the live calls metrics generated (top like).

I guess there is a problem with my prometheus binding, but I cannot see how to debug this:

prometheus URL: prometheus-server.monitioring

control-plane deployment:

helm install linkerd-control-plane -n linkerd \
  --set-file identityTrustAnchorsPEM=ca.crt \
  --set identity.issuer.scheme=kubernetes.io/tls \
  --set prometheus.enabled=false \
  --set prometheusUrl=http://prometheus-server.monitoring \
  --set proxyInit.runAsRoot=true \
  --version 1.9.4 \
  linkerd/linkerd-control-plane

viz values:

prometheus:
  enabled: false

prometheusUrl: "http://prometheus-server.monitoring"

Finally, a proxy sidecar log:

[     0.002900s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.003443s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.003453s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.003456s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.003458s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.003461s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.uidigestv2.serviceaccount.identity.linkerd.cluster.local
[     0.003463s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003466s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.047287s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=default.uidigestv2.serviceaccount.identity.linkerd.cluster.local

What could I check to understand what is going wrong ?

All the checks from linkerd say everything is fine but…

Thank you

Answered by fredczj

Jan 6, 2023

Resolved: I missed a gotemplate replacement in the prometheus scrape configuration for linkerd-proxy:

#10086 (comment)

View full answer

fredczj · 2023-01-02T12:49:33Z

fredczj
Jan 2, 2023
Author

By the way, querying linkerd into prometheus returned no results at all…

0 replies

fredczj · 2023-01-02T15:30:46Z

fredczj
Jan 2, 2023
Author

I have checked the linkerd scrape configuration. Everything seems fine. Prometheus even indicates the scrape configuration is OK:

0 replies

fredczj · 2023-01-04T12:35:14Z

fredczj
Jan 4, 2023
Author

kubectl:

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:51:43Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:29:58Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}

linkerd version

Client version: stable-2.12.2
linServer version: stable-2.12.3

Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all node podCIDRs
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2023-02-02T03:56:52Z
    see https://linkerd.io/2.12/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.12.2 but the latest stable version is 2.12.3
    see https://linkerd.io/2.12/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
‼ control plane and cli versions match
    control plane running stable-2.12.3 but cli running stable-2.12.2
    see https://linkerd.io/2.12/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
‼ control plane proxies and cli versions match
    linkerd-destination-5b6f4d4f78-7cbvq running stable-2.12.3 but cli running stable-2.12.2
    see https://linkerd.io/2.12/checks/#l5d-cp-proxy-cli-version for hints

Linkerd extensions checks
=========================

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-86d75ffc69-gc7h9 (stable-2.12.2)
	* tap-8577b8b69-zpqfc (stable-2.12.2)
	* tap-injector-b7b9fc9f6-tdv5p (stable-2.12.2)
	* web-66b6b55fb7-qswl7 (stable-2.12.2)
    see https://linkerd.io/2.12/checks/#l5d-viz-proxy-cp-version for hints
√ viz extension proxies and cli versions match
√ can initialize the client
√ viz extension self-check

Status check results are √

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-86d75ffc69-gc7h9 (stable-2.12.2)
	* tap-8577b8b69-zpqfc (stable-2.12.2)
	* tap-injector-b7b9fc9f6-tdv5p (stable-2.12.2)
	* web-66b6b55fb7-qswl7 (stable-2.12.2)
    see https://linkerd.io/2.12/checks/#l5d-viz-proxy-cp-version for hints
√ viz extension proxies and cli versions match
√ can initialize the client
√ viz extension self-check

0 replies

fredczj · 2023-01-05T09:04:05Z

fredczj
Jan 5, 2023
Author

Therefore, I try to look where the metrics are.

Using linkerd diagnostics proxy-metrics returns data. But they seem not in prometheus, even if scrape configuration sounds correct.

For instance, I cannot find tcp_read_bytes_total, but I can get http_client_in_flight_requests of linkerd-controller

Few lines from diagnostic that are not present in prometheus (when I search tcp_write_bytes_total for instance)

# HELP tcp_write_bytes_total Total count of bytes written to peers
# TYPE tcp_write_bytes_total counter
tcp_write_bytes_total{direction="outbound",peer="dst",authority="uidigest-request-cip.uidigestv2.svc.cluster.local:8081",target_addr="10.2.2.246:8081",target_ip="10.2.2.246",target_port="8081",tls="true",server_id="default.uidigestv2.serviceaccount.identity.linkerd.cluster.local",dst_control_plane_ns="linkerd",dst_deployment="uidigest-request-dep",dst_namespace="uidigestv2",dst_pod="uidigest-request-dep-8ff5567cc-hngnz",dst_pod_template_hash="8ff5567cc",dst_service="uidigest-request-cip",dst_serviceaccount="default"} 3507
tcp_write_bytes_total{direction="outbound",peer="dst",authority="uidigest-request-cip.uidigestv2.svc.cluster.local:8081",target_addr="10.2.3.126:8081",target_ip="10.2.3.126",target_port="8081",tls="true",server_id="default.uidigestv2.serviceaccount.identity.linkerd.cluster.local",dst_control_plane_ns="linkerd",dst_deployment="uidigest-request-dep",dst_namespace="uidigestv2",dst_pod="uidigest-request-dep-8ff5567cc-7bwz5",dst_pod_template_hash="8ff5567cc",dst_service="uidigest-request-cip",dst_serviceaccount="default"} 3736
tcp_write_bytes_total{direction="outbound",peer="src",target_addr="10.3.200.251:8081",target_ip="10.3.200.251",target_port="8081",tls="no_identity",no_tls_reason="loopback"} 1148004
tcp_write_bytes_total{direction="outbound",peer="dst",authority="parcontrol-request-cip.parcontrolv3.svc.cluster.local:8081",target_addr="10.2.2.21:8081",target_ip="10.2.2.21",target_port="8081",tls="true",server_id="default.parcontrolv3.serviceaccount.identity.linkerd.cluster.local",dst_control_plane_ns="linkerd",dst_deployment="parcontrol-request-dep",dst_namespace="parcontrolv3",dst_pod="parcontrol-request-dep-5b67fbd695-bqxtp",dst_pod_template_hash="5b67fbd695",dst_service="parcontrol-request-cip",dst_serviceaccount="default"} 30255
tcp_write_bytes_total{direction="outbound",peer="src",target_addr="10.3.53.42:6379",target_ip="10.3.53.42",target_port="6379",tls="no_identity",no_tls_reason="loopback"} 28567936
tcp_write_bytes_total{direction="inbound",peer="src",target_addr="10.2.3.171:8081",target_ip="10.2.3.171",target_port="8081",tls="true",client_id="contour-envoy.contour.serviceaccount.identity.linkerd.cluster.local",srv_group="",srv_kind="default",srv_name="all-unauthenticated"} 1201336

0 replies

fredczj · 2023-01-05T09:04:55Z

fredczj
Jan 5, 2023
Author

After discussion on Slack, I move this discussion to issue

0 replies

fredczj · 2023-01-05T10:23:47Z

fredczj
Jan 5, 2023
Author

#10086

0 replies

fredczj · 2023-01-06T07:41:24Z

fredczj
Jan 6, 2023
Author

Resolved: I missed a gotemplate replacement in the prometheus scrape configuration for linkerd-proxy:

#10086 (comment)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Incomplete install #10067

Uh oh!

{{title}}

Uh oh!

Replies: 7 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Incomplete install #10067

Uh oh!

fredczj Jan 2, 2023

Replies: 7 comments

Uh oh!

fredczj Jan 2, 2023 Author

Uh oh!

fredczj Jan 2, 2023 Author

Uh oh!

fredczj Jan 4, 2023 Author

Uh oh!

fredczj Jan 5, 2023 Author

Uh oh!

fredczj Jan 5, 2023 Author

Uh oh!

fredczj Jan 5, 2023 Author

Uh oh!

fredczj Jan 6, 2023 Author

fredczj
Jan 2, 2023

fredczj
Jan 2, 2023
Author

fredczj
Jan 2, 2023
Author

fredczj
Jan 4, 2023
Author

fredczj
Jan 5, 2023
Author

fredczj
Jan 5, 2023
Author

fredczj
Jan 5, 2023
Author

fredczj
Jan 6, 2023
Author