Target DOWN - Context deadline exceeded.

[maverick2808]

I was able to setup Prometheus using kube-prometheus in our Tanzu cluster but I see Kube-scheduler and kube-controller-manager targets are DOWN reporting.

Get "https://192.168.4.90:10259/metrics": context deadline exceeded

Get "https://192.168.4.90:10257/metrics": context deadline exceeded

Tried increasing the scrape timeout to 1 min as well and tried changing the ports of the kube scheduler to 10251 & 10257, but nothing helped.
From one of the Prometheus pod, I'm able to telnet to the "kubelet IP and port", i.e 192.168.4.90:10250 but not the kube-scheduler endpoint, not sure where to check the logs for this problem.

kube-prometheus v0.9

NAME                                                         STATUS   ROLES                  AGE   VERSION
mycluster-tanzu-control-plane-mjirt                          Ready    control-plane,master   30d   v1.23.8+vmware.3
mycluster-09jhy6gdg8-68bg7                                   Ready    <none>                 30d   v1.23.8+vmware.3
mycluster-09jhy6gdg8-f7zjb                                   Ready    <none>                 30d   v1.23.8+vmware.3
mycluster-09jhy6gdg8-fr9gg                                   Ready    <none>                 30d   v1.23.8+vmware.3
mycluster-09jhy6gdg8-k2677                                   Ready    <none>                 30d   v1.23.8+vmware.3


apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: kube-prometheus
    app.kubernetes.io/part-of: kube-prometheus
    component: kube-scheduler
  name: kube-scheduler
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 10s
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: component
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      component: kube-scheduler
	  
apiVersion: v1
kind: Service
metadata:
  labels:
    component: kube-scheduler
  name: kube-scheduler
  namespace: kube-system
spec:
  clusterIP: None
  ports:
    - name: http-metrics
      port: 10259
      protocol: TCP
      targetPort: 10259
  selector:
    component: kube-scheduler

[shixinlishixinli]

i meet the same bug, do you have solution