New target in the prometheus.yaml

[anil-venkata]

Hi team,

How can we add new target/exporter to the prometheus.yaml.

Actual requirement is , need to install telegraf for the logs and will be deploying as a daemonset in the cluster. And we will use prometheus as a output plugin for telegraf. and let say it will run with the port : 9273 in all the hosts like node_exporter.

So how can i add all the telegraf agents running in all the nodes as a target in the prometheus.

[olegmayko]

Use ServiceMonitor or PodMonitor, you may find more info in the getting-started

[anil-venkata]

@olegmayko , thanks for the suggestion, and sure will try that.

[SamYuan1990]

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: main
          image: nginx
          imagePullPolicy: Always
          ports:
            - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
    - name: operations
      port: 80
      protocol: TCP
  selector:
    app: nginx

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  endpoints:
  - port: operations

@anil-venkata , @olegmayko and team, I had tried to apply yaml above to monitoring namespace after

kubectl create -f manifests/setup
kubectl get po -n monitoring
kubectl create -f manifests/
kubectl get po -n monitoring

but I can't find the target listed at

http://localhost:9090/targets

please advise.

[SamYuan1990]

yuanyi@192 kube-prometheus % kubectl get ServiceMonitor -n monitoring | grep nginx
nginx                     14m

I suppose even if there some kind of settings not applied at nginx side. But there should be an item there listing as nginx with unhealthy?

[SamYuan1990]

ok, I got a sample for it. the reason for mistake seems related to my svc missed with label app: org0-orderer1

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: monitor-org0-orderer1
  namespace: monitoring
  labels:
    release: mypro  # I am still investigating on this.
spec:
  namespaceSelector: # the name space for pod
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app: org0-orderer1
  endpoints:
  - port: operations