Prometheus target is added but not discovered (0/0 up)

[efotopoulou]

I successfully deploy deploy my app at kubernetes. The ServiceMonitor is also added but no targets are discovered (0/0 up) .
The application is an nginx server with the relevant nginx-prometheus-exporter sidecar.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-example-v3
  labels:
    app: nginx-example-v3
spec:
  selector:
    matchLabels:
      app: nginx-example-v3
  template:
    metadata:
      labels:
        app: nginx-example-v3
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            memory: "128Mi"
            cpu: "100m"
        ports:
        - name: http
          containerPort: 8080
        volumeMounts:
        - name: "config"
          mountPath: "/etc/nginx/nginx.conf"
          subPath: "nginx.conf"
      - name: exporter
        image: nginx/nginx-prometheus-exporter:0.8.0
        ports:
        - containerPort: 9113
      volumes:
        - name: "config"
          configMap:
            name: "nginx-example-v2-config"
            
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: nginx-example-v3
  name: nginx-example-v3
spec:
  type: LoadBalancer
  selector:
    app: nginx-example-v3
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: http-exporter
    port: 9113
    targetPort: 9113

After that i can see the nginx custom metrics exposed at the /metrics API:

Then i apply the monitoring service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-example-v3
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: web
  selector:
    matchLabels:
      app: nginx-example-v3

i can see that the service is successfully added at prometheus "Service Discovery" section:

BUT no targets are discovered (0/0 up) on behalf of prometheus

What i am missing???
Any help is really apreciated since i have been stuck on this many days!
Thank you very much in advance. :-)

[efotopoulou]

I slightly changed the service manifest and i am able to get the metrics now.
sorry for the new discussion. i hope my manifests help other persons to configure their services.
This is the updated manifest.
:-)

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-example-v3
  name: nginx-example-v3
spec:
  type: LoadBalancer
  selector:
    app: nginx-example-v3
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: web
    port: 9113
    targetPort: 9113

[seva-ramin]

Is this all you needed to do? I am basically doing the same thing, but not able to see the service or the target. I think my issue might be that prometheus is running in a different namespace and my app/services are in default namespace.

[xen22]

I had exactly the same issue: the service was missing the 'app' label (like the author, I had the app label in the deployment but not in the service). As soon as I added it, it started working. It makes sense because the ServiceMonitor in this example, and in mine, looks at that label to decide whether to scrape the service.
Thank you so much!