Getting application metrics to prometheus

[ratnadeepb]

I have a Java application (Kafka consumer) running in the default namespace of Kubernetes. It exposes a prometheus gauge metric which is exposed through HTTP.

I have tried adding all namespaces and to follow along the example app to get it going. But to no avail so far.

Generally the process looks like this so far:

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd /kube-prometheus/
kubectl create -f manifests/setup
kubectl wait \
	--for condition=Established \
	--all CustomResourceDefinition \
	--namespace=monitoring
cd jsonnet/kube-prometheus/
wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/examples/all-namespaces.jsonnet
jb install
jsonnet -J vendor all-namespaces.jsonnet -m . > json-files-list
mkdir manifests
# move the grafana files
for f in $(cat json-files-list | grep grafana); do mv $f manifests/$f.json; done
# convert and move the non-grafana files
for f in $(cat json-files-list | grep -v grafana); do yq r -P $f > manifests/$f.yaml; done
kubectl create -f manifests/

My service yaml looks like this:

apiVersion: v1
kind: Service
metadata:
  name: consumer
spec:
  ports: 
  - port: 9102
    name: metrics
    protocol: TCP
    targetPort: 9102
  selector:
    app: consumer
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: consumer
  name: consumer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      containers:
      - image: ratnadeepb/java-kafka-consumer:latest
        name: consumer
        ports:
        - containerPort: 9102

And the service monitor looks like this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: consumer
  namespace: monitoring
  labels:
    app: consumer
spec:
  selector:
    matchLabels:
      app: consumer
  endpoints:
  - port: metrics
    interval: 2s

The setup is on a Linux server with no GUI. So I am accessing the Prometheus dashboard from my laptop using port-forwarding:

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

ssh -L 9090:localhost:9090 node0

What am I missing? Any help is appreciated!

[CoalChris]

I managed to get my application's metrics showing by adding the scraping job config as Additional Scrape Configuration.