Kube-Prometheus-Stack and securing external node-exporter

[m6f13]

Hello,
i would like to secure my external node-exporter with tls and authentication, so that in my network, not everyone is able to access the metrics exposed by the node-exporter.

on prometheus side i have service, Servicemonitor and endpoint:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
    prometheus.io/scrape: 'true'
  labels:
    app: node-exporter-vm-agent
    jobLabel: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  externalName: 192.168.1.72
  ports:
  - name: metrics
    port: 9100
    protocol: TCP
    targetPort: 9100
  selector:
    app: vm-agent
    release: prom00
  type: ExternalName

---

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
  labels:
    app: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  endpoints:
    - port: metrics
      scheme: https
      tlsConfig:
        insecureSkipVerify: true
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: node-exporter-vm-agent
      release: prom00

---

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: node-exporter-vm-agent
    jobLabel: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
subsets:
- addresses:
  - ip: 192.168.1.72
    nodeName: 192.168.1.72
  ports:
  - name: metrics
    port: 9100
    protocol: TCP

on the external node-exporter i have the web.yml with the generated crt and key file and the password generated with htpasswd.

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key
  # basic_auth_users:
  # prometheus: $2y$10$V2RmZ2wKC7S8jhEz1OXRKOLkq1UHw4qlgpHT.hMg7B447dJQl7RqS

I can use the self-generated certificate by using insecureSkipVerify: true.
if i enable basic_auth_users with the user: prometheus and password, it works when i try to access the node-exporter and enter the user/password.

But how to implement the basic_auth into yaml to create the credential / or what is the correct command.?
Is there any better way to secure the external node-exporter, if prometheus is deploy with helm?

thanks for your help!

[paulfantom]

In ServiceMonitor Endpoint you can add basicAuth and/or tlsConfig. Sth like:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
  labels:
    app: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  endpoints:
    - port: metrics
      scheme: https
      tlsConfig:
        ca: <CA_CONFIGMAP_OR_SECRET_REF>
        cert: <CERT_CONFIGMAP_OR_SECRET_REF>
        keySecret: <KEY_SECRET_REF>
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: node-exporter-vm-agent
      release: prom00

Alternatively with basicAuth:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
  labels:
    app: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  endpoints:
    - port: metrics
      scheme: https
      basicAuth:
        username: <SECRET_REF>
        password: <SECRET_REF>
      tlsConfig:
        insecureSkipVerify: false
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: node-exporter-vm-agent
      release: prom00

*_REF follows spec of https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#secretkeyselector-v1-core

For additional data you can check prometheus-operator API docs for ServiceMonitorin in https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#servicemonitor

[m6f13]

Hello,
thanks for your feedback.
to complete this, below the notes for this topic:
........
spec:
endpoints:
- port: metrics
scheme: https
basicAuth:
username:
key: username
name: basic-auth
password:
key: password
name: basic-auth
tlsConfig:
insecureSkipVerify: true
......
for the secret: i have create it with:
kubectl -n monitoring-dev create secret generic basic-auth --from-literal=username='prometheus' --from-literal=password='password'

on the node-exporter, i have created a web-config.yml file with:

tls_server_config:
cert_file: ../cert/prom_node_exp.crt
key_file: ../cert/prom_node_expnopass.key
basic_auth_users:
prometheus: $2y$10$W.nywLSnmQjagtmT6k4uLedGhk1sWMMG3Rspv2r6Z0CzGmLJUveFC

--> the password for the user is created with: htpasswd -nBC 10 "" | tr -d ':\n'

thanks for your support.

[subhranil05]

@m6f13 Can you please share the steps how you created web-config.yml ,started the node-exporter with config file in node-exporter in k8
Do I have to login to pod and follow those steps?
Plz share