How to: Custom prometheus.yml for prometheus-prometheus.yaml

[jpradoar]

Hi all, I have been working in my custom config for prometheus and I have a strange problem.
I added my own custom prometheus.yml file and mount it in a volume in prometheus-prometheus.yaml, but when prometheus pod start and running, the config is not the same
example:
I depoyed configmap with custom prometheus config (custom-prometheus-config.yaml) and add a volume in prometheus-prometheus.yaml to use this config, but when I access to prometheus using kubectl -n monitoring port-forward svc/prometheus-k8s 9090 and access to http://127.0.0.1:9090/config I dont see my custom config, in fact I see so different to my file, and then if a access to pod kubectl -n monitoring exec -it pod/prometheus- k8s-0 -- sh
and execute cat /etc/prometheus/prometheus.yml I see the same as my configmap file has

I saw a similar discussion in (discussions/1323) but is not the same problem that I have.
Please, someone can show the correct way or maybe some documentation that how to do it.

Thx a lot!, ...and sorry for my bad english

custom-prometheus-config.yaml (my custom prometheus config file)

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: prometheus-master-conf
    namespace: monitoring
  data:
    prometheus.yml: |
       # 
       # ORIGINAL PROM CONFIG <---------- im not changed it
       #
       global:
          scrape_interval: 15s 
          evaluation_interval: 15s 
        scrape_configs:
          - job_name: 'prometheus'
            static_configs:
            - targets: ['localhost:9090']
          #
          #  PART OF MY CUSTOM CONFIG <---------- add it to original config     
          # 
          - job_name: "my-custom-endpoint"
            static_configs:
              - targets: ["remote-endpoint:8080"]
          #
          - job_name: 'federate'
            scrape_interval: 15s
            honor_labels: true
            metrics_path: '/federate'
            params:
              'match[]':
                - '{job="prometheus"}'
                - '{__name__=~"job:.*"}'
            static_configs:
             - targets:
               - ["remote-prometheus-01:9090", "remote-prometheus-02:9090", "remote-prometheus-N:9090",]
       # 
       # END CUSTOM CONFIG <---------- 
       #

prometheus-prometheus.yaml (this is "origian" file in this repo.)

# prometheus-prometheus.yaml 
 ...
 ...
 ...
version: 2.34.0
# ADD VOLUME TO SAVE CUSTOM CONFIG  <----------- I added it below
volumes:
- name: prometheus-config-volume      # PROMETHEUS CUSTOM CONFIG FILE
  configMap:
    name: prometheus-master-conf
containers:
- name: prometheus   
  volumeMounts:
  - name: prometheus-config-volume    # PROMETHEUS CUSTOM CONFIG FILE
    mountPath: /etc/prometheus/   

[pyrodex]

I did this with an additionalScrapeConfigs model. Here is a snippet from my json:

    prometheus+:: {
      prometheus+: {
        spec+: {
          additionalScrapeConfigs: {
            name: 'additional-scrape-configs',
            key: 'prometheus-additional.yaml',
          }, // additionalScrapeConfigs
          retention: '365d',
          storage: {
            volumeClaimTemplate: {
              apiVersion: 'v1',
              kind: 'PersistentVolumeClaim',
              spec: {
                accessModes: ['ReadWriteOnce'],
                resources: { requests: { storage: '100Gi' } },
                storageClassName: 'longhorn',
              },
            },
          },  // storage
        },  // spec+
      },  // prometheus+
    },  // prometheus++

I then crafted a simple script to help update it in the system without modifying the original prometheus.yaml to keep it cleaner.

#!/bin/sh
# Convert to needed secret format
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -n monitoring -oyaml > additional-scrape-configs.yaml
# Load the new format
kubectl apply -f additional-scrape-configs.yaml -n monitoring
# Reload prometheus
curl -v -X POST http://192.168.15.122:9090/-/reload

This will take a file called prometheus-additional.yaml and has the following as an example:

- job_name: "smartprom"
  scrape_interval:     300s
  static_configs:
  - targets:
     - 192.168.14.8:9902
    labels:
       name: dns01.lan
       network: LAN
       hardware: RPI4 4GB
       os: Debian 11
       location: Server Room
       function: DNS Server
  - targets:
     - 192.168.14.9:9902
    labels:
       name: dns02.lan
       network: LAN
       hardware: RPI4 4GB
       os: Debian 11
       location: HVAC Room
       function: DNS Server
  - targets:
       - 192.168.14.40:9902
    labels:
       name: monitoring.lan
       network: LAN
       hardware: physical
       os: Ubuntu 20.04
       location: Server Room
       function: Monitioring server
  - targets:
       - 192.168.14.122:9902
    labels:
       name: office4khtpc.lan
       network: LAN
       hardware: physical
       os: Ubuntu 20.10
       location: Richie's Office
       function: Richie's 4K HTPC
  - targets:
       - 192.168.14.32:9902
    labels:
       name: seedbox.lan
       network: LAN
       hardware: physical
       os: Ubuntu 20.04
       location: Server Room
       function: Seedbox server
  - targets:
       - 192.168.14.31:9902
    labels:
       name: tardis.lan
       network: LAN
       hardware: physical
       os: TrueNAS SCALE
       location: Server Room
       function: Storage server

I like this model since I can put in labels and stuff in there.

The script once run will also reload prometheus and the new targets will eventually show up.

[jpradoar]

Hey @pyrodex, thanks a lot for your help! to be honest, I read you explication but I cudn't resolv my problem, because I deploy all using kubectl, not using Json configs.
Of course you nice explication helped me to search the correct way (in my case) to solve my problem.
after search some information about your idea and how to fix, I found the exact solution for my case.

I added this three lines to my prometheus-prometheus.yaml

  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

Ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

Thanks so much!!!! :D

[pyrodex]

Yea I decided against the raw yaml so I don't get stagnant over time and have to compare them :) Glad I could help.

[jpradoar]

If someone has the same problem in any time, I offer my custom prometheus-prometheus.yaml file as an example.

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: encrypted-gp2
  namespace: monitoring
provisioner: kubernetes.io/aws-ebs # Only for AWS  EBS
allowVolumeExpansion: true
parameters:
  type: gp2
  fsType: ext4
  encrypted: "true"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: prometheus-pvc
  namespace: monitoring
  labels:
      prometheus: k8s
  annotations:
      prometheus.io/scrape: "true"
spec:
  storageClassName: encrypted-gp2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 400Gi

---      
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.35.0
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: alertmanager-main
      namespace: monitoring
      port: web
  enableFeatures: []
  externalLabels: 
    cluster: ${cluster_name}   # SUPER IMPORTANT TO USE FEDERATION
  image: quay.io/prometheus/prometheus:v2.35.0
  nodeSelector:
    kubernetes.io/os: linux
  podMetadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.35.0
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 1    # IMPORTANT:  in aws EBS dont support more than 1 pod attached to PV () 
  retention: 2y  # IMPORTANT:  retention on prometheus DB  2 years
  resources:
    requests:
      memory: 400Mi
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  additionalScrapeConfigs:             #----------------
    name: additional-scrape-configs    #  My problem: https://github.com/prometheus-operator/kube-prometheus/discussions/1713
    key: prometheus-additional.yaml    #  The solution:  https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md
  serviceMonitorNamespaceSelector: {}  #----------------
  serviceMonitorSelector: {}
  version: 2.35.0
  volumes:
  - name: prometheus-k8s-data
    persistentVolumeClaim:
      claimName: prometheus-pvc
  containers:
  - name: prometheus   
    volumeMounts:
    - mountPath: /prometheus
      name: prometheus-k8s-data