Add ServiceMonitor (and PodMonitor) resources instead of JUST `prometheus.io/` annotations

[frittentheke]

Proposed change

Full disclose: This has been asked for >3 years ago in #2029 already. But I strongly believe this is a (still) valid idea and a great improvement to this powerful chart to run JupyterHub on K8s.

I'd like to propose adding ServiceMonitor (https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitor) and potentially also PodMonitor (https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.PodMonitor) resources to integrate the exposed metrics with Kubernetes environments using the Prometheus Operator (https://github.com/prometheus-operator/prometheus-operator)

While I understand that you don't want to support all sorts of individual configurations, the usage of those Custom Resources to
configure Prometheus is really really common.

If you look across the Kubernetes ecosystem e.g. the Prometheus-Community repo with charts for various exporters, but also also other applications meant to run on Kubernetes ...

• Kube-State-Metrics https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-state-metrics/templates/servicemonitor.yaml
• Grafana https://github.com/grafana/helm-charts/blob/main/charts/grafana/templates/servicemonitor.yaml
• NGINX Ingress Controller https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/templates/controller-servicemonitor.yaml
• qdrant vector DB https://github.com/qdrant/qdrant-helm/blob/main/charts/qdrant/templates/servicemonitor.yaml
• OpenSearch feat: Add serviceMonitor resource opensearch-project/helm-charts#537
• Cloud Native PG PostgreSQL Operator https://cloudnative-pg.io/documentation/1.17/monitoring/
• TopoLVM CSI https://github.com/topolvm/topolvm/blob/main/charts/topolvm/templates/node/podmonitor.yaml
• RabbitMQ Operator https://www.rabbitmq.com/kubernetes/operator/operator-monitoring#prom-operator
• [...] https://github.com/search?q=repo%3Aprometheus-community%2Fhelm-charts+.Values.serviceMonitor.&type=code

The Prometheus Operator has also been adopted by lots of managed solutions, be it with the hyperscalers (Amazon EKS, Google Kubernetes Engine (GKE) Azure Kubernetes Service (AKS), ...) or on premise (VMware Tanzu Kubernetes Grid (TKG), OpenShift, Rancher)

Alternative options

The current set of annotations the likes of prometheus.io/scrape: true could remain in place and a new option in values.yaml could simply switch from those to dedicated ServiceMonitor and PodMonitor custom resources.

Who would use this feature?

Everybody using the Prometheus Operator to run their monitoring stack.

(Optional): Suggest a solution

see comment #2029 (comment) by @dmpe including a code snippet.

I gladly push a PR, if you somewhat agree that (optionally) providing Service- and PodMonitors (or active probes even ;-) ) is a good idea.

[manics]

I'm more inclined to support something like #3134 which allows arbitrary additional manifests without any expectation of support for the contents.

[frittentheke]

I'm more inclined to support something like #3134 which allows arbitrary additional manifests without any expectation of support for the contents.

@manics While it's certainly worth discussing if a particular functionality should come with a chart (e.g. the integration to popular monitoring stacks) or if it's a rather special case / addition which should be up to the user and individual installation to add, please kindly see me comment about the parent + subchart pattern #3134 (comment).

I still believe that providing a ServiceMonitor with the base application chart makes sense, since all of the potential variables (ports, additional labels, ...) are already available and, as I tried to explain in my initial comment, it's quite common among other charts. To me it's kinda like adding an Ingress resource. It's just really really common and somewhat of a "batteries included" approach, even though there might be people using service meshes that would require different resource types to configure that properly.

[korolenkowork]

Hey @manics, any updates on this? I can work on it if you guys are ready to support it

[manics]

My preference is to only support official kubernetes resources and avoid external CRDs in this chart. However if other maintainers think we should add it I won't block it.