Kubernetes-powered leader election for High availability

[snipa14]

Hi,

I’d like to ask if there are any plans to support high availability in some form. Also, I’m curious if the maintainers would be open to accepting a community implementation based on k8s.io/client-go/tools/leaderelection?

Thanks and happy to contribute if this aligns with the project's direction!

[npdgm]

Hi,
High availability can already be achieved by scaling up the Deployment to several replicas and then handling the deduplication of Prometheus series, depending on how your metrics stack is implemented (pure Prometheus, Thanos, etc.).
However, you might be considering a leader election mechanism so that only one instance would build the cache? This is the only scenario where I see an advantage.
Before proceeding with an implementation, could you please clarify the following:

• What specific problem would introducing a leader election solve?
• Why can't this issue be handled by the scraper?
• How will the leader be utilized by Prometheus? Will it manage a Pod label?

This information will help assess whether this is a viable direction. We are not inclined to dismiss work that has already been done.
The project is primarily driven by a KISS aproach to minimize the dependencies surface and avoid niche features that could introduce excessive maintenance or testing challenges. For example, a leader election mechanism would likely not be enabled by default and would be used by very few users.
Cheers

[snipa14]

Hi,

We have around 4,500 namespaces on one of our main clusters, with even more secrets to scan. We've already had to tweak kubeApiRateLimits to avoid overloading the cluster's API server, which is hosted on Gardener. So yes, our initial idea was to implement some form of leader election to prevent multiple X.509 exporters from building their cache simultaneously, which would further strain the API server.

As far as we can tell, deduplicating Prometheus metrics in a “soft” way isn’t something we’d prefer. One alternative we explored was using session affinity to have Prometheus scrape the first exporter pod in most cases — as long as it's available. However, Prometheus itself doesn’t really support automatic failover between multiple targets without introducing duplicated metrics. Tools like Thanos can handle that better, but we’re not using Thanos in this case.

To address your questions:
What specific problem would introducing a leader election solve?
It would help eliminate the need for static or soft configuration workarounds in Prometheus and, more importantly, reduce the load on the Kubernetes API server by allowing only one pod to build and maintain the cache.

Why can't this issue be handled by the scraper?
Prometheus doesn’t natively deduplicate metrics coming from multiple identical exporters. If two exporters expose the same series, Prometheus treats them as duplicates, which leads to inconsistencies unless you handle it manually with PromQL.

How will the leader be utilized by Prometheus? Will it manage a Pod label?
No — the idea is that only the leader pod would actively scrape and expose metrics. The others would stay idle (not start the exporter logic) until they acquire leadership, thereby preventing unnecessary load and avoiding duplicated data at the source.

The final approach I'd like to highlight involves introducing NGINX in front of the two X.509 Certificate Exporter pods. NGINX can be configured to prioritize the first pod and fall back to the second when necessary. This method requires no changes to the X.509 Certificate Exporter itself.

In case you believe we should consider a different approach — such as simply increasing the number of deployments, adjusting the rate limits, and potentially introducing session affinity — I’m also open to hearing that.