mgr/cephadm: use ip_hash to enforce the same prometheues upstream

Implemented ip_hash in Nginx to ensure consistent upstream selection
for Prometheus in HA scenarios. This helps maintain query consistency
in Grafana and reduces gaps caused by Prometheus instance failovers.

Fixes: https://tracker.ceph.com/issues/70352?

Signed-off-by: Redouane Kachach <[email protected]>

Author: rkachach

src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2

@@ -52,6 +52,7 @@ http {
 
 {% if prometheus_endpoints %}
     upstream prometheus_servers {
+     ip_hash;
      {% for ep in prometheus_endpoints %}
      server {{ ep }};
      {% endfor %}

src/pybind/mgr/cephadm/tests/test_services.py

@@ -4004,6 +4004,7 @@ def get_services_endpoints(name):
                                              }
 
                                              upstream prometheus_servers {
+                                              ip_hash;
                                               server 192.168.100.100:9095;
                                               server 192.168.100.101:9095;
                                              }
@@ -4256,6 +4257,7 @@ def get_services_endpoints(name):
                                              }
 
                                              upstream prometheus_servers {
+                                              ip_hash;
                                               server 192.168.100.100:9095;
                                               server 192.168.100.101:9095;
                                              }