From ce30ef9517a6eca63f6d9e4724b0ecc60c8096aa Mon Sep 17 00:00:00 2001 From: Pavan Gudiwada Date: Wed, 12 Feb 2025 14:41:29 +0530 Subject: [PATCH 1/4] Adding cluster label docs --- docs/setup-robusta/additional-settings.rst | 25 ++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/setup-robusta/additional-settings.rst b/docs/setup-robusta/additional-settings.rst index 3fdd531eb..dc3db0f10 100644 --- a/docs/setup-robusta/additional-settings.rst +++ b/docs/setup-robusta/additional-settings.rst @@ -1,5 +1,3 @@ -:hide-toc: - Additional Settings ======================= @@ -221,3 +219,26 @@ Add this to Robusta's Helm values: memory: 2048Mi limits: memory: 2048Mi + +Adding a Cluster Label to Alerts +--------------------------------------------- +Robusta uses the ``cluster_name`` you set during installation to identify which cluster an alert belongs to when forwarding alerts via the Robusta API webhook. This is useful when forwarding alerts to Robusta from external services like Grafana/Grafana cloud. + +You can add this label to all of your metrics using ``prometheus.prometheusSpec.additionalScrapeConfigs`` as mentioned below. Add the following config to your Helm values file. + +.. code-block:: yaml + + prometheus: + prometheusSpec: + additionalScrapeConfigs: + - job_name: "cluster-name-to-metric" + kubernetes_sd_configs: + - role: pod + metric_relabel_configs: + - target_label: cluster_name + replacement: "YOUR_ROBUSTA_CLUSTER_NAME" + +.. note:: + + 1. ``prometheus.prometheusSpec.externalLabels.cluster`` does not work for cases when you need ``cluster_name`` label in Grafana. + 2. ``cluster_name`` label will be added only to metrics after you add this config. i.e Previously scraped metrics will not have ``cluster_name`` label. \ No newline at end of file From e74cdb0e52c3af6b216b396e96d28d8509684fa5 Mon Sep 17 00:00:00 2001 From: Pavan Gudiwada Date: Wed, 12 Feb 2025 15:46:52 +0530 Subject: [PATCH 2/4] Updated based on comments --- docs/setup-robusta/additional-settings.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/setup-robusta/additional-settings.rst b/docs/setup-robusta/additional-settings.rst index dc3db0f10..db6ad892e 100644 --- a/docs/setup-robusta/additional-settings.rst +++ b/docs/setup-robusta/additional-settings.rst @@ -222,7 +222,9 @@ Add this to Robusta's Helm values: Adding a Cluster Label to Alerts --------------------------------------------- -Robusta uses the ``cluster_name`` you set during installation to identify which cluster an alert belongs to when forwarding alerts via the Robusta API webhook. This is useful when forwarding alerts to Robusta from external services like Grafana/Grafana cloud. +When using Robusta's built-in Prometheus stack, Robusta uses the ``cluster_name`` you set during installation to identify which alerts belong to which cluster. + +If you forward external alerts to Robusta (e.g., from Grafana/Grafana Cloud), you will need to pass the ``cluster_name`` metadata manually. For example, if you use Grafana alerting, ensure that all your metrics and alerts have a ``cluster_name`` label. You can add this label to all of your metrics using ``prometheus.prometheusSpec.additionalScrapeConfigs`` as mentioned below. Add the following config to your Helm values file. @@ -236,9 +238,10 @@ You can add this label to all of your metrics using ``prometheus.prometheusSpec. - role: pod metric_relabel_configs: - target_label: cluster_name - replacement: "YOUR_ROBUSTA_CLUSTER_NAME" + replacement: "YOUR_ROBUSTA_CLUSTER_NAME" # This is the cluster name you set in the Helm values during Robusta installation .. note:: - 1. ``prometheus.prometheusSpec.externalLabels.cluster`` does not work for cases when you need ``cluster_name`` label in Grafana. - 2. ``cluster_name`` label will be added only to metrics after you add this config. i.e Previously scraped metrics will not have ``cluster_name`` label. \ No newline at end of file + 1. ``cluster_name`` label will be added only to metrics after you add this config. i.e Previously scraped metrics will not have ``cluster_name`` label. **You will need to wait a few hours after adding this configuration for the label to show up on your alerts and be forwarded correctly.** + 2. ``prometheus.prometheusSpec.externalLabels.cluster`` does not work for cases when you need ``cluster_name`` label in Grafana. + \ No newline at end of file From 680bc67afa89b5b2bc6db0a77647e15343d24219 Mon Sep 17 00:00:00 2001 From: Pavan Gudiwada Date: Wed, 12 Feb 2025 16:00:50 +0530 Subject: [PATCH 3/4] Added reference from Grafana docs and fixed some text --- .../alertmanager-integration/grafana-alert-manager.rst | 2 +- docs/setup-robusta/additional-settings.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configuration/alertmanager-integration/grafana-alert-manager.rst b/docs/configuration/alertmanager-integration/grafana-alert-manager.rst index f7324d60c..7c4c42ec9 100644 --- a/docs/configuration/alertmanager-integration/grafana-alert-manager.rst +++ b/docs/configuration/alertmanager-integration/grafana-alert-manager.rst @@ -14,7 +14,7 @@ After completing this tutorial, we recommend configuring a metrics integration a Prerequisite ================================= -* A label in the following format ``cluster_name: YourClusterName`` added to each alert, with the cluster name as it appears in Robusta ``generated_values.yaml``. +* A label in the following format ``cluster_name: YourClusterName`` added to each alert, with the cluster name as it appears in Robusta ``generated_values.yaml``. Follow the instructions :ref:`here `. Send Alerts to Robusta ============================ diff --git a/docs/setup-robusta/additional-settings.rst b/docs/setup-robusta/additional-settings.rst index db6ad892e..d73a8b00b 100644 --- a/docs/setup-robusta/additional-settings.rst +++ b/docs/setup-robusta/additional-settings.rst @@ -222,7 +222,7 @@ Add this to Robusta's Helm values: Adding a Cluster Label to Alerts --------------------------------------------- -When using Robusta's built-in Prometheus stack, Robusta uses the ``cluster_name`` you set during installation to identify which alerts belong to which cluster. +When using Kube-Prometheus-Stack, Robusta uses the ``cluster_name`` you set during installation to identify which alerts belong to which cluster. If you forward external alerts to Robusta (e.g., from Grafana/Grafana Cloud), you will need to pass the ``cluster_name`` metadata manually. For example, if you use Grafana alerting, ensure that all your metrics and alerts have a ``cluster_name`` label. From fc01230d8199d6bc026c5caf417685e499945c36 Mon Sep 17 00:00:00 2001 From: Robusta Runner Date: Wed, 12 Feb 2025 13:28:03 +0200 Subject: [PATCH 4/4] minor tweaks --- docs/setup-robusta/additional-settings.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/setup-robusta/additional-settings.rst b/docs/setup-robusta/additional-settings.rst index d73a8b00b..ca1273864 100644 --- a/docs/setup-robusta/additional-settings.rst +++ b/docs/setup-robusta/additional-settings.rst @@ -222,11 +222,11 @@ Add this to Robusta's Helm values: Adding a Cluster Label to Alerts --------------------------------------------- -When using Kube-Prometheus-Stack, Robusta uses the ``cluster_name`` you set during installation to identify which alerts belong to which cluster. +When using the built-in Prometheus stack, Robusta uses the ``cluster_name`` you set during installation to identify which alerts belong to which cluster. If you forward external alerts to Robusta (e.g., from Grafana/Grafana Cloud), you will need to pass the ``cluster_name`` metadata manually. For example, if you use Grafana alerting, ensure that all your metrics and alerts have a ``cluster_name`` label. -You can add this label to all of your metrics using ``prometheus.prometheusSpec.additionalScrapeConfigs`` as mentioned below. Add the following config to your Helm values file. +The easiest way to do this with Prometheus is by setting `metric_relabel_configs `_. How you apply that configuration depends on how Prometheus was installed. If you use kube-prometheus-stack, there is a Helm value for it: .. code-block:: yaml