You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cockpit does not support Grafana-managed alerting. It integrates with Grafana to visualize metrics, but alerts are managed through the Scaleway alert manager. You should use Grafana only to define alert rules, not to evaluate or receive alert notifications. Once the conditions of your alert rule are met, the Scaleway alert manager evaluates the rule and sends a notification to the contact points you have configured in the Scaleway console or in Grafana.
20
+
Cockpit does not support the Grafana alert manager nor Grafana-managed alert rules. You should only use the Grafana interface to define alert rules, not to evaluate or receive alert notifications. Alerting rules are evaluated at the data source level and notifications are handled and sent out by the Scaleway alert manager only.
21
+
22
+
Once the conditions of your alert rule are met, the rule evaluation engine of your data source forwards the firing alert to the Scaleway alert manager, which then sends a notification to the contacts you have configured in the Scaleway console or in Grafana.
23
+
24
+
You can also create alerting rules on your custom data sources.
21
25
22
26
This page shows you how to create alert rules in Grafana for monitoring Scaleway resources integrated with Cockpit, such as Instances, Object Storage, and Kubernetes. These alerts rely on Scaleway-provided metrics, which are preconfigured and available in the **Metrics browser** drop-down when using the **Scaleway Metrics data source** in the Grafana interface. This page explains how to use the `Scaleway Metrics` data source, interpret metrics, set alert conditions, and activate alerts.
23
27
@@ -27,12 +31,12 @@ This page shows you how to create alert rules in Grafana for monitoring Scaleway
27
31
-[Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
28
32
- Scaleway resources you can monitor
29
33
-[Created Grafana credentials](/cockpit/how-to/retrieve-grafana-credentials/) with the **Editor** role
30
-
-[Enabled](/cockpit/how-to/enable-alert-manager/) the Scaleway alert manager
31
-
-[Created](/cockpit/how-to/add-contact-points/)a contact point in the Scaleway console or in Grafana (with the `Scaleway Alerting` alert manager of the same region as your `Scaleway Metrics` data source), otherwise alerts will not be delivered
34
+
-[Enabled](/cockpit/how-to/enable-alert-manager/) the Scaleway alert manager in the same region as the resources you want to be alerted for
35
+
-[Added](/cockpit/how-to/add-contact-points/)contacts in the Scaleway console or contact points in Grafana (with the `Scaleway Alerting` alert manager of the same region as your `Scaleway Metrics` data source), otherwise alerts will not be delivered
32
36
33
37
## Switch to the data source-managed tab
34
38
35
-
Data source managed alert rules allow you to configure alerts managed by the data source of your choice, instead of using Grafana's managed alerting system which is not supported by Cockpit.
39
+
Data source managed alert rules allow you to configure alerts managed by the data source of your choice, instead of using Grafana's managed alerting system **which is not supported by Cockpit**.
36
40
37
41
1.[Log in to Grafana](/cockpit/how-to/access-grafana-and-managed-dashboards/) using your credentials.
38
42
2. Click the Grafana icon in the top left side of your screen to open the menu.
@@ -45,7 +49,7 @@ Data source managed alert rules allow you to configure alerts managed by the dat
45
49
8. In the **Rule type** subsection, click the **Data source-managed** tab.
46
50
<Lightboximage={DataSourceManaged}alt="" />
47
51
<Messagetype="important">
48
-
Data source managed alert rules allow you to configure alerts managed by the data source of your choice, instead of using Grafana's managed alerting system which is not supported by Cockpit.
52
+
Data source managed alert rules allow you to configure alerts managed by the data source of your choice, instead of using Grafana's managed alerting system **which is not supported by Cockpit**.
49
53
This step is **mandatory** because Cockpit does not support Grafana’s built-in alerting system, but only alerts configured and evaluated by the data source itself.
50
54
</Message>
51
55
@@ -64,16 +68,19 @@ Switch between the tabs below to create alerts for a Scaleway Instance, an Objec
64
68
<Messagetype="tip">
65
69
The `instance_server_cpu_seconds_total` metric records how many seconds of CPU time your Instance has used in total. It is helpful to detect unexpected CPU usage spikes.
66
70
</Message>
67
-
2. In the **Set alert evaluation behavior** section, specify how long the condition must be true before triggering the alert.
68
-
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert, and optionally, add annotations.
71
+
2. In the **Set alert evaluation behavior** section, specify how long the condition must be met before triggering the alert.
72
+
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert rules. Rules that share the same group will use the same configuration, including the evaluation interval which determines how often the rule is evaluated (by default: every 1 minute). You can modify this interval later in the group settings.
73
+
<Messagetype="note">
74
+
The evaluation interval is different from the pending period set in step 2. The evaluation interval controls how often the rule is checked, while the pending period defines how long the condition must be continuously met before the alert fires.
75
+
</Message>
69
76
4. In the **Configure labels and notifications** section, click **+ Add labels**. A pop-up appears.
70
-
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contact points you may already have created in the Scaleway console.
77
+
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contacts you may already have created in the Scaleway console.
71
78
<Messagetype="note">
72
79
In Grafana, notifications are sent by matching alerts to notification policies based on labels. This step is about deciding how alerts will reach you or your team (Slack, email, etc.) based on labels you attach to them. Then, you can set up rules that define who receives notifications in the **Notification policies** page.
73
-
For example, if an alert has the label `team = instances-team`, you are telling Grafana to send a notification to the Instances team when your alert`alert-for-high-cpu-usage` gets triggered. Find out how to [configure notification policies in Grafana](/tutorials/configure-slack-alerting/#configuring-a-notification-policy).
80
+
For example, if your alert named `alert-for-high-cpu-usage`has the label `team = instances-team`, you are telling Grafana to send a notification to the Instances team when the alert gets triggered. Find out how to [configure notification policies in Grafana](/tutorials/configure-slack-alerting/#configuring-a-notification-policy).
74
81
</Message>
75
82
6. Click **Save rule and exit** in the top right corner of your screen to save and activate your alert.
76
-
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contact point](/cockpit/concepts/#contact-points).
83
+
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contacts](/cockpit/concepts/#contact-points).
77
84
</TabsTab>
78
85
<TabsTablabel="Object Storage bucket">
79
86
The steps below explain how to create the metric selection and configure an alert condition that triggers when **the object count in your bucket exceeds a specific threshold**.
@@ -85,16 +92,19 @@ Switch between the tabs below to create alerts for a Scaleway Instance, an Objec
85
92
<Messagetype="tip">
86
93
The `object_storage_bucket_objects_total` metric indicates the total number of objects stored in a given Object Storage bucket. It is useful to monitor and control object growth in your bucket and avoid hitting thresholds.
87
94
</Message>
88
-
2. In the **Set alert evaluation behavior** section, specify how long the condition must be true before triggering the alert.
89
-
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert, and optionally, add annotations.
95
+
2. In the **Set alert evaluation behavior** section, specify how long the condition must be met before triggering the alert.
96
+
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert rules. Rules that share the same group will use the same configuration, including the evaluation interval which determines how often the rule is evaluated (by default: every 1 minute). You can modify this interval later in the group settings.
97
+
<Messagetype="note">
98
+
The evaluation interval is different from the pending period set in step 2. The evaluation interval controls how often the rule is checked, while the pending period defines how long the condition must be continuously met before the alert fires.
99
+
</Message>
90
100
4. In the **Configure labels and notifications** section, click **+ Add labels**. A pop-up appears.
91
-
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contact points you may already have created in the Scaleway console.
101
+
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contacts you may already have created in the Scaleway console.
92
102
<Messagetype="note">
93
103
In Grafana, notifications are sent by matching alerts to notification policies based on labels. This step is about deciding how alerts will reach you or your team (Slack, email, etc.) based on labels you attach to them. Then, you can set up rules that define who receives notifications in the **Notification policies** page.
94
104
For example, if an alert has the label `team = object-storage-team`, you are telling Grafana to send a notification to the Object Storage team when your alert is firing. Find out how to [configure notification policies in Grafana](/tutorials/configure-slack-alerting/#configuring-a-notification-policy).
95
105
</Message>
96
106
6. Click **Save rule and exit** in the top right corner of your screen to save and activate your alert.
97
-
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contact point](/cockpit/concepts/#contact-points).
107
+
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contacts](/cockpit/concepts/#contact-points).
98
108
</TabsTab>
99
109
<TabsTablabel="Kubernetes pod">
100
110
The steps below explain how to create the metric selection and configure an alert condition that triggers when **no new pod activity occurs, which could mean your cluster is stuck or unresponsive.**
@@ -107,15 +117,18 @@ Switch between the tabs below to create alerts for a Scaleway Instance, an Objec
107
117
The `kubernetes_cluster_k8s_shoot_nodes_pods_usage_total` metric represents the total number of pods currently running across all nodes in your Kubernetes cluster. It is helpful to monitor current pod consumption per node pool or cluster, and help track resource saturation or unexpected workload spikes.
108
118
</Message>
109
119
2. In the **Set alert evaluation behavior** field, specify how long the condition must be true before triggering the alert.
110
-
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert, and optionally, add annotations.
120
+
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert rules. Rules that share the same group will use the same configuration, including the evaluation interval which determines how often the rule is evaluated (by default: every 1 minute). You can modify this interval later in the group settings.
121
+
<Messagetype="note">
122
+
The evaluation interval is different from the pending period set in step 2. The evaluation interval controls how often the rule is checked, while the pending period defines how long the condition must be continuously met before the alert fires.
123
+
</Message>
111
124
4. In the **Configure labels and notifications** section, click **+ Add labels**. A pop-up appears.
112
-
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contact points you may already have created in the Scaleway console.
125
+
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contacts you may already have created in the Scaleway console.
113
126
<Messagetype="note">
114
127
In Grafana, notifications are sent by matching alerts to notification policies based on labels. This step is about deciding how alerts will reach you or your team (Slack, email, etc.) based on labels you attach to them. Then, you can set up rules that define who receives notifications in the **Notification policies** page.
115
128
For example, if an alert has the label `team = kubernetes-team`, you are telling Grafana to send a notification to the Kubernetes team when your alert is firing. Find out how to [configure notification policies in Grafana](/tutorials/configure-slack-alerting/#configuring-a-notification-policy).
116
129
</Message>
117
130
6. Click **Save rule and exit** in the top right corner of your screen to save and activate your alert.
118
-
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contact point](/cockpit/concepts/#contact-points).
131
+
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contacts](/cockpit/concepts/#contact-points).
119
132
</TabsTab>
120
133
<TabsTablabel="Cockpit logs">
121
134
The steps below explain how to create the metric selection and configure an alert condition that triggers when **no logs are stored for 5 minutes, which may indicate your app or system is broken**.
@@ -128,22 +141,25 @@ Switch between the tabs below to create alerts for a Scaleway Instance, an Objec
128
141
The `observability_cockpit_loki_chunk_store_stored_chunks_total:increase5m` metric represents the number of chunks (log storage blocks) that have been written over the last 5 minutes for a specific resource. It is useful to monitor log ingestion activity and detect issues such as a crash of the logging agent, or your application not producing logs.
129
142
</Message>
130
143
2. In the **Set alert evaluation behavior** field, specify how long the condition must be true before triggering the alert.
131
-
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert, and optionally, add annotations.
144
+
3. Enter a name in the **Namespace** and **Group** fields to categorize and manage your alert rules. Rules that share the same group will use the same configuration, including the evaluation interval which determines how often the rule is evaluated (by default: every 1 minute). You can modify this interval later in the group settings.
145
+
<Messagetype="note">
146
+
The evaluation interval is different from the pending period set in step 2. The evaluation interval controls how often the rule is checked, while the pending period defines how long the condition must be continuously met before the alert fires.
147
+
</Message>
132
148
4. In the **Configure labels and notifications** section, click **+ Add labels**. A pop-up appears.
133
-
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contact points you may already have created in the Scaleway console.
149
+
5. Enter a label and value name and click **Save**. You can skip this step if you want your alerts to be sent to the contacts you may already have created in the Scaleway console.
134
150
<Messagetype="note">
135
151
In Grafana, notifications are sent by matching alerts to notification policies based on labels. This step is about deciding how alerts will reach you or your team (Slack, email, etc.) based on labels you attach to them. Then, you can set up rules that define who receives notifications in the **Notification policies** page.
136
152
For example, if an alert has the label `team = cockpit-team`, you are telling Grafana to send a notification to the Cockpit team when your alert is firing. Find out how to [configure notification policies in Grafana](/tutorials/configure-slack-alerting/#configuring-a-notification-policy).
137
153
</Message>
138
154
6. Click **Save rule and exit** in the top right corner of your screen to save and activate your alert.
139
-
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contact point](/cockpit/concepts/#contact-points).
155
+
7. Optionally, check that your configuration works by temporarily lowering the threshold. This will trigger the alert and notify your [contacts](/cockpit/concepts/#contact-points).
140
156
</TabsTab>
141
157
</Tabs>
142
158
143
159
**You can configure up to a maximum of 10 alerts** for the `Scaleway Metrics` data source.
144
160
145
161
<Messagetype="tip">
146
-
You can also build your alerts from the Grafana-managed tab, with the metrics explorer (book icon) and **+ Operations** button. However, remember to click the **Data source-managed** tab once you are done building them to receive alert notifications.
162
+
You can also build your PromQL expressions and find the right metrics from the Grafana-managed tab, with the metrics explorer (book icon) and **+ Operations** button. However, remember to click the **Data source-managed** tab once you are done building them to proceed with the final configuration steps.
0 commit comments