Skip to content

Commit 1e369b7

Browse files
bburt-rhbburt-rh
committed
RHDEVDOCS-4601-how-to-resize-prometheus-storage-volumes
1 parent c7acae6 commit 1e369b7

File tree

2 files changed

+207
-0
lines changed

2 files changed

+207
-0
lines changed

‎modules/monitoring-resizing-a-persistent-storage-volume.adoc

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/configuring-the-monitoring-stack.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="resizing-a-persistent-storage-volume_{context}"]
7+
= Resizing a persistent storage volume
8+
9+
{product-title} does not support resizing an existing persistent storage volume used by `StatefulSet` resources, even if the underlying `StorageClass` resource used supports persistent volume sizing.
10+
Therefore, even if you update the `storage` field for an existing persistent volume claim (PVC) with a larger size, this setting will not be propagated to the associated persistent volume (PV).
11+
12+
However, resizing a PV is still possible by using a manual process. If you want to resize a PV for a monitoring component such as Prometheus, Thanos Ruler, or Alertmanager, you can update the appropriate config map in which the component is configured. Then, patch the PVC, and delete and orphan the pods.
13+
Orphaning the pods recreates the `StatefulSet` resource immediately and automatically updates the size of the volumes mounted in the pods with the new PVC settings.
14+
No service disruption occurs during this process.
15+
16+
.Prerequisites
17+
18+
* You have installed the OpenShift CLI (`oc`).
19+
* *If you are configuring core {product-title} monitoring components*:
20+
** You have access to the cluster as a user with the `cluster-admin` role.
21+
** You have created the `cluster-monitoring-config` `ConfigMap` object.
22+
** You have configured at least one PVC for core {product-title} monitoring components.
23+
* *If you are configuring components that monitor user-defined projects*:
24+
** You have access to the cluster as a user with the `cluster-admin` role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
25+
** You have created the `user-workload-monitoring-config` `ConfigMap` object.
26+
** You have configured at least one PVC for components that monitor user-defined projects.
27+
28+
.Procedure
29+
30+
. Edit the `ConfigMap` object:
31+
** *To resize a PVC for a component that monitors core {product-title} projects*:
32+
.. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
33+
+
34+
[source,terminal]
35+
----
36+
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
37+
----
38+
39+
.. Add a new storage size for the PVC configuration for the component under `data/config.yaml`:
40+
+
41+
[source,yaml]
42+
----
43+
apiVersion: v1
44+
kind: ConfigMap
45+
metadata:
46+
name: cluster-monitoring-config
47+
namespace: openshift-monitoring
48+
data:
49+
config.yaml: |
50+
<component>: <1>
51+
volumeClaimTemplate:
52+
spec:
53+
storageClassName: <storage_class> <2>
54+
resources:
55+
requests:
56+
storage: <amount_of_storage> <3>
57+
----
58+
<1> Specify the core monitoring component.
59+
<2> Specify the storage class.
60+
<3> Specify the new size for the storage volume.
61+
+
62+
The following example configures a PVC that sets the local persistent storage to 100 gigabytes for the Prometheus instance that monitors core {product-title} components:
63+
+
64+
[source,yaml]
65+
----
66+
apiVersion: v1
67+
kind: ConfigMap
68+
metadata:
69+
name: cluster-monitoring-config
70+
namespace: openshift-monitoring
71+
data:
72+
config.yaml: |
73+
prometheusK8s:
74+
volumeClaimTemplate:
75+
spec:
76+
storageClassName: local-storage
77+
resources:
78+
requests:
79+
storage: 100Gi
80+
----
81+
+
82+
The following example configures a PVC that sets the local persistent storage for Alertmanager to 40 gigabytes:
83+
+
84+
[source,yaml]
85+
----
86+
apiVersion: v1
87+
kind: ConfigMap
88+
metadata:
89+
name: cluster-monitoring-config
90+
namespace: openshift-monitoring
91+
data:
92+
config.yaml: |
93+
alertmanagerMain:
94+
volumeClaimTemplate:
95+
spec:
96+
storageClassName: local-storage
97+
resources:
98+
requests:
99+
storage: 40Gi
100+
----
101+
102+
** *To resize a PVC for a component that monitors user-defined projects*:
103+
+
104+
[NOTE]
105+
====
106+
You can resize the volumes for the Thanos Ruler and Prometheus instances that monitor user-defined projects.
107+
====
108+
+
109+
.. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project:
110+
+
111+
[source,terminal]
112+
----
113+
$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
114+
----
115+
116+
.. Update the PVC configuration for the monitoring component under `data/config.yaml`:
117+
+
118+
[source,yaml]
119+
----
120+
apiVersion: v1
121+
kind: ConfigMap
122+
metadata:
123+
name: user-workload-monitoring-config
124+
namespace: openshift-user-workload-monitoring
125+
data:
126+
config.yaml: |
127+
<component>: <1>
128+
volumeClaimTemplate:
129+
spec:
130+
storageClassName: <storage_class> <2>
131+
resources:
132+
requests:
133+
storage: <amount_of_storage> <3>
134+
----
135+
<1> Specify the core monitoring component.
136+
<2> Specify the storage class.
137+
<3> Specify the new size for the storage volume.
138+
+
139+
The following example configures the PVC size to 100 gigabytes for the Prometheus instance that monitors user-defined projects:
140+
+
141+
[source,yaml]
142+
----
143+
apiVersion: v1
144+
kind: ConfigMap
145+
metadata:
146+
name: user-workload-monitoring-config
147+
namespace: openshift-user-workload-monitoring
148+
data:
149+
config.yaml: |
150+
prometheus:
151+
volumeClaimTemplate:
152+
spec:
153+
storageClassName: local-storage
154+
resources:
155+
requests:
156+
storage: 100Gi
157+
----
158+
+
159+
The following example sets the PVC size to 20 gigabytes for Thanos Ruler:
160+
+
161+
[source,yaml]
162+
----
163+
apiVersion: v1
164+
kind: ConfigMap
165+
metadata:
166+
name: user-workload-monitoring-config
167+
namespace: openshift-user-workload-monitoring
168+
data:
169+
config.yaml: |
170+
thanosRuler:
171+
volumeClaimTemplate:
172+
spec:
173+
storageClassName: local-storage
174+
resources:
175+
requests:
176+
storage: 20Gi
177+
----
178+
+
179+
[NOTE]
180+
====
181+
Storage requirements for the `thanosRuler` component depend on the number of rules that are evaluated and how many samples each rule generates.
182+
====
183+
184+
. Save the file to apply the changes. The pods affected by the new configuration restart automatically.
185+
+
186+
[WARNING]
187+
====
188+
When you save changes to a monitoring config map, the pods and other resources in the related project might be redeployed. The monitoring processes running in that project might also be restarted.
189+
====
190+
191+
. Manually patch every PVC with the updated storage request. The following example resizes the storage size for the Prometheus component in the `openshift-monitoring` namespace to 100Gi:
192+
+
193+
[source,terminal]
194+
----
195+
$ for p in $(oc -n openshift-monitoring get pvc -l app.kubernetes.io/name=prometheus -o jsonpath='{range .items[*]}{.metadata.name} {end}'); do \
196+
oc -n openshift-monitoring patch pvc/${p} --patch '{"spec": {"resources": {"requests": {"storage":"100Gi"}}}}'; \
197+
done
198+
199+
----
200+
201+
. Delete the underlying StatefulSet with the `--cascade=orphan` parameter:
202+
+
203+
[source,terminal]
204+
----
205+
$ oc delete statefulset -l app.kubernetes.io/name=prometheus --cascade=orphan
206+
----

‎monitoring/configuring-the-monitoring-stack.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ include::modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.ado
7979
// Configuring persistent storage
8080
include::modules/monitoring-configuring-persistent-storage.adoc[leveloffset=+1]
8181
include::modules/monitoring-configuring-a-local-persistent-volume-claim.adoc[leveloffset=+2]
82+
include::modules/monitoring-resizing-a-persistent-storage-volume.adoc[leveloffset=+2]
8283
include::modules/monitoring-modifying-retention-time-and-size-for-prometheus-metrics-data.adoc[leveloffset=+2]
8384
include::modules/monitoring-modifying-the-retention-time-for-thanos-ruler-metrics-data.adoc[leveloffset=+2]
8485

0 commit comments

Comments
 (0)