Skip to content

Commit 88ed1b6

Browse files
weltekialexellis
welteki
authored and
alexellis
committed
Add tutorial for long term metrics retention
Walk through different options to export OpenFaaS Prometheus metrics for long long term retention. Signed-off-by: Han Verstraete (OpenFaaS Ltd) <[email protected]>
1 parent 56b73a3 commit 88ed1b6

File tree

2 files changed

+212
-0
lines changed

2 files changed

+212
-0
lines changed

‎docs/tutorials/long-term-metrics-retention.md

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# Long term OpenFaaS metrics retention
2+
3+
OpenFaaS exposes metrics in Prometheus format from its different components. These metrics are collected by a built-in Prometheus server which is deployed via the OpenFaaS Helm chart.
4+
5+
The built-in Prometheus instance is designed to be a "black box", an internal component of OpenFaaS, not to be used for long term storage or custom settings. By default there is no persistence for the Prometheus metrics and the metrics retention period is only 15 days.
6+
7+
But what if you would like to enable long-term retention of Prometheus metrics?
8+
9+
In this tutorial we will show you how metrics can be exported using [Grafana Alloy](https://grafana.com/docs/alloy/latest/) or [Prometheus Federation](https://prometheus.io/docs/prometheus/latest/federation/) for long term retention.
10+
11+
## Configure retention for OpenFaaS built-in Prometheus
12+
13+
You can configure an optional PVC for Prometheus to persist data for the OpenFaaS Prometheus deployment.
14+
15+
Without the PVC, the data held by Prometheus is erased if:
16+
17+
- The node where Prometheus is running gets reclaimed or recycled
18+
- The Helm chart is upgraded (something we should be doing often for fixes and features)
19+
- You update the configuration for Prometheus for custom autoscaling rules etc
20+
21+
You can configure this feature through the [OpenFaaS Helm chart](https://github.com/openfaas/faas-netes/tree/master/chart/openfaas). By default it is turned off, but we recommend turning it on.
22+
23+
```yaml
24+
prometheus:
25+
# Set to true to enable persistent storage for the Prometheus Pod
26+
# otherwise, the data will be lost when the Pod is restarted
27+
pvc:
28+
enabled: false
29+
# You may want to set this higher for production, or lower for development/staging.
30+
size: 30Gi
31+
# Leave the storageClassName blank for the default storage class
32+
# using the string "default" does not necessarily mean the default
33+
# storage class
34+
storageClassName:
35+
```
36+
37+
The retention period can also be changed;:
38+
39+
```yaml
40+
prometheus:
41+
retention:
42+
time: 15d
43+
```
44+
45+
## Prometheus metrics federation
46+
47+
Federation allows Prometheus to scrape selected time series from another Prometheus server. You can configure your own Prometheus deployment to scrape metrics from the OpenFaaS Prometheus server.
48+
49+
Add the following job to your Prometheus scrape configuration:
50+
51+
```yaml
52+
scrape_configs:
53+
- job_name: 'openfaas'
54+
scrape_interval: 60s
55+
56+
honor_labels: true
57+
metrics_path: "/federate"
58+
59+
params:
60+
'match[]':
61+
- '{job=~\".*\"}'
62+
- '{__name__=~\"^job:.*\"}'
63+
64+
static_configs:
65+
- targets:
66+
- 'prometheus.openfaas.svc.cluster.local:9090'
67+
```
68+
69+
This config assumes your Prometheus deployment is running in the same cluster as OpenFaaS. If this is not the case we recommend using [Grafana Alloy](https://grafana.com/docs/alloy/latest/) to collect metrics and forward them to a Prometheus-compatible database.
70+
71+
## Collect Prometheus metrics with Grafana Alloy
72+
73+
We will be deploying and configuring Grafana Alloy through Helm and configure it to collect metrics from the OpenFaaS Prometheus server. Those metrics are then forwarded to a remote Prometheus server using the [Prometheus Remote Write protocol](https://prometheus.io/docs/specs/remote_write_spec/).
74+
75+
We will show you how to configure different remote write targets like [Grafana Cloud](https://grafana.com/products/cloud/) and [Amazon Managed Service for Prometheus](https://aws.amazon.com/prometheus/).
76+
77+
### Deploy Grafana Alloy with Helm
78+
79+
Create a `values.yaml` file to deploy and configure Grafana Alloy for remote write to an external Prometheus deployment with the Grafana Alloy Helm chart.
80+
81+
We configure Alloy directly using the Helm configuration but you can create a separate ConfigMap from a file if that is preferred. See [Configure Grafana Alloy on Kubernetes](https://grafana.com/docs/alloy/latest/configure/kubernetes/) for more info.
82+
83+
```sh
84+
export PROM_USERNAME=""
85+
export PROM_PASSWORD="./prom-password.txt"
86+
export REMOTE_WRITE_ENDPOINT=""
87+
88+
cat <<EOF > values.yaml
89+
alloy:
90+
configMap:
91+
content: |-
92+
// Collect metrics from the OpenFaaS Prometheus instance and forward them
93+
// to the prometheus remote write component for long term retention.
94+
prometheus.scrape "openfaas" {
95+
targets = [{
96+
__address__ = "prometheus.openfaas.svc.cluster.local:9090",
97+
}]
98+
honor_labels = true
99+
metrics_path = "/federate"
100+
101+
params = {
102+
"match[]" = ["{job=~\".*\"}", "{__name__=~\"^job:.*\"}"],
103+
}
104+
105+
scrape_interval = "60s"
106+
107+
forward_to = [prometheus.remote_write.metrics_service.receiver]
108+
}
109+
110+
prometheus.remote_write "metrics_service" {
111+
endpoint {
112+
url = ${REMOTE_WRITE_ENDPOINT}
113+
114+
basic_auth {
115+
username = ${PROM_USERNAME}
116+
password = $(cat $PROM_PASSWORD)
117+
}
118+
}
119+
}
120+
EOF
121+
```
122+
123+
You can leave out the `basic_auth` section if your Prometheus deployment has no authentication enabled.
124+
125+
!!! Note
126+
Check out the Alloy [values.yaml](https://raw.githubusercontent.com/grafana/alloy/main/operations/helm/charts/alloy/values.yaml) for more configuration options.
127+
128+
The target Prometheus server should be running with the flag `--web.enable-remote-write-receiver=true` to accept remote write requests.
129+
130+
Deploy Grafana Alloy in the `openfaas` namespace:
131+
132+
```sh
133+
helm upgrade grafana-alloy \
134+
--install grafana/alloy \
135+
--namespace openfaas \
136+
-f ./values.yaml
137+
```
138+
139+
Optionally verify the configuration through the Alloy UI.
140+
141+
```sh
142+
kubectl port-forward -n openfaas svc/grafana-alloy 12345:12345
143+
```
144+
145+
### Grafana Cloud
146+
147+
The configuration for pushing metrics to Grafana Cloud is exactly the same as for a self-hosted Prometheus instance. You will need to get the remote write endpoint, username (instance ID) and password (API access token) from your Grafana Cloud organization.
148+
149+
### Amazon Managed Service for Prometheus (from EKS cluster or EC2)
150+
151+
It is possible to ingest OpenFaaS Prometheus metrics in [Amazon Managed Service for Prometheus](https://aws.amazon.com/prometheus/) for clusters running on Amazon EKS or self-managed cluster running on Amazon EC2.
152+
153+
1. Create a new Amazon Managed Service for Prometheus workspace in your desired region.
154+
2. [Set up IAM roles for service accounts](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-existing-Prometheus.html#AMP-onboard-existing-Prometheus-IRSA)
155+
156+
Make sure you have created an IAM role `amp-iamproxy-ingest-role` before continuing.
157+
158+
This IAM role must be associated with the Kubernetes service account used by Grafana Alloy to give it the appropriate permissions to push metrics to the AWS Prometheus service.
159+
160+
Append the parameters for the service account to your Helm configuration file for Grafana Alloy:
161+
162+
- Replace `${IAM_PROXY_PROMETHEUS_ROLE_ARN}` with the ARN of the `amp-iamproxy-ingest-role` that you created.
163+
164+
```yaml
165+
serviceAccount:
166+
annotations:
167+
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
168+
```
169+
170+
Add a new remote write target to the Alloy config:
171+
172+
- Replace `${WORKSPACE_ID}` with your Amazon Managed Service for Prometheus workspace ID.
173+
- Replace `${REGION}` with the Region of the Amazon Managed Service for Prometheus workspace (such as us-west-2).
174+
175+
```
176+
prometheus.remote_write "aws" {
177+
endpoint {
178+
url = "https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write"
179+
}
180+
181+
// Configure AWS Signature Verification 4 for authenticating to the endpoint.
182+
sigv4 {
183+
region = "${AWS_REGION}"
184+
}
185+
186+
// Configuration for how metrics are batched before sending.
187+
queue_config {
188+
max_samples_per_send = 1000
189+
max_shards = 200
190+
capacity = 2500
191+
}
192+
}
193+
```
194+
195+
Change the `prometheus.scrape "openfaas"` component to forward metrics to the correct target:
196+
197+
```diff
198+
prometheus.scrape "openfaas" {
199+
- forward_to = [prometheus.remote_write.metrics_service.receiver]
200+
+ forward_to = [prometheus.remote_write.aws.receiver]
201+
}
202+
```
203+
204+
Create or update the Grafana Alloy deployment:
205+
206+
```sh
207+
helm upgrade grafana-alloy \
208+
--install grafana/alloy \
209+
--namespace openfaas \
210+
-f ./values.yaml
211+
```

‎mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,7 @@ nav:
209209
- Local Ingress with KinD: ./tutorials/local-kind-ingress.md
210210
- Local Registry with KinD: ./tutorials/local-kind-registry.md
211211
- Featured: ./tutorials/featured.md
212+
- Metrics retention: ./tutorials/long-term-metrics-retention.md
212213
- Contributing:
213214
- Get Started: ./contributing/get-started.md
214215
- Code Repositories: ./contributing/code-repositories.md

0 commit comments

Comments
 (0)