Skip to content

Commit f3081b9

Browse files
christian-stephenchristian-stephen
authored
Add TempoMonolithic deployment (#57)
1 parent 3ec1fb4 commit f3081b9

File tree

9 files changed

+280
-8
lines changed

9 files changed

+280
-8
lines changed

‎.circleci/config.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,14 @@ jobs:
206206
command: |
207207
helm install server-monitoring-stack ./target/server-monitoring-stack*.tgz \
208208
-n server-monitoring --set global.enabled=false --set prometheusOperator.installCRDs=true
209+
- run:
210+
name: Install Grafana Tempo operator
211+
command: |
212+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.18.0/cert-manager.yaml
213+
kubectl wait --for=condition=available --timeout=30s deployment/cert-manager-webhook -n cert-manager
214+
215+
kubectl apply -f https://github.com/grafana/tempo-operator/releases/download/v0.16.0/tempo-operator.yaml
216+
kubectl wait --for=condition=available --timeout=30s deployment/tempo-operator-controller -n tempo-operator-system
209217
- run:
210218
name: Install Helm chart
211219
command: helm upgrade --install server-monitoring-stack . --reset-values --timeout=2m -n server-monitoring

‎README.md‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,21 @@ $ helm repo update
5151

5252
### 3. Install Dependencies
5353

54-
Before installing the full chart, you must first install the dependency subcharts, including the Prometheus Custom Resource Definitions (CRDs) and the Grafana operator chart. This assumes you are installing it in the same namespace as your CircleCI server installation:
54+
Before installing the full chart, you must first install the dependency subcharts and operators.
55+
56+
#### 3.1 Install Prometheus CRDs and Grafana Operator
57+
58+
Install the Prometheus Custom Resource Definitions (CRDs) and the Grafana operator chart. This assumes you are installing it in the same namespace as your CircleCI server installation:
5559

5660
```bash
5761
$ helm install server-monitoring-stack server-monitoring-stack/server-monitoring-stack --set global.enabled=false --set prometheusOperator.installCRDs=true --version 0.1.0-alpha.4 -n <your-server-namespace>
5862
```
59-
> **_NOTE:_** It's possible to install the monitoring stack in a different namespace than the CircleCI server installation. If you do so, set the `prometheus.serviceMonitor.selectorNamespaces` value with the target namespace.
63+
64+
> **_NOTE:_** It's possible to install the monitoring stack in a different namespace than the CircleCI server installation. If you do so, set the `prometheus.serviceMonitor.selectorNamespaces` value with the target namespace.
65+
66+
#### 3.2 Install Tempo Operator (Optional)
67+
68+
If you plan to enable distributed tracing with Tempo (`tempo.enabled=true`), you must manually install the Tempo Operator by following the official documentation at https://grafana.com/docs/tempo/latest/setup/operator/#installation[]. There is currently no official Helm chart available for the Tempo Operator or its CRDs.
6069

6170
### 4. Install the Helm Chart
6271

@@ -211,6 +220,16 @@ Dashboards are provisioned directly from CRDs, which means any manual edits will
211220
| prometheusOperator.prometheusConfigReloader.image.repository | string | `"quay.io/prometheus-operator/prometheus-config-reloader"` | Image repository for Prometheus Config Reloader. |
212221
| prometheusOperator.prometheusConfigReloader.image.tag | string | `"v0.81.0"` | Tag for the Prometheus Config Reloader image. |
213222
| prometheusOperator.replicas | int | `1` | Number of Prometheus Operator replicas to deploy. |
223+
| tempo.enabled | string | `"-"` | Enable Tempo distributed tracing Requires manual installation of Tempo Operator Set to true to enable, false to disable, "-" to use global default |
224+
| tempo.resources | object | `{"limits":{"cpu":"1000m","memory":"2Gi"},"requests":{"cpu":"500m","memory":"1Gi"}}` | Resource requirements for Tempo pods Adjust based on your trace volume and cluster capacity |
225+
| tempo.resources.limits.cpu | string | `"1000m"` | Maximum CPU Tempo pods can use |
226+
| tempo.resources.limits.memory | string | `"2Gi"` | Maximum memory Tempo pods can use |
227+
| tempo.resources.requests.cpu | string | `"500m"` | Minimum CPU guaranteed to Tempo pods |
228+
| tempo.resources.requests.memory | string | `"1Gi"` | Minimum memory guaranteed to Tempo pods |
229+
| tempo.retentionPeriod | string | `"24h"` | Trace retention period How long to keep trace data before deletion |
230+
| tempo.storage | object | `{"traces":{"backend":"memory","size":"20Gi"}}` | Storage configuration for trace data |
231+
| tempo.storage.traces.backend | string | `"memory"` | Storage backend for traces Default: in-memory storage (traces lost on pod restart) Suitable for development/testing environments only |
232+
| tempo.storage.traces.size | string | `"20Gi"` | Storage volume size For memory/pv: actual volume size For cloud backends: size of WAL (Write-Ahead Log) volume Increase for higher trace volumes or longer retention |
214233

215234
## Releases
216235

‎README.md.gotmpl‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,21 @@ $ helm repo update
4545

4646
### 3. Install Dependencies
4747

48-
Before installing the full chart, you must first install the dependency subcharts, including the Prometheus Custom Resource Definitions (CRDs) and the Grafana operator chart. This assumes you are installing it in the same namespace as your CircleCI server installation:
48+
Before installing the full chart, you must first install the dependency subcharts and operators.
49+
50+
#### 3.1 Install Prometheus CRDs and Grafana Operator
51+
52+
Install the Prometheus Custom Resource Definitions (CRDs) and the Grafana operator chart. This assumes you are installing it in the same namespace as your CircleCI server installation:
4953

5054
```bash
5155
$ helm install {{ template "chart.name" . }} {{ template "chart.name" . }}/{{ template "chart.name" . }} --set global.enabled=false --set prometheusOperator.installCRDs=true --version {{ template "chart.version" . }} -n <your-server-namespace>
5256
```
53-
> **_NOTE:_** It's possible to install the monitoring stack in a different namespace than the CircleCI server installation. If you do so, set the `prometheus.serviceMonitor.selectorNamespaces` value with the target namespace.
57+
58+
> **_NOTE:_** It's possible to install the monitoring stack in a different namespace than the CircleCI server installation. If you do so, set the `prometheus.serviceMonitor.selectorNamespaces` value with the target namespace.
59+
60+
#### 3.2 Install Tempo Operator (Optional)
61+
62+
If you plan to enable distributed tracing with Tempo (`tempo.enabled=true`), you must manually install the Tempo Operator by following the official documentation at https://grafana.com/docs/tempo/latest/setup/operator/#installation[]. There is currently no official Helm chart available for the Tempo Operator or its CRDs.
5463

5564
### 4. Install the Helm Chart
5665

‎do‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,21 @@ version() {
1515
help_kubeconform="Run helm kubeconform"
1616
kubeconform() {
1717
check-helm
18-
1918
install-plugin kubeconform https://github.com/jtyr/kubeconform-helm
2019

21-
helm kubeconform --ignore-missing-schema --verbose --summary --strict "$@" \
20+
# Add schemas not included in any catalogs
21+
schemas_dir="./target/schemas"
22+
mkdir -p "${schemas_dir}"
23+
cd "${schemas_dir}"
24+
curl -s https://raw.githubusercontent.com/yannh/kubeconform/master/scripts/openapi2jsonschema.py | \
25+
python3 - https://raw.githubusercontent.com/grafana/tempo-operator/refs/heads/main/config/crd/bases/tempo.grafana.com_tempomonolithics.yaml
26+
cd -
27+
28+
helm kubeconform --verbose --summary --strict "$@" \
2229
--schema-location default \
23-
--schema-location https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json \
24-
. --skip GrafanaDashboard # https://github.com/yannh/kubeconform/issues/300
30+
--schema-location "https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json" \
31+
--schema-location "${schemas_dir}/{{ .ResourceKind }}_{{.ResourceAPIVersion}}.json" \
32+
.
2533
}
2634

2735
# This variable is used, but shellcheck can't tell.

‎templates/_helpers.tpl‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,12 @@ Compute if grafana is enabled.
3636
(eq (.Values.grafana.enabled | toString) "true")
3737
(and (eq (.Values.grafana.enabled | toString) "-") (eq (.Values.global.enabled | toString) "true"))) -}}
3838
{{- end -}}
39+
40+
{{/*
41+
Compute if tempo is enabled.
42+
*/}}
43+
{{- define "tempo.enabled" -}}
44+
{{- $_ := set . "tempoEnabled" (or
45+
(eq (.Values.tempo.enabled | toString) "true")
46+
(and (eq (.Values.tempo.enabled | toString) "-") (eq (.Values.global.enabled | toString) "true"))) -}}
47+
{{- end -}}

‎templates/tempo/tempomonolithic.yaml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
{{- include "tempo.enabled" . -}}
2+
{{- if .tempoEnabled }}
3+
{{- $name := include "server-monitoring.fullname" . -}}
4+
5+
apiVersion: tempo.grafana.com/v1alpha1
6+
kind: TempoMonolithic
7+
metadata:
8+
name: {{ $name }}-tempo
9+
namespace: {{ .Release.Namespace }}
10+
labels:
11+
app: {{ $name }}-tempo
12+
spec:
13+
{{- with .Values.tempo.resources }}
14+
resources:
15+
{{- toYaml . | nindent 4 }}
16+
{{- end }}
17+
18+
{{- with .Values.tempo.storage.traces }}
19+
storage:
20+
traces:
21+
backend: {{ .backend }}
22+
size: {{ .size }}
23+
24+
{{- if eq .backend "s3" }}
25+
s3:
26+
secret: {{ .s3.secret }}
27+
{{- end }}
28+
29+
{{- if eq .backend "gcs" }}
30+
gcs:
31+
secret: {{ .gcs.secret }}
32+
{{- end }}
33+
{{- end }}
34+
35+
# TODO: https://circleci.atlassian.net/browse/ONPREM-2157
36+
# observability:
37+
# grafana:
38+
# dataSource:
39+
# enabled: true
40+
41+
extraConfig:
42+
tempo:
43+
limits_config:
44+
retention_period: {{ .Values.tempo.retentionPeriod }}
45+
compactor:
46+
ring:
47+
kvstore:
48+
store: inmemory
49+
ingester:
50+
lifecycler:
51+
ring:
52+
kvstore:
53+
store: inmemory
54+
55+
{{- end }}

‎tests/kubeconform/pvc-values.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,8 @@ grafana:
77
persistence:
88
enabled: true
99
storageClass: "custom-storage"
10+
11+
tempo:
12+
storage:
13+
traces:
14+
backend: pv

‎tests/tempo/tempomonolithic_test.yaml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
suite: test tempo
2+
templates:
3+
- tempo/tempomonolithic.yaml
4+
tests:
5+
- it: should work with basic configuration
6+
set:
7+
tempo.enabled: true
8+
tempo.resources.requests.cpu: 250m
9+
tempo.resources.requests.memory: 512Mi
10+
tempo.resources.limits.cpu: 500m
11+
tempo.resources.limits.memory: 1Gi
12+
tempo.storage.traces.size: 20Gi
13+
asserts:
14+
- isKind:
15+
of: TempoMonolithic
16+
- matchRegex:
17+
path: metadata.name
18+
pattern: tempo
19+
- equal:
20+
path: spec.resources.requests.cpu
21+
value: 250m
22+
- equal:
23+
path: spec.resources.requests.memory
24+
value: 512Mi
25+
- equal:
26+
path: spec.resources.limits.cpu
27+
value: 500m
28+
- equal:
29+
path: spec.resources.limits.memory
30+
value: 1Gi
31+
- equal:
32+
path: spec.storage.traces.size
33+
value: 20Gi
34+
35+
- it: should configure memory backend
36+
set:
37+
tempo.enabled: true
38+
tempo.storage.traces.backend: memory
39+
tempo.storage.traces.size: 2Gi
40+
asserts:
41+
- equal:
42+
path: spec.storage.traces.backend
43+
value: memory
44+
- equal:
45+
path: spec.storage.traces.size
46+
value: 2Gi
47+
48+
- it: should configure s3 backend
49+
set:
50+
tempo.enabled: true
51+
tempo.storage.traces.backend: s3
52+
tempo.storage.traces.s3.secret: my-s3-secret
53+
tempo.storage.traces.size: 100Gi
54+
asserts:
55+
- equal:
56+
path: spec.storage.traces.backend
57+
value: s3
58+
- equal:
59+
path: spec.storage.traces.s3.secret
60+
value: my-s3-secret
61+
- equal:
62+
path: spec.storage.traces.size
63+
value: 100Gi
64+
65+
- it: should configure gcs backend
66+
set:
67+
tempo.enabled: true
68+
tempo.storage.traces.backend: gcs
69+
tempo.storage.traces.gcs.secret: my-gcs-secret
70+
tempo.storage.traces.size: 100Gi
71+
asserts:
72+
- equal:
73+
path: spec.storage.traces.backend
74+
value: gcs
75+
- equal:
76+
path: spec.storage.traces.gcs.secret
77+
value: my-gcs-secret
78+
- equal:
79+
path: spec.storage.traces.size
80+
value: 100Gi
81+
82+
- it: should configure pv backend
83+
set:
84+
tempo.enabled: true
85+
tempo.storage.traces.backend: pv
86+
tempo.storage.traces.size: 50Gi
87+
asserts:
88+
- equal:
89+
path: spec.storage.traces.backend
90+
value: pv
91+
- equal:
92+
path: spec.storage.traces.size
93+
value: 50Gi
94+
95+
- it: should set retention period
96+
set:
97+
tempo.enabled: true
98+
tempo.retentionPeriod: 7d
99+
asserts:
100+
- equal:
101+
path: spec.extraConfig.tempo.limits_config.retention_period
102+
value: 7d
103+
104+
- it: should not create resource when disabled
105+
set:
106+
tempo.enabled: false
107+
asserts:
108+
- hasDocuments:
109+
count: 0

‎values.yaml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,3 +149,53 @@ grafana:
149149
dashboards:
150150
# -- The directory containing JSON files for Grafana dashboards.
151151
jsonDirectory: dashboards
152+
153+
tempo:
154+
# -- Enable Tempo distributed tracing
155+
# Requires manual installation of Tempo Operator
156+
# Set to true to enable, false to disable, "-" to use global default
157+
enabled: "-"
158+
# -- Resource requirements for Tempo pods
159+
# Adjust based on your trace volume and cluster capacity
160+
resources:
161+
requests:
162+
# -- Minimum CPU guaranteed to Tempo pods
163+
cpu: 500m
164+
# -- Minimum memory guaranteed to Tempo pods
165+
memory: 1Gi
166+
limits:
167+
# -- Maximum CPU Tempo pods can use
168+
cpu: 1000m
169+
# -- Maximum memory Tempo pods can use
170+
memory: 2Gi
171+
# -- Storage configuration for trace data
172+
storage:
173+
traces:
174+
# -- Storage backend for traces
175+
# Default: in-memory storage (traces lost on pod restart)
176+
# Suitable for development/testing environments only
177+
backend: memory
178+
179+
# -- Persistent Volume storage
180+
# Stores traces on persistent disk attached to the pod
181+
# backend: pv
182+
183+
# -- Cloud storage backends
184+
# Choose based on your CircleCI server deployment location:
185+
186+
# -- AWS S3 (use if CircleCI server is on AWS)
187+
# s3:
188+
# secret: tempo-s3-secret
189+
190+
# -- Google Cloud Storage (use if CircleCI server is on GCP)
191+
# gcs:
192+
# secret: tempo-gcs-secret
193+
194+
# -- Storage volume size
195+
# For memory/pv: actual volume size
196+
# For cloud backends: size of WAL (Write-Ahead Log) volume
197+
# Increase for higher trace volumes or longer retention
198+
size: 20Gi
199+
# -- Trace retention period
200+
# How long to keep trace data before deletion
201+
retentionPeriod: 24h

0 commit comments

Comments
 (0)