Skip to content

Commit d50b911

Browse files
siggysiggy
authored
Introduce Thanos demo (#244)
This environment demonstrates deploying Linkerd across 4 Kubernetes cluster providers, and aggregating metrics into a single Thanos Querier. Signed-off-by: Andrew Seigner <[email protected]>
1 parent 0b368bb commit d50b911

11 files changed

+6102
-0
lines changed

‎thanos-demo/README.md‎

Lines changed: 280 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
# Thanos Demo
2+
3+
This environment demonstrates deploying [Linkerd](https://linkerd.io) and sample
4+
apps across 4 cluster providers, and aggregating metrics into a single
5+
[Thanos](https://github.com/improbable-eng/thanos) Querier.
6+
7+
These configs assume the following:
8+
- Working Kubernetes clusters in 4 cloud providers:
9+
- Amazon EKS
10+
- Azure Kubernetes Service (AKS)
11+
- DigitalOcean
12+
- Google Kubernetes Engine (GKE)
13+
- Block storage configured and accessible from each Kubernetes cluster
14+
15+
## Config files
16+
17+
- `cluster-*.yaml`: `Namespace`, `Service`, and `ConfigMap` objects
18+
to enable persistent IP addresses and provide access to object stores,
19+
specific to each cluster provider. For more information, see
20+
[Object Storage](https://github.com/improbable-eng/thanos/blob/master/docs/storage.md)
21+
in the [Thanos repo](https://github.com/improbable-eng/thanos).
22+
- `linkerd-install-*.yaml`: modified `linkerd install` configs to support Thanos
23+
integration.
24+
- [`thanos-querier.yaml`](thanos-querier.yaml): The Thanos Querier and Grafana,
25+
to aggregate metrics from all clusters.
26+
27+
### Linkerd / Thanos integration
28+
29+
To enable Linkerd integration with Thanos, 4 changes are required to the default
30+
`linkerd install` output. These changes have already been made in the
31+
`linkerd-install-*.yaml` files:
32+
33+
1. In `linkerd-prometheus` Deployment, introduce a `thanos-config` volume. This
34+
references the `ConfigMap` defined in `cluster-*.yaml`, enabling object
35+
storage:
36+
```yaml
37+
kind: Deployment
38+
name: linkerd-prometheus
39+
spec:
40+
template:
41+
spec:
42+
volumes:
43+
- configMap:
44+
name: thanos-config
45+
name: thanos-config
46+
```
47+
48+
2. In `ConfigMap/linkerd-prometheus-config`, introduce an `external_labels`
49+
field to the Prometheus config file, indicating the cluster:
50+
```yaml
51+
kind: ConfigMap
52+
metadata:
53+
name: linkerd-prometheus-config
54+
data:
55+
prometheus.yml: |-
56+
global:
57+
external_labels:
58+
cluster: aks # or do, eks, gke
59+
```
60+
61+
3. In the `linkerd-prometheus` container, set
62+
`--storage.tsdb.max-block-duration=2h` and
63+
`--storage.tsdb.min-block-duration=2h`:
64+
```yaml
65+
kind: Deployment
66+
name: linkerd-prometheus
67+
spec:
68+
template:
69+
spec:
70+
containers:
71+
- args:
72+
- --storage.tsdb.max-block-duration=2h
73+
- --storage.tsdb.min-block-duration=2h
74+
```
75+
76+
4. In `linkerd-prometheus` pod, introduce `thanos-sidecar` and `thanos-store`:
77+
```yaml
78+
kind: Deployment
79+
name: linkerd-prometheus
80+
spec:
81+
template:
82+
spec:
83+
containers:
84+
- name: thanos-sidecar
85+
image: improbable/thanos:v0.3.2
86+
args:
87+
- sidecar
88+
- --tsdb.path=/data
89+
- --prometheus.url=http://localhost:9090
90+
- --cluster.disable
91+
- --objstore.config-file=/etc/thanos/bucket.yml
92+
- --grpc-address=0.0.0.0:10901
93+
- --http-address=0.0.0.0:10902
94+
ports:
95+
- name: http-sidecar
96+
containerPort: 10902
97+
- name: grpc
98+
containerPort: 10901
99+
volumeMounts:
100+
- mountPath: /data
101+
name: data
102+
- mountPath: /etc/prometheus
103+
name: prometheus-config
104+
readOnly: true
105+
- mountPath: /etc/thanos
106+
name: thanos-config
107+
readOnly: true
108+
- name: thanos-store
109+
image: improbable/thanos:v0.3.2
110+
args:
111+
- store
112+
- --data-dir=/data
113+
- --cluster.disable
114+
- --objstore.config-file=/etc/thanos/bucket.yml
115+
- --grpc-address=0.0.0.0:10911
116+
- --http-address=0.0.0.0:10912
117+
- --index-cache-size=500MB
118+
- --chunk-pool-size=500MB
119+
ports:
120+
- name: store-http
121+
containerPort: 10912
122+
- name: store-grpc
123+
containerPort: 10911
124+
volumeMounts:
125+
- mountPath: /etc/thanos
126+
name: thanos-config
127+
readOnly: true
128+
```
129+
130+
## Install Linkerd + Thanos sidecars
131+
132+
```bash
133+
# define kubectl contexts
134+
export AKS=[AKS CONTEXT]
135+
export DO=[DIGITAL OCEAN CONTEXT]
136+
export EKS=[AMAZON EKS CONTEXT]
137+
export GKE=[GKE CONTEXT]
138+
139+
# namespaces, services, and block storage access
140+
cat cluster-aks.yaml | kubectl --context $AKS apply -f -
141+
cat cluster-do.yaml | kubectl --context $DO apply -f -
142+
cat cluster-eks.yaml | kubectl --context $EKS apply -f -
143+
cat cluster-gke.yaml | kubectl --context $GKE apply -f -
144+
145+
# linkerd and thanos sidecars
146+
cat linkerd-install-aks.yaml | kubectl --context $AKS apply -f -
147+
cat linkerd-install-do.yaml | kubectl --context $DO apply -f -
148+
cat linkerd-install-eks.yaml | kubectl --context $EKS apply -f -
149+
cat linkerd-install-gke.yaml | kubectl --context $GKE apply -f -
150+
```
151+
152+
### Validate Linkerd is installed
153+
154+
```bash
155+
for CLUSTER in $AKS $DO $EKS $GKE
156+
do
157+
printf "\n$CLUSTER\n"
158+
linkerd --context $CLUSTER version
159+
done
160+
```
161+
162+
```bash
163+
linkerd --context $AKS check
164+
```
165+
166+
## Install sample apps
167+
168+
```bash
169+
for CLUSTER in $AKS $DO $EKS $GKE
170+
do
171+
curl -s https://run.linkerd.io/emojivoto.yml |
172+
kubectl --context $CLUSTER apply -f -
173+
done
174+
```
175+
176+
### View sample app
177+
178+
```bash
179+
kubectl --context $AKS -n emojivoto port-forward svc/web-svc 8080:80
180+
```
181+
182+
### Inject sample apps with Linkerd
183+
184+
```bash
185+
for CLUSTER in $AKS $DO $EKS $GKE
186+
do
187+
curl -s https://run.linkerd.io/emojivoto.yml |
188+
linkerd inject - |
189+
kubectl --context $CLUSTER apply -f -
190+
done
191+
```
192+
193+
Validate proxy injected
194+
195+
```bash
196+
for CLUSTER in $AKS $DO $EKS $GKE
197+
do
198+
printf "\n$CLUSTER\n"
199+
linkerd --context $CLUSTER -n emojivoto stat deploy
200+
done
201+
```
202+
203+
```bash
204+
linkerd --context $AKS dashboard
205+
```
206+
207+
## Install Thanos Querier + Grafana on AKS
208+
209+
The [`thanos-querier.yaml`](thanos-querier.yaml) deployment reads from all 4
210+
Thanos Sidecars. Addresses from each sidecar must be provided to the Thanos
211+
Querier via command line flags.
212+
213+
### Static Addresses
214+
215+
Obtain static addresses for all 4 sidecars:
216+
217+
```bash
218+
for CLUSTER in AKS DO GKE
219+
do
220+
echo "$CLUSTER"_ADDRESS=$(
221+
kubectl --context ${!CLUSTER} -n linkerd get svc/thanos-sidecar \
222+
-o jsonpath='{.spec.loadBalancerIP}'
223+
)
224+
done
225+
226+
# EKS Load Balancers are configured slightly differently
227+
echo EKS_ADDRESS=$(
228+
kubectl --context $EKS \
229+
-n linkerd get svc/thanos-sidecar \
230+
-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
231+
)
232+
```
233+
234+
#### Update `store` values in `thanos-querier.yaml`
235+
236+
Each cluster has a Thanos Sidecar and a Thanos Store. Thanos Sidecar reads from
237+
Prometheus and writes into the object store. Thanos Store reads from the object
238+
store are provides historical data to Thanos Querier. Configure Thanos Querier
239+
to read from both sidecar and store on each cluster:
240+
241+
```yaml
242+
kind: Deployment
243+
metadata:
244+
name: thanos-querier
245+
spec:
246+
template:
247+
spec:
248+
containers:
249+
- name: thanos
250+
args:
251+
- query
252+
- --store=[AKS_ADDRESS]:10901 # AKS sidecar
253+
- --store=[AKS_ADDRESS]:10911 # AKS store
254+
- --store=[DO_ADDRESS]:10901 # DO sidecar
255+
- --store=[DO_ADDRESS]:10911 # DO store
256+
- --store=[EKS_ADDRESS]:10901 # EKS sidecar
257+
- --store=[EKS_ADDRESS]:10911 # EKS store
258+
- --store=[GKE_ADDRESS]:10901 # GKE sidecar
259+
- --store=[GKE_ADDRESS]:10911 # GKE store
260+
```
261+
262+
### Deploy
263+
264+
In this example, we deploy Thanos Querier into AKS:
265+
266+
```bash
267+
cat thanos-querier.yaml | kubectl --context $AKS apply -f -
268+
```
269+
270+
### View Thanos Querier Dashboard
271+
272+
```bash
273+
kubectl --context $AKS -n thanos-demo port-forward svc/thanos-querier 10902
274+
```
275+
276+
### View Thanos Querier Grafana
277+
278+
```bash
279+
kubectl --context $AKS -n thanos-demo port-forward svc/grafana 3000
280+
```

‎thanos-demo/cluster-aks.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
kind: Namespace
3+
apiVersion: v1
4+
metadata:
5+
name: linkerd
6+
---
7+
kind: Service
8+
apiVersion: v1
9+
metadata:
10+
name: thanos-sidecar
11+
namespace: linkerd
12+
spec:
13+
type: LoadBalancer
14+
# loadBalancerIP: XXX
15+
selector:
16+
linkerd.io/control-plane-component: prometheus
17+
ports:
18+
- port: 10901
19+
targetPort: grpc
20+
name: grpc
21+
- port: 10911
22+
targetPort: store-grpc
23+
name: store-grpc
24+
---
25+
kind: ConfigMap
26+
apiVersion: v1
27+
metadata:
28+
name: thanos-config
29+
namespace: linkerd
30+
data:
31+
bucket.yml: |-
32+
type: AZURE
33+
config:
34+
storage_account: [AKS_ACCOUNT]
35+
storage_account_key: [AKS_KEY]
36+
container: thanos-demo

‎thanos-demo/cluster-do.yaml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
kind: Namespace
3+
apiVersion: v1
4+
metadata:
5+
name: linkerd
6+
---
7+
kind: Service
8+
apiVersion: v1
9+
metadata:
10+
name: thanos-sidecar
11+
namespace: linkerd
12+
spec:
13+
type: LoadBalancer
14+
# loadBalancerIP: XXX
15+
selector:
16+
linkerd.io/control-plane-component: prometheus
17+
ports:
18+
- port: 10901
19+
targetPort: grpc
20+
name: grpc
21+
- port: 10911
22+
targetPort: store-grpc
23+
name: store-grpc
24+
---
25+
kind: ConfigMap
26+
apiVersion: v1
27+
metadata:
28+
name: thanos-config
29+
namespace: linkerd
30+
data:
31+
bucket.yml: |-
32+
type: S3
33+
config:
34+
bucket: "thanos-demo"
35+
endpoint: "sfo2.digitaloceanspaces.com"
36+
access_key: [DO_ACCESS_KEY]
37+
secret_key: [DO_SECRET_KEY]
38+
insecure: false
39+
signature_version2: false
40+
encrypt_sse: false
41+
put_user_metadata: {}
42+
http_config:
43+
idle_conn_timeout: 0s
44+
insecure_skip_verify: false

0 commit comments

Comments
 (0)