Skip to content

Commit 715dced

Browse files
networkandcodenetworkandcode
committed
Tracing demo with AWS X-Ray and Grafana
1 parent 2bcb456 commit 715dced

9 files changed

+529
-1
lines changed

‎_config.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ twitter_username: networkandcode
3131
github_username: networkandcode
3232

3333
# Build settings
34-
theme: minima
34+
theme: architect
3535
plugins:
3636
- jekyll-feed
3737

‎_posts/2024-04-26/2023-02-17-query-harperdbs-rest-api-via-apollo-graphql.md‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
---
2+
canonical_url: https://dev.to/aws-builders/tracing-demo-with-aws-x-ray-and-grafana-1pb5
3+
date: 2024-04-26
4+
title: Tracing demo with AWS X-Ray and Grafana
5+
categories: aws, grafana, otel, xray
6+
cover_image: https://source.unsplash.com/featured?trace
7+
tags: aws, grafana, kubernetes, otel
8+
---
9+
10+
**This post first appeared on [dev.to](https://dev.to/aws-builders/tracing-demo-with-aws-x-ray-and-grafana-1pb5)**
11+
12+
## Introduction
13+
Hello :wave:, In this post we'll see about sending traces from a demo app to AWS X-Ray via the ADOT(AWS Distro for OpenTelemetry) collector. We would then visualize this on Grafana. Note that we'd deploy the workloads on a kubernetes cluster.
14+
15+
Here is a picture of what we are trying to accomplish:
16+
![Block diagram for the lab](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d79dr2bjzdphe7d2w3de.png)
17+
18+
Alright, let's get started!!!
19+
20+
## Namespace
21+
We shall deploy the workloads on a separate namespace. Let's create one.
22+
```
23+
kubectl create ns adot-traces-demo
24+
```
25+
26+
## Credentials
27+
Store the AWS credentials as a kubernetes secret.
28+
```
29+
kubectl create secret generic aws-credentials \
30+
--from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
31+
--from-literal=AWS_SECRET_ACCESS_KEY=<aws-secret-access-key> \
32+
-n adot-traces-demo
33+
```
34+
35+
## ADOT Config
36+
Set the ADOT config in a file.
37+
```
38+
$ cat adot-config.yaml
39+
exporters:
40+
awsxray:
41+
region: ap-south-2
42+
receivers:
43+
otlp:
44+
protocols:
45+
grpc:
46+
endpoint: 0.0.0.0:4317
47+
service:
48+
pipelines:
49+
traces:
50+
exporters:
51+
- awsxray
52+
receivers:
53+
- otlp
54+
```
55+
56+
And create a config map with this file.
57+
```
58+
kubectl create configmap adot-config --from-file=adot-config.yaml -n adot-traces-demo
59+
```
60+
61+
## ADOT Deployment
62+
Setup deployment spec in a file, that injects the secret we created earlier as environment variables and the configmap as a volume.
63+
```
64+
$ cat adot-deploy.yaml
65+
apiVersion: apps/v1
66+
kind: Deployment
67+
metadata:
68+
labels:
69+
app: adot-collector
70+
name: adot-collector
71+
spec:
72+
replicas: 1
73+
selector:
74+
matchLabels:
75+
app: adot-collector
76+
template:
77+
metadata:
78+
labels:
79+
app: adot-collector
80+
spec:
81+
containers:
82+
- args:
83+
- '--config=/etc/adot-config.yaml'
84+
envFrom:
85+
- secretRef:
86+
name: aws-credentials
87+
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
88+
name: adot-collector
89+
volumeMounts:
90+
- mountPath: /etc/adot-config.yaml
91+
name: config-volume
92+
subPath: adot-config.yaml
93+
volumes:
94+
- configMap:
95+
name: adot-config
96+
name: config-volume
97+
```
98+
99+
Create the deployment.
100+
```
101+
kubectl create -f adot-deploy.yaml -n adot-traces-demo
102+
```
103+
104+
The pod in the deployment should be running.
105+
```
106+
$ kubectl get po -n adot-traces-demo
107+
NAME READY STATUS RESTARTS AGE
108+
adot-collector-7cbf849b89-b4bkl 1/1 Running 0 3m26s
109+
```
110+
111+
## ADOT Service
112+
We can expose the ADOT deployment with a service spec that exposes the grpc port 4317 as follows.
113+
```
114+
$ cat adot-svc.yaml
115+
apiVersion: v1
116+
kind: Service
117+
metadata:
118+
name: adot-collector-service
119+
spec:
120+
selector:
121+
app: adot-collector
122+
ports:
123+
- protocol: TCP
124+
port: 4317
125+
targetPort: 4317
126+
```
127+
128+
We can now create the service.
129+
```
130+
kubectl create -f adot-svc.yaml -n adot-traces-demo
131+
```
132+
133+
The endpoint IP should match with the pod IP.
134+
```
135+
$ kubectl get ep -n adot-traces-demo
136+
NAME ENDPOINTS AGE
137+
adot-collector-service 10.1.3.187:4317 22s
138+
139+
$ kubectl get po -n adot-traces-demo -o wide
140+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
141+
adot-collector-7cbf849b89-b4bkl 1/1 Running 0 7m11s 10.1.3.187 docker-desktop <none> <none>
142+
```
143+
144+
## Demo app
145+
We can now deploy the sample demo app which can send traces to ADOT collector, with the following manifest.
146+
```
147+
$ cat k6-tracing-deploy.yaml
148+
apiVersion: apps/v1
149+
kind: Deployment
150+
metadata:
151+
name: xk6-tracing
152+
spec:
153+
replicas: 1
154+
selector:
155+
matchLabels:
156+
app: xk6-tracing
157+
template:
158+
metadata:
159+
labels:
160+
app: xk6-tracing
161+
spec:
162+
containers:
163+
- env:
164+
- name: ENDPOINT
165+
value: adot-collector-service:4317
166+
image: ghcr.io/grafana/xk6-client-tracing:v0.0.2
167+
name: xk6-tracing
168+
```
169+
170+
Let's create the deployment.
171+
```
172+
kubectl create -f k6-tracing-deploy.yaml -n adot-traces-demo
173+
```
174+
175+
Both the ADOT collector and k6-tracing pods should now be running.
176+
```
177+
$ kubectl get po -n adot-traces-demo
178+
NAME READY STATUS RESTARTS AGE
179+
adot-collector-7cbf849b89-b4bkl 1/1 Running 0 14m
180+
xk6-tracing-69b48fcfd9-bjzbd 1/1 Running 0 24s
181+
```
182+
183+
## X-Ray
184+
We can now headover to AWS X-Ray, in `ap-south-2` region that we mentioned in the adot-config.
185+
![Traces in AWS X-Ray](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lzblt5bwa2ikustn0h6h.png)
186+
187+
The nodes(services) shown in the screenshot belong to our demo application. We could filter for traces that passes through a particular service name for ex. article service, like below.
188+
![Traces ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/793iwmcy6cl1qweessew.png)
189+
190+
If we click on a single trace we should be able to see a complete service map for that trace, that shows all the services that trace traverses.
191+
![Service map for a trace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/37vtbbx0fkowsnpxtjty.png)
192+
193+
If we go a further down on this we should be able to see the details for segments/spans with in this trace.
194+
![Spans in trace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ib655esdlfxoxeyr4cac.png)
195+
196+
## Grafana
197+
So we far we were able to see the traces in AWS X-Ray, we can do a similar exercise on Grafana. I am using a Grafana Cloud Free subscription for this lab.
198+
199+
Go to Connections, Add a new connection and search for X-Ray and install it.
200+
![Install X-Ray](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lvks2wrj6m13x11lpsxx.png)
201+
202+
You can then go to datasources, add a new X-Ray datasource with the access key id, secret access key, and default region(I have chosen ap-south-2 which matches with adot config).
203+
204+
![Add X-Ray datasource](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5z1wgvj4cuvlb28tuzkj.png)
205+
206+
All good, we can try adding a new panel, go to dashboards > new dashboard and a new visualization with table as panel type and a sample query for ex. `service(id(name: "article-service" ))`
207+
![Traces in tabular format](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o31cgl9cpatnaw8jnhin.png)
208+
209+
We can click on one of the traces we should take us to the explore view where we can see the node graph(service map)
210+
![Node graph for trace in grafana](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/65b4n0id03356ftff66f.png)
211+
212+
We should also see the trace explorer that shows the individual spans.
213+
![Trace explorer in Grafana](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wikqutl9srogd6t62qm4.png)
214+
215+
Okay so we reached this far, that was some fun exploring traces on AWS and Grafana with Open Telemetry. Thank you for reading !!!
216+
217+
## CleanUp
218+
Just delete the namespace with `kubectl delete ns adot-traces-demo` and it should remove the workloads we deployed from kubernetes and stop sending any new data to the cloud.

‎_posts/2024-04-26/adot-config.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
exporters:
2+
awsxray:
3+
region: ap-south-2
4+
receivers:
5+
otlp:
6+
protocols:
7+
grpc:
8+
endpoint: 0.0.0.0:4317
9+
service:
10+
pipelines:
11+
traces:
12+
exporters:
13+
- awsxray
14+
receivers:
15+
- otlp

‎_posts/2024-04-26/adot-deploy.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: adot-collector
6+
name: adot-collector
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
app: adot-collector
12+
template:
13+
metadata:
14+
labels:
15+
app: adot-collector
16+
spec:
17+
containers:
18+
- args:
19+
- '--config=/etc/adot-config.yaml'
20+
envFrom:
21+
- secretRef:
22+
name: aws-credentials
23+
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
24+
name: adot-collector
25+
volumeMounts:
26+
- mountPath: /etc/adot-config.yaml
27+
name: config-volume
28+
subPath: adot-config.yaml
29+
volumes:
30+
- configMap:
31+
name: adot-config
32+
name: config-volume

‎_posts/2024-04-26/adot-svc.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: adot-collector-service
5+
spec:
6+
selector:
7+
app: adot-collector
8+
ports:
9+
- protocol: TCP
10+
port: 4317
11+
targetPort: 4317

‎_posts/2024-04-26/k6-tracing-deploy.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: xk6-tracing
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
app: xk6-tracing
10+
template:
11+
metadata:
12+
labels:
13+
app: xk6-tracing
14+
spec:
15+
containers:
16+
- env:
17+
- name: ENDPOINT
18+
value: http://adot-collector-service:4317
19+
image: ghcr.io/grafana/xk6-client-tracing:v0.0.2
20+
name: xk6-tracing

0 commit comments

Comments
 (0)