Skip to content

Commit 26371d5

Browse files
bonclay7bonclay7
authored
Improve tracing (#195)
* Dropping daemonset option * Update adot loglevel config * Enable tracing by default * Init tracing docs * Add tracing docs * Update tracing.md * Update collector endpoint * Fix tracing instructions
1 parent e6083b1 commit 26371d5

File tree

5 files changed

+218
-92
lines changed

5 files changed

+218
-92
lines changed

‎docs/eks/tracing.md

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Tracing on Amazon EKS
2+
3+
[Distributed tracing](https://aws-observability.github.io/observability-best-practices/signals/traces/)
4+
helps you have end-to-end visibility between transactions in distributed nodes.
5+
The `eks-monitoring` module is configured by default to collect traces into
6+
[AWS X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html).
7+
8+
The AWS Distro for OpenTelemetry collector is configured to receive traces
9+
in the OTLP format (OTLP receiver), using the OpenTelemetry SDK or
10+
auto-instrumentation agents.
11+
12+
!!! note
13+
To disable the tracing configuration, set up `enable_tracing = false` in
14+
the [module configuration](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/eks-monitoring#input_enable_tracing)
15+
16+
17+
## Instrumentation
18+
19+
Let's take a [sample application](https://github.com/aws-observability/aws-otel-community/tree/master/sample-apps/go-sample-app)
20+
that is already instrumented with the OpenTelemetry SDK.
21+
22+
!!! note
23+
To learn more about instrumenting with OpenTelemetry, please visit the
24+
[OpenTelemetry documentation](https://opentelemetry.io/docs/instrumentation/)
25+
for your programming language.
26+
27+
Cloning the repo
28+
29+
```console
30+
git clone https://github.com/aws-observability/aws-otel-community.git
31+
cd aws-otel-community/sample-apps/go-sample-app
32+
```
33+
34+
35+
Highlighting code sections
36+
37+
38+
## Deploying on Amazon EKS
39+
40+
Using the sample application, we will build a container image, create and push
41+
an image on Amazon ECR. We will use a Kubernetes manifest to deploy to an EKS
42+
cluster.
43+
44+
!!! warning
45+
The following steps require that you have an EKS cluster ready. To deploy
46+
an EKS cluster, please visit [our example](https://aws-observability.github.io/terraform-aws-observability-accelerator/helpers/new-eks-cluster/).
47+
48+
### Building container image
49+
50+
51+
=== "amd64 linux"
52+
53+
``` console
54+
docker build -t go-sample-app .
55+
```
56+
57+
=== "cross platform build"
58+
59+
``` bash
60+
docker buildx build -t go-sample-app . --platform=linux/amd64
61+
```
62+
63+
### Publishing on Amazon ECR
64+
65+
66+
=== "using docker"
67+
68+
``` console
69+
export ECR_REPOSITORY_URI=$(aws ecr create-repository --repository go-sample-app --query repository.repositoryUri --output text)
70+
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
71+
docker tag go-sample-app:latest "${ECR_REPOSITORY_URI}:latest"
72+
docker push "${ECR_REPOSITORY_URI}:latest"
73+
```
74+
75+
76+
## Deploying on Amazon EKS
77+
78+
79+
``` yaml title="eks.yaml" linenums="1"
80+
apiVersion: apps/v1
81+
kind: Deployment
82+
metadata:
83+
name: go-sample-app
84+
namespace: default
85+
spec:
86+
replicas: 2
87+
selector:
88+
matchLabels:
89+
app: go-sample-app
90+
template:
91+
metadata:
92+
labels:
93+
app: go-sample-app
94+
spec:
95+
containers:
96+
- name: go-sample-app
97+
image: "${ECR_REPOSITORY_URI}:latest" # make sure to replace this variable
98+
imagePullPolicy: Always
99+
env:
100+
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
101+
value: adot-collector.adot-collector-kubeprometheus.svc.cluster.local:4317
102+
resources:
103+
limits:
104+
cpu: 300m
105+
memory: 300Mi
106+
requests:
107+
cpu: 100m
108+
memory: 180Mi
109+
ports:
110+
- containerPort: 8080
111+
---
112+
apiVersion: v1
113+
kind: Service
114+
metadata:
115+
name: go-sample-app
116+
namespace: default
117+
labels:
118+
app: go-sample-app
119+
spec:
120+
ports:
121+
- protocol: TCP
122+
port: 8080
123+
targetPort: 8080
124+
selector:
125+
app: go-sample-app
126+
---
127+
apiVersion: v1
128+
kind: Service
129+
metadata:
130+
name: go-sample-app
131+
namespace: default
132+
spec:
133+
type: ClusterIP
134+
selector:
135+
app: go-sample-app
136+
ports:
137+
- protocol: TCP
138+
port: 8080
139+
targetPort: 8080
140+
```
141+
142+
### Deploying and testing
143+
144+
With the Kubernetes manifest ready, run:
145+
146+
```bash
147+
kubectl apply -f eks.yaml
148+
```
149+
150+
You should see the pods running with the command:
151+
152+
```console
153+
kubectl get pods
154+
NAME READY STATUS RESTARTS AGE
155+
go-sample-app-67c48ff8c6-bdw74 1/1 Running 0 4s
156+
go-sample-app-67c48ff8c6-t6k2j 1/1 Running 0 4s
157+
```
158+
159+
To simulate some traffic you can forward the service port to your local host
160+
and test a few queries
161+
162+
```console
163+
kubectl port-forward deployment/go-sample-app 8080:8080
164+
```
165+
166+
Test a few endpoints
167+
168+
```
169+
curl http://localhost:8080/
170+
curl http://localhost:8080/outgoing-http-call
171+
curl http://localhost:8080/aws-sdk-call
172+
curl http://localhost:8080/outgoing-sampleapp
173+
```
174+
175+
## Visualizing traces
176+
177+
As this is a basic example, the service map doesn't have a lot of nodes,
178+
but this shows you how to setup tracing in your application and deploying
179+
it on Amazon EKS using the `eks-monitoring` module.
180+
181+
With Flux and Grafana Operator, the `eks-monitoring` module configures
182+
an AWS X-Ray data source on your provided Grafana workspace. Open the
183+
Grafana explorer view and select the X-Ray data source. If you type the query
184+
below, and select `Trace List` for **Query Type**, you should see the list
185+
of traces occured in the selected timeframe.
186+
187+
<img width="1721" alt="Screenshot 2023-07-20 at 21 42 30" src="https://github.com/aws-observability/terraform-aws-observability-accelerator/assets/10175027/bd992a77-05fb-47d2-8ed4-af05d96e951d">
188+
189+
You can add the service map to a dashbaord, for example a service focused
190+
dashbaord. You can click on any of the traces to view a node map and the traces
191+
details.
192+
193+
There is a button that can take you the CloudWatch console to view the same
194+
data. If your logs are stored on CloudWatch Logs, this page can present
195+
all the logs in the trace details page. The CloudWatch Log Group name should
196+
be added to the trace as an attribute.
197+
Read more about this in our [One Observability Workshop](https://catalog.workshops.aws/observability/en-US/use-cases/trace-to-logs-java-instrumentation/concepts)
198+
199+
![CloudWatch service map](https://user-images.githubusercontent.com/10175027/254973349-1028f428-c2ef-4bd2-8114-0d0961d7cdd8.png)
200+
201+
202+
## Resoures
203+
204+
- [AWS Observability Best Practices](https://aws-observability.github.io/observability-best-practices/)
205+
- [One Observability Workshop](https://catalog.workshops.aws/observability/en-US/)
206+
- [AWS Distro for OpenTelemetry documentation](https://aws-otel.github.io/docs/introduction)
207+
- [AWS X-Ray user guide](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html)
208+
- [OpenTelemetry documentation](https://opentelemetry.io/docs/what-is-opentelemetry/)

‎mkdocs.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ nav:
3232
- Nginx: eks/nginx.md
3333
- Istio: eks/istio.md
3434
- Viewing logs: eks/logs.md
35+
- Tracing: eks/tracing.md
3536
- Teardown: eks/destroy.md
3637
- Monitoring Managed Service for Prometheus Workspaces: workloads/managed-prometheus.md
3738
- Supporting Examples:
@@ -47,6 +48,8 @@ markdown_extensions:
4748
- codehilite
4849
- footnotes
4950
- pymdownx.critic
51+
- pymdownx.tabbed:
52+
alternate_style: true
5053
- pymdownx.superfences:
5154
custom_fences:
5255
- name: mermaid

‎modules/eks-monitoring/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
6666

6767
| Name | Description | Type | Default | Required |
6868
|------|-------------|------|---------|:--------:|
69-
| <a name="input_adot_loglevel"></a> [adot\_loglevel](#input\_adot\_loglevel) | Verbosity level for ADOT collector logs | `string` | `"warn"` | no |
69+
| <a name="input_adot_loglevel"></a> [adot\_loglevel](#input\_adot\_loglevel) | Verbosity level for ADOT collector logs. This accepts (detailed\|normal\|basic), see https://aws-otel.github.io/docs/components/misc-exporters for mor infos. | `string` | `"normal"` | no |
7070
| <a name="input_custom_metrics_config"></a> [custom\_metrics\_config](#input\_custom\_metrics\_config) | Configuration object to enable custom metrics collection | <pre>map(object({<br> enableBasicAuth = bool<br> path = string<br> basicAuthUsername = string<br> basicAuthPassword = string<br> ports = string<br> droppedSeriesPrefixes = string<br> }))</pre> | `null` | no |
7171
| <a name="input_eks_cluster_id"></a> [eks\_cluster\_id](#input\_eks\_cluster\_id) | EKS Cluster Id | `string` | n/a | yes |
7272
| <a name="input_enable_alerting_rules"></a> [enable\_alerting\_rules](#input\_enable\_alerting\_rules) | Enables or disables Managed Prometheus alerting rules | `bool` | `true` | no |
@@ -84,7 +84,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
8484
| <a name="input_enable_nginx"></a> [enable\_nginx](#input\_enable\_nginx) | Enable NGINX workloads monitoring, alerting and default dashboards | `bool` | `false` | no |
8585
| <a name="input_enable_node_exporter"></a> [enable\_node\_exporter](#input\_enable\_node\_exporter) | Enables or disables Node exporter. Disabling this might affect some data in the dashboards | `bool` | `true` | no |
8686
| <a name="input_enable_recording_rules"></a> [enable\_recording\_rules](#input\_enable\_recording\_rules) | Enables or disables Managed Prometheus recording rules | `bool` | `true` | no |
87-
| <a name="input_enable_tracing"></a> [enable\_tracing](#input\_enable\_tracing) | (Experimental) Enables tracing with AWS X-Ray. This changes the deploy mode of the collector to daemon set. Requirement: adot add-on <= 0.58-build.0 | `bool` | `false` | no |
87+
| <a name="input_enable_tracing"></a> [enable\_tracing](#input\_enable\_tracing) | Enables tracing with OTLP traces receiver to X-Ray | `bool` | `true` | no |
8888
| <a name="input_flux_config"></a> [flux\_config](#input\_flux\_config) | FluxCD configuration | <pre>object({<br> create_namespace = bool<br> k8s_namespace = string<br> helm_chart_name = string<br> helm_chart_version = string<br> helm_release_name = string<br> helm_repo_url = string<br> helm_settings = map(string)<br> helm_values = map(any)<br> })</pre> | <pre>{<br> "create_namespace": true,<br> "helm_chart_name": "flux2",<br> "helm_chart_version": "2.7.0",<br> "helm_release_name": "observability-fluxcd-addon",<br> "helm_repo_url": "https://fluxcd-community.github.io/helm-charts",<br> "helm_settings": {},<br> "helm_values": {},<br> "k8s_namespace": "flux-system"<br>}</pre> | no |
8989
| <a name="input_flux_gitrepository_branch"></a> [flux\_gitrepository\_branch](#input\_flux\_gitrepository\_branch) | Flux GitRepository Branch | `string` | `"main"` | no |
9090
| <a name="input_flux_gitrepository_name"></a> [flux\_gitrepository\_name](#input\_flux\_gitrepository\_name) | Flux GitRepository name | `string` | `"aws-observability-accelerator"` | no |

0 commit comments

Comments
 (0)