Skip to content

Commit 7ec2427

Browse files
committed
Repo migration
1 parent b5b9736 commit 7ec2427

File tree

13 files changed

+802
-0
lines changed

13 files changed

+802
-0
lines changed
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
title: 'Lab: Application Performance Monitoring'
3+
linkTitle: Microserverices APM
4+
weight: 6
5+
---
6+
## Goals
7+
8+
1. Understand GDI path for APM for common tech stacks (Docker, K8s).
9+
1. Be able to instrument an app from scratch (traces, custom metadata).
10+
1. Understand how distributed tracing works across tech stacks (header propagation, …)
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
---
2+
title: Using OpenTelementry in Instrumentation
3+
weight: 9
4+
---
5+
The business teams want to add the service version, the customer profile that is defined by a color and the name of the analyzed file.
6+
7+
Switch to the provided milestone `12microservices-k8s-autoi` with the instructions from "Getting Started".
8+
9+
Implement the requested changes to the `public-api` microservice.
10+
11+
The Kubernetes manifests are located in the `k8s` folder. Add the service version by [configuring the OpenTelemetry resource attributes][splunk-py-otel-cfg].
12+
13+
The customer profile and the file name vary by request. Create attributes and assign them to the current span with the [OpenTelemetry Python API][otel-py-api].
14+
15+
[splunk-py-otel-cfg]: https://github.com/signalfx/splunk-otel-python/blob/main/docs/advanced-config.md#trace-configuration
16+
17+
[otel-py-api]: https://opentelemetry-python.readthedocs.io/en/stable/faq-and-cookbook.html
18+
19+
You can use a random function to generate the customer profile (e.g. red, blue, green) with this snippet:
20+
21+
```python
22+
color = random.choice(['red', 'blue', 'green'])
23+
```
24+
25+
Note 1: Do not use a temporary variable to retrieve the current span. Use the `trace` directly.
26+
27+
Note 2: Make sure to import modules.
28+
29+
Rebuild the container images for the private registry:
30+
31+
{{< tabpane >}}
32+
{{< tab header="Shell Command" lang="bash" >}}
33+
docker-compose build{{< /tab >}}
34+
{{< /tabpane >}}
35+
36+
Push the images to the private registry:
37+
38+
{{< tabpane >}}
39+
{{< tab header="Shell Command" lang="bash" >}}
40+
docker-compose push{{< /tab >}}
41+
{{< /tabpane >}}
42+
43+
Delete the `public-api` deployment:
44+
45+
{{< tabpane >}}
46+
{{< tab header="Shell Command" lang="bash" >}}
47+
kubectl delete deploy public-api{{< /tab >}}
48+
{{< /tabpane >}}
49+
50+
Redeploy to the cluster with
51+
52+
{{< tabpane >}}
53+
{{< tab header="Shell Command" lang="bash" >}}
54+
kubectl apply -f k8s{{< /tab >}}
55+
{{< /tabpane >}}
56+
57+
Test the service with
58+
59+
{{< tabpane >}}
60+
{{< tab header="Shell Command" lang="bash" >}}
61+
ENDPOINT=$(kubectl get service/public-api -o jsonpath='{.spec.clusterIP}')
62+
curl http://$ENDPOINT:8000/api -F text=@hamlet.txt{{< /tab >}}
63+
{{< /tabpane >}}
64+
65+
Verify in Splunk APM that traces contain the desired informations: TODO screenshot
66+
67+
[Create a new indexed span tag][index-span-tag] so that the business team is able to breakdown performance per customer profile.
68+
69+
The milestone for this task is `13custom-instr`. It adds the described custom instrumentation.
70+
71+
[span-tag]: (https://docs.splunk.com/Observability/apm/span-tags/index-span-tags.html#index-a-new-span-tag)
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
title: Instrumentation in Kubernetes
3+
weight: 7
4+
---
5+
TODO Note on .env being overwritten
6+
7+
TODO change name of environment from YOURENV to something else.
8+
9+
The development team has started using Kubernetes for container orchestration. Switch to the provided milestone `12microservices-k8s` with the instructions from "Getting Started".
10+
11+
The Kubernetes manifests are located in the `k8s` folder. Add auto-instrumentation to the `public_api` microservice `deployment` by configuring the [Splunk distribution of OpenTelemetry Python][splunk-otel-python]. The `Dockerfile` has already been prepared.
12+
13+
Install the OpenTelemetry Collector to the environment using [Splunk's helm chart][splunk-otel-helm] and use the provided `values.yaml`:
14+
15+
{{< tabpane >}}
16+
{{< tab header="Shell Command" lang="bash" >}}
17+
helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart
18+
19+
helm install my-splunk-otel-collector --set="splunkObservability.realm=${SPLUNK_REALM},splunkObservability.accessToken=${SPLUNK_ACCESS_TOKEN},clusterName=${CLUSTER_NAME}" splunk-otel-collector-chart/splunk-otel-collector -f values.yaml{{< /tab >}}{{< /tabpane >}}
20+
21+
Rebuild the container images for the private registry:
22+
23+
{{< tabpane >}}
24+
{{< tab header="Shell Command" lang="bash" >}}
25+
docker-compose build{{< /tab >}}
26+
{{< /tabpane >}}
27+
28+
Push the images to the private registry:
29+
30+
{{< tabpane >}}
31+
{{< tab header="Shell Command" lang="bash" >}}
32+
docker-compose push{{< /tab >}}
33+
{{< /tabpane >}}
34+
35+
Deploy to the cluster with
36+
37+
{{< tabpane >}}
38+
{{< tab header="Shell Command" lang="bash" >}}
39+
kubectl apply -f k8s{{< /tab >}}
40+
{{< /tabpane >}}
41+
42+
Test the service with
43+
44+
{{< tabpane >}}
45+
{{< tab header="Shell Command" lang="bash" >}}
46+
ENDPOINT=$(kubectl get service/public-api -o jsonpath='{.spec.clusterIP}')
47+
curl http://$ENDPOINT:8000/api -F text=@hamlet.txt{{< /tab >}}
48+
{{< /tabpane >}}
49+
50+
The milestone for this task is `12microservices-k8s-autoi`. It has auto-instrumentation applied for *all* microservices.
51+
52+
[splunk-otel-helm]: https://github.com/signalfx/splunk-otel-collector-chart
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
---
2+
title: Kubernetes
3+
weight: 21
4+
---
5+
The development team has started using [Kubernetes][kubernetes] for container orchestration. Switch to the provided milestone `09k8s` with the instructions from "Getting Started".
6+
7+
Rebuild the container images for the private registry:
8+
9+
{{< tabpane >}}
10+
{{< tab header="Shell Command" lang="bash" >}}
11+
docker-compose build{{< /tab >}}
12+
{{< /tabpane >}}
13+
14+
Push the images to the private registry:
15+
16+
{{< tabpane >}}
17+
{{< tab header="Shell Command" lang="bash" >}}
18+
docker-compose push
19+
{{< /tab >}}
20+
{{< /tabpane >}}
21+
22+
Then deploy the services into the cluster:
23+
24+
{{< tabpane >}}
25+
{{< tab header="Shell Command" lang="bash" >}}
26+
kubectl apply -f k8s
27+
{{< /tab >}}
28+
{{< /tabpane >}}
29+
30+
Test the service with
31+
32+
{{< tabpane >}}
33+
{{< tab header="Shell Command" lang="bash" >}}
34+
ENDPOINT=$(kubectl get service/wordcount -o jsonpath='{.spec.clusterIP}')
35+
curl http://$ENDPOINT:8000/wordcount -F text=@hamlet.txt
36+
{{< /tab >}}
37+
{{< /tabpane >}}
38+
39+
Configure and install an OpenTelemetry Collector using [Splunk\'s helm chart][splunk-otel-helm]:
40+
41+
1. Review the [configuration how-to][otel-docs] and the [advanced configuration][otel-adv-cfg] to create a `values.yaml` that adds the required receivers for redis and prometheus.
42+
43+
1. Use the environment variables for realm,token and cluster name and pass them to `helm` as arguments.
44+
45+
The milestone for this task is `09k8s-otel`.
46+
47+
[kubernetes]: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
48+
[splunk-otel-helm]: https://github.com/signalfx/splunk-otel-collector-chart
49+
[otel-docs]: https://github.com/signalfx/splunk-otel-collector-chart#how-to-install
50+
[otel-adv-cfg]: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/advanced-configuration.md
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: Monitor Containerized Services
3+
weight: 19
4+
---
5+
The development team has started using other containerized services with docker compose. Switch to the provided milestone `08docker-compose-redis` with the instructions from "Getting Started".
6+
7+
Add the [redis monitor][redis-mon] to the OpenTelemetry Collector configuration in `collector.yaml` to get metrics from the [redis cache].
8+
9+
Rebuild the docker-compose stack and run it.
10+
11+
Check that you are getting data in the Redis dashboard:
12+
13+
![Redis dashboard](../../../images/redis-dashboard.png)
14+
15+
The milestone for this task is `08docker-compose-redis-otel`.
16+
17+
[redis]: https://redis.io/
18+
[redis-mon]: https://docs.splunk.com/Observability/gdi/redis/redis.html
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
title: Add Prometheus metrics
3+
weight: 7
4+
---
5+
We need visibility into performance - let us add metrics with [Prometheus][prometheus].
6+
7+
Install the [Python Prometheus client][py-prom] as a dependency:
8+
9+
{{< tabpane >}}
10+
{{< tab header="Shell Command" lang="bash" >}}
11+
echo "prometheus-client" >> requirements.txt
12+
python3 -m venv .venv
13+
source .venv/bin/activate
14+
.venv/bin/pip install -r requirements.txt
15+
{{< /tab >}}
16+
{{< /tabpane >}}
17+
18+
Import the modules by editing `app.py`. These imports go towards the top of the file:
19+
20+
```python
21+
import prometheus_client
22+
from prometheus_client.exposition import CONTENT_TYPE_LATEST
23+
from prometheus_client import Counter
24+
```
25+
26+
Define a metrics endpoint before `@app.route('/wordcount', methods=['POST'])`:
27+
28+
```python
29+
@app.route('/metrics')
30+
def metrics():
31+
return Response(prometheus_client.generate_latest(), mimetype=CONTENT_TYPE_LATEST)
32+
```
33+
34+
And use this python snippet after `app = Flask(__name__)` to define a new counter metric:
35+
36+
```python
37+
c_recv = Counter('characters_recv', 'Number of characters received')
38+
```
39+
40+
Increase the counter metric after `data = request.files['text'].read().decode('utf-8')`:
41+
42+
```python
43+
c_recv.inc(len(data))
44+
```
45+
46+
Test that the application exposes metrics by hitting the endpoint while the app is running:
47+
48+
{{< tabpane >}}
49+
{{< tab header="Shell Command" lang="bash" >}}
50+
curl http://127.0.0.1:5000/metrics{{< /tab >}}
51+
{{< /tabpane >}}
52+
53+
The milestone for this task is `02service-metrics`.
54+
55+
[prometheus]: https://prometheus.io/docs/introduction/overview/#architecture
56+
[py-prom]: https://pypi.org/project/prometheus-client/
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
title: Adding Notes and Dashboard Layout
3+
linkTitle: Notes and layout
4+
weight: 13
5+
---
6+
7+
## 1. Adding Notes
8+
9+
Often on dashboards it makes sense to place a short "instruction" pane that helps users of a dashboard.
10+
11+
Lets add one now by clicking on the {{% labelbutton color="ui-button-blue" %}}New Text Note{{% /labelbutton %}}
12+
Button.
13+
14+
![three charts](../../../images/M-Notes-0.png)
15+
16+
This will open the notes editor.
17+
18+
![Notes 1](../../../images/M-Notes-1.png)
19+
20+
To allow you to add more then just text to you notes, Splunk is allowing you to use Markdown in these notes/panes.
21+
Markdown is a lightweight markup language for creating formatted text using plain-text often used in Webpages.
22+
23+
This includes (but not limited to):
24+
25+
* Headers. (in various sizes)
26+
* Emphasis styles.
27+
* Lists and Tables.
28+
* Links. These can be external webpages (for documentation for example) or directly to other Splunk IMT Dashboards
29+
30+
Below is an example of above Markdown options you can use in your note.
31+
32+
{{< tabpane >}}
33+
{{< tab header="Sample Markdown text" lang="markdown" >}}
34+
35+
# h1 Big headings
36+
37+
###### h6 To small headings
38+
39+
##### Emphasis
40+
41+
**This is bold text**, *This is italic text* , ~~Strikethrough~~
42+
43+
##### Lists
44+
45+
Unordered
46+
47+
+ Create a list by starting a line with `+`, `-`, or `*`
48+
- Sub-lists are made by indenting 2 spaces:
49+
- Marker character change forces new list start:
50+
* Ac tristique libero volutpat at
51+
+ Facilisis in pretium nisl aliquet
52+
* Very easy!
53+
54+
Ordered
55+
56+
1. Lorem ipsum dolor sit amet
57+
2. Consectetur adipiscing elit
58+
3. Integer molestie lorem at massa
59+
60+
##### Tables
61+
62+
| Option | Description |
63+
| ------ | ----------- |
64+
| chart | path to data files to supply the data that will be passed into templates. |
65+
| engine | engine to be used for processing templates. Handlebars is the default. |
66+
| ext | extension to be used for dest files. |
67+
68+
#### Links
69+
70+
[link to webpage](https://www.splunk.com)
71+
72+
[link to dashboard with title](https://app.eu0.signalfx.com/#/dashboard/EaJHrbPAEAA?groupId=EaJHgrsAIAA&configId=EaJHsHzAEAA "Link to the Sample chart Dashboard!")
73+
{{< /tab >}}
74+
{{< /tabpane >}}
75+
76+
Copy the above by using the copy button and paste it in the *Edit* box.
77+
the preview will show you how it will look.
78+
79+
---
80+
81+
## 2. Saving our chart
82+
83+
Give the Note chart a name, in our example we used *Example text chart*, then press the {{< labelbutton >}}Save And Close{{< /labelbutton >}} Button.
84+
85+
![saving note](../../../images/M-Notes-2.png)
86+
87+
This will bring you back to you Dashboard, that now includes the note.
88+
89+
![three charts and note](../../../images/M-Notes-3.png)
90+
91+
---
92+
93+
## 3. Ordering & sizing of charts
94+
95+
If you do not like the default order and sizes of your charts you can simply use window dragging technique to move and size them to the desired location.
96+
97+
Grab the **top** border of a chart and you should see the mouse pointer change to a drag icon (see picture below).
98+
99+
![dragging charts](../../../images/M-Notes-4.png)
100+
101+
Now drag the **Latency vs Load** chart to sit between the **Latency History** Chart and the **Example text chart**.
102+
103+
![sizing](../../../images/M-Notes-5.png)
104+
105+
You can also resize windows by dragging from the left, right and bottom edges.
106+
107+
As a last exercise reduce the width of the note chart to about a third of the other charts. The chart will automatically snap to one of the sizes it supports. Widen the 3 other charts to about a third of the Dashboard. Drag the notes to the right of the others and resize it to match it to the 3 others.
108+
109+
Set the time to -1 h hour and you should have the following dashboard!
110+
111+
![TaDA!](../../../images/M-Notes-6.png)
112+
113+
On to Detectors!
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Working with Detectors
3+
linkTitle: Detectors
4+
weight: 5
5+
description: >
6+
**10 minutes**
7+
---

0 commit comments

Comments
 (0)