Skip to content

Commit 9cc8525

Browse files
alhendricksonalhendrickson
authored
Creating prod documentation (#4)
* Refactor files * Refactor advanced notes
1 parent 5737d4a commit 9cc8525

File tree

6 files changed

+148
-144
lines changed

6 files changed

+148
-144
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# Custom ignores
22
observability/examples/simple/observability-simple
3+
observability/examples/full/cogstack-observability
34
_build
45

56
# Python ignores

‎docs/observability/customization/alerts-customization.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Alerts Customization
2+
3+
You can further setup and customize alerts in the stack.
4+
5+
## Customize Alert Contact points
6+
If you want alerts to go to a different contact, for example an Email address instead of slack, you can customize where alerts are sent by defining a new contact point in Grafana:
7+
8+
```
9+
notifiers:
10+
- name: "custom-contact"
11+
type: "slack"
12+
settings:
13+
url: "https://hooks.slack.com/services/..."
14+
```
15+
16+
Mount this file into:
17+
18+
```
19+
/etc/grafana/provisioning/alerting/custom-contact.yml
20+
```
21+
22+
Then update the environment variable:
23+
24+
```
25+
ALERTING_DEFAULT_CONTACT=custom-contact
26+
```
27+
28+
**Note** to be only mount the exact file, and not override the whole provisioning folder in the image, as this is already used to contain the defaults.
29+
30+
---
31+
32+
## Add Custom Alerts
33+
You can make custom alerts.
34+
35+
To define additional alert rules, create files in:
36+
37+
```
38+
/etc/grafana/provisioning/alerting/
39+
```
40+
41+
Grafana will automatically load these at startup.
42+
43+
---
44+

‎docs/observability/customization/blackbox-exporter-config.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Prober Customizations
2+
3+
## How to add Auth to the prober or further configurations?
4+
5+
To define how a probe behaves (e.g., add basic auth, headers, timeout, method), we will configure a module in the Blackbox Exporter config.
6+
7+
#### Create a Blackbox Exporter Config file
8+
You will need to create a new file, and then mount it over the existing provided vconfig
9+
10+
11+
1. Create a new file:
12+
13+
```
14+
prometheus/blackbox-exporter/custom-blackbox-config.yml
15+
```
16+
17+
2. Add the existing defaults
18+
19+
```
20+
modules:
21+
http_get_200:
22+
prober: http
23+
timeout: 5s
24+
http:
25+
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
26+
valid_status_codes: [200] # Defaults to 2xx
27+
method: GET
28+
preferred_ip_protocol: "ip4" # defaults to "ip6"
29+
tls_config:
30+
insecure_skip_verify: true
31+
```
32+
33+
3. Add your own module to the modules in that file
34+
```
35+
http_2xx_custom:
36+
prober: http
37+
timeout: 5s
38+
http:
39+
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
40+
valid_status_codes: [200] # Defaults to 2xx
41+
method: GET
42+
preferred_ip_protocol: "ip4" # defaults to "ip6"
43+
tls_config:
44+
insecure_skip_verify: true
45+
basic_auth:
46+
username: my-user
47+
password: example-pass
48+
```
49+
50+
This example adds a module named `http_2xx_custom` that adds some basic auth credentials
51+
52+
---
53+
54+
#### Reference the new module in your prober config
55+
56+
In your probe YAML file, reference the module in the `module` field of the `labels` section:
57+
58+
```
59+
- targets:
60+
- https://myservice.example.com/health
61+
labels:
62+
name: my-service
63+
module: http_2xx_custom # Optional - overrides the default Blackbox module
64+
```
65+
66+
#### Mount the config file
67+
You lastly need to mount the new config file and refer to it in docker compose
68+
69+
```
70+
blackbox-exporter:
71+
image: cogstacksystems/cogstack-observability-blackbox-exporter:latest
72+
restart: unless-stopped
73+
networks:
74+
- observability
75+
volumes:
76+
- ./prometheus/blackbox-exporter:/config
77+
command:
78+
- "--config.file=/config/custom-blackbox-config.yml"
79+
```
80+
81+
---

‎docs/observability/setup/alerting.md‎

Lines changed: 1 addition & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -57,48 +57,10 @@ SLACK_WEBHOOK_URL=https://hooks.slack.com/services/your-webhook
5757
---
5858

5959
## Advanced Customization
60-
### Customize Alert Contact points
60+
See [Alerts Customization](../customization/alerts-customization.md) to further customize alerts, for example setting up a different contact, or a new rule.
6161

62-
You can customize where alerts are sent by defining a new contact point in Grafana:
63-
64-
```
65-
notifiers:
66-
- name: "custom-contact"
67-
type: "slack"
68-
settings:
69-
url: "https://hooks.slack.com/services/..."
70-
```
71-
72-
Mount this file into:
73-
74-
```
75-
/etc/grafana/provisioning/alerting/custom-contact.yml
76-
```
77-
78-
Then update the environment variable:
79-
80-
```
81-
ALERTING_DEFAULT_CONTACT=custom-contact
82-
```
83-
84-
**Note** to be only mount the exact file, and not override the whole provisioning folder in the image, as this is already used to contain the defaults.
85-
86-
---
87-
88-
### Add Custom Alerts
89-
To define additional alert rules, create files in:
90-
91-
```
92-
/etc/grafana/provisioning/alerting/
93-
```
94-
95-
Grafana will automatically load these at startup.
96-
97-
---
9862

9963
## Further Reading
10064

10165
* [Grafana Alerting Provisioning](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/)
10266
* [Google SRE – Burn Rate Alerting](https://sre.google/workbook/alerting-on-slos/#4-alert-on-burn-rate)
103-
104-
Let me know if you'd like to split this into multiple focused guides, e.g., one for basic uptime, one for SLO-based alerts.

‎docs/observability/setup/probing.md‎

Lines changed: 1 addition & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -35,86 +35,7 @@ To add a new prober target:
3535
---
3636

3737
## Advanced Setup
38-
39-
### How to add Auth to the prober or further configurations
40-
41-
To define how a probe behaves (e.g., add basic auth, headers, timeout, method), we will configure a module in the Blackbox Exporter config.
42-
43-
#### Create a Blackbox Exporter Config file
44-
You will need to create a new file, and then mount it over the existing provided vconfig
45-
46-
47-
1. Create a new file:
48-
49-
```
50-
prometheus/blackbox-exporter/custom-blackbox-config.yml
51-
```
52-
53-
2. Add the existing defaults
54-
55-
```
56-
modules:
57-
http_get_200:
58-
prober: http
59-
timeout: 5s
60-
http:
61-
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
62-
valid_status_codes: [200] # Defaults to 2xx
63-
method: GET
64-
preferred_ip_protocol: "ip4" # defaults to "ip6"
65-
tls_config:
66-
insecure_skip_verify: true
67-
```
68-
69-
3. Add your own module to the modules in that file
70-
```
71-
http_2xx_custom:
72-
prober: http
73-
timeout: 5s
74-
http:
75-
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
76-
valid_status_codes: [200] # Defaults to 2xx
77-
method: GET
78-
preferred_ip_protocol: "ip4" # defaults to "ip6"
79-
tls_config:
80-
insecure_skip_verify: true
81-
basic_auth:
82-
username: my-user
83-
password: example-pass
84-
```
85-
86-
This example adds a module named `http_2xx_custom` that adds some basic auth credentials
87-
88-
---
89-
90-
#### Reference the new module in your prober config
91-
92-
In your probe YAML file, reference the module in the `module` field of the `labels` section:
93-
94-
```
95-
- targets:
96-
- https://myservice.example.com/health
97-
labels:
98-
name: my-service
99-
module: http_2xx_custom # Optional - overrides the default Blackbox module
100-
```
101-
102-
#### Mount the config file
103-
You lastly need to mount the new config file and refer to it in docker compose
104-
105-
```
106-
blackbox-exporter:
107-
image: cogstacksystems/cogstack-observability-blackbox-exporter:latest
108-
restart: unless-stopped
109-
networks:
110-
- observability
111-
volumes:
112-
- ./prometheus/blackbox-exporter:/config
113-
command:
114-
- "--config.file=/config/custom-blackbox-config.yml"
115-
```
116-
117-
---
38+
See [Prober Customization](../customization/blackbox-exporter-config.md) to do any advanced setup, for example adding Basic Auth to allow the prober to call endpoints that need a username and password
11839

11940
## Notes
12041

‎docs/observability/setup/production-setup.md‎

Lines changed: 20 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
# Production Setup Tutorial
2-
//In Progress
32
This tutorial guides you through setting up the **CogStack Observability Stack** for production use.
43

54
If you're new, we recommend completing the [Quickstart Tutorial](../quickstart.md) first to get a simplified setup running.
@@ -63,41 +62,37 @@ This will launch Prometheus, Grafana, and all required services with
6362

6463

6564
## Step 4: Create Site-Specific Config Files
66-
6765
You must provide your own scrape and recording rules to tell Prometheus what to monitor.
6866

69-
* Exporters: Targets like Elasticsearch or Docker
70-
→ Add files in `scrape-configs/exporters/*.yml`
67+
This is probably the hardest step: You will actually need to know what is running, and where it is! Building out these config files will give you that inventory, and give a real definition of what is running where.
7168

72-
* Probers: HTTP endpoints you want to monitor for availability
73-
→ Add files in `scrape-configs/probers/*.yml`
69+
- Probers: HTTP endpoints you want to monitor for availability
70+
- Add files in `scrape-configs/probers/*.yml`
71+
- [Configure Probers](./probing.md)
72+
73+
- Exporters: Targets like Elasticsearch or Docker
74+
- Add files in `scrape-configs/exporters/*.yml`
75+
- [Add Exporters](./telemetry.md)
7476

75-
* Recording Rules: Define uptime goals or custom aggregations
76-
→ Add files in `recording-rules/*.yml`
77-
78-
Refer to the following How-To guides for creating each config:
79-
80-
* [Configure Probers](./probing.md)
81-
* [Add Exporters](./telemetry.md)
82-
* [Enable Alerting](./alerting.md)
83-
* [Customise Setup](../customization/_index.md)
84-
85-
---
77+
- Recording Rules: Define uptime goals or custom aggregations
78+
- Add files in `recording-rules/*.yml`
79+
- [Enable Alerting](./alerting.md)
8680

81+
## Step 5: Run Exporters Everywhere
82+
The exporters need to be run on each VM that you want information from. It's a pull model, not push.
8783

8884

8985
---
9086

9187
## What’s Next?
9288

93-
Your observability stack is now monitoring your own services.
94-
95-
Continue with:
89+
Your observability stack is now monitoring your services, and you have a production ready project setup
9690

97-
* [Grafana Dashboards](./dashboards.md)
98-
* [Set up Alerts](./alerting.md)
99-
* [Create custom views](../customization/_index.md)
91+
You can now setup prometheus with any telemetry or probers required following the remaining steps in [Setup](./_index.md)
10092

101-
---
93+
For the last steps, you can
10294

103-
Let me know if you'd like to add code snippets for `.yml` examples in each folder.
95+
- Run the exporters on all the VMs that you want access to
96+
- Deploy the stack in produciton
97+
- Fully customise with [Customization](../customization/_index.md)
98+
- Look further into understanding the concepts and details in [Reference](../reference/_index.md)

0 commit comments

Comments
 (0)