Begin writing documentation for public users

alhendrickson · alhendrickson · commit 5ac71a89f660 · 2025-06-03T14:12:57.000Z
diff --git a/.readthedocs.yml b/.readthedocs.yml
diff --git a/docs/observability/_index.md b/docs/observability/_index.md
@@ -16,6 +16,8 @@ See the [Quickstart](./get-started/quickstart.md) to see how to easily run this
 :maxdepth: 2
 
 get-started/_index
-
+setup/_index
+customization/_index
+reference/_index
 
 ```
diff --git a/docs/observability/customization/_index.md b/docs/observability/customization/_index.md
@@ -0,0 +1,19 @@
+# Customization
+
+```{include} custom-dashboards.md
+:heading-offset: 1
+```
+
+```{include} custom-prometheus-configs.md
+:heading-offset: 1
+```
+
+
+```{toctree}
+:titlesonly:
+:hidden:
+
+custom-prometheus-configs.md
+custom-dashboards.md
+
+```
diff --git a/docs/observability/customization/custom-dashboards.md b/docs/observability/customization/custom-dashboards.md
@@ -0,0 +1,14 @@
+# Custom Dashboards
+
+Grafana is setup with preconfigured dashboards, datasource, and alerting. These will work when prometheus is run in this stack, and is dependent on all the metrics following defined rules. 
+
+it is advised that any edits or new configs get committed back into your git repository, and stick with grafana provisioning instead of allowing manual edits
+
+
+## How to add a new dashboard with provisioning 
+
+- Mount new dashboard files in the `/etc/grafana/provisioning/dashboards/site` directory
+- To remove or change the existing, mount over the existing files there
+
+For more info see [Grafana Alerting Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards)
+
diff --git a/docs/observability/customization/custom-prometheus-configs.md b/docs/observability/customization/custom-prometheus-configs.md
@@ -0,0 +1,19 @@
+# Custom Prometheus Configuration
+
+
+You can add compeltely custom prometheus scrape configs and recording rules by mounting in docker.
+
+
+
+- `site/prometheus/scrape-configs/*.yml`. This is for advanced configuration. Any yml file put in this directory will be used as standard promethues scrape configs. This will give full flexibility over what metrics are collected and all features in prometheus. Add any further configs that you want prometheus to use.
+
+```yaml
+# Custom scrape config definition
+scrape_configs:
+  - job_name: custom-scrape-config # Scrape configuration to get metrics from elasticsearch, eg index size.
+    static_configs:
+      - targets:
+          - my-custom-target:9114
+        labels:
+          custom_label: custom # (Optional)
+```
diff --git a/docs/observability/get-started/_index.md b/docs/observability/get-started/_index.md
@@ -3,6 +3,5 @@
 ```{toctree}
 :maxdepth: 2
 quickstart
-setup-tutorial
 userguide-tutorial
 ```
diff --git a/docs/observability/get-started/quickstart.md b/docs/observability/get-started/quickstart.md
@@ -22,13 +22,13 @@ Now go to "http://localhost/grafana" to see the dashboards
 
 Thats everything. The stack is running and you can see the availability.
 
+If you can't use the script, see the [Manual Quickstart](../advanced-usage/quickstart-manual.md) to setup your own files. 
+
 
 ### Optional Step: Probe your own web page
 Now you can look at getting monitoring of your own page
 
-In your current folder, edit the file `prometheus/scrape-configs/probers/probe-simple.yml` that you downloaded from git.
-
-Add the following yml to the bottom of the file:
+1. In your current folder, in the file `prometheus/scrape-configs/probers/probe-simple.yml` add the following yml to the bottom of the file:
 
 ```yaml
 - targets:
@@ -38,22 +38,24 @@ Add the following yml to the bottom of the file:
     job: probe-my-own-site
 ```
 
+Note to be careful of the indentation in yml, this target must be at the same depth as the existing contents. 
 
-The change should get applied automatically, but if you dont want to wait then run
+2. Restart the containers with:
 ```
 docker compose restart
 ```
 
 Now refresh the grafana dashboard, and you can see the availability of google.com, it's probably 100%!
 
-
 ## Next steps
 This is the end of this quickstart tutorial, that enables probing availability of endpoints.
 
 For the next steps we can:
+
 - Productionise our deployment to enable further features
-- Enable *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters
+- Configure *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters
 - Enable *Alerting* based on our availability and a defined Service Level Objective (SLO)
+- Setup further *Probing* of our running services to get availability metrics
 - Look further into the available dashboards
 - Fully customize the stack with our own dashboards, recording rules and metrics
 
diff --git a/docs/observability/get-started/setup-tutorial.md b/docs/observability/get-started/setup-tutorial.md
diff --git a/docs/observability/get-started/userguide-tutorial.md b/docs/observability/get-started/userguide-tutorial.md
@@ -1,3 +1,11 @@
 # Dashboard User Guide
 
-This is my dashboard user guide
+This is my dashboard user guide
+
+
+## Grafana Dashboards
+
+- Availability
+- Elasticsearch
+- VM Metrics (Memory use, CPU etc)
+- Docker Metrics (Running containers)
diff --git a/docs/observability/reference/_index.md b/docs/observability/reference/_index.md
@@ -0,0 +1,8 @@
+# Reference
+
+```{toctree}
+:maxdepth: 2
+
+quickstart-manual.md
+
+```
diff --git a/docs/observability/reference/quickstart-manual.md b/docs/observability/reference/quickstart-manual.md
diff --git a/docs/observability/setup/_index.md b/docs/observability/setup/_index.md
@@ -0,0 +1,12 @@
+# Setup
+
+```{toctree}
+:maxdepth: 2
+
+full-installation.md
+probing.md
+telemetry.md
+alerts.md
+
+
+```
diff --git a/docs/observability/setup/alerts.md b/docs/observability/setup/alerts.md
@@ -0,0 +1,38 @@
+# Alerting
+TODO
+
+The alerts are paused by default.
+
+Alerting is based on either pure availability on 5 minutes or 6 hours, as well as a burn rate implementation. 
+
+See [Google SRE Guide](https://sre.google/workbook/alerting-on-slos/#4-alert-on-burn-rate) which explains burn rate alerting. The alerting setup here follows the recommendations in the SRE handbook for Multiwindow, Multiburn rate alerting.
+
+For burn rate alerting, ensure that a recording rule is setup to create a record for `slo_target_over_30_days`, with a job label that matches your probe job labels. See the prometheus readme in this project. 
+
+## Define a SLO
+To enable the burn rate alerting feature, create prometheus recording rule file with the following contents.
+
+```yaml
+groups:
+  - name:  slo-target-rules
+    rules:
+      - record: slo_target_over_30_days # (Dont change)
+        expr: 0.95 # Mandatory - Specify the SLO you want to target, for example 0.95 for 95% uptime over 30 days
+        labels:
+          job: "probe-cogstack-availability" #Mandatory - name the job, which must match the job in the probe targets defined
+```
+
+In docker, mount the file in `site/prometheus/recording-rules/slo.yml`.
+
+
+## Turn on alerting
+- Enable/Disable alerts using environment variables 
+- By default alerts will send to slack. Provide the env variable `SLACK_WEBHOOK_URL` to go there
+
+
+## Configuration
+- To change where the alerts are sent: create and mount custom a custom contact point in `/etc/grafana/provisioning/alerting/custom-contact.yml`. Then change the environment variable `ALERTING_DEFAULT_CONTACT` to use that name
+- Add custom alerts by mounting alert files in `/etc/grafana/provisioning/alerting/`.
+
+For more info see [Grafana Provisioning](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/)
+
diff --git a/docs/observability/setup/full-installation.md b/docs/observability/setup/full-installation.md
@@ -0,0 +1,41 @@
+# Production Setup
+This page shows how to setup the stack for a production deployment.
+
+If you havent already done it, do see the quickstart tutorial
+
+
+## Run the Observability Stack using Docker Compose
+
+See the /examples folder for a working example of running this
+
+To setup the stack for your deployment:
+1. Create prometheus configurations as listed below
+1. Copy this docker-compose.yml file
+1. Mount your site config files into `/etc/prometheus/cogstack/site` 
+1. Run with docker compose
+
+To collect metrics from VMs to fill out the dashboards
+- Run the Exporters on each VM as detailed below
+
+
+## Prometheus Configurations
+
+Prometheus is preconfigured with rules and configs, though needs additional site specific configs to scrape your stack. For example, it needs to be told exactly what hostname and port elasticsearch is running on, in order to check availability. 
+
+This is managed through using different prometheus configuration files, which include different scrape config files.
+
+Prometheus is used to get metrics from targets. It can target anything exporting metrics.
+
+To use this project, you need to define your own targets in yml files and mount in docker following this folder structure:
+
+```
+  site/
+    scrape-configs/exporters/*.yml
+    scrape-configs/probers/*.yml
+    scrape-configs/*.yml
+    recording-rules/*yml
+```
+
+Note that all the URLs need to be accessible from the host running prometheus
+
+
diff --git a/docs/observability/setup/probing.md b/docs/observability/setup/probing.md
@@ -0,0 +1,19 @@
+# Availability Probing
+
+
+
+- `site/prometheus/scrape-configs/probers/*.yml`. 
+Add yaml files to this folder as probe targets. Any yml files put into this directory, for example "probe.example.yml", will be used as targets to probe for availability using blackbox exporter. Add any URLs that you want to measure the availability of. 
+
+```yaml 
+# Prober yml
+- targets:
+    - https://google.com/something
+  labels:
+    name: google-homepage # Mandatory - the name of the service being probed
+    job: override_job # (Optional. Default is "probe-cogstack-availability") Customise a job to enable grouping in the dashboard
+    ip_address: "123.0.0.1" # (Optional) The IP address
+    host: a_hostname # (Optional) A readable hostname
+    custom_label: a_custom_label # (Optional)  Any other label
+    
+```
diff --git a/docs/observability/setup/telemetry.md b/docs/observability/setup/telemetry.md
@@ -0,0 +1,34 @@
+# Telemetry
+We can get telemetry from our services and VMs displayed in our dashboards. This telemetry gives us things like memory usage, and running container versions.
+
+Using telemetry lets us get feedback from the stack, diagnose problems, and predict issues before they occur.
+
+## Prometheus Exporters
+Prometheus gets metrics from "Exporters". These need to be run on each VM you want to get metrics from
+
+Run these exporters on each virtual machine.
+
+- Node Exporter: This gives host metrics eg disk usage, memory
+- Elastic Search Exporter: Get ES metrics like index size
+- CAdvisor: This gives docker metrics, eg what containers are running
+
+ Then add the host and IPs into a yaml file in `scrape-configs/exporters/` to tell prometheus where to scrape.
+
+
+## How to run Exporters
+
+## Add Exporters to Prometheus
+- `scrape-configs/exporters/` -
+ Add yaml files into this folder. These file should contain all exporter prometheus metrics, for example from node_exporter or CAdvisor. Add any hosts and ip addresses you want to collect /metrics from will be retrieved
+
+```yaml 
+# Exporter example yml
+- targets:
+    - 123.0.0.1:9100 # Enter your IP address and port of a target
+  labels:
+    job: node_exporter # Mandatory  - Enter the type of metric being collected
+    host: my-host-name  # (Optional) A readable hostname
+    custom_label: a_custom_label # (Optional)
+    # __metrics_path__: /path/metrics  # Optionally override the metrics path, the default is just /metrics
+# ... add all targets
+```
diff --git a/observability/README.md b/observability/README.md
