Refactor advanced notes2

Author: alhendrickson

docs/observability/customization/alerts-customization.md

@@ -0,0 +1,44 @@
+# Alerts Customization
+
+You can further setup and customize alerts in the stack.
+
+## Customize Alert Contact points
+If you want alerts to go to a different contact, for example an Email address instead of slack, you can customize where alerts are sent by defining a new contact point in Grafana:
+
+```
+notifiers:
+  - name: "custom-contact"
+    type: "slack"
+    settings:
+      url: "https://hooks.slack.com/services/..."
+```
+
+Mount this file into:
+
+```
+/etc/grafana/provisioning/alerting/custom-contact.yml
+```
+
+Then update the environment variable:
+
+```
+ALERTING_DEFAULT_CONTACT=custom-contact
+```
+
+**Note** to be only mount the exact file, and not override the whole provisioning folder in the image, as this is already used to contain the defaults. 
+
+---
+
+## Add Custom Alerts
+You can make custom alerts.
+
+To define additional alert rules, create files in:
+
+```
+/etc/grafana/provisioning/alerting/
+```
+
+Grafana will automatically load these at startup.
+
+---
+

docs/observability/customization/blackbox-exporter-config.md

@@ -0,0 +1,81 @@
+# Prober Customizations
+
+## How to add Auth to the prober or further configurations?
+
+To define how a probe behaves (e.g., add basic auth, headers, timeout, method), we will configure a module in the Blackbox Exporter config.
+
+#### Create a Blackbox Exporter Config file
+You will need to create a new file, and then mount it over the existing provided vconfig
+
+
+1. Create a new file:
+
+   ```
+    prometheus/blackbox-exporter/custom-blackbox-config.yml
+   ```
+
+2. Add the existing defaults
+
+```  
+modules:
+    http_get_200:
+        prober: http
+        timeout: 5s
+        http:
+        valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
+        valid_status_codes: [200]  # Defaults to 2xx
+        method: GET
+        preferred_ip_protocol: "ip4" # defaults to "ip6"
+        tls_config:
+            insecure_skip_verify: true
+```
+
+3. Add your own module to the modules in that file
+```
+  http_2xx_custom:
+    prober: http
+    timeout: 5s
+    http:
+      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
+      valid_status_codes: [200]  # Defaults to 2xx
+      method: GET
+      preferred_ip_protocol: "ip4" # defaults to "ip6"
+      tls_config:
+        insecure_skip_verify: true
+      basic_auth:
+        username: my-user
+        password: example-pass
+```
+
+This example adds a module named `http_2xx_custom` that adds some basic auth credentials
+
+---
+
+#### Reference the new module in your prober config
+
+In your probe YAML file, reference the module in the `module` field of the `labels` section:
+
+```
+    - targets:
+        - https://myservice.example.com/health
+      labels:
+        name: my-service
+        module: http_2xx_custom      # Optional - overrides the default Blackbox module
+```
+
+#### Mount the config file
+You lastly need to mount the new config file and refer to it in docker compose
+
+```
+  blackbox-exporter:
+    image: cogstacksystems/cogstack-observability-blackbox-exporter:latest
+    restart: unless-stopped
+    networks:
+      - observability
+    volumes:
+      - ./prometheus/blackbox-exporter:/config
+    command:
+      - "--config.file=/config/custom-blackbox-config.yml" 
+```
+
+---

docs/observability/setup/production-setup.md

@@ -62,8 +62,10 @@ This will launch Prometheus, Grafana, and all required services with
 
 
 ## Step 4: Create Site-Specific Config Files
-
 You must provide your own scrape and recording rules to tell Prometheus what to monitor.
+
+This is probably the hardest step: You will actually need to know what is running, and where it is! Building out these config files will give you that inventory, and give a real definition of what is running where.
+
 - Probers: HTTP endpoints you want to monitor for availability
   - Add files in `scrape-configs/probers/*.yml`
   - [Configure Probers](./probing.md)
@@ -76,6 +78,9 @@ You must provide your own scrape and recording rules to tell Prometheus what to
   - Add files in `recording-rules/*.yml`
   - [Enable Alerting](./alerting.md)
 
+## Step 5: Run Exporters Everywhere
+The exporters need to be run on each VM that you want information from. It's a pull model, not push.
+
 
 ---
 
@@ -87,5 +92,7 @@ You can now setup prometheus with any telemetry or probers required following th
 
 For the last steps, you can 
 
+- Run the exporters on all the VMs that you want access to 
+- Deploy the stack in produciton
 - Fully customise with [Customization](../customization/_index.md)
 - Look further into understanding the concepts and details in [Reference](../reference/_index.md)