Improve Grafana Alloy useability (#8)

* Improve alloy docker image. Cleanup folders

* Update documentation

* Move dashboards into folders

Author: alhendrickson

docs/observability/customization/custom-prometheus-configs.md

@@ -12,7 +12,7 @@ Mount these files under `site/prometheus/scrape-configs/exporters/*.yml` in dock
 
 
 ### Add Exporters to Prometheus
-- `prometheus/scrape-configs/exporters/` -
+- `prometheus/scrape-configs/exporters/`
  Add yaml files into this folder. These file should contain all exporter prometheus metrics, for example from node_exporter or CAdvisor. Add any hosts and ip addresses you want to collect /metrics from will be retrieved
 
 ```yaml 
@@ -26,6 +26,9 @@ Mount these files under `site/prometheus/scrape-configs/exporters/*.yml` in dock
     # __metrics_path__: /path/metrics  # Optionally override the metrics path, the default is just /metrics
 # ... add all targets
 ```
+
+Note that this project is setup to run Grafana Alloy to push metrics from individual VMs. The config in `prometheus/scrape-configs/exporters/` is an alternative way, allowing you to pull metrics from any other services.
+
 ## Custom Prometheus Scrape Configs
 
 You can add compeltely custom prometheus scrape configs and recording rules by mounting in docker.

docs/observability/get-started/quickstart.md

@@ -28,7 +28,7 @@ If you can't use the script, see the [Manual Quickstart](../reference/quickstart
 ### Optional Step: Probe your own web page
 Now you can look at getting monitoring of your own page
 
-1. In your current folder, in the file `prometheus/scrape-configs/probers/probe-simple.yml` add the following yml to the bottom of the file:
+1. In your current folder, in the file `alloy/probers/probe-observability.yml` add the following yml to the bottom of the file:
 
 ```yaml
 - targets:
@@ -52,11 +52,7 @@ This is the end of this quickstart tutorial, that enables probing availability o
 
 For the next steps we can:
 - Look deeper into the observability dashboards, on [Dashboards Userguide](./userguide-tutorial.md)
-- Productionise our deployment to enable further features
-- Configure *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters
-- Enable *Alerting* based on our availability and a defined Service Level Objective (SLO)
-- Setup further *Probing* of our running services to get availability metrics
-- Fully customize the stack with our own dashboards, recording rules and metrics
+- Productionise our deployment to enable further features  by following [Production Setup](../setup/production-setup.md)
 
 
 

docs/observability/setup/production-setup.md

@@ -6,6 +6,12 @@ If you're new, we recommend completing the [Quickstart Tutorial](../get-started/
 
 By the end of the tutorial, you will have a complete stack offering all the observability features, customized to your usage. 
 
+We will run the stack and then:
+- Configure *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters
+- Enable *Alerting* based on our availability and a defined Service Level Objective (SLO)
+- Setup further *Probing* of our running services to get availability metrics
+
+
 ---
 
 ## Step 1: Understand the Folder Structure
@@ -44,7 +50,7 @@ Downloads the example docker compose files:
 
 Downloads the configurations:
 - [alloy/probers/probe-external.yml](../../../observability/examples/full/alloy/probers/probe-external.yml)
-- [alloy/probers/probe-internal.yml ](../../../observability/examples/full/alloy/probers/probe-internal.yml)
+- [alloy/probers/probe-observability.yml ](../../../observability/examples/full/alloy/probers/probe-observability.yml)
 - [prometheus/scrape-configs/exporters/exporters.yml](../../../observability/examples/full/prometheus/scrape-configs/exporters/exporters.yml)
 - [prometheus/scrape-configs/recording-rules/slo.yml](../../../observability/examples/full/prometheus/scrape-configs/recording-rules/slo.yml)
 
@@ -58,10 +64,10 @@ The files come with basic defaults, so we can now run the stack
 
    ```
    docker compose up -d
-   docker compose -f exporters.docker-compose.yml up -d
    ```
 
-This will launch Prometheus, Grafana, and all required services with 
+This will launch Prometheus, Grafana, and Alloy
+
 
 
 ## Step 4: Create Site-Specific Config Files
@@ -70,21 +76,24 @@ You must provide your own scrape and recording rules to tell Prometheus what to
 This is probably the hardest step: You will actually need to know what is running, and where it is! Building out these config files will give you that inventory, and give a real definition of what is running where.
 
 - Probers: HTTP endpoints you want to monitor for availability
-  - Add files in `scrape-configs/probers/*.yml`
+  - Add files in `alloy/probers/*.yml`
   - [Configure Probers](./probing.md)
-  
-- Exporters: Targets like Elasticsearch or Docker
-  - Add files in `scrape-configs/exporters/*.yml`
-  - [Add Exporters](./telemetry.md)
+
+- Telemetry: Run Grafana Alloy on every VM you want telemetry from
+    - [Configure Telemetry](./probing.md)
 
 - Recording Rules: Define uptime goals or custom aggregations
   - Add files in `recording-rules/*.yml`
   - [Enable Alerting](./alerting.md)
 
-## Step 5: Run Exporters Everywhere
-The exporters need to be run on each VM that you want information from. It's a pull model, not push.
+## Step 5: Run Grafana Alloy on every VM
+The Grafana Alloy image needs to be run on each VM that you want to get information from. 
 
+Use the example docker compose file in [exporters.docker-compose.yml](../../../observability/examples/full/exporters.docker-compose.yml) which will start up alloy and get metrics
 
+   ```
+   docker compose -f exporters.docker-compose.yml up -d
+   ```
 ---
 
 ## What’s Next?
@@ -93,7 +102,7 @@ Your observability stack is now monitoring your services, and you have a product
 
 You can now setup prometheus with any telemetry or probers required following the remaining steps in [Setup](./_index.md)
 
-For the last steps, you can 
+For the last steps, you can:
 
 - Run the exporters on all the VMs that you want access to 
 - Deploy the stack in produciton

docs/observability/setup/telemetry.md

@@ -10,7 +10,6 @@ Grafana Alloy is used to get telemetry. These features are configured by default
 - Elastic Search Exporter: Get ES metrics like index size
 - CAdvisor: This gives docker metrics, eg what containers are running
 
-
 ## How to get Telemetry
 
 - Copy this docker compose file: (exporters.docker-compose.yml)[observability/examples/full/exporters.docker-compose.yml]

observability/examples/alloy/docker-compose.yml

@@ -0,0 +1,12 @@
+- targets:
+    - cogstack-observability-traefik-1/grafana/api/health
+  labels:
+    name: grafana
+    job: probe-observability-stack
+    host: localhost
+- targets:
+    - cogstack-observability-traefik-1/prometheus/-/healthy
+  labels:
+    name: prometheus
+    job: probe-observability-stack
+    host: localhost

observability/examples/alloy/prometheus/scrape-configs/probers/probe-internal.yml

@@ -6,6 +6,8 @@ services:
     image: cogstacksystems/cogstack-observability-alloy:latest
     ports:
       - "12345:12345"
+    networks:
+      - observability
     volumes:
       - ${BASE_DIR-.}/alloy/probers:/etc/alloy/probers
       # CAdvisor