Update documentation

Author: alhendrickson

docs/observability/customization/custom-prometheus-configs.md

@@ -12,7 +12,7 @@ Mount these files under `site/prometheus/scrape-configs/exporters/*.yml` in dock
 
 
 ### Add Exporters to Prometheus
-- `prometheus/scrape-configs/exporters/` -
+- `prometheus/scrape-configs/exporters/`
  Add yaml files into this folder. These file should contain all exporter prometheus metrics, for example from node_exporter or CAdvisor. Add any hosts and ip addresses you want to collect /metrics from will be retrieved
 
 ```yaml 
@@ -26,6 +26,9 @@ Mount these files under `site/prometheus/scrape-configs/exporters/*.yml` in dock
     # __metrics_path__: /path/metrics  # Optionally override the metrics path, the default is just /metrics
 # ... add all targets
 ```
+
+Note that this project is setup to run Grafana Alloy to push metrics from individual VMs. The config in `prometheus/scrape-configs/exporters/` is an alternative way, allowing you to pull metrics from any other services.
+
 ## Custom Prometheus Scrape Configs
 
 You can add compeltely custom prometheus scrape configs and recording rules by mounting in docker.

docs/observability/setup/production-setup.md

@@ -44,7 +44,7 @@ Downloads the example docker compose files:
 
 Downloads the configurations:
 - [alloy/probers/probe-external.yml](../../../observability/examples/full/alloy/probers/probe-external.yml)
-- [alloy/probers/probe-internal.yml ](../../../observability/examples/full/alloy/probers/probe-internal.yml)
+- [alloy/probers/probe-observability.yml ](../../../observability/examples/full/alloy/probers/probe-observability.yml)
 - [prometheus/scrape-configs/exporters/exporters.yml](../../../observability/examples/full/prometheus/scrape-configs/exporters/exporters.yml)
 - [prometheus/scrape-configs/recording-rules/slo.yml](../../../observability/examples/full/prometheus/scrape-configs/recording-rules/slo.yml)
 
@@ -58,10 +58,9 @@ The files come with basic defaults, so we can now run the stack
 
    ```
    docker compose up -d
-   docker compose -f exporters.docker-compose.yml up -d
    ```
 
-This will launch Prometheus, Grafana, and all required services with 
+This will launch Prometheus, Grafana, and Alloy
 
 
 ## Step 4: Create Site-Specific Config Files
@@ -70,21 +69,21 @@ You must provide your own scrape and recording rules to tell Prometheus what to
 This is probably the hardest step: You will actually need to know what is running, and where it is! Building out these config files will give you that inventory, and give a real definition of what is running where.
 
 - Probers: HTTP endpoints you want to monitor for availability
-  - Add files in `scrape-configs/probers/*.yml`
+  - Add files in `alloy/probers/*.yml`
   - [Configure Probers](./probing.md)
 
-- Exporters: Targets like Elasticsearch or Docker
-  - Add files in `scrape-configs/exporters/*.yml`
-  - [Add Exporters](./telemetry.md)
-
 - Recording Rules: Define uptime goals or custom aggregations
   - Add files in `recording-rules/*.yml`
   - [Enable Alerting](./alerting.md)
 
-## Step 5: Run Exporters Everywhere
-The exporters need to be run on each VM that you want information from. It's a pull model, not push.
+## Step 5: Run Grafana Alloy on every VM
+The Grafana Alloy image needs to be run on each VM that you want to get information from. 
 
+Use the example docker compose file in [exporters.docker-compose.yml](../../../observability/examples/full/exporters.docker-compose.yml) which will start up alloy and get metrics
 
+   ```
+   docker compose -f exporters.docker-compose.yml up -d
+   ```
 ---
 
 ## What’s Next?

observability/examples/full/full-quickstart.sh

@@ -10,6 +10,7 @@ download_to() {
   curl -fsSL -o "$path" "$url"
 }
 
+mkdir -p cogstack-observability/alloy/probers
 mkdir -p cogstack-observability/prometheus/scrape-configs/probers
 mkdir -p cogstack-observability/prometheus/scrape-configs/exporters
 mkdir -p cogstack-observability/prometheus/scrape-configs/recording-rules

observability/examples/simple/alloy/probers/probe-simple.yml

@@ -9,8 +9,8 @@ curl -fsSL -o docker-compose.yml \
   https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/docker-compose.yml
 
 echo "Downloading probe-simple.yml into alloy/probers/..."
-curl -fsSL -o probers/probe-simple.yml \
-  https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/probers/probe-simple.yml
+curl -fsSL -o probers/probe-observability.yml \
+  https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/probers/probe-observability.yml
 
 echo "Setup complete in observability-simple/"