fix: update all of the monitoring docs (#1316)

All of the monitoring docs were out of date, this updates them.

Signed-off-by: Kevin Carter <[email protected]>

Author: cloudnull

docs/monitoring-getting-started.md

@@ -1,56 +1,91 @@
-# Getting started with genestack monitoring
+# Getting Started with Genestack Monitoring
 
-In order to begin monitoring your genestack deployment we first need to deploy the core prometheus components
+This guide walks you through setting up a complete monitoring stack for your Genestack deployment. The monitoring system consists of three main layers: metrics collection, visualization, and alerting.
 
-## Install the Prometheus stack
+## Overview
 
-Install [Prometheus](prometheus.md) which is part of the kube-prometheus-stack and includes:
+The Genestack monitoring stack includes:
 
-* Prometheus and the Prometheus operator to manage the Prometheus cluster deployment
-* AlertManager which allows for alerting configurations to be set in order to notify various services like email or PagerDuty for specified alerting thresholds
+- **Prometheus** - Time-series database and metrics collection engine
+- **Grafana** - Visualization and dashboards
+- **AlertManager** - Alert routing and notification management
+- **Metric Exporters** - Service-specific metrics collection for OpenStack components
 
-The [Prometheus](prometheus.md) kube-prometheus-stack will also deploy a couple core metric exporters as part of the stack, those include:
+## Prerequisites
 
-* Node Exporter(Hardware metrics)
-* Kube State Exporter(Kubernetes cluster metrics)
+Before proceeding, ensure you have:
 
-## Install Grafana
+- A running Genestack deployment
+- Helm 3.x installed
+- Access to your Kubernetes cluster with appropriate permissions
 
-We can then deploy our visualization dashboard Grafana
+## Step 1: Install the Prometheus Stack
 
-* [Install Grafana](grafana.md)
+The kube-prometheus-stack is the foundation of your monitoring infrastructure. It deploys and manages the core monitoring components.
 
-Grafana is used to visualize various metrics provided by the monitoring system as well as alerts and logs, take a look at the [Grafana](https://grafana.com/) documentation for more information
+Install Prometheus, which includes:
 
-## Install the metric exporters and pushgateway
+- **Prometheus Operator** - Manages the Prometheus cluster deployment lifecycle
+- **Prometheus Server** - Collects and stores metrics from configured targets
+- **AlertManager** - Handles alerts sent by Prometheus and routes them to notification channels (email, PagerDuty, Slack, etc.)
+- **Node Exporter** - Collects hardware and OS-level metrics from cluster nodes
+- **Kube State Metrics** - Exposes Kubernetes cluster state metrics
 
-Now let's deploy our exporters and pushgateway!
+See the [Prometheus installation guide](prometheus.md) for detailed setup instructions.
 
-* [Mysql Exporter](prometheus-mysql-exporter.md)
-* [RabbitMQ Exporter](prometheus-rabbitmq-exporter.md)
-* [Postgres Exporter](prometheus-postgres-exporter.md)
-* [Memcached Exporter](prometheus-memcached-exporter.md)
-* [Openstack Exporter](prometheus-openstack-metrics-exporter.md)
-* [Pushgateway](prometheus-pushgateway.md)
+## Step 2: Install Grafana
 
-## Next steps
+Grafana provides visualization dashboards for your metrics, alerts, and logs.
 
-### Configure alert manager
+Install Grafana to:
 
-Configure the alert manager to send the specified alerts to slack as an example, see: [Slack Alerts](alertmanager-slack.md)
+- Create custom dashboards for monitoring OpenStack services
+- Visualize metrics collected by Prometheus
+- Set up alert notifications and integrations
+- Analyze logs and trace data
 
-... and more ...
+For more information about Grafana's capabilities, visit the [Grafana](grafana.md).
 
-### Update alerting rules
+## Step 3: Deploy Service-Specific Metric Exporters
 
-Within the genestack repo we can update our custom alerting rules via the alerting_rules.yaml to fit our needs
+With the core monitoring stack in place, deploy exporters to collect metrics from your OpenStack services and infrastructure components. All exporters are available for easy deployment.
 
-View alerting_rules.yaml in:
+## Step 4: Configure AlertManager
 
-``` shell
+Configure AlertManager to send notifications when alerts are triggered. Available integrations include:
+
+- [Slack Alerts](alertmanager-slack.md) - Send alerts to Slack channels
+- Email notifications
+- PagerDuty integration
+- Webhook receivers
+
+## Step 5: Customize Alerting Rules
+
+### Custom Alerting Rules
+
+Genestack includes default alerting rules that can be customized for your environment. To view or modify the custom rules:
+
+```shell
 less /etc/genestack/helm-configs/prometheus/alerting_rules.yaml
 ```
 
-However, many opreators comes with ServiceMonitor and PodMonitor services. These services expose, scrape endpoints
-out of the box. These operators will also provide alerting rules curated for the specific service. See specific
-service install for any monitoring rules. Example: [RabbitMQ Operator Monitoring](infrastructure-rabbitmq.md#rabbitmq-operator-monitoring)
+Edit this file to add, modify, or remove alerting rules based on your operational requirements.
+
+### Operator-Provided Alerting Rules
+
+Many Genestack operators come with built-in ServiceMonitor and PodMonitor resources that automatically:
+
+- Expose scrape endpoints for metrics collection
+- Provide pre-configured alerting rules tailored to the specific service
+
+These operator-managed rules are curated for best practices and don't require manual configuration. For service-specific monitoring details, refer to the individual service documentation. For example: [RabbitMQ Operator Monitoring](infrastructure-rabbitmq.md#rabbitmq-operator-monitoring).
+
+## Next Steps
+
+Once your monitoring stack is deployed:
+
+1. **Access Grafana** - Log in to Grafana and explore the pre-built dashboards
+2. **Verify Metrics Collection** - Check that Prometheus is successfully scraping all targets
+3. **Test Alerting** - Trigger a test alert to verify AlertManager configuration
+4. **Create Custom Dashboards** - Build dashboards specific to your operational needs
+5. **Tune Alert Thresholds** - Adjust alerting rules based on your environment's baseline behavior

docs/prometheus-blackbox-exporter.md

@@ -3,13 +3,12 @@
 Using the blackbox exporter we can gather metrics around uptime, latency, cert expiry and more for our public endpoints.
 The blackbox exporter ideally would be ran outside the cluster but can still provide useful information when deployed within it when combined with alerting and visualizations.
 
+## Installation
 
-#### Install Blackbox Exporter Helm Chart
+??? example "`/opt/genestack/bin/install-prometheus-blackbox-exporter.sh`"
 
+    ``` shell
+    --8<-- "bin/install-prometheus-blackbox-exporter.sh"
+    ```
 
-``` shell
-bin/install-prometheus-blackbox-exporter.sh
-```
-
-!!! success
-    If the installation is successful, you should see the related blackbox exporter pods in the prometheus namespace.
+If the installation is successful, you should see the related Blackbox exporter pods in the prometheus namespace.

docs/prometheus-custom-node-metrics.md

@@ -6,21 +6,25 @@ For more information visit: [Node Exporter Textfile Collectors](https://github.c
 
 You can also view example scripts here: [Textfile Collector Scripts](https://github.com/prometheus-community/node-exporter-textfile-collector-scripts)
 
-
-#### Example custom exporter playbook
+## Example custom exporter playbook
 
 ``` shell
 ansible-playbook custom_exporters.yml
 ```
 
-#### Example custom exporter playbook with overrides
+## Example custom exporter playbook with overrides
 
-Confirm `inventory.yaml` matches what is in /etc/genestack/inventory. If it does not match update the command to match the file names.
+Confirm `inventory.yaml` matches what is in `/etc/genestack/inventory`. If it does not match update the command to match the file names.
 
 ``` shell
 # Example overriding things on the CLI
 source /opt/genestack/scripts/genestack.rc
-ansible-playbook custom_exporters.yml --private-key ${HOME}/.ssh/openstack-keypair.key
 ```
 
+!!! example "Run the playbook"
+
+    ``` shell
+    ansible-playbook custom_exporters.yml --private-key ${HOME}/.ssh/openstack-keypair.key
+    ```
+
 Once the scripts run the node exporter will collect your metrics and supply them to prometheus for you to view.

docs/prometheus-envoy-gateway.md

@@ -2,7 +2,7 @@
 
 Envoy Gateway exposes metrics that can be used to monitor the behavior and health of the Envoy Gateway.
 
-Following the deployment of the [Envoy Gateway](infrastructure-envoy-gateway-api.md) the metrics will be served and the service monitor will be created. 
+Following the deployment of the [Envoy Gateway](infrastructure-envoy-gateway-api.md) the metrics will be served and the service monitor will be created.
 
 If you need to deploy the service monitor independently you may apply the file directly with the following directions.
 
@@ -11,4 +11,3 @@ If you need to deploy the service monitor independently you may apply the file d
 ``` shell
 kubectl apply -f /etc/genestack/kustomize/envoyproxy-gateway/base/envoy-service-monitor.yaml
 ```
-

docs/prometheus-kube-event-exporter.md

@@ -1,21 +1,21 @@
 # Kubernetes Event Exporter
 
-Kubernetes Event Exporter is used to expose kubernetes events which provides useful information regarding the operation of 
-the kubernetes system. 
+Kubernetes Event Exporter is used to expose kubernetes events which provides useful information regarding the operation of
+the kubernetes system.
 
 !!! note
 
     To deploy metric exporters you will first need to deploy the Prometheus Operator, see: ([Deploy Prometheus](prometheus.md)).
 
-## Update the Kubernetes Event Exporter recievers
+## Installation
 
-Edit the Helm overrides file for the event exporter at `/opt/genestack/base-helm-configs/prometheus-kube-event-exporter/values.yaml` 
+Edit the Helm overrides file for the event exporter at `/etc/genestack/helm-configs/kubernetes-event-exporter/values.yaml`
 to add any event notification receivers you may wish to use. View the examples at [Kubernetes Event Exporter](https://github.com/resmoio/kubernetes-event-exporter).
 
-Once the changes have been made, apply them by running the  `/opt/genestack/bin/install-event-exporter.sh` script:
+Once the changes have been made, apply them by running the  `/opt/genestack/bin/install-kubernetes-event-exporter.sh` script:
 
-!!! example "`/opt/genestack/bin/install-event-exporter.sh`"
+??? example "`/opt/genestack/bin/install-kubernetes-event-exporter.sh`"
 
     ``` shell
-    --8<-- "bin/install-event-exporter.sh"
+    --8<-- "bin/install-kubernetes-event-exporter.sh"
     ```

docs/prometheus-kube-ovn.md

@@ -3,7 +3,6 @@
 Kube-OVN exposes a lot of important metrics about the controller, pinger and cni plugin. We simply
 create a service monitor to pull these metrics into Prometheus.
 
-
 ## Installation
 
 ``` shell

docs/prometheus-memcached-exporter.md

@@ -19,9 +19,9 @@ metrics:
     enabled: true
 ```
 
-Once the changes have been made, apply the changes to the memcached deployment with the `/opt/genestack/bin/install-memcached.sh` script:
+Once the changes have been made, apply the changes to the memcached deployment with the `/opt/genestack/bin/install-memcached.sh` script
 
-!!! example "`/opt/genestack/bin/install-memcached.sh`"
+??? example "`/opt/genestack/bin/install-memcached.sh`"
 
     ``` shell
     --8<-- "bin/install-memcached.sh"

docs/prometheus-monitoring-overview.md

@@ -1,7 +1,6 @@
 # Prometheus Monitoring Overview
 
-Genestack utilizes Prometheus for monitoring, alerting and metrics collection. To read more about Prometheus
-please take a look at the [upstream docs](https://prometheus.io).
+Genestack utilizes Prometheus for monitoring, alerting and metrics collection. To read more about Prometheus [Prometheus](prometheus.md)
 
 Components used to monitor and provide alerting and visualization mechanisms for genestack include:
 

docs/prometheus-mysql-exporter.md

@@ -3,23 +3,26 @@
 Mysql Exporter is used to expose metrics from a running mysql/mariadb server. The type of metrics exposed is controlled
 by the exporter and expressed in values.yaml file.
 
-!!! note
-
-    To deploy metric exporters you will first need to deploy the Prometheus Operator, see: ([Deploy Prometheus](prometheus.md)).
+To deploy metric exporters you will first need to deploy the Prometheus Operator, see: ([Deploy Prometheus](prometheus.md)).
 
 ## Installation
 
-First create secret containing password for monitoring user
+!!! note "Information about the secretes used"
 
-``` shell
-kubectl --namespace openstack \
-        create secret generic mariadb-monitoring \
-        --type Opaque \
-        --from-literal=username="monitoring" \
-        --from-literal=password="$(< /dev/urandom tr -dc _A-Za-z0-9 | head -c${1:-64};echo;)"
-```
+    Manual secret generation is only required if you haven't run the `create-secrets.sh` script located in `/opt/genestack/bin`.
+
+    ??? example "Example secret generation"
+
+        ``` shell
+        kubectl --namespace openstack \
+                create secret generic mariadb-monitoring \
+                --type Opaque \
+                --from-literal=username="monitoring" \
+                --from-literal=password="$(< /dev/urandom tr -dc _A-Za-z0-9 | head -c${1:-64};echo;)"
+        ```
+
+Add the config to a secret that'll be used within the container for our shared services
 
-Then add the config to a secret that'll be used within the container for our shared services
 ``` shell
 kubectl -n openstack create secret generic mariadb-monitor --type Opaque --from-literal=my.cnf="[client.mariadb-monitor]
 user=monitoring
@@ -28,11 +31,14 @@ password=$(kubectl --namespace openstack get secret mariadb-monitoring -o jsonpa
 
 Next, install the exporter
 
-``` shell
-bin/install-prometheus-mysql-exporter.sh
-```
+??? example "`/opt/genestack/bin/install-prometheus-mysql-exporter.sh`"
+
+    ``` shell
+    --8<-- "bin/install-prometheus-mysql-exporter.sh"
+    ```
+
+!!! note
 
-!!! note "Helm chart versions are defined in (opt)/genestack/helm-chart-versions.yaml and can be overridden in (etc)/genestack/helm-chart-versions.yaml"
+    Helm chart versions are defined in `opt/genestack/helm-chart-versions.yaml` and can be overridden in `/etc/genestack/helm-chart-versions.yaml`.
 
-!!! success
-    If the installation is successful, you should see the exporter pod in the openstack namespace.
+If the installation is successful, you should see the exporter pod in the openstack namespace.