Monitoring & Observability Setup Guide

Monitoring & Observability Setup Guide To ensure high availability and stable performance in production environments, Sendium includes a comprehensive monitoring stack. We utilize Micrometer to expose internal application metrics, Prometheus to scrape and store this time-series data, and Grafana for visualization.

This guide outlines how the metrics are generated and provides the necessary configuration files and scripts to deploy the monitoring infrastructure locally or on your server.

1. Exposing Application Metrics (Quarkus) Sendium leverages the Micrometer extension within the Quarkus framework to automatically expose JVM metrics, HTTP traffic statistics, and custom business metrics.

By default, the metrics are exposed and available for scraping at the following endpoint: http://:8080/q/metrics

2. Configuring Prometheus Prometheus acts as the central metrics server. It needs to be configured to know where to find the Sendium application and how often to collect the data.

Create a file named prometheus.yml in your deployment directory and paste the following configuration. This setup configures a 3-second scrape interval for near real-time monitoring.

YAML

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'sendium-metrics'
    metrics_path: '/q/metrics'
    scrape_interval: 3s
    static_configs:
      - targets: ['sendium.cloud']

3. Deploying the Monitoring Stack (Docker) To simplify the deployment for operations teams, we have provided automated Bash scripts that spin up both Prometheus and Grafana using Docker. These scripts also handle the creation of persistent volumes to ensure your metrics and dashboards are not lost upon container restarts.

3.1 Deploy Prometheus Create a file named prometheus.sh in the same directory as your prometheus.yml file:

Bash

#!/bin/bash

# we assume that this script is in the same directory as the prometheus.yml
# and that in the same directory we need the persistent mounted data directory

FULL_PATH_TO_SCRIPT="$(realpath "${BASH_SOURCE[-1]}")"
SCRIPT_DIRECTORY="$(dirname "$FULL_PATH_TO_SCRIPT")"

# now cd to base directory
cd "${SCRIPT_DIRECTORY}/../.." || exit 1

mkdir -p "${SCRIPT_DIRECTORY}/prometheus-data"
chmod 777 "${SCRIPT_DIRECTORY}/prometheus-data"

docker rm -fv mcore-metrics-prometheus
docker run -d --name mcore-metrics-prometheus -p 9090:9090 -v "${SCRIPT_DIRECTORY}/prometheus-data":/prometheus -v "${SCRIPT_DIRECTORY}":/etc/prometheus prom/prometheus --config.file=/etc/prometheus/prometheus.yml

3.2 Deploy Grafana Create a file named grafana.sh in the same directory:

Bash

#!/bin/bash

# we assume that this script is in the same directory as the prometheus.yml
# and that in the same directory we need the persistent mounted data directory

FULL_PATH_TO_SCRIPT="$(realpath "${BASH_SOURCE[-1]}")"
SCRIPT_DIRECTORY="$(dirname "$FULL_PATH_TO_SCRIPT")"

# now cd to base directory
cd "${SCRIPT_DIRECTORY}/../.." || exit 1

mkdir -p "${SCRIPT_DIRECTORY}/grafana-data"
chmod 777 "${SCRIPT_DIRECTORY}/grafana-data"

docker rm -fv mcore-metrics-grafana
docker run -d --name mcore-metrics-grafana -v "${SCRIPT_DIRECTORY}/grafana-data":/var/lib/grafana -p 3000:3000 grafana/grafana

3.3 Execution Instructions To start the monitoring stack, make the scripts executable and run them from your terminal:

chmod +x prometheus.sh grafana.sh

./prometheus.sh

./grafana.sh

Once deployed:

Access Prometheus at: http://localhost:9090

Access Grafana at: http://localhost:3000 (Default credentials are usually admin / admin)