Monitoring with Prometheus & Grafana.md

Monitoring with Prometheus & Grafana

DockFlare can export native Prometheus metrics from the managed cloudflared agent, enabling detailed monitoring of your tunnel's health, traffic, and performance. This guide explains how to enable this feature and how to set up a monitoring stack to visualize the data.

Enabling Metrics

To enable the metrics endpoint, you need to set one environment variable in your .env file.

Variable	Required	Default	Description
CLOUDFLARED_METRICS_PORT	No	(unset)	If set to a port number (e.g., 2000), this enables the Prometheus metrics endpoint on the managed cloudflared agent. The agent will listen on 0.0.0.0:<PORT>, and the port will be exposed to the Docker host. If unset, the metrics endpoint is disabled.

Example .env configuration:

# Enables the Prometheus metrics endpoint on port 2000
CLOUDFLARED_METRICS_PORT=2000

DockFlare's built-in reconciliation logic will automatically handle the rest. If you add, change, or remove this variable and restart DockFlare, the cloudflared agent container will be automatically recreated with the correct settings.

Available Metrics

cloudflared exposes a wide range of metrics. Key metrics to monitor include:

• Tunnel Traffic: cloudflared_tunnel_total_requests, cloudflared_tunnel_request_errors
• Performance: cloudflared_tunnel_request_duration_seconds_bucket (used to calculate latency percentiles like P99, P95, etc.)
• Connections: cloudflared_tunnel_ha_connections (connections to the Cloudflare edge), cloudflared_tunnel_concurrent_requests
• HTTP Details: cloudflared_tunnel_response_by_code (breakdown of HTTP status codes), cloudflared_tunnel_requests_per_protocol
• Internal Process Health: Advanced metrics like process_cpu_seconds_total and go_memstats_alloc_bytes for debugging the agent's resource usage.

Visualizing with Grafana

Using the Pre-built Dashboard

The easiest way to get started is by using the pre-configured dashboard provided in the DockFlare repository.

1. Find the Dashboard: The dashboard file is located at examples/dashboard.json.
2. Import to Grafana:
   • Navigate to your Grafana instance.
   • On the left menu, go to Dashboards.
   • Click New -> Import.
   • Either paste the JSON content or upload the dashboard.json file.
   • Select your Prometheus data source and click Import.

Example: Setting Up a Local Prometheus & Grafana Stack

If you don't have a monitoring stack, you can use the following guide to set one up with Docker Compose.

1. Directory Structure

Create these folders and files next to your main docker-compose.yml file:

.
├── docker-compose.yml   # Your main compose file
├── prometheus.yml         # New file
└── grafana-provisioning/  # New folder
    └── datasources/       # New sub-folder
        └── datasource.yml # New file

2. docker-compose.yml Services

Add the following prometheus and grafana services to your existing docker-compose.yml:

services:
  # ... your existing dockflare and other services ...

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_data:/prometheus # Persistent data for Prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    networks:
      - your-dockflare-network # <-- IMPORTANT: Use the same network as DockFlare
    labels:
      - "dockflare.enable=true"
      - "dockflare.hostname=prometheus.your-domain.com"
      - "dockflare.service=http://prometheus:9090"

  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    restart: unless-stopped
    volumes:
      - ./grafana_data:/var/lib/grafana # Persistent data for Grafana
      - ./grafana-provisioning:/etc/grafana/provisioning
    networks:
      - your-dockflare-network # <-- IMPORTANT: Use the same network as DockFlare
    labels:
      - "dockflare.enable=true"
      - "dockflare.hostname=metrics.your-domain.com" # Exposes Grafana
      - "dockflare.service=http://grafana:3000"

Permissions Troubleshooting

If Grafana or Prometheus fail to start with a "Permission denied" error, you must set the correct ownership on the host directories before starting the containers.

# Stop the stack first
docker-compose down

# Set ownership for Grafana (user ID 472)
sudo chown -R 472:472 ./grafana_data

# Set ownership for Prometheus (user ID 65534)
sudo chown -R 65534:65534 ./prometheus_data

# Start the stack again
docker-compose up -d

3. prometheus.yml Configuration

Create the prometheus.yml file. This tells Prometheus where to find your cloudflared agent.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'cloudflared'
    static_configs:
      - targets: ['your-cloudflared-agent-name:2000']
        # --- IMPORTANT ---
        # 1. Replace 'your-cloudflared-agent-name' with the name of your agent container (e.g., 'cloudflared-agent-green-bern').
        # 2. Replace '2000' with the port you set for CLOUDFLARED_METRICS_PORT.

4. Grafana datasource.yml Configuration

Create the grafana-provisioning/datasources/datasource.yml file. This automatically adds Prometheus as a data source in Grafana.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

5. Launch & Verify

1. Run docker-compose up -d.
2. Navigate to your Prometheus URL (e.g., http://prometheus.your-domain.com).
3. Go to Status -> Targets. The cloudflared endpoint should be UP.
4. Navigate to your Grafana URL, log in (default: admin/admin), and import the pre-built dashboard. You should see your tunnel's metrics