Skip to content

update prometheus, coralogix, and grafana docs #933

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+631 −214
Open

update prometheus, coralogix, and grafana docs #933

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0ce72b3
update prometheus docs
aantn Sep 3, 2025
1ac5b65
more docs
aantn Sep 4, 2025
ce5e8c2
Merge branch 'master' into docs-prometheus-fixes
aantn Sep 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 177 additions & 24 deletions docs/data-sources/builtin-toolsets/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,85 @@ Connect HolmesGPT to Prometheus for metrics analysis and query generation. This

## Configuration

```yaml-toolset-config
toolsets:
prometheus/metrics:
enabled: true
config:
=== "CLI"

Create or edit your `~/.holmes/config.yaml`:

```yaml
toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<your-prometheus-service>:9090

# Optional authentication:
#headers:
# Authorization: "Basic <base_64_encoded_string>"

# Optional SSL/TLS settings:
#prometheus_ssl_enabled: true # Set to false to disable SSL verification (default: true)

# Optional label filtering:
#additional_labels: # Add extra label selectors to all Prometheus queries
# cluster: "production"
# region: "us-west-2"
```

💡 **Alternative**: Set the `PROMETHEUS_URL` environment variable instead of using the config file:
```bash
export PROMETHEUS_URL="http://your-prometheus:9090"
holmes ask "show me CPU usage"
```

=== "HolmesGPT Helm Chart"

Configure Prometheus in your `values.yaml`:

```yaml
holmes:
toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<your-prometheus-service>:9090

# Optional:
# Optional authentication:
#headers:
# Authorization: "Basic <base_64_encoded_string>"

# Optional SSL/TLS settings:
#prometheus_ssl_enabled: true # Set to false to disable SSL verification (default: true)

# Optional label filtering:
#additional_labels: # Add extra label selectors to all Prometheus queries
# cluster: "production"
# region: "us-west-2"
```

=== "Robusta Helm Chart"

When using Robusta's integrated HolmesGPT, configure Prometheus in your `values.yaml`:

```yaml
globalConfig:
custom_toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<your-prometheus-service>:9090

# Optional authentication:
#headers:
# Authorization: "Basic <base_64_encoded_string>"
```

# Optional SSL/TLS settings:
#prometheus_ssl_enabled: true # Set to false to disable SSL verification (default: true)

💡 **Alternative**: Set the `PROMETHEUS_URL` environment variable instead of using the config file.
# Optional label filtering:
#additional_labels: # Add extra label selectors to all Prometheus queries
# cluster: "production"
# region: "us-west-2"
```

## Validation

Expand Down Expand Up @@ -65,28 +130,91 @@ This will print all possible Prometheus service URLs in your cluster. Pick the o

- **Connection refused**: Check if the Prometheus URL is accessible from HolmesGPT.
- **Authentication errors**: Verify the headers configuration for secured Prometheus endpoints.
- **No metrics returned**: Ensure that Prometheus is scraping your targets.
- **SSL certificate errors**:
- For self-signed certificates, set `prometheus_ssl_enabled: false` to disable verification
- Or provide a custom CA certificate via the `CERTIFICATE` environment variable (see [Custom SSL Certificates](../../ai-providers/openai-compatible.md#custom-ssl-certificates))


## Advanced Configuration

You can further customize the Prometheus toolset with the following options:

```yaml
toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<prometheus-host>:9090
healthcheck: "-/healthy" # Path for health checking (default: -/healthy)
headers:
Authorization: "Basic <base_64_encoded_string>"
metrics_labels_time_window_hrs: 48 # Time window (hours) for fetching labels (default: 48)
metrics_labels_cache_duration_hrs: 12 # How long to cache labels (hours, default: 12)
fetch_labels_with_labels_api: false # Use labels API instead of series API (default: false)
fetch_metadata_with_series_api: false # Use series API for metadata (default: false)
tool_calls_return_data: true # If false, disables returning Prometheus data (default: true)
```
=== "CLI"

```yaml
toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<prometheus-host>:9090
healthcheck: "-/healthy" # Path for health checking (default: -/healthy)
headers:
Authorization: "Basic <base_64_encoded_string>"
metrics_labels_time_window_hrs: 48 # Time window (hours) for fetching labels (default: 48)
metrics_labels_cache_duration_hrs: 12 # How long to cache labels (hours, default: 12)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix incorrect healthcheck path (-/healthy → /-/healthy).

Prometheus uses /-/healthy. The missing leading slash will break health checks and mislead users copying the snippet. Update all occurrences and the default comment.

Apply these diffs:

-          healthcheck: "-/healthy"  # Path for health checking (default: -/healthy)
+          healthcheck: "/-/healthy"  # Path for health checking (default: /-/healthy)
-            healthcheck: "-/healthy"  # Path for health checking (default: -/healthy)
+            healthcheck: "/-/healthy"  # Path for health checking (default: /-/healthy)
-            healthcheck: "-/healthy"  # Path for health checking (default: -/healthy)
+            healthcheck: "/-/healthy"  # Path for health checking (default: /-/healthy)
-- `healthcheck`: Path used for health checking Prometheus or Mimir/Cortex endpoint. Defaults to `-/healthy` for Prometheus, use `/ready` for Grafana Mimir.
+- `healthcheck`: Path used for health checking Prometheus or Mimir/Cortex endpoint. Defaults to `/-/healthy` for Prometheus, use `/ready` for Grafana Mimir.

Also applies to: 173-176, 196-199, 221-223

🤖 Prompt for AI Agents 
In docs/data-sources/builtin-toolsets/prometheus.md around lines 150-154 (and
also apply the same change to ranges 173-176, 196-199, and 221-223), the
healthcheck path is missing the leading slash ("-/healthy") and the default
comment is wrong; update each occurrence to "/-/healthy" and adjust the inline
default comment to "(default: /-/healthy)" so examples and comments consistently
show the correct Prometheus healthcheck path.

fetch_labels_with_labels_api: false # Use labels API instead of series API (default: false)
fetch_metadata_with_series_api: false # Use series API for metadata (default: false)
tool_calls_return_data: true # If false, disables returning Prometheus data (default: true)
prometheus_ssl_enabled: true # Set to false to disable SSL verification (default: true)
additional_labels: # Add extra label selectors to all Prometheus queries (optional)
cluster: "production"
region: "us-west-2"
```

=== "HolmesGPT Helm Chart"

```yaml
holmes:
toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<prometheus-host>:9090
healthcheck: "-/healthy" # Path for health checking (default: -/healthy)
headers:
Authorization: "Basic <base_64_encoded_string>"
metrics_labels_time_window_hrs: 48 # Time window (hours) for fetching labels (default: 48)
metrics_labels_cache_duration_hrs: 12 # How long to cache labels (hours, default: 12)
fetch_labels_with_labels_api: false # Use labels API instead of series API (default: false)
fetch_metadata_with_series_api: false # Use series API for metadata (default: false)
tool_calls_return_data: true # If false, disables returning Prometheus data (default: true)
prometheus_ssl_enabled: true # Set to false to disable SSL verification (default: true)
additional_labels: # Add extra label selectors to all Prometheus queries (optional)
cluster: "production"
region: "us-west-2"
```

=== "Robusta Helm Chart"

```yaml
globalConfig:
custom_toolsets:
prometheus/metrics:
enabled: true
config:
prometheus_url: http://<prometheus-host>:9090
healthcheck: "-/healthy" # Path for health checking (default: -/healthy)
headers:
Authorization: "Basic <base_64_encoded_string>"
metrics_labels_time_window_hrs: 48 # Time window (hours) for fetching labels (default: 48)
metrics_labels_cache_duration_hrs: 12 # How long to cache labels (hours, default: 12)
fetch_labels_with_labels_api: false # Use labels API instead of series API (default: false)
fetch_metadata_with_series_api: false # Use series API for metadata (default: false)
tool_calls_return_data: true # If false, disables returning Prometheus data (default: true)
prometheus_ssl_enabled: true # Set to false to disable SSL verification (default: true)
additional_labels: # Add extra label selectors to all Prometheus queries (optional)
cluster: "production"
region: "us-west-2"

# Note: Some Robusta versions require additional RBAC permissions:
customClusterRoleRules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "daemonsets", "statefulsets"]
verbs: ["get", "list", "watch"]
```

**Config option explanations:**

Expand All @@ -98,6 +226,31 @@ toolsets:
- `fetch_labels_with_labels_api`: Use the Prometheus labels API to fetch labels (can improve performance, but increases HTTP calls).
- `fetch_metadata_with_series_api`: Use the series API for metadata (only set to true if the metadata API is disabled or not working).
- `tool_calls_return_data`: If `false`, disables returning Prometheus data to HolmesGPT (useful if you hit token limits).
- `prometheus_ssl_enabled`: Enable/disable SSL certificate verification. Set to `false` for self-signed certificates (default: `true`).
- `additional_labels`: Dictionary of labels to add to all Prometheus queries. Useful for filtering metrics in multi-cluster or multi-tenant environments.

## SSL/TLS Configuration

### Self-Signed Certificates

If your Prometheus instance uses self-signed certificates, you have two options:

**Option 1: Disable SSL verification** (less secure, but simpler)
```yaml
prometheus/metrics:
config:
prometheus_ssl_enabled: false
```

**Option 2: Provide custom CA certificate** (more secure)
```yaml
# Set the CERTIFICATE environment variable with your base64-encoded CA certificate
additionalEnvVars:
- name: CERTIFICATE
value: "LS0tLS1CRUdJTi..." # Your base64-encoded CA certificate
```

The `CERTIFICATE` environment variable applies globally to all HTTPS connections made by Holmes, including Prometheus, AI providers, and other integrations. See [Custom SSL Certificates](../../ai-providers/openai-compatible.md#custom-ssl-certificates) for more details.

## Capabilities

Expand Down
Loading