Skip to content

Fix bug with prometheus embeds #1044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
+13 −5
Draft

Fix bug with prometheus embeds #1044

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions holmes/plugins/toolsets/prometheus/prometheus_instructions.jinja2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@
* Use Prometheus to query metrics from the alert promql
* Use prometheus to execute promql queries with the tools `execute_prometheus_instant_query` and `execute_prometheus_range_query`
* To create queries, use 'start_timestamp' and 'end_timestamp' as graphs start and end times
* ALWAYS embed the execution results into your answer
* You only need to embed the partial result in your response. Include the "tool_name" and "random_key". For example: << {"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "92jf2hf"} >>
* Use these tools to generate charts that users can see. Here are standard metrics but you can use different ones:
* Here are standard metrics but you can use different ones:
** For memory consumption: `container_memory_working_set_bytes`
** For CPU usage: `container_cpu_usage_seconds_total`
** For CPU throttling: `container_cpu_cfs_throttled_periods_total`
Expand Down Expand Up @@ -61,8 +59,17 @@
- Example: `count(count by (pod) (container_cpu_usage_seconds_total{namespace="example"}))`
- If count > 10, use topk() in your range query
* When doing queries, always extend the time range, to 15 min before and after the alert start time
* ALWAYS embed the execution results into your answer
* ALWAYS embed a Prometheus graph in the response. The graph should visualize data related to the incident.

## Embedding Graphs in Responses
* You are running in a GUI and can show users graphs so they can interpret the data themselves and confirm what you are saying
* When relevant to the user question OR if you think it can help users visually confirm what you are saying, ALWAYS embed a graph of Prometheus results into your answer so that users can see it as follows:
* Embed a graph by inserting the following format:

<< {"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "92jf2hf"} >>

* ALWAYS include the "tool_name" and "random_key"
* You may ONLY embed graphs for execute_prometheus_range_query (and NOT execute_prometheus_instant_query)

Comment on lines +63 to +72
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

🧩 Analysis chain

Tighten embed requirements: require type, unique random_key, and prior tool execution

  • Make “type” required (parser often relies on it).
  • State random_key must be unique per embed.
  • Require executing execute_prometheus_range_query with the exact PromQL before inserting the embed.
 ## Embedding Graphs in Responses
 * You are running in a GUI and can show users graphs so they can interpret the data themselves and confirm what you are saying
-* When relevant to the user question OR if you think it can help users visually confirm what you are saying, ALWAYS embed a graph of Prometheus results into your answer so that users can see it as follows:
+* When relevant to the user question OR if you think it can help users visually confirm what you are saying, ALWAYS embed a graph of Prometheus results into your answer so that users can see it as follows:
 * Embed a graph by inserting the following format:
 
   << {"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "92jf2hf"} >>
 
-* ALWAYS include the "tool_name" and "random_key"
-* You may ONLY embed graphs for execute_prometheus_range_query (and NOT execute_prometheus_instant_query)
+* ALWAYS include "type", "tool_name", and a unique "random_key" per embed
+* You may ONLY embed graphs for execute_prometheus_range_query (and NOT execute_prometheus_instant_query)
+* ALWAYS execute execute_prometheus_range_query with the exact PromQL you want graphed before adding the embed placeholder

Please verify the embed parser tolerates the whitespace shown in the examples (spaces after ‘<<’ and before ‘>>’) and that no other templates allow instant-query embeds. Run:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "1) Check parser patterns (tolerance for whitespace around << >> and required fields):"
rg -nP -C2 '<<\s*\{.*"type"\s*:\s*"promql".*"tool_name"\s*:\s*"execute_prometheus_range_query".*"random_key"\s*:\s*".+?"\s*\}\s*>>'

echo
echo "2) Search for any embed placeholders that reference execute_prometheus_instant_query:"
rg -nP '<<\s*\{[^}]*"tool_name"\s*:\s*"execute_prometheus_instant_query"'

echo
echo "3) Search for code that parses/prompts for promql embeds to confirm whitespace tolerance:"
rg -nP -C3 '<<\s*\{.*"type".*"tool_name".*"random_key".*\}\s*>>'

Length of output: 20947

Tighten PromQL embed requirements

  • Require each embed placeholder to include "type", "tool_name", and a unique "random_key".
  • Always execute execute_prometheus_range_query with the exact PromQL before inserting the embed placeholder.
  • Whitespace around << and >> is already tolerated—no changes needed to example spacing.
  • Remove or update any execute_prometheus_instant_query embeds (e.g. in FEATURES.md) so only range-query embeds remain.
🤖 Prompt for AI Agents 
holmes/plugins/toolsets/prometheus/prometheus_instructions.jinja2 lines 63-72:
tighten embed handling so every Prometheus graph embed placeholder includes the
keys "type", "tool_name", and a unique "random_key"; ensure the code path always
executes execute_prometheus_range_query with the exact PromQL string prior to
inserting the placeholder (do not insert the placeholder unless that range query
was run); and find/replace any execute_prometheus_instant_query embeds
(including those in FEATURES.md) to use execute_prometheus_range_query
equivalents or remove them so only range-query embeds remain.

* Embed at most 2 graphs
* When embedding multiple graphs, always add line spacing between them
For example:
Expand All @@ -72,6 +79,7 @@
<<{"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "IKtq"}>>

{%- if config and config.additional_labels and config.additional_labels.keys()|list|length > 0 %}
## Labels to ALWAYS add to PromQL queries
* ALWAYS add the following additional labels to ALL PromQL queries:
{%- for key, value in config.additional_labels.items() %}
* {{ key }}="{{ value }}"
Expand Down
Loading