Skip to content

Commit 9a6f025

Browse files
hjethjet
authored
Add analyse ruler, dashboard, rule-file commands (#201)
Adds the following: analyse ruler analyse dashboards (JSON dashboard input) analyse rule-files (YAML rule file input) Modifies analyse prometheus to accept input from analyse ruler and analyse rule-file output Fixes a bug to iterate over sub panels of a row type Panel Adds two simple tests for extracting metrics Adds metric extraction from templating sections of dashboard files
1 parent f3b169f commit 9a6f025

15 files changed

+2956
-97
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,4 @@ dist/
1212
/cortextool
1313
/metrics-in-grafana.json
1414
/prometheus-metrics.json
15+
/metrics-in-ruler.json

‎README.md

Lines changed: 150 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -195,11 +195,12 @@ This lets you generate the header which can then be used to enforce access contr
195195

196196
#### Analyse
197197

198-
Run analysis against your Prometheus, Grafana and Cortex to see which metrics being used and exported.
198+
Run analysis against your Prometheus, Grafana and Cortex to see which metrics being used and exported. Can also extract metrics
199+
from dashboard JSON and rules YAML files.
199200

200-
##### grafana-analyse
201+
##### `analyse grafana`
201202

202-
This command will be run against your Grafana instance and it will download the dashboards and pick the Prometheus metrics that are used in the queries. The output is a JSON file.
203+
This command will run against your Grafana instance and will download its dashboards and then extract the Prometheus metrics used in its queries. The output is a JSON file.
203204

204205
###### Configuration
205206

@@ -211,28 +212,166 @@ This command will be run against your Grafana instance and it will download the
211212

212213
###### Running the command
213214

214-
```
215+
```shell
215216
cortextool analyse grafana --address=<grafana-address> --key=<API-Key>
216217
```
217218

218-
##### prometheus-analyse
219+
###### Sample output
220+
221+
```json
222+
{
223+
"metricsUsed": [
224+
"apiserver_request:availability30d",
225+
"workqueue_depth",
226+
"workqueue_queue_duration_seconds_bucket",
227+
...
228+
],
229+
"dashboards": [
230+
{
231+
"slug": "",
232+
"uid": "09ec8aa1e996d6ffcd6817bbaff4db1b",
233+
"title": "Kubernetes / API server",
234+
"metrics": [
235+
"apiserver_request:availability30d",
236+
"apiserver_request_total",
237+
"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile",
238+
"workqueue_depth",
239+
"workqueue_queue_duration_seconds_bucket",
240+
...
241+
],
242+
"parse_errors": null
243+
}
244+
]
245+
}
246+
```
247+
248+
##### `analyse ruler`
249+
250+
This command will run against your Grafana Cloud Prometheus instance and will fetch its rule groups. It will then extract the Prometheus metrics used in the rule queries. The output is a JSON file.
251+
252+
###### Configuration
253+
254+
| Env Variables | Flag | Description |
255+
| ----------------- | --------- | ------------------------------------------------------------------------------------------------------------- |
256+
| CORTEX_ADDRESS | `address` | Address of the Prometheus instance. |
257+
| CORTEX_TENANT_ID | `id` | If you're using Grafana Cloud this is your instance ID. |
258+
| CORTEX_API_KEY | `key` | If you're using Grafana Cloud this is your API Key. |
259+
| __ | `output` | The output file path. metrics-in-ruler.json by default. |
260+
261+
###### Running the command
262+
263+
```shell
264+
cortextool analyse ruler --address=https://prometheus-blocks-prod-us-central1.grafana.net --id=<1234> --key=<API-Key>
265+
```
266+
267+
###### Sample output
268+
269+
```json
270+
{
271+
"metricsUsed": [
272+
"apiserver_request_duration_seconds_bucket",
273+
"container_cpu_usage_seconds_total",
274+
"scheduler_scheduling_algorithm_duration_seconds_bucket"
275+
...
276+
],
277+
"ruleGroups": [
278+
{
279+
"namspace": "prometheus_rules",
280+
"name": "kube-apiserver.rules",
281+
"metrics": [
282+
"apiserver_request_duration_seconds_bucket",
283+
"apiserver_request_duration_seconds_count",
284+
"apiserver_request_total"
285+
],
286+
"parse_errors": null
287+
},
288+
...
289+
}
290+
```
291+
292+
##### `analyse prometheus`
219293

220-
This command will be run against your Prometheus / GrafanaCloud instance, then it will use the output from `grafana-analyse` show you how many series in the Prometheus server are actually being used in dashboards. Also, it'll show which metrics exist in Grafana Cloud that are **not** in dashboards. The output is a JSON file
294+
This command will run against your Prometheus / Cloud Prometheus instance. It will then use the output from `analyse grafana` and `analyse ruler` to show you how many series in the Prometheus server are actually being used in dashboards and rules. Also, it'll show which metrics exist in Grafana Cloud that are **not** in dashboards or rules. The output is a JSON file.
221295

222296
###### Configuration
223297

224298
| Env Variables | Flag | Description |
225299
| ----------------- | --------- | ------------------------------------------------------------------------------------------------------------- |
226300
| CORTEX_ADDRESS | `address` | Address of the Prometheus instance. |
227-
| CORTEX_TENANT_ID | `id` | If you're using GrafanaCloud this is your instance ID. |
228-
| CORTEX_API_KEY | `key` | If you're using GrafanaCloud this is your API Key. |
229-
| __ | `grafana-metrics-file` | The input file path. metrics-in-grafana.json by default. |
230-
| __ | `output` | The output file path. prometheus-metrics.json by default. |
301+
| CORTEX_TENANT_ID | `id` | If you're using Grafana Cloud this is your instance ID. |
302+
| CORTEX_API_KEY | `key` | If you're using Grafana Cloud this is your API Key. |
303+
| __ | `grafana-metrics-file` | The dashboard metrics input file path. `metrics-in-grafana.json` by default. |
304+
| __ | `ruler-metrics-file` | The rules metrics input file path. `metrics-in-ruler.json` by default. |
305+
| __ | `output` | The output file path. `prometheus-metrics.json` by default. |
231306

232307
###### Running the command
233308

309+
```shell
310+
cortextool analyse prometheus --address=https://prometheus-blocks-prod-us-central1.grafana.net --id=<1234> --key=<API-Key> --log.level=debug
311+
```
312+
313+
###### Sample output
314+
315+
```json
316+
{
317+
"total_active_series": 38184,
318+
"in_use_active_series": 14047,
319+
"additional_active_series": 24137,
320+
"in_use_metric_counts": [
321+
{
322+
"metric": "apiserver_request_duration_seconds_bucket",
323+
"count": 11400,
324+
"job_counts": [
325+
{
326+
"job": "apiserver",
327+
"count": 11400
328+
}
329+
]
330+
},
331+
{
332+
"metric": "apiserver_request_total",
333+
"count": 684,
334+
"job_counts": [
335+
{
336+
"job": "apiserver",
337+
"count": 684
338+
}
339+
]
340+
},
341+
...
342+
],
343+
"additional_metric_counts": [
344+
{
345+
"metric": "etcd_request_duration_seconds_bucket",
346+
"count": 2688,
347+
"job_counts": [
348+
{
349+
"job": "apiserver",
350+
"count": 2688
351+
}
352+
]
353+
},
354+
...
234355
```
235-
cortextool analyse prometheus --address=https://prometheus-us-central1.grafana.net/api/prom --username=<1234> --password=<API-Key> --log.level=debug
356+
357+
##### `analyse dashboard`
358+
359+
This command accepts Grafana dashboard JSON files as input and extracts Prometheus metrics used in the queries. The output is a JSON file compatible with `analyse prometheus`.
360+
361+
###### Running the command
362+
363+
```shell
364+
cortextool analyse dashboard ./dashboard_one.json ./dashboard_two.json ...
365+
```
366+
367+
##### `analyse rule-file`
368+
369+
This command accepts Prometheus rule YAML files as input and extracts Prometheus metrics used in the queries. The output is a JSON file compatible with `analyse prometheus`.
370+
371+
###### Running the command
372+
373+
```shell
374+
cortextool analyse rule-file ./rule_file_one.yaml ./rule_file_two.yaml ...
236375
```
237376

238377
## chunktool

‎pkg/analyse/grafana.go

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
package analyse
22

33
type MetricsInGrafana struct {
4-
MetricsUsed []string `json:"metricsUsed"`
5-
Dashboards []DashboardMetrics `json:"dashboards"`
4+
MetricsUsed []string `json:"metricsUsed"`
5+
OverallMetrics map[string]struct{} `json:"-"`
6+
Dashboards []DashboardMetrics `json:"dashboards"`
67
}
78

89
type DashboardMetrics struct {

‎pkg/analyse/ruler.go

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
package analyse
2+
3+
type MetricsInRuler struct {
4+
MetricsUsed []string `json:"metricsUsed"`
5+
OverallMetrics map[string]struct{} `json:"-"`
6+
RuleGroups []RuleGroupMetrics `json:"ruleGroups"`
7+
}
8+
9+
type RuleGroupMetrics struct {
10+
Namespace string `json:"namspace"`
11+
GroupName string `json:"name"`
12+
Metrics []string `json:"metrics"`
13+
ParseErrors []string `json:"parse_errors"`
14+
}

‎pkg/commands/analyse.go

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ func (cmd *AnalyseCommand) Register(app *kingpin.Application) {
3030
prometheusAnalyseCmd.Flag("grafana-metrics-file", "The path for the input file containing the metrics from grafana-analyse command").
3131
Default("metrics-in-grafana.json").
3232
StringVar(&paCmd.grafanaMetricsFile)
33+
prometheusAnalyseCmd.Flag("ruler-metrics-file", "The path for the input file containing the metrics from ruler-analyse command").
34+
Default("metrics-in-ruler.json").
35+
StringVar(&paCmd.rulerMetricsFile)
3336
prometheusAnalyseCmd.Flag("output", "The path for the output file").
3437
Default("prometheus-metrics.json").
3538
StringVar(&paCmd.outputFile)
@@ -51,4 +54,41 @@ func (cmd *AnalyseCommand) Register(app *kingpin.Application) {
5154
grafanaAnalyseCmd.Flag("output", "The path for the output file").
5255
Default("metrics-in-grafana.json").
5356
StringVar(&gaCmd.outputFile)
57+
58+
raCmd := &RulerAnalyseCommand{}
59+
rulerAnalyseCmd := analyseCmd.Command("ruler", "Analyse and extract the metrics used in Cortex rules").
60+
Action(raCmd.run)
61+
rulerAnalyseCmd.Flag("address", "Address of the Prometheus/Cortex instance, alternatively set $CORTEX_ADDRESS.").
62+
Envar("CORTEX_ADDRESS").
63+
Required().
64+
StringVar(&raCmd.ClientConfig.Address)
65+
rulerAnalyseCmd.Flag("id", "Username to use when contacting Prometheus/Cortex, alternatively set $CORTEX_TENANT_ID.").
66+
Envar("CORTEX_TENANT_ID").
67+
Default("").
68+
StringVar(&raCmd.ClientConfig.ID)
69+
rulerAnalyseCmd.Flag("key", "Password to use when contacting Prometheus/Cortex, alternatively set $CORTEX_API_KEY.").
70+
Envar("CORTEX_API_KEY").
71+
Default("").
72+
StringVar(&raCmd.ClientConfig.Key)
73+
rulerAnalyseCmd.Flag("output", "The path for the output file").
74+
Default("metrics-in-ruler.json").
75+
StringVar(&raCmd.outputFile)
76+
77+
daCmd := &DashboardAnalyseCommand{}
78+
dashboardAnalyseCmd := analyseCmd.Command("dashboard", "Analyse and output the metrics used in Grafana dashboard files").Action(daCmd.run)
79+
dashboardAnalyseCmd.Arg("files", "Dashboard files").
80+
Required().
81+
ExistingFilesVar(&daCmd.DashFilesList)
82+
dashboardAnalyseCmd.Flag("output", "The path for the output file").
83+
Default("metrics-in-grafana.json").
84+
StringVar(&daCmd.outputFile)
85+
86+
rfCmd := &RuleFileAnalyseCommand{}
87+
ruleFileAnalyseCmd := analyseCmd.Command("rule-file", "Analyse and output the metrics used in Prometheus rules files").Action(rfCmd.run)
88+
ruleFileAnalyseCmd.Arg("files", "Rules files").
89+
Required().
90+
ExistingFilesVar(&rfCmd.RuleFilesList)
91+
ruleFileAnalyseCmd.Flag("output", "The path for the output file").
92+
Default("metrics-in-ruler.json").
93+
StringVar(&rfCmd.outputFile)
5494
}

‎pkg/commands/analyse_dashboards.go

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
package commands
2+
3+
import (
4+
"encoding/json"
5+
"fmt"
6+
"os"
7+
8+
"github.com/grafana-tools/sdk"
9+
"gopkg.in/alecthomas/kingpin.v2"
10+
11+
"github.com/grafana/cortex-tools/pkg/analyse"
12+
)
13+
14+
type DashboardAnalyseCommand struct {
15+
DashFilesList []string
16+
outputFile string
17+
}
18+
19+
func (cmd *DashboardAnalyseCommand) run(k *kingpin.ParseContext) error {
20+
output := &analyse.MetricsInGrafana{}
21+
output.OverallMetrics = make(map[string]struct{})
22+
23+
for _, file := range cmd.DashFilesList {
24+
var board sdk.Board
25+
buf, err := loadFile(file)
26+
if err != nil {
27+
return err
28+
}
29+
if err = json.Unmarshal(buf, &board); err != nil {
30+
fmt.Fprintf(os.Stderr, "%s for %s\n", err, file)
31+
continue
32+
}
33+
parseMetricsInBoard(output, board)
34+
}
35+
36+
err := writeOut(output, cmd.outputFile)
37+
if err != nil {
38+
return err
39+
}
40+
return nil
41+
}
42+
43+
func loadFile(filename string) ([]byte, error) {
44+
file, err := os.Open(filename)
45+
if err != nil {
46+
return nil, err
47+
}
48+
defer file.Close()
49+
50+
fileinfo, err := file.Stat()
51+
if err != nil {
52+
return nil, err
53+
}
54+
55+
filesize := fileinfo.Size()
56+
buffer := make([]byte, filesize)
57+
58+
_, err = file.Read(buffer)
59+
if err != nil {
60+
return nil, err
61+
}
62+
63+
return buffer, nil
64+
}

0 commit comments

Comments
 (0)