Skip to content

Commit 836773a

Browse files
authored
Add node-observ-lib (#25)
* Add node-observ-lib * Remove trends support (not in 10.0 schema) * Make filteringSelector for logs dashboard configurable * Temp change dependency (until PR is merged for commonlib) * Refactor config * Update jsonnetfile.json * Update README * Add separate loki example * Add sep file example
1 parent 720a114 commit 836773a

File tree

467 files changed

+85040
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

467 files changed

+85040
-0
lines changed

docs/node-observ-lib/README.md

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Node exporter observability lib
2+
3+
This jsonnet observability lib can be used to generate observability package for node exporter.
4+
5+
## Import
6+
7+
```sh
8+
jb init
9+
jb install https://github.com/grafana/node_exporter/docs/node-observ-lib
10+
```
11+
12+
## Examples
13+
14+
### Example 1: Basic example
15+
16+
You can use observ-lib to fill in monitoring-mixin structure:
17+
18+
```jsonnet
19+
// mixin.libsonnet file
20+
local nodelib = import 'node-observ-lib/main.libsonnet';
21+
22+
local linux =
23+
nodelib.new()
24+
+ nodelib.withConfigMixin({
25+
filteringSelector: 'job=~".*node.*"',
26+
groupLabels: ['job'],
27+
instanceLabels: ['instance'],
28+
dashboardNamePrefix: 'Node exporter / ',
29+
dashboardTags: ['node-exporter-mixin'],
30+
uid: 'node',
31+
// enable loki logs
32+
enableLokiLogs: true,
33+
});
34+
35+
{
36+
grafanaDashboards+:: linux.grafana.dashboards,
37+
prometheusAlerts+:: linux.prometheus.alerts,
38+
prometheusRules+:: linux.prometheus.recordingRules,
39+
}
40+
41+
```
42+
43+
### Example 2: Fill in monitoring-mixin with default config values and enable loki logs:
44+
45+
46+
```jsonnet
47+
// mixin.libsonnet file
48+
local nodelib = import 'node-observ-lib/main.libsonnet';
49+
50+
local linux =
51+
nodelib.new()
52+
+ nodelib.withConfigMixin({
53+
enableLokiLogs: true,
54+
});
55+
56+
{
57+
grafanaDashboards+:: linux.grafana.dashboards,
58+
prometheusAlerts+:: linux.prometheus.alerts,
59+
prometheusRules+:: linux.prometheus.recordingRules,
60+
}
61+
62+
```
63+
64+
### Example 3: Override some of default config values from file:
65+
66+
67+
```jsonnet
68+
// overrides.libsonnet
69+
{
70+
// Memory utilzation (%) level on which to trigger the
71+
// 'NodeMemoryHighUtilization' alert.
72+
memoryHighUtilizationThreshold: 80,
73+
74+
// Threshold for the rate of memory major page faults to trigger
75+
// 'NodeMemoryMajorPagesFaults' alert.
76+
memoryMajorPagesFaultsThreshold: 1000,
77+
78+
// Disk IO queue level above which to trigger
79+
// 'NodeDiskIOSaturation' alert.
80+
diskIOSaturationThreshold: 20,
81+
}
82+
83+
// mixin.libsonnet file
84+
local configOverride = import './overrides.libsonnet';
85+
local nodelib = import 'node-observ-lib/main.libsonnet';
86+
87+
local linux =
88+
nodelib.new()
89+
+ nodelib.withConfigMixin(configOverride);
90+
91+
{
92+
grafanaDashboards+:: linux.grafana.dashboards,
93+
prometheusAlerts+:: linux.prometheus.alerts,
94+
prometheusRules+:: linux.prometheus.recordingRules,
95+
}
96+
97+
```
98+
99+
### Example 4: Modify specific panel before rendering dashboards
100+
101+
```jsonnet
102+
local g = import './g.libsonnet';
103+
// mixin.libsonnet file
104+
local nodelib = import 'node-observ-lib/main.libsonnet';
105+
106+
local linux =
107+
nodelib.new()
108+
+ nodelib.withConfigMixin({
109+
filteringSelector: 'job=~".*node.*"',
110+
groupLabels: ['job'],
111+
instanceLabels: ['instance'],
112+
dashboardNamePrefix: 'Node exporter / ',
113+
dashboardTags: ['node-exporter-mixin'],
114+
uid: 'node',
115+
})
116+
+ {
117+
grafana+: {
118+
panels+: {
119+
networkSockstatAll+:
120+
+ g.panel.timeSeries.fieldConfig.defaults.custom.withDrawStyle('bars')
121+
}
122+
}
123+
};
124+
125+
{
126+
grafanaDashboards+:: linux.grafana.dashboards,
127+
prometheusAlerts+:: linux.prometheus.alerts,
128+
prometheusRules+:: linux.prometheus.recordingRules,
129+
}
130+
131+
```
132+
133+
## Collectors used:
134+
135+
Grafana Agent or combination of node_exporter/promtail can be used in order to collect data required.
136+
137+
### Logs collection
138+
139+
Loki logs are used to populate logs dashboard and also for annotations.
140+
141+
To use logs, you need to opt-in, with setting `enableLokiLogs: true` in config.
142+
143+
See example above.
144+
145+
The following scrape snippet can be used in grafana-agent/promtail:
146+
147+
```yaml
148+
- job_name: integrations/node_exporter_journal_scrape
149+
journal:
150+
max_age: 24h
151+
labels:
152+
instance: '<your-instance-name>'
153+
job: integrations/node_exporter
154+
relabel_configs:
155+
- source_labels: ['__journal__systemd_unit']
156+
target_label: 'unit'
157+
- source_labels: ['__journal__boot_id']
158+
target_label: 'boot_id'
159+
- source_labels: ['__journal__transport']
160+
target_label: 'transport'
161+
- source_labels: ['__journal_priority_keyword']
162+
target_label: 'level'
163+
```

0 commit comments

Comments
 (0)