|
| 1 | +# Hello world observ lib |
| 2 | + |
| 3 | +This lib can be used as a starter template of modular observ lib. Modular observ libs are highly reusable observability package containing grafana dashboards and prometheus alerts/rules. They can be used in as a replacement or in conjuction to monitoring-mixins format. |
| 4 | + |
| 5 | + |
| 6 | +## Import |
| 7 | + |
| 8 | +```sh |
| 9 | +jb init |
| 10 | +jb install https://github.com/grafana/jsonnet-libs/helloworld-observ-lib |
| 11 | +``` |
| 12 | + |
| 13 | +## Modular observability lib format |
| 14 | + |
| 15 | +```jsonnet |
| 16 | +
|
| 17 | +{ |
| 18 | + config: { |
| 19 | + //common options |
| 20 | + }, |
| 21 | +
|
| 22 | + grafana: { |
| 23 | + |
| 24 | + // grafana templated variables to reuse across mutltiple dashboards |
| 25 | + variables: { |
| 26 | + datasources: { |
| 27 | + prometheus: {}, |
| 28 | + loki: {}, |
| 29 | + }, |
| 30 | + multiInstance: [], |
| 31 | + singleInstace: [], |
| 32 | + queriesSelector: "", |
| 33 | + }, |
| 34 | +
|
| 35 | + // grafana targets (queries) to attach to panels |
| 36 | + targets: { |
| 37 | + target1: <target1>, |
| 38 | + target3: <target2>, |
| 39 | + ... |
| 40 | + targetN: <targetN>, |
| 41 | + }, |
| 42 | +
|
| 43 | + // grafana panels |
| 44 | + panels: { |
| 45 | + panel1: <panel1>, |
| 46 | + panel2: <panel2>, |
| 47 | + ... |
| 48 | + panelN: <panelN>, |
| 49 | + }, |
| 50 | +
|
| 51 | + // grafana dashboards |
| 52 | + dashboards: { |
| 53 | + dashboard1: <dashboard1>, |
| 54 | + dashboard2: <dashboard2>, |
| 55 | + ... |
| 56 | + dashboardN: <dashboardN>, |
| 57 | + }, |
| 58 | +
|
| 59 | + // grafana annotations |
| 60 | + annotations: { |
| 61 | + annotation1: <annotation1>, |
| 62 | + ... |
| 63 | + }, |
| 64 | + |
| 65 | + // grafana dashboard links |
| 66 | + links: {}, |
| 67 | + }, |
| 68 | +
|
| 69 | + prometheus: { |
| 70 | + alerts: {}, |
| 71 | + rules: {}, |
| 72 | + }, |
| 73 | +} |
| 74 | +
|
| 75 | +``` |
| 76 | + |
| 77 | +## Pros of using modular observabilty format |
| 78 | + |
| 79 | +- Uses (jsonnet)[https://jsonnet.org/learning/tutorial.html], jsonnet-bundler, and grafonnet[https://github.com/grafana/grafonnet] |
| 80 | +- Highly customizable and flexible: |
| 81 | + |
| 82 | +Any object like `panel`, `target` (query) can be easily referenced by key and then overriden before output of the lib is provided by using jsonnet (patching)[https://tanka.dev/tutorial/environments#patching] technique: |
| 83 | + |
| 84 | +```jsonnet |
| 85 | +local helloworldlib = import './main.libsonnet'; |
| 86 | +
|
| 87 | +local helloworld = |
| 88 | + helloworldlib.new( |
| 89 | + filteringSelector='job="integrations/helloworld"', |
| 90 | + uid='myhelloworld', |
| 91 | + groupLabels=['environment', 'cluster'], |
| 92 | + instanceLabels=['host'], |
| 93 | + ) |
| 94 | + + |
| 95 | + { |
| 96 | + grafana+: { |
| 97 | + panels+: { |
| 98 | + panel1+: |
| 99 | + g.panel.timeSeries.withDescription("My new description for panel1") |
| 100 | + } |
| 101 | + } |
| 102 | + }; |
| 103 | +``` |
| 104 | + |
| 105 | +- Due to high decomposition level, not only dashboards but single panels can be imported ('cherry-picked') from the library to be used in other dashboards |
| 106 | +- Format introduces mandatory arguments that each library should have: `filteringSelector`, `instanceLabels`, `groupLabels`, `uid`. Proper use of those parameters ensures library can be used to instantiate multiple copies of the observability package in the same enviroment without `ids` conflicts or timeSeries overlapping. |
| 107 | + |
| 108 | +## Examples |
| 109 | + |
| 110 | +### Example 1: Monitoring-Mixin example |
| 111 | + |
| 112 | +You can use lib to fill in [monitoring-mixin](https://monitoring.mixins.dev/) structure: |
| 113 | + |
| 114 | +```jsonnet |
| 115 | +// mixin.libsonnet file |
| 116 | +
|
| 117 | +local helloworldlib = import 'helloworld-observ-lib/main.libsonnet'; |
| 118 | +
|
| 119 | +local helloworld = |
| 120 | + helloworldlib.new( |
| 121 | + filteringSelector='job="integrations/helloworld"', |
| 122 | + uid='myhelloworld', |
| 123 | + groupLabels=['environment', 'cluster'], |
| 124 | + instanceLabels=['host'], |
| 125 | + ); |
| 126 | +
|
| 127 | +// populate monitoring-mixin: |
| 128 | +{ |
| 129 | + grafanaDashboards+:: helloworld.grafana.dashboards, |
| 130 | + prometheusAlerts+:: helloworld.prometheus.alerts, |
| 131 | + prometheusRules+:: helloworld.prometheus.recordingRules, |
| 132 | +} |
| 133 | +``` |
| 134 | + |
| 135 | +### Example 2: Changing specific panel before rendering dashboards |
| 136 | + |
| 137 | +We can point to any object (i.e grafana.panels.panel1) and modify it by using (jsonnnet mixins)[https://jsonnet.org/learning/tutorial.html]. |
| 138 | + |
| 139 | +For example, let's modify panel's default draw style to bars by mutating it with (grafonnet)[https://grafana.github.io/grafonnet/API/panel/timeSeries/index.html#fn-fieldconfigdefaultscustomwithdrawstyle] |
| 140 | + |
| 141 | +``` |
| 142 | +local g = import './g.libsonnet'; |
| 143 | +local helloworldlib = import 'helloworld-observ-lib/main.libsonnet'; |
| 144 | +
|
| 145 | +local helloworld = |
| 146 | + helloworldlib.new( |
| 147 | + filteringSelector='job="integrations/helloworld"', |
| 148 | + uid='myhelloworld', |
| 149 | + groupLabels=['environment', 'cluster'], |
| 150 | + instanceLabels=['host'], |
| 151 | + ) |
| 152 | + + { |
| 153 | + grafana+: { |
| 154 | + panels+: { |
| 155 | + networkSockstatAll+: |
| 156 | + + g.panel.timeSeries.fieldConfig.defaults.custom.withDrawStyle('bars') |
| 157 | + } |
| 158 | + } |
| 159 | + }; |
| 160 | +
|
| 161 | +// populate monitoring-mixin: |
| 162 | +{ |
| 163 | + grafanaDashboards+:: helloworld.grafana.dashboards, |
| 164 | + prometheusAlerts+:: helloworld.prometheus.alerts, |
| 165 | + prometheusRules+:: helloworld.prometheus.recordingRules, |
| 166 | +} |
| 167 | +
|
| 168 | +``` |
| 169 | + |
| 170 | +### Example 3: Optional logs collection |
| 171 | + |
| 172 | +Grafana Loki datasource is used to populate logs dashboard and also for quering annotations. |
| 173 | + |
| 174 | +To opt-out, you can set `enableLokiLogs: false` in config: |
| 175 | + |
| 176 | +``` |
| 177 | +local helloworldlib = import 'helloworld-observ-lib/main.libsonnet'; |
| 178 | +
|
| 179 | +local helloworld = |
| 180 | + helloworldlib.new( |
| 181 | + filteringSelector='job="integrations/helloworld"', |
| 182 | + uid='myhelloworld', |
| 183 | + groupLabels=['environment', 'cluster'], |
| 184 | + instanceLabels=['host'], |
| 185 | + ) |
| 186 | + + helloworldlib.withConfigMixin( |
| 187 | + { |
| 188 | + // disable loki logs |
| 189 | + enableLokiLogs: false, |
| 190 | + } |
| 191 | + ); |
| 192 | +
|
| 193 | +// populate monitoring-mixin: |
| 194 | +{ |
| 195 | + grafanaDashboards+:: helloworld.grafana.dashboards, |
| 196 | + prometheusAlerts+:: helloworld.prometheus.alerts, |
| 197 | + prometheusRules+:: helloworld.prometheus.recordingRules, |
| 198 | +} |
| 199 | +``` |
| 200 | + |
| 201 | +## Recommended dev environment |
| 202 | + |
| 203 | +To speed up developing observability libs as-code, we recommend to work in the following dev enviroment: |
| 204 | + |
| 205 | +- Setup Vscode with [Jsonnet Language Server][https://marketplace.visualstudio.com/items?itemName=Grafana.vscode-jsonnet] |
| 206 | +- Setup format on save in vscode to lint jsonnet automatically. |
0 commit comments