You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Both options provide efficient storage of test results while providing high-precision queries.
52
+
53
+
Note that k6 aggregates trend metric data before sending it to Prometheus in both options. The reasons for aggregating data are:
54
+
55
+
- Prometheus stores data in a millisecond precision (`ms`), but k6 metrics collect data points with higher accuracy, nanosecond (`ns`).
56
+
- A load test could generate vast amounts of data points. High-precision raw data could quickly become expensive and complex to scale and is unnecessary when analyzing performance trends.
57
+
58
+
### Counter and Gauges
59
+
60
+
By default, Prometheus supports [Counter and Gauge Metric types](https://prometheus.io/docs/concepts/metric_types/). Therefore, this option is the default of this output and converts all the k6 `Trend` metrics to Counter and Gauges Prometheus metrics.
61
+
62
+
You can configure how to convert all the k6 trend metrics with the [`K6_PROMETHEUS_RW_TREND_STATS` option](#options) that accepts a comma-separated list of stats functions: `count`, `sum`, `min`, `max`, `avg`, `med`, `p(x)`.
63
+
64
+
65
+
Given the list of stats functions, k6 converts all trend metrics to the respective math functions as Prometheus metrics.
66
+
67
+
68
+
For example, `K6_PROMETHEUS_RW_TREND_STATS=p(90),p(95),max` transforms each trend metric into three Prometheus metrics as follows:
69
+
70
+
-`k6_*_p90`
71
+
-`k6_*_p95`
72
+
-`k6_*_max`
73
+
74
+
This option provides a configurable solution to represent `Trend` metrics in Prometheus but has the following drawbacks:
75
+
- Convert a k6 `Trend` metric to several Prometheus metrics.
45
76
- It is impossible to aggregate some gauge values (especially percentiles).
46
77
- It uses a memory-expensive k6 data structure.
47
78
48
-
To resolve these limitations, you can map a trend as a [Prometheus Native Histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).
49
-
To do this,
50
-
you can use the `K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true` environment variable
51
-
(or one of the other ways).
52
-
The output then converts all the trend types into a dedicated Native Histogram.
79
+
### Prometheus Native histogram
53
80
54
-
Mapping trends as native histograms gives you more efficient storage and more precise queries.
55
-
The drawback is that **the feature is experimental, released in Prometheus v2.40.0**.
56
81
57
-
<Blockquotemod="note"title="">
82
+
To resolve the previous drawbacks, you can convert k6 trend metrics to high-fidelity histograms using [Prometheus Native Histogram](https://prometheus.io/docs/concepts/metric_types/#histogram). Each k6 trend metric maps to its respective Prometheus histogram metric: `k6_*`.
58
83
59
-
Native Histogram is an experimental feature, so it has to be enabled ([--enable-feature=native-histograms](https://prometheus.io/docs/prometheus/latest/feature_flags/#native-histograms)).
60
-
Other remote write implementations might not support it yet.
61
84
62
-
</Blockquote>
85
+
<Blockquotemod=""title="">
63
86
64
-
### Naming conventions
87
+
To learn the benefits and outcomes of using Histograms, watch [High-resolution Histograms in the Prometheus TSDB](https://www.youtube.com/watch?v=F72Tk8iaWeA).
65
88
66
-
The output maps the metrics into time series with Name labels.
67
-
As much as possible, k6 respects the [naming best practices](https://prometheus.io/docs/practices/naming) that the Prometheus project defines:
89
+
</Blockquote>
68
90
69
-
* All time series are prefixed with the `k6_` namespace.
70
-
* All time series are suffixed with the base unit of the sample value (if k6 knows what the base unit is).
71
-
* Trends and rates have the relative suffixes, to make them more discoverable.
91
+
Note that Native Histogram is an experimental feature released in Prometheus v2.40.0, and other remote write implementations might not support it yet. In the future, when Prometheus makes this feature stable, k6 will consider using it as the default conversion method for Trend metrics.
72
92
73
-
### Stale trend metrics
93
+
The additional settings to use Native Histogram types are:
94
+
95
+
1. Enable the feature flag [--enable-feature=native-histograms](https://prometheus.io/docs/prometheus/latest/feature_flags/#native-histograms) in Prometheus.
96
+
2. Run the k6 test enabling the `K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true` environment variable.
74
97
75
-
This k6 output can mark the time series at the end of the test as stale.
76
-
To enable the stale marker option, set the `K6_PROMETHEUS_RW_STALE_MARKERS` environment variable to `true`.
77
98
78
-
By default, the metrics are active for 5 minutes after the last flushed sample.
79
-
They are automatically marked as stale after.
80
-
For details about staleness, refer to the [Prometheus docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
81
99
82
100
## Send test metrics to a remote write endpoint
83
101
84
-
<Blockquotemod="note"title="Prometheus as remote write agent">
102
+
To use remote write in Prometheus 2.x:
85
103
86
-
To use remote write in Prometheus 2.x, use the `--web.enable-remote-write-receiver` flag.
87
-
The [xk6 extension](https://github.com/grafana/xk6-output-prometheus-remote) repository has some docker compose examples in the [example](https://github.com/grafana/xk6-output-prometheus-remote/tree/main/example) directory.
88
-
For remote write storage options, refer to the [Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write).
104
+
- Enable the feature flag [--web.enable-remote-write-receiver](https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver). For remote write storage options, refer to the [Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write).
105
+
106
+
- Optionally, enable the feature flag [--enable-feature=native-histograms](https://prometheus.io/docs/prometheus/latest/feature_flags/#native-histograms) in Prometheus 2.40.0 or higher to use [Prometheus Native Histogram](#prometheus-native-histogram).
89
107
90
-
</Blockquote>
91
108
92
-
To send k6 metrics to a remote write endpoint, follow these steps:
109
+
To send k6 metrics to a remote write endpoint:
93
110
1. Set up a running remote write endpoint with an endpoint that k6 can reach.
94
-
1. Run your k6 script, using the `--out` flag with `experimental-prometheus-rw` as the argument:
111
+
2. Run your k6 script, using the `--out` flag with `experimental-prometheus-rw` as the argument:
All the time series have a [`k6_` prefix](#metrics-mapping).
146
+
In the Metric Explorer UI in Grafana, it looks something like this:
147
+
148
+
149
+
116
150
117
-
</CodeGroup>
118
151
119
-
Feel free to request more authentication methods or provide your experience in the [issue tracker](https://github.com/grafana/xk6-output-prometheus-remote/issues).
152
+
The [`xk6-output-prometheus-remote` extension](https://github.com/grafana/xk6-output-prometheus-remote) repository has some docker compose examples. Feel free to request more authentication methods or provide your experience in the [issue tracker](https://github.com/grafana/xk6-output-prometheus-remote/issues).
120
153
121
154
## Options
122
155
123
156
k6 has special options for remote write output.
124
157
125
-
| Name | Type | Default | Description |
126
-
| ---- | ---- | ------- | ----------- |
127
-
|`K6_PROMETHEUS_RW_SERVER_URL`|`string`|`http://localhost:9090/api/v1/write`| URL of the Prometheus remote write implementation's endpoint. |
128
-
|`K6_PROMETHEUS_RW_HEADERS_<here-the-header-key>`| list of `string`|| Additional headers to include in the HTTP requests. `K6_PROMETHEUS_RW_HEADERS_X-MY-HEADER=foo`|
129
-
|`K6_PROMETHEUS_RW_USERNAME`|`string`|| User for the HTTP Basic authentication at the Prometheus remote write endpoint. |
130
-
|`K6_PROMETHEUS_RW_PASSWORD`|`string`|| Password for the HTTP Basic authentication at the Prometheus remote write endpoint. |
131
-
|`K6_PROMETHEUS_RW_PUSH_INTERVAL`|`string`|`5s`| Interval of the metrics' aggregation and upload to the endpoint. |
132
-
|`K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM`|`boolean`| false | If true, it maps the all defined trend metrics as [Native Histograms](#trend). |
133
-
|`K6_PROMETHEUS_RW_TREND_STATS`| list of `string`|`p(99)`| If Native Histogram is not enabled then it defines the stats functions to map for the all defined trend metrics. It's a comma-separated list of stats functions to include (e.g. `p(90),avg,sum`). Check the trend section to know the entire set of the supported stats. |
134
-
|`K6_PROMETHEUS_RW_INSECURE_SKIP_TLS_VERIFY`|`boolean`| false | If true, the HTTP client skips TLS verification on the endpoint. |
135
-
|`K6_PROMETHEUS_RW_STALE_MARKERS`|`boolean`| false | If true, the output at the end of the test marks all the seen time series as stale. |
158
+
| Name | Type | Description |
159
+
| ---- | ---- | ----------- |
160
+
|`K6_PROMETHEUS_RW_SERVER_URL`|`string`| URL of the Prometheus remote write implementation's endpoint. Default is ``http://localhost:9090/api/v1/write``|
161
+
|`K6_PROMETHEUS_RW_HEADERS_<here-the-header-key>`| list of `string`| Additional headers to include in the HTTP requests. `K6_PROMETHEUS_RW_HEADERS_X-MY-HEADER=foo`|
162
+
|`K6_PROMETHEUS_RW_USERNAME`|`string`| User for the HTTP Basic authentication at the Prometheus remote write endpoint. |
163
+
|`K6_PROMETHEUS_RW_PASSWORD`|`string`| Password for the HTTP Basic authentication at the Prometheus remote write endpoint. |
164
+
|`K6_PROMETHEUS_RW_PUSH_INTERVAL`|`string`| Interval of the metrics' aggregation and upload to the endpoint. Default is `5s`|
165
+
|`K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM`|`boolean`| If true, it maps the all defined trend metrics as [Native Histograms](#prometheus-native-histogram). Default is `false`. |
166
+
|`K6_PROMETHEUS_RW_TREND_STATS`| list of `string`| If Native Histogram is not enabled then it defines the stats functions to map for the all defined trend metrics. It's a comma-separated list of stats functions to include (e.g. `p(90),avg,sum`). Check the trend section to know the entire set of the supported stats. Default is `p(99)`|
167
+
|`K6_PROMETHEUS_RW_INSECURE_SKIP_TLS_VERIFY`|`boolean`| If true, the HTTP client skips TLS verification on the endpoint. Default is `false`. |
168
+
|`K6_PROMETHEUS_RW_STALE_MARKERS`|`boolean`| If true, the output at the end of the test marks all the seen time series as stale. Default is `false`. |
169
+
170
+
### Stale trend metrics
171
+
172
+
This k6 output can mark the time series at the end of the test as stale.
173
+
To enable the stale marker option, set the `K6_PROMETHEUS_RW_STALE_MARKERS` environment variable to `true`.
174
+
175
+
By default, the metrics are active for 5 minutes after the last flushed sample.
176
+
They are automatically marked as stale after.
177
+
For details about staleness, refer to the [Prometheus docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
0 commit comments