|
| 1 | +# Benchtool |
| 2 | + |
| 3 | +The `benchtool` is a small load testing utility to generate load for Prometheus |
| 4 | +remote-write and query API endpoints. It uses a statically configured YAML |
| 5 | +workload file to describe the series and queries used for the generate load. |
| 6 | + |
| 7 | +**warning:** the specifications outlined here are subject to change. It is |
| 8 | +likely the specs in this file will require some adjustment to accomodate |
| 9 | +expanded functionality and this should not be consided stable. |
| 10 | + |
| 11 | +## Workload file |
| 12 | + |
| 13 | +The workload file can be configured as a follows: |
| 14 | + |
| 15 | +```yaml |
| 16 | +replicas: "<int>" |
| 17 | +queries: |
| 18 | + - expr_template: "<string>" |
| 19 | + interval: "<duration>" |
| 20 | + num_queries: "<int>" |
| 21 | + series_type: "<enum series-type>" |
| 22 | + regex: "<bool>" |
| 23 | +series: |
| 24 | + - name: "<string>" |
| 25 | + type: "<enum series-type>" |
| 26 | + static_labels: |
| 27 | + "<string>": "<string>" |
| 28 | + labels: |
| 29 | + - name: "<string>" |
| 30 | + unique_values: "<int>" |
| 31 | + value_prefix: "<string>" |
| 32 | +write_options: |
| 33 | + batch_size: "<int>" |
| 34 | + interval: "<duration>" |
| 35 | +``` |
| 36 | +
|
| 37 | +### Series |
| 38 | +
|
| 39 | +The series configuration section of the workload is used to configure the |
| 40 | +metrics generated by the workload. It is a list of series objects that allow the |
| 41 | +workload author to declare name and characteristics of each series the workload |
| 42 | +will generate. |
| 43 | +
|
| 44 | +- **name** is the metric name of the series that will generated. The name will |
| 45 | + be used as the reserved `__name__` label for each series generated using this |
| 46 | + series config. |
| 47 | +- **type** references the type of series that will be generated and the |
| 48 | + characteristics of it's underlying data. See the [series type |
| 49 | + section](#series-type) for more information. |
| 50 | +- **static_labels** is a map of string values to string keys. These will be |
| 51 | + added as labels to each series generated from this series description. |
| 52 | +- **labels** is field that allows for the input of a list of label descriptions. |
| 53 | + The series generated from this series description will be based on the result |
| 54 | + of all possible combinations of the provided label descriptions. |
| 55 | + - **name** refers to the label name of the generated label. |
| 56 | + - **unique_values** referes to the number of values to generate for this |
| 57 | + label. |
| 58 | + - **value_prefix** refers to the prefix that will used for each generated |
| 59 | + label value. The resultant label value will be the prefix appended with |
| 60 | + `-<integer>`, where the integer is a value between `0` and the configured |
| 61 | + `unique_values`. |
| 62 | + |
| 63 | +#### Series types |
| 64 | + |
| 65 | +Series types are used in both the series and queries section of the workload. |
| 66 | +The denote the character and properties of the data of the generated or queried |
| 67 | +series. The following series types are supported. |
| 68 | + |
| 69 | +- *gauge-zero*: correlates to a gauge series that will contantly be zero. This |
| 70 | + is a commonly seen pattern in Prometheus info metrics. |
| 71 | +- *gauge-random*: correlates to a gauge series that constant changes to a random |
| 72 | + value. The values is chosen using `rand.Float64`. |
| 73 | +- *counter-one* correlates to a counter that increases by 1 at every interval. |
| 74 | +- *counter-random* correlates to a counter that increases by a random amount |
| 75 | + every interval. The random amount is not constant and is currently chosen |
| 76 | + using `rand.Int()`. |
| 77 | + |
| 78 | +### Global options |
| 79 | + |
| 80 | +- **replicas**: Replicas is meant to be a stand in for the host label. For each |
| 81 | +value between 0 and the configured replica value, a `bench_replica` label will |
| 82 | +be added and appended to each generated series. This label will also be used for |
| 83 | +queries configured to use regular expressions. |
| 84 | + |
| 85 | +### Queries |
| 86 | + |
| 87 | +The query workload is made up of a list of configured queries. Queries have a |
| 88 | +variety of options that can be configured to get the desired behavior. |
| 89 | + |
| 90 | +- **series_type:** Series type to use for this query. A series name generated |
| 91 | + from the `series` section of the Cortex config will be chosen and used for |
| 92 | + each generated query. |
| 93 | +- **expr_template** is used to configure the query expression that will be run. |
| 94 | + It uses a modified [Go text template](https://golang.org/pkg/text/template/) |
| 95 | + to allow for the name of a series and a matcher to be injected into the query. |
| 96 | + The name will be sourced from the series name and the matcher will be a match |
| 97 | + on the `bench_replicas` label. |
| 98 | +- **regex** is a boolean value that, if enabled, will cause the injected |
| 99 | + `bench_replica` matcher on the query to be a regex match on a single replica |
| 100 | + instead of an exact matcher. |
| 101 | +- **interval:** Interval at which each instantiation of this query description |
| 102 | + will run. A random jitter no greater than half the configured interval is |
| 103 | + applied before the first run of each query to ensure that queries are not all |
| 104 | + scheduled to run simultaneously. |
| 105 | +- **num_queries:** Number of replicas to run of this configured query. Each |
| 106 | + instance of the generated query will select a random series with the same |
| 107 | + series type. Each query will also apply a unique random jitter on the interval |
| 108 | + before it's first run. |
| 109 | + |
| 110 | +### Write options |
| 111 | + |
| 112 | +- **batch_size** determines the number of samples send in each remote-write |
| 113 | + request. |
| 114 | +- **interval** is the duration period between when each batch of remote-write |
| 115 | + requests are generated and queued to be sent. |
| 116 | + |
| 117 | +### Example workload file |
| 118 | + |
| 119 | +```yaml |
| 120 | +--- |
| 121 | +queries: |
| 122 | + - expr_template: sum(<<.Name>>{<<.Matchers>>}) |
| 123 | + interval: 1m |
| 124 | + num_queries: 3 |
| 125 | + series_type: gauge-random |
| 126 | + - expr_template: sum(<<.Name>>{<<.Matchers>>}) |
| 127 | + interval: 1m |
| 128 | + num_queries: 5 |
| 129 | + series_type: gauge-zero |
| 130 | + time_range: 2h |
| 131 | +replicas: 5 |
| 132 | +series: |
| 133 | + - labels: |
| 134 | + - name: label_01 |
| 135 | + unique_values: 5 |
| 136 | + value_prefix: label_value_01 |
| 137 | + - name: label_02 |
| 138 | + unique_values: 20 |
| 139 | + value_prefix: label_value_02 |
| 140 | + name: metric_gauge_random_01 |
| 141 | + static_labels: |
| 142 | + static: "true" |
| 143 | + type: gauge-random |
| 144 | + - labels: |
| 145 | + - name: label_01 |
| 146 | + unique_values: 5 |
| 147 | + value_prefix: label_value_01 |
| 148 | + - name: label_02 |
| 149 | + unique_values: 20 |
| 150 | + value_prefix: label_value_02 |
| 151 | + name: metric_gauge_zero_01 |
| 152 | + static_labels: |
| 153 | + static: "true" |
| 154 | + type: gauge-zero |
| 155 | +write_options: |
| 156 | + batch_size: 1000 |
| 157 | + interval: 15s |
| 158 | +``` |
| 159 | + |
| 160 | +## Consistency |
| 161 | + |
| 162 | +To ensure results are consistent between runs, the seed of the random number |
| 163 | +generator used by the workload is determined by the configured `-bench.id` flag |
| 164 | +value of the benchtool instance. |
| 165 | + |
| 166 | +This will ensure two `benchtool` processes run with the same id and workload |
| 167 | +config file will result in the same behavior between runs. |
0 commit comments