Skip to content

Commit a189b41

Browse files
TofelTofel
committed
more docs
1 parent 827b282 commit a189b41

File tree

17 files changed

+309
-62
lines changed

17 files changed

+309
-62
lines changed

‎book/src/SUMMARY.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,11 +77,13 @@
7777
- [Custom Loki metrics](./libs/wasp/benchspy/loki_custom.md)
7878
- [Standard Prometheus metrics](./libs/wasp/benchspy/prometheus_std.md)
7979
- [Custom Prometheus metrics](./libs/wasp/benchspy/prometheus_custom.md)
80-
- [Defining a new report]()
81-
- [Adding new QueryExecutor]()
80+
- [Reports](./libs/wasp/benchspy/reports/overview.md)
81+
- [Standard Report](./libs/wasp/benchspy/reports/standard_report.md)
82+
- [Adding new QueryExecutor](./libs/wasp/benchspy/reports/new_executor.md)
83+
- [Adding new standard load metric]()
84+
- [Adding new standard resource metric]()
85+
- [Defining a new report](./libs/wasp/benchspy/reports/new_report.md)
8286
- [Adding new storage]()
83-
- [Adding new standard load metric]()
84-
- [Adding new standard resource metric]()
8587
- [How to](./libs/wasp/how-to/overview.md)
8688
- [Start local observability stack](./libs/wasp/how-to/start_local_observability_stack.md)
8789
- [Try it out quickly](./libs/wasp/how-to/run_included_tests.md)

‎book/src/libs/wasp/benchspy/first_test.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ baseLineReport, err := benchspy.NewStandardReport(
3737
// random hash, this should be commit or hash of the Application Under Test (AUT)
3838
"e7fc5826a572c09f8b93df3b9f674113372ce924",
3939
// use built-in queries for an executor that fetches data directly from the WASP generator
40-
benchspy.WithStandardQueryExecutorType(benchspy.StandardQueryExecutor_Generator),
40+
benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Generator),
4141
// WASP generators
4242
benchspy.WithGenerators(gen),
4343
)
@@ -85,7 +85,7 @@ defer cancelFn()
8585
currentReport, previousReport, err := benchspy.FetchNewStandardReportAndLoadLatestPrevious(
8686
fetchCtx,
8787
"e7fc5826a572c09f8b93df3b9f674113372ce925",
88-
benchspy.WithStandardQueryExecutorType(benchspy.StandardQueryExecutor_Generator),
88+
benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Generator),
8989
benchspy.WithGenerators(newGen),
9090
)
9191
require.NoError(t, err, "failed to fetch current report or load the previous one")

‎book/src/libs/wasp/benchspy/loki_std.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ defer cancelFn()
4646
baseLineReport, err := benchspy.NewStandardReport(
4747
"c2cf545d733eef8bad51d685fcb302e277d7ca14",
4848
// notice the different standard executor type
49-
benchspy.WithStandardQueryExecutorType(benchspy.StandardQueryExecutor_Loki),
49+
benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Loki),
5050
benchspy.WithGenerators(gen),
5151
)
5252
require.NoError(t, err, "failed to create original report")

‎book/src/libs/wasp/benchspy/prometheus_std.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Just like in previous examples we will use built-in Prometheus metrics and fetch
2222
```go
2323
baseLineReport, err := benchspy.NewStandardReport(
2424
"91ee9e3c903d52de12f3d0c1a07ac3c2a6d141fb",
25-
benchspy.WithQueryExecutorType(benchspy.StandardQueryExecutor_Prometheus),
25+
benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Prometheus),
2626
benchspy.WithPrometheusConfig(promConfig),
2727
// needed even if we don't query Loki or Generator,
2828
// because we calculate test time range based on

‎book/src/libs/wasp/benchspy/reports/new_executor.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# BenchSpy - Adding new QueryExecutor
2+
3+
As mentioned previously the `StandardReport` comes with support of three different data types:
4+
* `WASP generator`
5+
* `Loki`
6+
* `Prometheus`
7+
8+
Each of them implements the `QueryExecutor` interface:
9+
```go
10+
type QueryExecutor interface {
11+
// Kind returns the type of the QueryExecutor
12+
Kind() string
13+
// Validate checks if the QueryExecutor has all the necessary data and configuration to execute the queries
14+
Validate() error
15+
// Execute executes the queries and populates the QueryExecutor with the results
16+
Execute(ctx context.Context) error
17+
// Results returns the results of the queries, where key is the name of the query and value is the result
18+
Results() map[string]interface{}
19+
// IsComparable checks whether both QueryExecutors can be compared (e.g. they have the same type, queries are the same, etc.), and returns an error (if any difference is found)
20+
IsComparable(other QueryExecutor) error
21+
// TimeRange sets the time range for the queries
22+
TimeRange(startTime, endTime time.Time)
23+
}
24+
```
25+
26+
Most of the functions that your new `QueryExecutor` should implement are self-explanatory and I will skip them and focus on two that might not be obvious.
27+
28+
## Kind
29+
Kind should return a name as string of your `QueryExecutor`. It needs to be unique, because `StandardReport` uses it, when unmarshalling JSON files with
30+
stored reports. That also means that you need to add support for your new executor in the `StandardReport.UnmarshallJSON` function.
31+
32+
> [!NOTE]
33+
> If your new `QueryExecutor` uses interfaces or `interface{}` or `any` types, or has some fields that should not/cannot be serialized,
34+
> remember to add custom `MarshallJSON` and `UnmarshallJSON` functions to it. Existing executors can serve as a good example.
35+
36+
## TimeRange
37+
`StandardReport` calls this method just before calling `Execute()` for each executor. It is used to set the time range for the query (required
38+
for Loki and Prometheus). This is done primarly to avoid the need for manual calculation of the time range for the test, because `StandardReport` does it automatically by
39+
analysing schedules of all generators and finding earliest start time and latest end time.

‎book/src/libs/wasp/benchspy/reports/new_report.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Defining a new report
2+
3+
Each `BenchSpy` report should implement the `Reporter` interface, which handles 3 responsibilities:
4+
* storage and retrival (`Storer` interface)
5+
* data fetching (`DataFetcher` interface)
6+
* comparator (`Comparator` interface)
7+
8+
### Definition
9+
```go
10+
type Reporter interface {
11+
Storer
12+
DataFetcher
13+
Comparator
14+
}
15+
16+
```
17+
18+
Comparison of actual performance data should not be part of the report and should be done independently from it,
19+
ideally using simple Go's `require` and `assert` statements.
20+
21+
# Storer interface
22+
## Definition
23+
```go
24+
type Storer interface {
25+
// Store stores the report in a persistent storage and returns the path to it, or an error
26+
Store() (string, error)
27+
// Load loads the report from a persistent storage and returns it, or an error
28+
Load(testName, commitOrTag string) error
29+
// LoadLatest loads the latest report from a persistent storage and returns it, or an error
30+
LoadLatest(testName string) error
31+
}
32+
```
33+
34+
If storing the reports on the local filesystem under Git fulfills your requirements you can reuse the `LocalStorage`
35+
implementation of `Storer`.
36+
37+
If you would like to store them in S3 or a database, you will need to implement the interface yourself.
38+
39+
# DataFetcher interface
40+
## Definition
41+
```go
42+
type DataFetcher interface {
43+
// Fetch populates the report with the data from the test
44+
FetchData(ctx context.Context) error
45+
}
46+
```
47+
This interface is only concerned with fetching the data from the data source and populating the results.
48+
49+
# Comparator interface
50+
## Definition
51+
```go
52+
type Comparator interface {
53+
// IsComparable checks whether both reports can be compared (e.g. test config is the same, app's resources are the same, queries or metrics used are the same, etc.), and an error if any difference is found
54+
IsComparable(otherReport Reporter) error
55+
}
56+
57+
```
58+
59+
This interface is only concerned with making sure that both report are comparable, for example by checking:
60+
* whether both use generators with identical configurations (such as load type, load characteristics)
61+
* whether both report feature the same data sources and queries

‎book/src/libs/wasp/benchspy/reports/overview.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# BenchSpy - Reports
2+
3+
`BenchSpy` was created with composability in mind. It comes with a `StandardReport` implementation that
4+
should cover the majority if needs and an ease of adding fully customised reports.
5+
6+
Let's look at the `StandardReport` in the [next chapter](./standard_report.md).

‎book/src/libs/wasp/benchspy/reports/standard_report.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# BenchSpy - Standard Report
2+
3+
`StandardReport` comes with built-in support for three types of data sources:
4+
* `WASP Generator`
5+
* `Loki`
6+
* `Prometheus`
7+
8+
Each of them allows you to both use pre-defined metrics or use your own.
9+
10+
## Pre-defined (standard) metrics
11+
12+
### WASP generator and Loki
13+
Both query executors focus on the characteristics of the load generated with WASP.
14+
The datasets they work on are almost identical, because the former allows you to query load-specific
15+
data before its sent to Loki. The latter offers you richer querying options (via `LogQL`) and access
16+
to actual load profile (as opposed to the configured one).
17+
18+
Both query executors have following predefined metrics:
19+
* median latency
20+
* 95th percentile latency
21+
* error rate
22+
23+
Latency is understood as the round time from making a request to receiving a response
24+
from the Application Under Test.
25+
26+
Error rate is the ratio of failed responses to the total number of responses. This include
27+
both requests that timed out or returned an error from `Gun` or `Vu` implementation.
28+
29+
### Prometehus
30+
On the other hand, these standard metrics focus on resource consumption by the application you are testing,
31+
instead on the load generation.
32+
33+
They include the following:
34+
* median CPU usage
35+
* 95th percentil of CPU usage
36+
* median memory usage
37+
* 95th percentil of memory usage
38+
39+
In both cases queries focus on `total` consumption, which consists of the sum of what the underlaying system and
40+
you appplication uses.
41+
42+
### How to use
43+
As mentioned in the examples, to use predefined metrics you should use the `NewStandardReport` method:
44+
```go
45+
report, err := benchspy.NewStandardReport(
46+
"91ee9e3c903d52de12f3d0c1a07ac3c2a6d141fb",
47+
// Query executor types for which standard metrics should be generated
48+
benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Prometheus, benchspy.StandardQueryExecutor_Loki),
49+
// Prometheus configuration is required if using standard Prometheus metrics
50+
benchspy.WithPrometheusConfig(benchspy.NewPrometheusConfig("node[^0]")),
51+
// WASP generators
52+
benchspy.WithGenerators(gen),
53+
)
54+
require.NoError(t, err, "failed to create the report")
55+
```
56+
57+
## Custom metrics
58+
### WASP Generator
59+
Since `WASP` stores AUT's responses in each generator you can create custom metrics that leverage them. Here's an example
60+
of adding a function that returns the number of responses that timed out:
61+
```go
62+
var generator *wasp.Generator
63+
64+
var timeouts = func(responses *wasp.SliceBuffer[wasp.Response]) (float64, error) {
65+
if len(responses.Data) == 0 {
66+
return 0, nil
67+
}
68+
69+
timeoutCount := 0.0
70+
inTimeCount := 0.0
71+
for _, response := range responses.Data {
72+
if response.Timeout {
73+
timeoutCount = timeoutCount + 1
74+
} else {
75+
inTimeCount = inTimeCount + 1
76+
}
77+
}
78+
79+
return timeoutCount / (timeoutCount + inTimeCount), nil
80+
}
81+
82+
generatorExectutor, err := NewGeneratorQueryExecutor(generator, map[string]GeneratorQueryFn{
83+
"timeout_ratio": timeouts,
84+
})
85+
require.NoError(t, err, "failed to create WASP Generator Query Executor")
86+
```
87+
88+
### Loki
89+
Using custom `LogQL` queries is even simpler as all you need to do is create a new instance of
90+
`NewLokiQueryExecutor` with a map of desired queries.
91+
```go
92+
var generator *wasp.Generator
93+
94+
lokiQueryExecutor := benchspy.NewLokiQueryExecutor(
95+
map[string]string{
96+
"responses_over_time": fmt.Sprintf("sum(count_over_time({my_label=~\"%s\", test_data_type=~\"responses\", gen_name=~\"%s\"} [1s])) by (node_id, go_test_name, gen_name)", label, gen.Cfg.GenName),
97+
},
98+
generator.Cfg.LokiConfig,
99+
)
100+
```
101+
> [!NOTE]
102+
> In order to effectively write `LogQL` queries for WASP you need to be familar with how to label
103+
> your generators and what `test_data_types` WASP uses.
104+
105+
### Prometheus
106+
Adding custom `PromQL` queries is equally straight-forward:
107+
```go
108+
promConfig := benchspy.NewPrometheusConfig()
109+
110+
prometheusExecutor, err := benchspy.NewPrometheusQueryExecutor(
111+
map[string]string{
112+
"cpu_rate_by_container": "rate(container_cpu_usage_seconds_total{name=~\"chainlink.*\"}[5m])[30m:1m]",
113+
},
114+
*promConfig,
115+
)
116+
require.NoError(t, err)
117+
```
118+
119+
### How to use with StandardReport
120+
Using custom queries with a `StandardReport` is rather simple. Instead of passing `StandardQueryExecutorType` with the
121+
functional option `WithStandardQueries` you should pass the `QueryExecutors` created above with `WithQueryExecutors` option:
122+
```go
123+
report, err := benchspy.NewStandardReport(
124+
"2d1fa3532656c51991c0212afce5f80d2914e34e",
125+
benchspy.WithQueryExecutors(generatorExectutor, lokiQueryExecutor, prometheusExecutor),
126+
benchspy.WithGenerators(gen),
127+
)
128+
require.NoError(t, err, "failed to create baseline report")

‎wasp/benchspy/generator.go‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,19 +20,26 @@ type GeneratorQueryExecutor struct {
2020
QueryResults map[string]interface{} `json:"query_results"`
2121
}
2222

23-
func NewGeneratorQueryExecutor(generator *wasp.Generator) (*GeneratorQueryExecutor, error) {
23+
func NewStandardGeneratorQueryExecutor(generator *wasp.Generator) (*GeneratorQueryExecutor, error) {
2424
g := &GeneratorQueryExecutor{
25-
KindName: string(StandardQueryExecutor_Generator),
26-
Generator: generator,
25+
KindName: string(StandardQueryExecutor_Generator),
2726
}
2827

2928
queries, err := g.generateStandardQueries()
3029
if err != nil {
3130
return nil, err
3231
}
3332

34-
g.Queries = queries
35-
g.QueryResults = make(map[string]interface{})
33+
return NewGeneratorQueryExecutor(generator, queries)
34+
}
35+
36+
func NewGeneratorQueryExecutor(generator *wasp.Generator, queries map[string]GeneratorQueryFn) (*GeneratorQueryExecutor, error) {
37+
g := &GeneratorQueryExecutor{
38+
KindName: string(StandardQueryExecutor_Generator),
39+
Generator: generator,
40+
Queries: queries,
41+
QueryResults: make(map[string]interface{}),
42+
}
3643

3744
return g, nil
3845
}
@@ -165,6 +172,10 @@ func (g *GeneratorQueryExecutor) standardQuery(standardMetric StandardLoadMetric
165172
return p95Fn, nil
166173
case ErrorRate:
167174
errorRateFn := func(responses *wasp.SliceBuffer[wasp.Response]) (float64, error) {
175+
if len(responses.Data) == 0 {
176+
return 0, nil
177+
}
178+
168179
failedCount := 0.0
169180
successfulCount := 0.0
170181
for _, response := range responses.Data {

0 commit comments

Comments
 (0)