rename Generator to Direct

Author: Tofel

book/src/libs/wasp/benchspy/first_test.md

@@ -2,7 +2,7 @@
 
 Let's start with a simplest case, which doesn't require you to have any of the observability stack, but only `WASP` and the application you are testing.
 `BenchSpy` comes with some built-in `QueryExecutors` each of which additionaly has predefined metrics that you can use. One of these executors is the
-`GeneratorQueryExecutor` that fetches metrics directly from `WASP` generators.
+`DirectQueryExecutor` that fetches metrics directly from `WASP` generators.
 
 Our first test will follow the following logic:
 * Run a simple load test
@@ -37,7 +37,7 @@ baseLineReport, err := benchspy.NewStandardReport(
     // random hash, this should be commit or hash of the Application Under Test (AUT)
     "e7fc5826a572c09f8b93df3b9f674113372ce924",
     // use built-in queries for an executor that fetches data directly from the WASP generator
-    benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Generator),
+    benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Direct),
     // WASP generators
     benchspy.WithGenerators(gen),
 )
@@ -85,7 +85,7 @@ defer cancelFn()
 currentReport, previousReport, err := benchspy.FetchNewStandardReportAndLoadLatestPrevious(
     fetchCtx,
     "e7fc5826a572c09f8b93df3b9f674113372ce925",
-    benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Generator),
+    benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Direct),
     benchspy.WithGenerators(newGen),
 )
 require.NoError(t, err, "failed to fetch current report or load the previous one")

book/src/libs/wasp/benchspy/loki_dillema.md

@@ -1,11 +1,11 @@
 # BenchSpy - To Loki or not to Loki?
 
-You might be asking yourself whether you should use `Loki` or `Generator` query executor if all you
+You might be asking yourself whether you should use `Loki` or `Direct` query executor if all you
 need are basic latency metrics.
 
 As a rule of thumb, if all you need is a single number that describes the median latency or error rate
 and you are not interested in directly comparing time series, minimum or maximum values or any kinds
-of more advanced calculation on raw data, then you should go with the `Generator`.
+of more advanced calculation on raw data, then you should go with the `Direct`.
 
 Why?
 

book/src/libs/wasp/benchspy/loki_std.md

@@ -101,4 +101,7 @@ compareMedian(string(benchspy.ErrorRate))
 
 We have used standard metrics, which are the same as in the first test, now let's see how you can use your custom LogQl queries.
 
+> [!NOTE]
+> Don't know whether to use `Loki` or `Direct` query executors? [Read this!](./loki_dillema.md)
+
 You can find the full example [here](...).

book/src/libs/wasp/benchspy/overview.md

@@ -6,10 +6,11 @@ It's main characteristics are:
 * three built-in data sources:
     * `Loki`
     * `Prometheus`
-    * `WASP generator`
+    * `Direct`
 * standard/pre-defined metrics for each data source
 * ease of extensibility with custom metrics
 * ability to load latest performance report based on Git history
+* 88% unit test coverage
 
 It doesn't come with any comparation logic, other than making sure that performance reports are comparable (e.g. they mesure the same metrics in the same way),
 leaving total freedom to the user.

book/src/libs/wasp/benchspy/reports/new_executor.md

@@ -1,7 +1,7 @@
 # BenchSpy - Adding new QueryExecutor
 
 As mentioned previously the `StandardReport` comes with support of three different data types:
-* `WASP generator`
+* `Direct`
 * `Loki`
 * `Prometheus`
 

book/src/libs/wasp/benchspy/reports/standard_report.md

@@ -1,15 +1,15 @@
 # BenchSpy - Standard Report
 
 `StandardReport` comes with built-in support for three types of data sources:
-* `WASP Generator`
+* `Direct`
 * `Loki`
 * `Prometheus`
 
 Each of them allows you to both use pre-defined metrics or use your own.
 
 ## Pre-defined (standard) metrics
 
-### WASP generator and Loki
+### Direct and Loki
 Both query executors focus on the characteristics of the load generated with WASP.
 The datasets they work on are almost identical, because the former allows you to query load-specific
 data before its sent to Loki. The latter offers you richer querying options (via `LogQL`) and access
@@ -79,10 +79,10 @@ var timeouts = func(responses *wasp.SliceBuffer[wasp.Response]) (float64, error)
     return timeoutCount / (timeoutCount + inTimeCount), nil
 }
 
-generatorExectutor, err := NewGeneratorQueryExecutor(generator, map[string]GeneratorQueryFn{
+directExectutor, err := NewDirectQueryExecutor(generator, map[string]DirectQueryFn{
     "timeout_ratio": timeouts,
 })
-require.NoError(t, err, "failed to create WASP Generator Query Executor")
+require.NoError(t, err, "failed to create Direct Query Executor")
 ```
 
 ### Loki
@@ -122,7 +122,7 @@ functional option `WithStandardQueries` you should pass the `QueryExecutors` cre
 ```go
 report, err := benchspy.NewStandardReport(
     "2d1fa3532656c51991c0212afce5f80d2914e34e",
-    benchspy.WithQueryExecutors(generatorExectutor, lokiQueryExecutor, prometheusExecutor),
+    benchspy.WithQueryExecutors(directExectutor, lokiQueryExecutor, prometheusExecutor),
     benchspy.WithGenerators(gen),
 )
 require.NoError(t, err, "failed to create baseline report")

book/src/libs/wasp/benchspy/simplest_metrics.md

@@ -14,8 +14,8 @@ so there's no dilemma, but with `Loki` and `Prometheus` things get a bit more co
 But first... since each of built-in `QueryExecutors` returns a different data type and we are using `interface{}` type to reflect that,
 we will use convenience functions to cast them to more usable types:
 ```go
-currentAsFloat64 := benchspy.MustAllGeneratorResults(currentReport)
-previousAsloat64 := benchspy.MustAllGeneratorResults(previousReport)
+currentAsFloat64 := benchspy.MustAllDirectResults(currentReport)
+previousAsloat64 := benchspy.MustAllDirectResults(previousReport)
 ```
 
 > [!NOTE]

wasp/benchspy/TO_DO.md

@@ -1,9 +1,9 @@
 Known things to do:
 - [ ] add logger
-- [ ] add unit tests for prometheus
-- [ ] add wasp test for prometheus only
+- [x] add unit tests for prometheus
+- [x] add wasp test for prometheus only
 - [ ] add e2e OCRv2 test with CTFv2
-- [ ] write documentation
+- [x] write documentation
 - [ ] add report builder (?)
-- [ ] add wrapper function for executing some code and then creating a report
+- [x] add wrapper function for executing some code and then creating a report
 - [ ] add helper method for a profile what would create a report based on all generators?

wasp/benchspy/generator.go wasp/benchspy/direct.gowasp/benchspy/generator.go renamed to wasp/benchspy/direct.go

@@ -11,31 +11,31 @@ import (
 	"github.com/smartcontractkit/chainlink-testing-framework/wasp"
 )
 
-type GeneratorQueryFn = func(responses *wasp.SliceBuffer[wasp.Response]) (float64, error)
+type DirectQueryFn = func(responses *wasp.SliceBuffer[wasp.Response]) (float64, error)
 
-type GeneratorQueryExecutor struct {
-	KindName     string                      `json:"kind"`
-	Generator    *wasp.Generator             `json:"generator_config"`
-	Queries      map[string]GeneratorQueryFn `json:"queries"`
-	QueryResults map[string]interface{}      `json:"query_results"`
+type DirectQueryExecutor struct {
+	KindName     string                   `json:"kind"`
+	Generator    *wasp.Generator          `json:"generator_config"`
+	Queries      map[string]DirectQueryFn `json:"queries"`
+	QueryResults map[string]interface{}   `json:"query_results"`
 }
 
-func NewStandardGeneratorQueryExecutor(generator *wasp.Generator) (*GeneratorQueryExecutor, error) {
-	g := &GeneratorQueryExecutor{
-		KindName: string(StandardQueryExecutor_Generator),
+func NewStandardDirectQueryExecutor(generator *wasp.Generator) (*DirectQueryExecutor, error) {
+	g := &DirectQueryExecutor{
+		KindName: string(StandardQueryExecutor_Direct),
 	}
 
 	queries, err := g.generateStandardQueries()
 	if err != nil {
 		return nil, err
 	}
 
-	return NewGeneratorQueryExecutor(generator, queries)
+	return NewDirectQueryExecutor(generator, queries)
 }
 
-func NewGeneratorQueryExecutor(generator *wasp.Generator, queries map[string]GeneratorQueryFn) (*GeneratorQueryExecutor, error) {
-	g := &GeneratorQueryExecutor{
-		KindName:     string(StandardQueryExecutor_Generator),
+func NewDirectQueryExecutor(generator *wasp.Generator, queries map[string]DirectQueryFn) (*DirectQueryExecutor, error) {
+	g := &DirectQueryExecutor{
+		KindName:     string(StandardQueryExecutor_Direct),
 		Generator:    generator,
 		Queries:      queries,
 		QueryResults: make(map[string]interface{}),
@@ -44,22 +44,22 @@ func NewGeneratorQueryExecutor(generator *wasp.Generator, queries map[string]Gen
 	return g, nil
 }
 
-func (g *GeneratorQueryExecutor) Results() map[string]interface{} {
+func (g *DirectQueryExecutor) Results() map[string]interface{} {
 	return g.QueryResults
 }
 
-func (l *GeneratorQueryExecutor) Kind() string {
+func (l *DirectQueryExecutor) Kind() string {
 	return l.KindName
 }
 
-func (g *GeneratorQueryExecutor) IsComparable(otherQueryExecutor QueryExecutor) error {
+func (g *DirectQueryExecutor) IsComparable(otherQueryExecutor QueryExecutor) error {
 	otherType := reflect.TypeOf(otherQueryExecutor)
 
 	if otherType != reflect.TypeOf(g) {
 		return fmt.Errorf("expected type %s, got %s", reflect.TypeOf(g), otherType)
 	}
 
-	otherGeneratorQueryExecutor := otherQueryExecutor.(*GeneratorQueryExecutor)
+	otherGeneratorQueryExecutor := otherQueryExecutor.(*DirectQueryExecutor)
 
 	if compareGeneratorConfigs(g.Generator.Cfg, otherGeneratorQueryExecutor.Generator.Cfg) != nil {
 		return errors.New("generators are not comparable")
@@ -68,7 +68,7 @@ func (g *GeneratorQueryExecutor) IsComparable(otherQueryExecutor QueryExecutor)
 	return g.compareQueries(otherGeneratorQueryExecutor.Queries)
 }
 
-func (l *GeneratorQueryExecutor) compareQueries(other map[string]GeneratorQueryFn) error {
+func (l *DirectQueryExecutor) compareQueries(other map[string]DirectQueryFn) error {
 	this := l.Queries
 	if len(this) != len(other) {
 		return fmt.Errorf("queries count is different. Expected %d, got %d", len(this), len(other))
@@ -83,7 +83,7 @@ func (l *GeneratorQueryExecutor) compareQueries(other map[string]GeneratorQueryF
 	return nil
 }
 
-func (g *GeneratorQueryExecutor) Validate() error {
+func (g *DirectQueryExecutor) Validate() error {
 	if g.Generator == nil {
 		return errors.New("generator is not set")
 	}
@@ -95,7 +95,7 @@ func (g *GeneratorQueryExecutor) Validate() error {
 	return nil
 }
 
-func (g *GeneratorQueryExecutor) Execute(_ context.Context) error {
+func (g *DirectQueryExecutor) Execute(_ context.Context) error {
 	if g.Generator == nil {
 		return errors.New("generator is not set")
 	}
@@ -130,12 +130,12 @@ func (g *GeneratorQueryExecutor) Execute(_ context.Context) error {
 	return nil
 }
 
-func (g *GeneratorQueryExecutor) TimeRange(_, _ time.Time) {
+func (g *DirectQueryExecutor) TimeRange(_, _ time.Time) {
 	// nothing to do here, since all responses stored in the generator are already in the right time range
 }
 
-func (g *GeneratorQueryExecutor) generateStandardQueries() (map[string]GeneratorQueryFn, error) {
-	standardQueries := make(map[string]GeneratorQueryFn)
+func (g *DirectQueryExecutor) generateStandardQueries() (map[string]DirectQueryFn, error) {
+	standardQueries := make(map[string]DirectQueryFn)
 
 	for _, metric := range standardLoadMetrics {
 		query, err := g.standardQuery(metric)
@@ -148,7 +148,7 @@ func (g *GeneratorQueryExecutor) generateStandardQueries() (map[string]Generator
 	return standardQueries, nil
 }
 
-func (g *GeneratorQueryExecutor) standardQuery(standardMetric StandardLoadMetric) (GeneratorQueryFn, error) {
+func (g *DirectQueryExecutor) standardQuery(standardMetric StandardLoadMetric) (DirectQueryFn, error) {
 	switch standardMetric {
 	case MedianLatency:
 		medianFn := func(responses *wasp.SliceBuffer[wasp.Response]) (float64, error) {
@@ -194,7 +194,7 @@ func (g *GeneratorQueryExecutor) standardQuery(standardMetric StandardLoadMetric
 	}
 }
 
-func (g *GeneratorQueryExecutor) MarshalJSON() ([]byte, error) {
+func (g *DirectQueryExecutor) MarshalJSON() ([]byte, error) {
 	// we need custom marshalling to only include query names, since the functions are not serializable
 	type QueryExecutor struct {
 		Kind         string                 `json:"kind"`
@@ -222,10 +222,10 @@ func (g *GeneratorQueryExecutor) MarshalJSON() ([]byte, error) {
 	})
 }
 
-func (g *GeneratorQueryExecutor) UnmarshalJSON(data []byte) error {
+func (g *DirectQueryExecutor) UnmarshalJSON(data []byte) error {
 	// helper struct with QueryExecutors as json.RawMessage and QueryResults as map[string]interface{}
 	// and as actual types
-	type Alias GeneratorQueryExecutor
+	type Alias DirectQueryExecutor
 	var raw struct {
 		Alias
 		GeneratorCfg wasp.Config            `json:"generator_config"`
@@ -238,7 +238,7 @@ func (g *GeneratorQueryExecutor) UnmarshalJSON(data []byte) error {
 		return err
 	}
 
-	queries := make(map[string]GeneratorQueryFn)
+	queries := make(map[string]DirectQueryFn)
 
 	// unmarshall only query names
 	for _, rawQuery := range raw.Queries {
@@ -256,7 +256,7 @@ func (g *GeneratorQueryExecutor) UnmarshalJSON(data []byte) error {
 		return conversionErr
 	}
 
-	*g = GeneratorQueryExecutor(raw.Alias)
+	*g = DirectQueryExecutor(raw.Alias)
 	g.Queries = queries
 	g.QueryResults = convertedTypes
 	g.Generator = &wasp.Generator{