|
| 1 | +# BenchSpy - Your First Test |
| 2 | + |
| 3 | +Let's start with the simplest case, which doesn't require any part of the observability stack—only `WASP` and the application you are testing. |
| 4 | +`BenchSpy` comes with built-in `QueryExecutors`, each of which also has predefined metrics that you can use. One of these executors is the `DirectQueryExecutor`, which fetches metrics directly from `WASP` generators, |
| 5 | +which means you can run it with Loki. |
| 6 | + |
| 7 | +> [!NOTE] |
| 8 | +> Not sure whether to use `Loki` or `Direct` query executors? [Read this!](./loki_dillema.md) |
| 9 | +
|
| 10 | +## Test Overview |
| 11 | + |
| 12 | +Our first test will follow this logic: |
| 13 | +- Run a simple load test. |
| 14 | +- Generate a performance report and store it. |
| 15 | +- Run the load test again. |
| 16 | +- Generate a new report and compare it to the previous one. |
| 17 | + |
| 18 | +We'll use very simplified assertions for this example and expect the performance to remain unchanged. |
| 19 | + |
| 20 | +### Step 1: Define and Run a Generator |
| 21 | + |
| 22 | +Let's start by defining and running a generator that uses a mocked service: |
| 23 | + |
| 24 | +```go |
| 25 | +gen, err := wasp.NewGenerator(&wasp.Config{ |
| 26 | + T: t, |
| 27 | + GenName: "vu", |
| 28 | + CallTimeout: 100 * time.Millisecond, |
| 29 | + LoadType: wasp.VU, |
| 30 | + Schedule: wasp.Plain(10, 15*time.Second), |
| 31 | + VU: wasp.NewMockVU(&wasp.MockVirtualUserConfig{ |
| 32 | + CallSleep: 50 * time.Millisecond, |
| 33 | + }), |
| 34 | +}) |
| 35 | +require.NoError(t, err) |
| 36 | +gen.Run(true) |
| 37 | +``` |
| 38 | + |
| 39 | +### Step 2: Generate a Baseline Performance Report |
| 40 | + |
| 41 | +With load data available, let's generate a baseline performance report and store it in local storage: |
| 42 | + |
| 43 | +```go |
| 44 | +baseLineReport, err := benchspy.NewStandardReport( |
| 45 | + // random hash, this should be the commit or hash of the Application Under Test (AUT) |
| 46 | + "v1.0.0", |
| 47 | + // use built-in queries for an executor that fetches data directly from the WASP generator |
| 48 | + benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Direct), |
| 49 | + // WASP generators |
| 50 | + benchspy.WithGenerators(gen), |
| 51 | +) |
| 52 | +require.NoError(t, err, "failed to create baseline report") |
| 53 | + |
| 54 | +fetchCtx, cancelFn := context.WithTimeout(context.Background(), 60*time.Second) |
| 55 | +defer cancelFn() |
| 56 | + |
| 57 | +fetchErr := baseLineReport.FetchData(fetchCtx) |
| 58 | +require.NoError(t, fetchErr, "failed to fetch data for baseline report") |
| 59 | + |
| 60 | +path, storeErr := baseLineReport.Store() |
| 61 | +require.NoError(t, storeErr, "failed to store baseline report", path) |
| 62 | +``` |
| 63 | + |
| 64 | +> [!NOTE] |
| 65 | +> There's a lot to unpack here, and you're encouraged to read more about the built-in `QueryExecutors` and the standard metrics they provide as well as about the `StandardReport` [here](./reports/standard_report.md). |
| 66 | +> |
| 67 | +> For now, it's enough to know that the standard metrics provided by `StandardQueryExecutor_Direct` include: |
| 68 | +> - Median latency |
| 69 | +> - P95 latency (95th percentile) |
| 70 | +> - Max latency |
| 71 | +> - Error rate |
| 72 | +
|
| 73 | +### Step 3: Run the Test Again and Compare Reports |
| 74 | + |
| 75 | +With the baseline report ready, let's run the load test again. This time, we'll use a wrapper function to automatically load the previous report, generate a new one, and ensure they are comparable. |
| 76 | + |
| 77 | +```go |
| 78 | +// define a new generator using the same config values |
| 79 | +newGen, err := wasp.NewGenerator(&wasp.Config{ |
| 80 | + T: t, |
| 81 | + GenName: "vu", |
| 82 | + CallTimeout: 100 * time.Millisecond, |
| 83 | + LoadType: wasp.VU, |
| 84 | + Schedule: wasp.Plain(10, 15*time.Second), |
| 85 | + VU: wasp.NewMockVU(&wasp.MockVirtualUserConfig{ |
| 86 | + CallSleep: 50 * time.Millisecond, |
| 87 | + }), |
| 88 | +}) |
| 89 | +require.NoError(t, err) |
| 90 | + |
| 91 | +// run the load |
| 92 | +newGen.Run(true) |
| 93 | + |
| 94 | +fetchCtx, cancelFn = context.WithTimeout(context.Background(), 60*time.Second) |
| 95 | +defer cancelFn() |
| 96 | + |
| 97 | +// currentReport is the report that we just created (baseLineReport) |
| 98 | +currentReport, previousReport, err := benchspy.FetchNewStandardReportAndLoadLatestPrevious( |
| 99 | + fetchCtx, |
| 100 | + // commit or tag of the new application version |
| 101 | + "v2.0.0", |
| 102 | + benchspy.WithStandardQueries(benchspy.StandardQueryExecutor_Direct), |
| 103 | + benchspy.WithGenerators(newGen), |
| 104 | +) |
| 105 | +require.NoError(t, err, "failed to fetch current report or load the previous one") |
| 106 | +``` |
| 107 | + |
| 108 | +> [!NOTE] |
| 109 | +> In a real-world case, once you've generated the first report, you should only need to use the `benchspy.FetchNewStandardReportAndLoadLatestPrevious` function. |
| 110 | +
|
| 111 | +### What's Next? |
| 112 | + |
| 113 | +Now that we have two reports, how do we ensure that the application's performance meets expectations? |
| 114 | +Find out in the [next chapter](./simplest_metrics.md). |
0 commit comments