You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Documentation~/index.md
+72-36Lines changed: 72 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,32 +1,46 @@
1
-
# Performance testing extension for Unity Test Runner
1
+
# Performance Testing Extension for Unity Test Runner
2
2
3
-
Extension provides a set of calls to make it easier to take measurements and record profiler markers. It also collects data about build and player settings which is useful when comparing data for separating different hardware and configurations.
3
+
The Unity Performance Testing Extension is a Unity Editor package that, when installed, provides an API and test case decorators to make it easier to take measurements/samples of Unity profiler markers, and other custom metrics outside of the profiler, within the Unity Editor and built players. It also collects configuration metadata, such as build and player settings, which is useful when comparing data against different hardware and configurations.
4
+
5
+
The Performance Testing Extension is intended to be used with, and complement, the Unity Test Runner framework.
6
+
7
+
**Important Note:** When tests are run with the Unity Test Runner, a development player is always built to support communication between the editor and player, effectively overriding the development build setting from the build settings UI or scripting API.
4
8
5
9
## Installing
6
-
To install this package, follow the instructions in the [Package Manager documentation](https://docs.unity3d.com/Packages/com.unity.package-manager-ui@latest/index.html).
7
10
8
-
And add `com.unity.test-framework.performance` your packages manifest.
9
-
YourProject/Packages/manifest.json
11
+
To install the Performance Testing Extension package
12
+
1. Open the manifest.json file for your Unity project (located in the YourProject/Packages directory) in a text editor
13
+
2. Add com.unity.test-framework.performance to the dependencies as seen below
14
+
3. Add com.unity.test-framework.performance to the testables section. If there is not a testables section in your manifest.json file, go ahead and add it.
15
+
4. Save the manifest.json file
16
+
5. Verify the Performance Testing Extension is now installed opening the Unity Package Manager window
17
+
6. Ensure you have created an Assembly Definition file in the same folder where your tests or scripts are that you’ll reference the Performance Testing Extension with. This Assembly Definition file needs to reference Unity.PerformanceTesting in order to use the Performance Testing Extension. Example of how to do this:
18
+
* Create a new folder for storing tests in ("Tests", for example)
19
+
* Create a new assembly definition file in the new folder using the context menu (right click/Create/Assembly definition) and name it "Tests" (or whatever you named the folder from step a. above)
20
+
* In inspector for the assembly definition file check "Test Assemblies", and then Apply.
21
+
* Open the assembly definition file in a text editor and add Unity.PerformanceTesting. To the references section. Save the file when you’re done doing this.
If you are using 2018.1 or 2018.2 the module dependencies are unnecessary.
28
42
29
-
Assembly definitions should reference `Unity.PerformanceTesting` in order to use it. Create a new folder for storing tests in and then create a new asset from context menu called `right click/Create/Assembly definition`. In inspector for the assembly file check "Test Assemblies and apply. Then open the file in text editor and add `Unity.PerformanceTesting`.
43
+
> Example: assembly definitionfile
30
44
31
45
```json
32
46
{
@@ -45,9 +59,6 @@ Assembly definitions should reference `Unity.PerformanceTesting` in order to use
45
59
}
46
60
```
47
61
48
-
How to test internals can be found in the following link:
More information on how to create and run tests please refer to [Unity Test Runner docs](https://docs.unity3d.com/Manual/testing-editortestsrunner.html).
52
63
53
64
@@ -61,29 +72,46 @@ More information on how to create and run tests please refer to [Unity Test Runn
61
72
62
73
## SampleGroupDefinition
63
74
64
-
**struct SampleGroupDefinition**
65
-
SampleGroupDefinition is used to define how a measurement is used in reporting and in regression detection.
75
+
**struct SampleGroupDefinition** - used to define how a measurement is used in reporting and in regression detection.
66
76
67
77
Required parameters
68
78
-**name** : Name of the measurement. Should be kept short and simple.
69
79
70
80
Optional parameters
71
-
-**sampleUnit** : Unit of the measurement.
81
+
-**sampleUnit** : Unit of the measurement to report samples in. Possible values are:
-**aggregationType** : Preferred aggregation (default is median)
74
-
-**percentile** : If aggregationType is Percentile, the percentile value used for the aggregation. i.e 0.95.
75
-
-**threshold** : Threshold used for regression detection. If current sample value is over the threshold different from the baseline results, the result is concidered as a regression or a progression. Default value is 0.15f.
76
-
-**increaseIsBetter** : Defines if an increase in the measurement value is concidered as a progression (better) or a regression. Default is false.
83
+
-**aggregationType** : Preferred aggregation (default is median). Possible values are:
84
+
- Median, Average, Min, Max, Percentile
85
+
-**percentile** : If aggregationType is Percentile, the percentile value used for the aggregation. e.g. 0.95.
86
+
-**increaseIsBetter** : Determines whether or not an increase in the measurement value should be considered a progression (performance improved) or a performance regression. Default is false. **NOTE:** This value is not used directly in the Performance Testing Extension, but recorded for later use in a reporting tool (such as the [Unity Performance Benchmark Reporter](https://github.com/Unity-Technologies/PerformanceBenchmarkReporter/wiki)) to determine whether or not a performance regression has occurred when used with a baseline result set.
87
+
-**threshold** : The threshold, as a percentage of the aggregated sample group value, to use for regression detection. Default value is 0.15f. **NOTE:** This value is not used directly in the Performance Testing Extension, but recorded for later use in a reporting tool (such as the [Unity Performance Benchmark Reporter](https://github.com/Unity-Technologies/PerformanceBenchmarkReporter/wiki)) to determine whether or not a performance regression has occurred when used with a baseline result set.
77
88
78
-
If unspecified a default SampleGroupDefinition will be used with the name of "Measure.Scope", it is recommended to specify a name that is descriptive of what it is measuring.
89
+
If unspecified a default SampleGroupDefinition will be used with the name of "Time", it is recommended to specify a name that is descriptive of what it is measuring.
79
90
80
91
## Taking measurements
81
92
82
-
Preferred way is to use `Measure.Method` or `Measure.Frames`. They both do a couple of warmup iterations which are then used to decide how many iterations per measurement should be used.
93
+
The Performance Testing Extension provides several API methods you can use to take measurements in your performance test, depending on what you need to measure and how you want to do it. They are:
The sections below detail the specifics of each measurement method with examples.
102
+
103
+
Preferred way is to use Measure.Method or Measure.Frames. They both do a couple of warmup iterations which are then used to decide how many iterations per measurement should be used.
104
+
83
105
84
106
**MethodMeasurement Method()**
85
107
86
-
It will execute provided method at least 3 times for warmup and 7 for measurements.
108
+
This will execute the provided method, sampling performance using the following additional properties/methods to control how the measurements are taken:
109
+
***WarmupCount(int n)** - number of times to to execute before measurements are collected. Default is 3 if not specified.
110
+
***MeasurementCount(int n)** - number of measurements to capture. Default is 7 if not specified.
111
+
***IterationsPerMeasurement(int n)** - number of iterations per measurement to use
112
+
***GC()** - if specified, will measure the Gargage Collection allocation value.
113
+
114
+
> Example 1: Simple method measurement using default values
87
115
88
116
```csharp
89
117
[PerformanceTest]
@@ -93,12 +121,7 @@ public void Test()
93
121
}
94
122
```
95
123
96
-
In cases where you feel the default values are not ideal you can specify custom iterations.
97
-
98
-
WarmupCount - how many iterations to run without measuring for warmup
99
-
MeasurementCount - how many measurements to take
100
-
IterationsPerMeasurement - how many iterations per measurement to take
101
-
GC - measures the amount of GC allocations
124
+
> Example 2: Customize Measure.Method properties
102
125
103
126
```
104
127
[PerformanceTest]
@@ -115,7 +138,15 @@ public void Test()
115
138
116
139
**FramesMeasurement Measure.Frames()**
117
140
118
-
Used to yield for frames. It will automatically select the number of warmup and runtime frames.
141
+
This will sample perf frame, records time per frame by default and provides additional properties/methods to control how the measurements are taken:
142
+
***WarmupCount(int n)** - number of times to to execute before measurements are collected. Default is 3 if not specified.
143
+
***MeasurementCount(int n)** - number of measurements to capture. Default is 7 if not specified.
***ProfilerMarkers(...)** - sample profile markers per frame
146
+
147
+
It will automatically select the number of warmup and runtime frames.
148
+
149
+
> Example 1: Simple frame time measurement
119
150
120
151
```csharp
121
152
[PerformanceUnityTest]
@@ -128,6 +159,8 @@ public IEnumerator Test()
128
159
```
129
160
130
161
In cases where you are measuring a system over frametime it is advised to disable frametime measurements and instead measure profiler markers for your system.
162
+
> Example 2: Sample profile markers per frame, disable frametime measurement
163
+
131
164
```csharp
132
165
[PerformanceUnityTest]
133
166
publicIEnumeratorTest()
@@ -142,6 +175,7 @@ public IEnumerator Test()
142
175
```
143
176
144
177
If you want more control, you can specify how many frames you want to measure.
178
+
> Example 3: Specify custom WarmupCount and MeasurementCount per frame
145
179
146
180
```csharp
147
181
[PerformanceUnityTest]
@@ -156,11 +190,11 @@ public IEnumerator Test()
156
190
}
157
191
```
158
192
159
-
When method or frame measurements are not enough you can use the following to measure. It will measure Scope, Frames, Markers or Cusom.
Records profiler samples for a scope. The name of sample group definition has to match profiler sample names.
228
+
When you want to record samples outside of frame time, method time, or profiler markers, use a custom measurement. It can be any double value. A sample group definition is required.
229
+
230
+
> Example 1: Use a custom measurement to capture total allocated memory
0 commit comments