com.unity.test-framework.performance@0.1.33-preview

## [0.1.33] - 2018-8-3

### Small fixes

Obsolete warnings, doc update with modules and internals, ValueSource fix

## [0.1.32] - 2018-7-9

### Add custom measurement/warmup counts

Method and Frames measurements can now specify custom warmup, measurement and iteration counts

## [0.1.31] - 2018-7-04

### mark metadata tests with performance category

Author: Unity Technologies

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## [0.1.33] - 2018-8-3
+
+### Small fixes
+
+Obsolete warnings, doc update with modules and internals, ValueSource fix
+
+## [0.1.32] - 2018-7-9
+
+### Add custom measurement/warmup counts
+
+Method and Frames measurements can now specify custom warmup, measurement and iteration counts
+
 ## [0.1.31] - 2018-7-04
 
 ### mark metadata tests with performance category

README.md

@@ -1,24 +1,30 @@
-# Performance testing extension for Unity Test Runner 
+# Performance testing extension for Unity Test Runner
 
 Extension provides a set of calls to make it easier to take measurements and record profiler markers. It also collects data about build and player settings which is useful when comparing data for separating different hardware and configurations.
 
 ## Installing
-To install this package, follow the instructions in the [Package Manager documentation](https://docs.unity3d.com/Packages/com.unity.package-manager-ui@latest/index.html). 
+To install this package, follow the instructions in the [Package Manager documentation](https://docs.unity3d.com/Packages/com.unity.package-manager-ui@latest/index.html).
 
 And add `com.unity.test-framework.performance` your packages manifest.
 YourProject/Packages/manifest.json
 
 ``` json
 {
-	"dependencies": {
-		"com.unity.test-framework.performance": "0.1.31-preview"
-	  },
-	  "testables": [
-			"com.unity.test-framework.performance"
-	  ]
+    "dependencies": {
+        "com.unity.test-framework.performance": "0.1.33-preview",
+        "com.unity.modules.jsonserialize": "1.0.0",
+        "com.unity.modules.unitywebrequestwww": "1.0.0",
+        "com.unity.modules.vr": "1.0.0"
+      },
+      "testables": [
+            "com.unity.test-framework.performance"
+      ],
+      "registry": "https://staging-packages.unity.com"
 }
 ```
 
+If you are using 2018.1 or 2018.2 the module dependencies are unnecessary.
+
 Assembly definitions should reference `Unity.PerformanceTesting` in order to use it. Create a new folder for storing tests in and then create a new asset from context menu called `right click/Create/Assembly definition`. In inspector for the assembly file check "Test Assemblies and apply. Then open the file in text editor and add `Unity.PerformanceTesting`.
 
 ``` json
@@ -38,11 +44,10 @@ Assembly definitions should reference `Unity.PerformanceTesting` in order to use
 }
 ```
 
-More information on how to create and run tests please refer to [Unity Test Runner docs](https://docs.unity3d.com/Manual/testing-editortestsrunner.html).
-
-## Saving results
+How to test internals can be found in the following link:
+https://q.unity3d.com/questions/992/how-to-test-internal-variables-in-the-editor-tests.html
 
-If you are on 2018.3+ version of unity you can launch the editor with command line argument `-performanceTestResults "path"` it will save test results as json to path.
+More information on how to create and run tests please refer to [Unity Test Runner docs](https://docs.unity3d.com/Manual/testing-editortestsrunner.html).
 
 
 ## Test Attributes
@@ -64,7 +69,7 @@ Required parameters
 Optional parameters
 - **sampleUnit** : Unit of the measurement.
     - Nanosecond, Microsecond, Millisecond, Second, Byte, Kilobyte, Megabyte, Gigabyte
-- **aggregationType** : Preferred aggregation (default is median) 
+- **aggregationType** : Preferred aggregation (default is median)
 - **percentile** : If aggregationType is Percentile, the percentile value used for the aggregation. i.e 0.95.
 - **threshold** : Threshold used for regression detection. If current sample value is over the threshold different from the baseline results, the result is concidered as a regression or a progression. Default value is 0.15f.
 - **increaseIsBetter** : Defines if an increase in the measurement value is concidered as a progression (better) or a regression. Default is false.
@@ -73,11 +78,11 @@ If unspecified a default SampleGroupDefinition will be used with the name of "Me
 
 ## Taking measurements
 
-Preferred way is to use `Measure.Method` or `Measure.Frames`. They both do a couple of warmup iterations which are then used to decide how many iterations should be measured. It is advised to keep measurements above 100 microseconds. 
+Preferred way is to use `Measure.Method` or `Measure.Frames`. They both do a couple of warmup iterations which are then used to decide how many iterations per measurement should be used.
 
-**void Method(int executions, Action action, SampleGroupDefinition sampleGroupDefinition)**
+**MethodMeasurement Method()**
 
-It will execute provided method multiple times.
+It will execute provided method at least 3 times for warmup and 7 for measurements.
 
 ``` csharp
 [PerformanceTest]
@@ -87,9 +92,27 @@ public void Test()
 }
 ```
 
-**IEnumerator Measure.Frames(int frameCount, SampleGroupdDefinition sampleGroupDefinition)**
+In cases where you feel the default values are not ideal you can specify custom iterations.
+
+WarmupCount - how many iterations to run without measuring for warmup
+MeasurementCount - how many measurements to take
+IterationsPerMeasurement - how many iterations per measurement to take
+
+```
+[PerformanceTest]
+public void Test()
+{
+    Measure.Method(() => { ... })
+        .WarmupCount(10)
+        .MeasurementCount(10)
+        .IterationsPerMeasurement(5)
+        .Run();
+}
+```
+
+**FramesMeasurement Measure.Frames()**
 
-Used to yield for a specified amount of frames. Records frame times.
+Used to yield for frames. It will automatically select the number of warmup and runtime frames.
 
 ``` csharp
 [PerformanceUnityTest]
@@ -101,20 +124,34 @@ public IEnumerator Test()
 }
 ```
 
-In cases where you are measuring a system over frametime it is advised to disable frametime measurements and instead measure profiler systems markers.
+In cases where you are measuring a system over frametime it is advised to disable frametime measurements and instead measure profiler markers for your system.
 ``` csharp
 [PerformanceUnityTest]
 public IEnumerator Test()
 {
     ...
 
-    // passing in false will run the frames but will not take frametime mesaurements
     yield return Measure.Frames()
         .ProfilerMarkers(...)
-        .Run(false);
+        .DontRecordFrametime()
+        .Run();
 }
 ```
 
+If you want more control, you can specify how many frames you want to measure.
+
+``` csharp
+[PerformanceUnityTest]
+public IEnumerator Test()
+{
+    ...
+
+    yield return Measure.Frames()
+        .WarmupCount(5)
+        .MeasurementCount(10)
+        .Run();
+}
+```
 
 When method or frame measurements are not enough you can use the following to measure. It will measure Scope, Frames, Markers or Cusom.
 
@@ -157,9 +194,9 @@ Records profiler samples for a scope. The name of sample group definition has to
 [PerformanceTest]
 public void Test()
 {
-    SampleGroupDefinition[] m_definitions = 
+    SampleGroupDefinition[] m_definitions =
     {
-        new SampleGroupDefinition("Instantiate"), 
+        new SampleGroupDefinition("Instantiate"),
         new SampleGroupDefinition("Instantiate.Copy"),
         new SampleGroupDefinition("Instantiate.Produce"),
         new SampleGroupDefinition("Instantiate.Awake")
@@ -240,7 +277,7 @@ Each performance test will have a performance test summary. Every sample group w
 
 ``` csharp
     // Records total and frame times for loading a scene async
-    
+
     [PerformanceUnityTest]
     public IEnumerator LoadAsync_SampleScene()
     {
@@ -255,9 +292,9 @@ Each performance test will have a performance test summary. Every sample group w
 ```
 
 ``` csharp
-    SampleGroupDefinition[] m_definitions = 
+    SampleGroupDefinition[] m_definitions =
     {
-        new SampleGroupDefinition("Instantiate"), 
+        new SampleGroupDefinition("Instantiate"),
         new SampleGroupDefinition("Instantiate.Copy"),
         new SampleGroupDefinition("Instantiate.Produce"),
         new SampleGroupDefinition("Instantiate.Awake")

Runtime/Attributes/PerformanceTestAttribute.cs

@@ -9,12 +9,8 @@
 namespace Unity.PerformanceTesting
 {
     [AttributeUsage(AttributeTargets.Method)]
-    public class PerformanceTestAttribute : CombiningStrategyAttribute, IImplyFixture, IWrapTestMethod
+    public class PerformanceTestAttribute : TestAttribute, IWrapTestMethod
     {
-        public PerformanceTestAttribute() : base(new UnityCombinatorialStrategy(), new ParameterDataSourceProvider())
-        {
-        }
-
         public TestCommand Wrap(TestCommand command)
         {
         #if UNITY_2018_2_OR_NEWER

Runtime/Measurements/FramesMeasurement.cs

@@ -48,7 +48,7 @@ public FramesMeasurement Definition(string name, SampleUnit sampleUnit, Aggregat
                 increaseIsBetter, failOnBaseline));
         }
 
-        public FramesMeasurement ExecutionCount(int count)
+        public FramesMeasurement MeasurementCount(int count)
         {
             m_Executions = count;
             return this;

Runtime/Measurements/MethodMeasurement.cs

@@ -19,6 +19,9 @@ public class MethodMeasurement
         private readonly List<SampleGroup> m_SampleGroups = new List<SampleGroup>();
 
         private SampleGroupDefinition m_Definition;
+        private int m_WarmupCount;
+        private int m_MeasurementCount;
+        private int m_IterationCount = 1;
 
         public MethodMeasurement(Action action)
         {
@@ -63,20 +66,44 @@ public MethodMeasurement Definition(string name, SampleUnit sampleUnit, Aggregat
             return Definition(new SampleGroupDefinition(name, sampleUnit, aggregationType, percentile, threshold,
                 increaseIsBetter, failOnBaseline));
         }
+
+        public MethodMeasurement WarmupCount(int count)
+        {
+            m_WarmupCount = count;
+            return this;
+        }
+
+        public MethodMeasurement IterationsPerMeasurement(int count)
+        {
+            m_IterationCount = count;
+            return this;
+        }
+
+        public MethodMeasurement MeasurementCount(int count)
+        {
+            m_MeasurementCount = count;
+            return this;
+        }
 
         public void Run()
         {
+            if (m_MeasurementCount > 0)
+            {
+                Warmup(m_WarmupCount);
+                RunForIterations(m_IterationCount, m_MeasurementCount);
+                return;
+            }
+            
             var iterations = Probing();
-
-            RunForIterations(iterations);
+            RunForIterations(iterations, k_MeasurementCount);
         }
 
-        private void RunForIterations(int iterations)
+        private void RunForIterations(int iterations, int measurements)
         {
             UpdateSampleGroupDefinition();
 
-            EnableProfilerMarkers();
-            for (int j = 0; j < k_MeasurementCount; j++)
+            EnableMarkers();
+            for (int j = 0; j < measurements; j++)
             {
                 var executionTime = Time.realtimeSinceStartup;
                 for (var i = 0; i < iterations; i++)
@@ -87,18 +114,18 @@ private void RunForIterations(int iterations)
                 Measure.Custom(m_Definition, Utils.ConvertSample(SampleUnit.Millisecond, m_Definition.SampleUnit, executionTime));
             }
 
-            MeasureProfilerMarkers();
+            DisableAndMeasureMarkers();
         }
 
-        private void EnableProfilerMarkers()
+        private void EnableMarkers()
         {
             foreach (var sampleGroup in m_SampleGroups)
             {
                 sampleGroup.Recorder.enabled = true;
             }
         }
 
-        private void MeasureProfilerMarkers()
+        private void DisableAndMeasureMarkers()
         {
             foreach (var sampleGroup in m_SampleGroups)
             {
@@ -119,10 +146,7 @@ private int Probing()
             while (executionTime < k_MinWarmupTimeMs)
             {
                 executionTime = Time.realtimeSinceStartup;
-                for (var i = 0; i < iterations; i++)
-                {
-                    m_Action.Invoke();
-                }
+                Warmup(iterations);
                 executionTime = (Time.realtimeSinceStartup - executionTime) * 1000f;
 
                 if (executionTime < k_MinWarmupTimeMs)
@@ -138,13 +162,21 @@ private int Probing()
 
                 return 1;
             }
-            
+
             var deisredIterationsCount =
                 Mathf.Clamp((int) (k_MinMeasurementTimeMs * iterations / executionTime), 1, k_MaxIterations);
-            
+
             return deisredIterationsCount;
         }
 
+        private void Warmup(int iterations)
+        {
+            for (var i = 0; i < iterations; i++)
+            {
+                m_Action.Invoke();
+            }
+        }
+
         private void UpdateSampleGroupDefinition()
         {
             if (m_Definition.Name == null)