com.unity.test-framework.performance@2.1.0-preview

## [2.1.0] - 2020-05-14

Add flexible horizontal splitter for report window
Fix date format

## [2.0.9] - 2020-03-23

Fix profiler measurements for method measurements
Throw exceptions when measuring NaN

Author: Unity Technologies

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## [2.1.0] - 2020-05-14
+
+Add flexible horizontal splitter for report window
+Fix date format
+
 ## [2.0.9] - 2020-03-23
 
 Fix profiler measurements for method measurements

Documentation~/images/example01.png

@@ -11,7 +11,7 @@ The Performance Testing Extension is intended to be used with, and complement, t
 
 To install the Performance Testing Extension package
 1. Open the `manifest.json` file for your Unity project (located in the YourProject/Packages directory) in a text editor
-2. Add `"com.unity.test-framework.performance": "2.0.9-preview",` to the dependencies
+2. Add `"com.unity.test-framework.performance": "2.1.0-preview",` to the dependencies
 3. Save the manifest.json file
 4. Verify the Performance Testing Extension is now installed opening the Unity Package Manager window
 
@@ -334,6 +334,131 @@ public static string GetScenePath(string name)
 }
 ```
 
+## Writing a simple test
+
+As an example, to measure the performance of Vector2 operations you can use `Measure.Method`. This executes the provided method and samples performance. You can increase `MeasurementCount` from the default of 7 to 20 to achieve better performance test stability. You can execute a `Vector2.MoveTowards` test in Edit or Play mode. 
+
+``` csharp
+[Test, Performance]
+public void Vector2_operations()
+{
+    var a = Vector2.one;
+    var b = Vector2.zero;
+
+    Measure.Method(() =>
+    {
+            Vector2.MoveTowards(a, b, 0.5f);
+            Vector2.ClampMagnitude(a, 0.5f);
+            Vector2.Reflect(a, b);
+            Vector2.SignedAngle(a, b);
+    })
+        .MeasurementCount(20)
+        .Run();
+}
+```
+
+To view the results, go to **Window > Analysis > Performance Test Report**.
+
+![Performance test example01](images/example01.png)
+
+In this example, the results show that the first method execution took five times longer than the subsequent methods, and those subsequent method executions were unstable. Also, you can’t tell from these test results how long it took for each Vector2 operation to execute.
+
+### Improving test stability
+
+You can improve this test in several ways. First, try to improve test stability. Test instability at the beginning of a test can occur for several reasons, such as entering Play mode or method initialization, because the tested method is quite fast and more sensitive to other running background processes. To improve this, add `WarmupCount(n)`. This allows you to execute methods before you start to record data, so Unity doesn’t record method initialization. The simplest way to increase test execution time is to repeat the method in a loop.
+Avoid having measurements that are less than 1ms because they are usually more sensitive to unstable environments.
+To help you track each operation's execution times, split the Vector2 operations into several tests. Usually, when writing a test, you use setup and clean up methods to isolate the test environment. However, in this case, methods are isolated and do not affect other methods, therefore they do not need a cleanup or setup.
+The following code example shows a performance test for the `Vector2.MoveTowards` operation. Other Vector2 performance tests are identical.
+
+``` csharp
+[Test, Performance]
+public void Vector2_MoveTowards()
+{
+
+    Measure.Method(() =>
+    {
+        Vector2.MoveTowards(Vector2.one, Vector2.zero, 0.5f);
+    })
+        .WarmupCount(5)
+        .IterationsPerMeasurement(10000)
+        .MeasurementCount(20)
+        .Run();
+}
+```
+
+With 100000 iterations in this test, there is a small fluctuation in method execution time, but the standard deviation is low, which means the test is reasonably stable.
+
+![Performance test example02](images/example02.png)
+
+As an example, if you want to measure a method that only runs in Play mode (for example `Physics.Raycast`), you can use `Measure.Frames()`, which records time per frame by default. To only measure `Physics.Raycast` time, you can disable frame time measurements with `DontRecordFrametime` and just measure the `Physics.Raycast` profiler marker.
+This test creates objects that you need to dispose of at the end of each test, because multiple unnecessary objects can affect the next test results. Use the SetUp method to create GameObjects, and the TearDown method to destroy the created GameObjects after each test. 
+
+``` csharp
+[SetUp]
+public void SetUp()
+{
+    for (int i = 0; i < 100; ++i)
+    {
+        GameObject raycaster = new GameObject("raycast", typeof(Raycast));
+        raycaster.transform.SetParent(_parent.transform);
+    }
+}
+
+[UnityTest, Performance]
+public IEnumerator Physics_RaycastTests()
+{
+    string profilierMarker = "Physics.Raycast";
+    
+    yield return Measure.Frames()
+        .WarmupCount(3)
+        .DontRecordFrametime()
+        .MeasurementCount(10)
+        .ProfilerMarkers(profilierMarker)
+        .Run();
+}
+
+public class Raycast : MonoBehaviour
+{
+    private void Update()
+    {
+        RaycastHit hit;
+        bool ray = Physics.Raycast(
+            transform.position, 
+            Vector3.forward, 
+            out hit, 
+            Mathf.Infinity);
+    }
+}
+
+[TearDown]
+public void TearDown()
+{
+    GameObject.DestroyImmediate(_parent);
+}
+```
+
+To record your own measurements, create a new sample group and record a custom metric. The following examples measures `Allocated` and `Reserved` memory.
+
+``` csharp
+[Test, Performance]
+public void Empty()
+{
+    var allocated = new SampleGroup("TotalAllocatedMemory", SampleUnit.Megabyte);
+    var reserved = new SampleGroup("TotalReservedMemory", SampleUnit.Megabyte);
+
+    using (Measure.Scope())
+    {
+        Measure.Custom(allocated, Profiler.GetTotalAllocatedMemoryLong() / 1048576f);
+        Measure.Custom(reserved, Profiler.GetTotalReservedMemoryLong() / 1048576f);
+    }
+}
+```
+
+Before you start to collect package performance data, make sure that the tests you run locally are stable (the data set deviation is <5%). In the **Performance Test Report** window check whether the test isn’t fluctuating and that the results between runs are similar. 
+
+Performance test results run on a local machine can be significantly different compared to  previous test runs because there could be other applications running in the background, or CPU overheating or CPU boosting is enabled. Make sure that CPU intensive applications are turned off where possible. You can disable CPU boost in the BIOS or by using third-parties software, such as Real Temp.
+
+For compairing performance data between runs use [Unity Performance Benchmark Reporter](https://github.com/Unity-Technologies/PerformanceBenchmarkReporter/wiki). Unity Performance Benchmark Reporter provides you with a graphical HTML report that enables you to compare performance metric baselines and subsequent performance metrics.
 
 ## More Examples
 

Documentation~/images/example02.png

@@ -37,6 +37,11 @@ public class TestReportWindow : EditorWindow
         private bool m_showSamples = true;
         private int[] m_columnWidth = new int[4];
 
+        private bool m_isResizing;
+        private float m_testListHeight = s_windowHeight / 4;
+        private Rect m_splitterRect;
+        private float m_windowHeight;
+
         [SerializeField]
         TreeViewState m_testListTreeViewState;
         [SerializeField]
@@ -260,9 +265,10 @@ private void Draw()
                 {
                     if (m_testListTable != null)
                     {
-                        Rect r = GUILayoutUtility.GetRect(position.width, s_windowHeight / 4, GUI.skin.box,
+                        Rect r = GUILayoutUtility.GetRect(position.width, m_testListHeight, GUI.skin.box,
                             GUILayout.ExpandWidth(true));
                         m_testListTable.OnGUI(r);
+                        Resize(r.y);
                     }
                 }
 
@@ -350,6 +356,41 @@ private void Draw()
             }
         }
 
+        private void Resize(float headerHeight)
+        {
+            m_splitterRect = new Rect(0, headerHeight + m_testListHeight, position.width, 5f);
+
+            EditorGUIUtility.AddCursorRect(m_splitterRect, MouseCursor.ResizeVertical);
+
+            if (Event.current.type == EventType.MouseDown)
+                m_isResizing = m_splitterRect.Contains(Event.current.mousePosition);
+
+            const float minListHeight = 80f;
+            var maxListHeight = position.height - 240f;
+            var needMoveSplitterAndRepaint = false;
+
+            if (m_isResizing && Event.current.type == EventType.MouseDrag)
+            {
+                m_testListHeight = Mathf.Clamp(Event.current.mousePosition.y - headerHeight, minListHeight, maxListHeight);
+                needMoveSplitterAndRepaint = true;
+            }
+            else if (Math.Abs(m_windowHeight - position.height) > float.Epsilon)
+            {
+                m_windowHeight = position.height;
+                m_testListHeight = Mathf.Clamp(m_testListHeight, minListHeight, maxListHeight);
+                needMoveSplitterAndRepaint = true;
+            }
+
+            if (needMoveSplitterAndRepaint)
+            {
+                m_splitterRect.Set(m_splitterRect.x, m_testListHeight, m_splitterRect.width, m_splitterRect.height);
+                Repaint();
+            }
+
+            if (Event.current.type == EventType.MouseUp)
+                m_isResizing = false;
+        }
+
         private void DrawBarGraph(float width, float height, List<double> samples, float min, float max, float median)
         {
             if (DrawStart(width, height))
@@ -395,7 +436,7 @@ private void DrawBarGraph(float width, float height, List<double> samples, float
                 {
                     float sample = (float)samples[i];
 
-                    string tooltip = string.Format("{0} (at sample {1} of {2})", sample, i, samples.Count);
+                    string tooltip = string.Format("{0} (at sample {1} of {2})", sample, i+1, samples.Count);
 
                     GUI.Label(new Rect(rect.x + x, rect.y + y, xAxisInc, height), new GUIContent("", tooltip));
 

Documentation~/images/example03.png

@@ -27,7 +27,7 @@ public void Setup()
             run.Dependencies = GetPackageDependencies();
             SetBuildSettings(run);
 
-            run.Date = (int)Utils.ConvertToUnixTimestamp(DateTime.Now);
+            run.Date = Utils.ConvertToUnixTimestamp(DateTime.Now);
 
             CreateResourcesFolder();
             CreatePerformanceTestRunJson(run);

Documentation~/index.md

@@ -8,7 +8,7 @@ namespace Unity.PerformanceTesting.Data
     public class Run
     {
         public string TestSuite;
-        public int Date;
+        public long Date;
         public Player Player;
         public Hardware Hardware;
         public Editor Editor;

Editor/TestReportGraph/TestReportWindow.cs

@@ -13,17 +13,16 @@ public static class Utils
         public const string TestRunInfo = "PerformanceTestRunInfo.json";
         public const string PlayerPrefKeyRunJSON = "PT_Run";
 
-        public static DateTime ConvertFromUnixTimestamp(int timestamp)
+        public static DateTime ConvertFromUnixTimestamp(long timestamp)
         {
-            DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
-            return origin.AddSeconds(timestamp);
+            var offset = DateTimeOffset.FromUnixTimeMilliseconds(timestamp);
+            return offset.UtcDateTime;
         }
 
-        public static int ConvertToUnixTimestamp(DateTime date)
+        public static long ConvertToUnixTimestamp(DateTime date)
         {
-            DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
-            TimeSpan diff = date.ToUniversalTime() - origin;
-            return (int)Math.Floor(diff.TotalSeconds);
+            var offset = new DateTimeOffset(date);
+            return offset.ToUnixTimeMilliseconds();
         }
 
         public class RatioUnit

Editor/TestRunBuilder.cs

@@ -1,7 +1,7 @@
 {
   "name": "com.unity.test-framework.performance",
   "displayName": "Performance testing API",
-  "version": "2.0.9-preview",
+  "version": "2.1.0-preview",
   "unity": "2019.3",
   "unityRelease": "0a10",
   "description": "Extension to the Unity Test framework package. Adds performance testing capabilities and collects configuration metadata.",
@@ -13,10 +13,12 @@
     "com.unity.test-framework": "1.1.0",
     "com.unity.nuget.newtonsoft-json": "2.0.0-preview"
   },
+  "upmCi": {
+    "footprint": "bffa451d2d70421572fe14efb2acbaa69f80eeff"
+  },
   "repository": {
-    "footprint": "ad9c5c26d28dc49453310f158baf84c774e4f333",
     "type": "git",
     "url": "https://github.cds.internal.unity3d.com/unity/com.unity.test-framework.performance.git",
-    "revision": "2c45c9785265b08de6329a51eee72914b21b4be8"
+    "revision": "4fcf08c79385aef3efa6862e69754565792ef757"
   }
 }