com.unity.test-framework.performance@2.4.1-preview

## [2.4.1] - 2020-11-05

Metadata collection was made public

## [2.4.0] - 2020-09-16

Upgrade json dependency to release version
Reduced overhead introduced when running tests
Performance Test Report window updates:
Added CSV export option.
Added monitoring of results file timestemp to support auto refresh when a new file is found.
Added display of timestamp of last loaded results file.
Added option to sort test report window by order the tests ran in (index). This is now the default.
Added min and max to the table.
Improve titles and tooltips on columns

## [2.3.1] - 2020-07-01

Fix overhead introduced with Measure.Method no longer calculates execution time of Setup and Cleanup changes

Author: Unity Technologies

CHANGELOG.md

@@ -11,12 +11,19 @@ The Performance Testing Extension is intended to be used with, and complement, t
 
 To install the Performance Testing Extension package
 1. Open the `manifest.json` file for your Unity project (located in the YourProject/Packages directory) in a text editor
-2. Add `"com.unity.test-framework.performance": "2.3.1-preview",` to the dependencies
+2. Add `"com.unity.test-framework.performance": "2.4.0",` to the dependencies
 3. Save the manifest.json file
 4. Verify the Performance Testing Extension is now installed opening the Unity Package Manager window
 
 To access performance testing apis add `Unity.PerformanceTesting` to your assembly definition references section.
 
+Performance package will need to be preserved in link.xml file if Managed stripping level is enabled in Player settings. This is nesesarry for console and mobile platforms.
+
+```xml
+<linker>
+       <assembly fullname="Unity.PerformanceTesting" preserve="all"/>
+</linker>
+```
 
 ## Test Attributes
 **[Performance]** - Use this with  `Test` and `UnityTest` attributes. It will initialize necessary test setup for performance tests.
@@ -61,6 +68,9 @@ This will execute the provided method, sampling performance using the following
 * **IterationsPerMeasurement(int n)** - number of method executions per measurement to use. If this value is not specified, the method will be executed as many times as possible until approximately 100 ms has elapsed.
 * **SampleGroup(string name)** - by default the measurement name will be "Time", this allows you to override it
 * **GC()** - if specified, will measure the total number of Garbage Collection allocation calls.
+* **SetUp(Action action)** - is called every iteration before executing the method. Setup time is not measured.
+* **CleanUp(Action action)** - is called every iteration after the execution of the method. Cleanup time is not measured.
+
 
 #### Example 1: Simple method measurement using default values
 
@@ -83,6 +93,8 @@ public void Test()
         .MeasurementCount(10)
         .IterationsPerMeasurement(5)
         .GC()
+        .SetUp(()=> {/*setup code*/})
+        .CleanUp(()=> {/*cleanup code*/})
         .Run();
 }
 ```
@@ -177,7 +189,7 @@ public void Test()
 
 ### Measure.ProfilerMarkers()
 
-Used to record profiler markers. Profiler marker timings will be recorded automatically and sampled within the scope of the `using` statement. Names should match profiler marker labels. Note that deep and editor profiling are not available. Profiler markers created using `Profiler.BeginSample()` are not supported, switch to `ProfilerMarker` if possible. 
+Used to record profiler markers. Profiler marker timings will be recorded automatically and sampled within the scope of the `using` statement. Names should match profiler marker labels. Profiler markers are sampled once per frame. Sampling the same profiler marker per frame will result in the sum of all invocations. Note that deep and editor profiling is not available. Profiler markers created using `Profiler.BeginSample()` are not supported, switch to `ProfilerMarker` if possible. 
 
 #### Example 1: Measuring profiler markers in a scope
 
@@ -560,4 +572,4 @@ For compairing performance data between runs use [Unity Performance Benchmark Re
         Measure.Custom(allocated, Profiler.GetTotalAllocatedMemoryLong() / 1048576f);
         Measure.Custom(reserved, Profiler.GetTotalReservedMemoryLong() / 1048576f);
     }
-```
+```

Documentation~/index.md

@@ -11,43 +11,57 @@ namespace Unity.PerformanceTesting
 {
     class TestListTableItem : TreeViewItem
     {
+        public int index;
         public TestResult test;
         public double deviation;
         public double standardDeviation;
         public double median;
+        public double min;
+        public double max;
 
         public TestListTableItem(int id, int depth, string displayName, TestResult test)
             : base(id, depth,
                 displayName)
         {
             this.test = test;
 
+            index = id;
             deviation = 0f;
             if (test != null)
             {
                 foreach (var sample in test.SampleGroups)
                 {
                     if (sample.Name == "Time")
                     {
-                        deviation = sample.StandardDeviation;
+                        standardDeviation = sample.StandardDeviation;
                         median = sample.Median;
+                        min = sample.Min;
+                        max = sample.Max;
+
+                        deviation = standardDeviation / median;
                         break;
                     }
 
                     if (sample.Samples.Count <= 1)
                     {
+                        standardDeviation = sample.StandardDeviation;
                         median = sample.Median;
+                        min = sample.Min;
+                        max = sample.Max;
+
+                        deviation = standardDeviation / median;
                         break;
                     }
 
-                    if (sample.StandardDeviation > deviation)
-                        standardDeviation = sample.StandardDeviation;
-
                     double thisDeviation = sample.StandardDeviation / sample.Median;
                     if (thisDeviation > deviation)
                     {
-                        deviation = thisDeviation;
+                        standardDeviation = sample.StandardDeviation;
                         median = sample.Median;
+                        min = sample.Min;
+                        max = sample.Max;
+
+                        deviation = thisDeviation;
                     }
                 }
             }
@@ -64,30 +78,39 @@ class TestListTable : TreeView
         // All columns
         public enum MyColumns
         {
+            Index,
             Name,
             SampleCount,
             StandardDeviation,
             Deviation,
-            Median
+            Median,
+            Min,
+            Max,
         }
 
         public enum SortOption
         {
+            Index,
             Name,
             SampleCount,
             StandardDeviation,
             Deviation,
-            Median
+            Median,
+            Min,
+            Max,
         }
 
         // Sort options per column
         SortOption[] m_SortOptions =
         {
+            SortOption.Index,
             SortOption.Name,
             SortOption.SampleCount,
             SortOption.StandardDeviation,
             SortOption.Deviation,
-            SortOption.Median
+            SortOption.Median,
+            SortOption.Min,
+            SortOption.Max,
         };
 
         public TestListTable(TreeViewState state, MultiColumnHeader multicolumnHeader,
@@ -201,6 +224,9 @@ void SortByMultipleColumns()
 
                 switch (sortOption)
                 {
+                    case SortOption.Index:
+                        orderedQuery = orderedQuery.ThenBy(l => l.index, ascending);
+                        break;
                     case SortOption.Name:
                         orderedQuery = orderedQuery.ThenBy(l => l.displayName, ascending);
                         break;
@@ -216,6 +242,12 @@ void SortByMultipleColumns()
                     case SortOption.Median:
                         orderedQuery = orderedQuery.ThenBy(l => l.median, ascending);
                         break;
+                    case SortOption.Min:
+                        orderedQuery = orderedQuery.ThenBy(l => l.min, ascending);
+                        break;
+                    case SortOption.Max:
+                        orderedQuery = orderedQuery.ThenBy(l => l.max, ascending);
+                        break;
                 }
             }
 
@@ -228,6 +260,8 @@ IOrderedEnumerable<TestListTableItem> InitialOrder(IEnumerable<TestListTableItem
             bool ascending = multiColumnHeader.IsSortedAscending(history[0]);
             switch (sortOption)
             {
+                case SortOption.Index:
+                    return myTypes.Order(l => l.index, ascending);
                 case SortOption.Name:
                     return myTypes.Order(l => l.displayName, ascending);
                 case SortOption.SampleCount:
@@ -238,13 +272,17 @@ IOrderedEnumerable<TestListTableItem> InitialOrder(IEnumerable<TestListTableItem
                     return myTypes.Order(l => l.standardDeviation, ascending);
                 case SortOption.Median:
                     return myTypes.Order(l => l.median, ascending);
+                case SortOption.Min:
+                    return myTypes.Order(l => l.min, ascending);
+                case SortOption.Max:
+                    return myTypes.Order(l => l.max, ascending);
                 default:
                     Assert.IsTrue(false, "Unhandled enum");
                     break;
             }
 
             // default
-            return myTypes.Order(l => l.displayName, ascending);
+            return myTypes.Order(l => l.index, ascending);
         }
 
         protected override void RowGUI(RowGUIArgs args)
@@ -264,11 +302,11 @@ void CellGUI(Rect cellRect, TestListTableItem item, MyColumns column, ref RowGUI
 
             switch (column)
             {
+                case MyColumns.Index:
+                    EditorGUI.LabelField(cellRect, string.Format("{0}", item.index));
+                    break;
                 case MyColumns.Name:
-                {
-                    args.rowRect = cellRect;
-                    base.RowGUI(args);
-                }
+                    EditorGUI.LabelField(cellRect, string.Format("{0}", item.displayName));
                     break;
                 case MyColumns.SampleCount:
                     EditorGUI.LabelField(cellRect, string.Format("{0}", item.test.SampleGroups.Count));
@@ -282,6 +320,12 @@ void CellGUI(Rect cellRect, TestListTableItem item, MyColumns column, ref RowGUI
                 case MyColumns.Median:
                     EditorGUI.LabelField(cellRect, string.Format("{0:f2}", item.median));
                     break;
+                case MyColumns.Min:
+                    EditorGUI.LabelField(cellRect, string.Format("{0:f2}", item.min));
+                    break;
+                case MyColumns.Max:
+                    EditorGUI.LabelField(cellRect, string.Format("{0:f2}", item.max));
+                    break;
             }
         }
 
@@ -293,37 +337,54 @@ protected override bool CanMultiSelect(TreeViewItem item)
             return false;
         }
 
+        struct HeaderData
+        {
+            public readonly GUIContent content;
+            public readonly float width;
+            public readonly float minWidth;
+            public readonly bool autoResize;
+            public readonly bool allowToggleVisibility;
+            public readonly bool ascending;
+
+            public HeaderData(string name, string tooltip = "", float width = 100, float minWidth = 50, bool autoResize = true, bool allowToggleVisibility = true, bool ascending = false)
+            {
+                content = new GUIContent(name, tooltip);
+                this.width = width;
+                this.minWidth = minWidth;
+                this.autoResize = autoResize;
+                this.allowToggleVisibility = allowToggleVisibility;
+                this.ascending = ascending;
+            }
+        }
+
         public static MultiColumnHeaderState CreateDefaultMultiColumnHeaderState(float treeViewWidth)
         {
             var columnList = new List<MultiColumnHeaderState.Column>();
-            columnList.Add(new MultiColumnHeaderState.Column
+            HeaderData[] headerData = new HeaderData[]
             {
-                headerContent = new GUIContent("Name"),
-                headerTextAlignment = TextAlignment.Left,
-                sortedAscending = true,
-                sortingArrowAlignment = TextAlignment.Left,
-                width = 600,
-                minWidth = 100,
-                autoResize = false,
-                allowToggleVisibility = false
-            });
-            string[] names = { "Sample Groups", "Standard Deviation", "Deviation", "Median" };
-            foreach (var name in names)
+                new HeaderData("Index", "Ordering from the test run", width : 40, minWidth : 50),
+                new HeaderData("Name", "Name of test", width : 500, minWidth : 100, autoResize : false, allowToggleVisibility : false, ascending : true),
+                new HeaderData("Groups", "Number of Sample Groups", width : 60, minWidth : 50),
+                new HeaderData("SD", "Standard Deviation"), //  (of sample group with largest deviation)
+                new HeaderData("Deviation", "Standard Deviation / Median"),
+                new HeaderData("Median", "Median value"),
+                new HeaderData("Min", "Min value"),
+                new HeaderData("Max", "Max value"),
+            };
+            foreach (var header in headerData)
             {
-                var column = new MultiColumnHeaderState.Column
+                columnList.Add(new MultiColumnHeaderState.Column
                 {
-                    headerContent = new GUIContent(name),
+                    headerContent = header.content,
                     headerTextAlignment = TextAlignment.Left,
-                    sortedAscending = true,
+                    sortedAscending = header.ascending,
                     sortingArrowAlignment = TextAlignment.Left,
-                    width = 100,
-                    minWidth = 50,
-                    autoResize = true
-                };
-                columnList.Add(column);
-            }
-
-            ;
+                    width = header.width,
+                    minWidth = header.minWidth,
+                    autoResize = header.autoResize,
+                    allowToggleVisibility = header.allowToggleVisibility
+                });
+            };
             var columns = columnList.ToArray();
 
             Assert.AreEqual(columns.Length, Enum.GetValues(typeof(MyColumns)).Length,
@@ -332,11 +393,14 @@ public static MultiColumnHeaderState CreateDefaultMultiColumnHeaderState(float t
             var state = new MultiColumnHeaderState(columns);
             state.visibleColumns = new int[]
             {
+                (int)MyColumns.Index,
                 (int)MyColumns.Name,
                 (int)MyColumns.SampleCount,
                 (int)MyColumns.Deviation,
                 (int)MyColumns.StandardDeviation,
-                (int)MyColumns.Median
+                (int)MyColumns.Median,
+                (int)MyColumns.Min,
+                (int)MyColumns.Max,
             };
             return state;
         }