Skip to content

Commit 835a624

Browse files
author
Unity Technologies
committed
com.unity.test-framework.performance@0.1.41-preview
## [0.1.41] - 2018-10-2 ### Test report graph ## [0.1.40] - 2018-9-17 ### Update documentation
1 parent 44eae0c commit 835a624

24 files changed

+1453
-218
lines changed

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Changelog
22

3+
## [0.1.41] - 2018-10-2
4+
5+
### Test report graph
6+
37
## [0.1.40] - 2018-9-17
48

59
### Update documentation

‎Documentation~/images/graphtool.png‎

151 KB
Loading

‎Documentation~/index.md‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ Median, Average, Min, Max, Percentile
8585

8686
## Taking measurements
8787

88-
The Performance Testing Extension provides several API methods you can use to take measurements in your performance test, depending on what you need to measure and how you want to do it.
88+
The Performance Testing Extension provides several API methods you can use to take measurements in your performance test, depending on what you need to measure and how you want to do it. In order to use this you have to include `using Unity.PerformanceTesting` at the top of your script.
8989

9090
They are:
9191
* Measure.Method
@@ -222,7 +222,7 @@ public void Test()
222222

223223
Used to record profiler markers. Profiler marker timings will be sampled within the scope of the `using` statement. Note that deep and editor profiling markers are not available.
224224

225-
#### Example 1: Use a custom measurement to capture total allocated memory
225+
#### Example 1: Measuring profiler markers in a scope
226226

227227
``` csharp
228228
[PerformanceTest]
@@ -267,6 +267,28 @@ When a test is selected in the Unity Test Runner window within the Unity Editor,
267267

268268
`Time Millisecond Median:53.59 Min:53.36 Max:62.10 Avg:54.07 Std:1.90 Zeroes:0 SampleCount: 19 Sum: 1027.34`
269269

270+
271+
## Viewing performance test report
272+
273+
The Performance Test Report window shows a detailed breakdown of individual test runs. This can be used to assess the stability of each test. It provides a visualisation of each individual sample recorded within a sample group along with summary statistics for the selected test. You can open the window by going to *Window > Analysis > Performance Test Report*.
274+
275+
The Performance Test Report is split into two views: the *test view* and the *sample group view*.
276+
277+
**Test View:** Provides a list of all tests. Each of the columns can be clicked to sort the view. Column values show the sample group with highest deviation.
278+
279+
* *Name* - name of the test.
280+
* *Deviation* - The deviation is a value calculated by dividing the standard deviation by the median for a sample group. It shows the sample group with the highest 'deviation' value. Useful for defining stability of the test.
281+
* *Standard Deviation* - Standard deviation of the samples in a sample group. It shows the sample group with the highest standard deviation.
282+
283+
**Sample Group View:** Visualizes sample groups for selected test in the Test View. Provides
284+
* Sample group summary displaying the min, max, and median values for a given sample group.
285+
* Samples displayed in a bar chart, ordered by time, with a blue line indicating the median.
286+
* Box plot showing upper (75%) and lower (25%) quartiles, min, max and median of the samples for a given sample group.
287+
288+
*Note: Performance Test Report is supported in Unity version 2018.3 or newer.*
289+
290+
![Performance test report](images/graphtool.png)
291+
270292
## More Examples
271293

272294
#### Example 1: Measure execution time to serialize simple object to JSON

‎Editor/EditmodeMetadataCollector.cs‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
using UnityEditor;
77
using UnityEngine;
88
using UnityEngine.TestTools;
9-
using UnityEngine.XR;
109

1110
[Category("Performance")]
1211
public class EditmodeMetadataCollector : IPrebuildSetup
@@ -73,11 +72,11 @@ private static PlayerSystemInfo GetSystemInfo()
7372
DeviceName = SystemInfo.deviceName,
7473
ProcessorType = SystemInfo.processorType,
7574
ProcessorCount = SystemInfo.processorCount,
76-
XrModel = XRDevice.model,
7775
GraphicsDeviceName = SystemInfo.graphicsDeviceName,
7876
SystemMemorySize = SystemInfo.systemMemorySize,
7977
#if ENABLE_VR
80-
XrDevice = XRSettings.loadedDeviceName
78+
XrModel = UnityEngine.XR.XRDevice.model,
79+
XrDevice = UnityEngine.XR.XRSettings.loadedDeviceName
8180
#endif
8281
};
8382
}

‎Editor/PerformanceTestRunSaver.cs‎

Lines changed: 50 additions & 186 deletions
Original file line numberDiff line numberDiff line change
@@ -1,187 +1,51 @@
11
#if UNITY_2018_3_OR_NEWER
2-
//using System;
3-
//using System.IO;
4-
//using System.Text;
5-
//using Unity.PerformanceTesting.Runtime;
6-
//using UnityEditor;
7-
//using UnityEditor.Networking.PlayerConnection;
8-
//using UnityEngine;
9-
//using UnityEditor.TestTools.TestRunner.Api;
10-
//using UnityEngine.Networking.PlayerConnection;
11-
//
12-
//namespace Unity.PerformanceTesting.Editor
13-
//{
14-
// [InitializeOnLoad]
15-
// public class TestRunnerInitializer
16-
// {
17-
// static TestRunnerInitializer()
18-
// {
19-
// var api = ScriptableObject.CreateInstance<TestRunnerApi>();
20-
// var obj = ScriptableObject.CreateInstance<PerformanceTestRunSaver>();
21-
// api.RegisterCallbacks(obj);
22-
// }
23-
// }
24-
//
25-
// [Serializable]
26-
// public class PerformanceTestRunSaver : ScriptableObject, ICallbacks
27-
// {
28-
// private PerformanceTestRun m_Run;
29-
//
30-
// public void OnEnable()
31-
// {
32-
// EditorConnection.instance.Initialize();
33-
// EditorConnection.instance.Register(Utils.k_sendTestDataToEditor, OnTestDataReceived);
34-
// EditorConnection.instance.Register(Utils.k_sendPlayerDataToEditor, OnPlayerDataReceived);
35-
// }
36-
//
37-
// public void OnDisable()
38-
// {
39-
// EditorConnection.instance.Unregister(Utils.k_sendTestDataToEditor, OnTestDataReceived);
40-
// EditorConnection.instance.Unregister(Utils.k_sendPlayerDataToEditor, OnPlayerDataReceived);
41-
// }
42-
//
43-
// private void OnTestDataReceived(MessageEventArgs args)
44-
// {
45-
// var json = Encoding.ASCII.GetString(args.data);
46-
// var test = JsonUtility.FromJson<PerformanceTest>(json);
47-
// AddTest(test);
48-
// }
49-
//
50-
// private void OnPlayerDataReceived(MessageEventArgs args)
51-
// {
52-
// var text = Encoding.ASCII.GetString(args.data);
53-
// var runData = JsonUtility.FromJson<PerformanceTestRun>(text);
54-
//
55-
// if (m_Run == null) InitializeTestRun();
56-
// m_Run.PlayerSystemInfo = runData.PlayerSystemInfo;
57-
// m_Run.QualitySettings = runData.QualitySettings;
58-
// m_Run.ScreenSettings = runData.ScreenSettings;
59-
// m_Run.TestSuite = "Playmode";
60-
// m_Run.BuildSettings.Platform = runData.BuildSettings.Platform;
61-
// }
62-
//
63-
// private void AddTest(PerformanceTest test)
64-
// {
65-
// if (m_Run == null) InitializeTestRun();
66-
// m_Run.Results.Add(test);
67-
// }
68-
//
69-
// private void InitializeTestRun()
70-
// {
71-
// m_Run = new PerformanceTestRun
72-
// {
73-
// StartTime = Utils.DateToInt(DateTime.Now),
74-
// EditorVersion = GetEditorInfo(),
75-
// PlayerSettings = GetPlayerSettings(),
76-
// BuildSettings = GetPlayerBuildInfo()
77-
// };
78-
// }
79-
//
80-
// private void SetHardwareAndQualitySettings()
81-
// {
82-
// m_Run.PlayerSystemInfo = Utils.GetSystemInfo();
83-
// m_Run.QualitySettings = Utils.GetQualitySettings();
84-
// m_Run.ScreenSettings = Utils.GetScreenSettings();
85-
// m_Run.TestSuite = Application.isPlaying ? "Playmode" : "Editmode";
86-
// m_Run.BuildSettings.Platform = Application.platform.ToString();
87-
// }
88-
//
89-
// private static EditorVersion GetEditorInfo()
90-
// {
91-
// return new EditorVersion
92-
// {
93-
// FullVersion = UnityEditorInternal.InternalEditorUtility.GetFullUnityVersion(),
94-
// DateSeconds = int.Parse(UnityEditorInternal.InternalEditorUtility.GetUnityVersionDate().ToString()),
95-
// Branch = GetEditorBranch(),
96-
// RevisionValue = int.Parse(UnityEditorInternal.InternalEditorUtility.GetUnityRevision().ToString())
97-
// };
98-
// }
99-
//
100-
// private static string GetEditorBranch()
101-
// {
102-
// foreach (var method in typeof(UnityEditorInternal.InternalEditorUtility).GetMethods())
103-
// {
104-
// if (method.Name.Contains("GetUnityBuildBranch"))
105-
// {
106-
// return (string) method.Invoke(null, null);
107-
// }
108-
// }
109-
//
110-
// return "null";
111-
// }
112-
//
113-
// private static PlayerSettings GetPlayerSettings()
114-
// {
115-
// var settings = new PlayerSettings
116-
// {
117-
// VrSupported = UnityEditor.PlayerSettings.virtualRealitySupported,
118-
// MtRendering = UnityEditor.PlayerSettings.MTRendering,
119-
// GpuSkinning = UnityEditor.PlayerSettings.gpuSkinning,
120-
// GraphicsJobs = UnityEditor.PlayerSettings.graphicsJobs,
121-
// GraphicsApi =
122-
// UnityEditor.PlayerSettings.GetGraphicsAPIs(EditorUserBuildSettings.activeBuildTarget)[0].ToString(),
123-
// ScriptingBackend = UnityEditor.PlayerSettings
124-
// .GetScriptingBackend(EditorUserBuildSettings.selectedBuildTargetGroup).ToString(),
125-
// StereoRenderingPath = UnityEditor.PlayerSettings.stereoRenderingPath.ToString(),
126-
// RenderThreadingMode = UnityEditor.PlayerSettings.graphicsJobs ? "GraphicsJobs" :
127-
// UnityEditor.PlayerSettings.MTRendering ? "MultiThreaded" : "SingleThreaded",
128-
// AndroidMinimumSdkVersion = UnityEditor.PlayerSettings.Android.minSdkVersion.ToString(),
129-
// Batchmode = Application.isBatchMode.ToString(),
130-
// ScriptingRuntimeVersion = UnityEditor.PlayerSettings.scriptingRuntimeVersion.ToString()
131-
// //playerSettings.StaticBatching = TODO
132-
// //playerSettings.DynamicBatching = TODO
133-
// };
134-
//
135-
// return settings;
136-
// }
137-
//
138-
// private static BuildSettings GetPlayerBuildInfo()
139-
// {
140-
// var buildSettings = new BuildSettings
141-
// {
142-
// BuildTarget = EditorUserBuildSettings.activeBuildTarget.ToString(),
143-
// DevelopmentPlayer = EditorUserBuildSettings.development,
144-
// AndroidBuildSystem = EditorUserBuildSettings.androidBuildSystem.ToString()
145-
// };
146-
// return buildSettings;
147-
// }
148-
//
149-
// void ICallbacks.RunStarted(ITest testsToRun)
150-
// {
151-
// m_Run = null;
152-
// }
153-
//
154-
// void ICallbacks.RunFinished(ITestResult result)
155-
// {
156-
// if (m_Run.Results.Count == 0) SetHardwareAndQualitySettings();
157-
//
158-
// m_Run.EndTime = Utils.DateToInt(DateTime.Now);
159-
//
160-
// if (Utils.VerifyTestRunMetadata(m_Run))
161-
// {
162-
// var path = Utils.GetArg("-performanceTestResults");
163-
// if (path != null)
164-
// {
165-
// File.WriteAllText(path, JsonUtility.ToJson(m_Run));
166-
// }
167-
// }
168-
//
169-
// m_Run = null;
170-
// }
171-
//
172-
// void ICallbacks.TestStarted(ITest test)
173-
// {
174-
// if (m_Run == null) InitializeTestRun();
175-
// PerformanceTest.OnTestEnded = () =>
176-
// {
177-
// SetHardwareAndQualitySettings();
178-
// AddTest(PerformanceTest.Active);
179-
// };
180-
// }
181-
//
182-
// void ICallbacks.TestFinished(ITestResult result)
183-
// {
184-
// }
185-
// }
186-
//}
187-
#endif
2+
using System;
3+
using System.IO;
4+
using UnityEditor;
5+
using UnityEngine;
6+
using UnityEditor.TestTools.TestRunner.Api;
7+
using UnityEditor.TestTools.TestRunner.CommandLineTest;
8+
9+
namespace Unity.PerformanceTesting.Editor
10+
{
11+
[InitializeOnLoad]
12+
public class TestRunnerInitializer
13+
{
14+
static TestRunnerInitializer()
15+
{
16+
var api = ScriptableObject.CreateInstance<TestRunnerApi>();
17+
var obj = ScriptableObject.CreateInstance<PerformanceTestRunSaver>();
18+
api.RegisterCallbacks(obj);
19+
}
20+
}
21+
22+
[Serializable]
23+
public class PerformanceTestRunSaver : ScriptableObject, ICallbacks
24+
{
25+
private PerformanceTestRun m_Run;
26+
27+
void ICallbacks.RunStarted(ITest testsToRun)
28+
{
29+
}
30+
31+
void ICallbacks.RunFinished(ITestResult result)
32+
{
33+
var resultWriter = new ResultsWriter();
34+
string xmlPath = Path.Combine(Application.streamingAssetsPath, "TestResults.xml");
35+
string jsonPath = Path.Combine(Application.streamingAssetsPath, "PerformanceTestResults.json");
36+
resultWriter.WriteResultToFile(result, xmlPath);
37+
var xmlParser = new TestResultXmlParser();
38+
var run = xmlParser.GetPerformanceTestRunFromXml(xmlPath);
39+
File.WriteAllText(jsonPath, JsonUtility.ToJson(run, true));
40+
}
41+
42+
void ICallbacks.TestStarted(ITest test)
43+
{
44+
}
45+
46+
void ICallbacks.TestFinished(ITestResult result)
47+
{
48+
}
49+
}
50+
}
51+
#endif

‎Editor/TestReportGraph.meta‎

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Editor/TestReportGraph/Gintas.TestRunner.Graph.asmdef‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"name": "Gintas.TestRunner.Graph",
3+
"references": [
4+
"Unity.PerformanceTesting"
5+
],
6+
"optionalUnityReferences": [],
7+
"includePlatforms": [
8+
"Editor"
9+
],
10+
"excludePlatforms": [],
11+
"allowUnsafeCode": false,
12+
"overrideReferences": false,
13+
"precompiledReferences": [],
14+
"autoReferenced": true,
15+
"defineConstraints": []
16+
}

‎Editor/TestReportGraph/Gintas.TestRunner.Graph.asmdef.meta‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Editor/TestReportGraph/SampleGroupAdditionalData.cs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
using System;
2+
3+
struct SampleGroupAdditionalData
4+
{
5+
public float min;
6+
public float lowerQuartile;
7+
public float median;
8+
public float upperQuartile;
9+
public float max;
10+
}

‎Editor/TestReportGraph/SampleGroupAdditionalData.cs.meta‎

Lines changed: 11 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)