Skip to content
This repository was archived by the owner on Jan 23, 2023. It is now read-only.

Commit fdd4e91

Browse files
brianrobbrianrob
authored
Enable Linux Performance Runs (#12115)
* Enable linux perf runs (#11905) - Removing `[assembly: MeasureInstructionsRetired]` from performance tests. - Due to this issue: microsoft/xunit-performance#231, we are blocked from running on performance tests on Linux. The tag is not used by infrastructure and it is passed during runtime when profiling on Windows. - Adding the option to execute CoreRun with the stability prefix. - Moving file to be archived by Jenkins and updating groovy file. - Setting the maximum number of iterations. * Updating version of xUnit-Performance-Api (#11987) - This update contains a fix for this issue: microsoft/xunit-performance#230 - Consolidate the Api version in two places: `dependencies.prop` and `PerfHarness.csproj` - Move performance log files to root directory where Jenkins can archive them - Fixed the output to console, so it is easier to see on the Jenkins job the sccript steps live and get an exact repro step if needed
1 parent c59f400 commit fdd4e91

File tree

82 files changed

+320
-388
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

82 files changed

+320
-388
lines changed

‎dependencies.props‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@
1414
<PropertyGroup>
1515
<CoreClrPackageVersion>2.0.0-preview2-25330-02</CoreClrPackageVersion>
1616
<XunitPackageVersion>2.2.0-beta2-build3300</XunitPackageVersion>
17+
<XUnitConsoleNetCoreVersion>1.0.2-prerelease-00177</XUnitConsoleNetCoreVersion>
18+
<XUnitPerformanceApiVersion>1.0.0-beta-build0007</XUnitPerformanceApiVersion>
19+
<MicrosoftDiagnosticsTracingLibraryVersion>1.0.3-alpha-experimental</MicrosoftDiagnosticsTracingLibraryVersion>
1720
</PropertyGroup>
1821

1922
<!-- Package dependency verification/auto-upgrade configuration. -->
@@ -72,8 +75,16 @@
7275
<Version>1.0.0-alpha-build0040</Version>
7376
</StaticDependency>
7477

78+
<XUnitPerformanceApiDependency Include="xunit.performance.api" />
79+
<XUnitPerformanceApiDependency Include="xunit.performance.core" />
80+
<XUnitPerformanceApiDependency Include="xunit.performance.execution" />
81+
<XUnitPerformanceApiDependency Include="xunit.performance.metrics" />
82+
<StaticDependency Include="@(XUnitPerformanceApiDependency)">
83+
<Version>$(XUnitPerformanceApiVersion)</Version>
84+
</StaticDependency>
85+
7586
<StaticDependency Include="xunit.console.netcore">
76-
<Version>1.0.2-prerelease-00177</Version>
87+
<Version>$(XUnitConsoleNetCoreVersion)</Version>
7788
</StaticDependency>
7889

7990
<DependencyBuildInfo Include="@(StaticDependency)">

‎perf.groovy‎

Lines changed: 54 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ def static getOSGroup(def os) {
2424
assert osGroup != null : "Could not find os group for ${os}"
2525
return osGroupMap[os]
2626
}
27+
2728
// Setup perflab tests runs
2829
[true, false].each { isPR ->
2930
['Windows_NT'].each { os ->
@@ -51,34 +52,34 @@ def static getOSGroup(def os) {
5152
}
5253
}
5354

54-
if (isPR)
55-
{
56-
parameters
55+
if (isPR)
5756
{
58-
stringParam('BenchviewCommitName', '\${ghprbPullTitle}', 'The name that you will be used to build the full title of a run in Benchview. The final name will be of the form <branch> private BenchviewCommitName')
57+
parameters
58+
{
59+
stringParam('BenchviewCommitName', '\${ghprbPullTitle}', 'The name that you will be used to build the full title of a run in Benchview. The final name will be of the form <branch> private BenchviewCommitName')
60+
}
5961
}
60-
}
61-
if (isSmoketest)
62-
{
63-
parameters
62+
if (isSmoketest)
6463
{
65-
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION', '2', 'Sets the number of iterations to two. We want to do this so that we can run as fast as possible as this is just for smoke testing')
66-
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION_INNER_SPECIFIED', '2', 'Sets the number of iterations to two. We want to do this so that we can run as fast as possible as this is just for smoke testing')
64+
parameters
65+
{
66+
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION', '2', 'Sets the number of iterations to two. We want to do this so that we can run as fast as possible as this is just for smoke testing')
67+
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION_INNER_SPECIFIED', '2', 'Sets the number of iterations to two. We want to do this so that we can run as fast as possible as this is just for smoke testing')
68+
}
6769
}
68-
}
69-
else
70-
{
71-
parameters
70+
else
7271
{
73-
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION', '21', 'Sets the number of iterations to twenty one. We are doing this to limit the amount of data that we upload as 20 iterations is enought to get a good sample')
74-
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION_INNER_SPECIFIED', '21', 'Sets the number of iterations to twenty one. We are doing this to limit the amount of data that we upload as 20 iterations is enought to get a good sample')
72+
parameters
73+
{
74+
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION', '21', 'Sets the number of iterations to twenty one. We are doing this to limit the amount of data that we upload as 20 iterations is enought to get a good sample')
75+
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION_INNER_SPECIFIED', '21', 'Sets the number of iterations to twenty one. We are doing this to limit the amount of data that we upload as 20 iterations is enought to get a good sample')
76+
}
7577
}
76-
}
77-
def configuration = 'Release'
78-
def runType = isPR ? 'private' : 'rolling'
79-
def benchViewName = isPR ? 'coreclr private %BenchviewCommitName%' : 'coreclr rolling %GIT_BRANCH_WITHOUT_ORIGIN% %GIT_COMMIT%'
80-
def uploadString = isSmoketest ? '' : '-uploadToBenchview'
81-
78+
def configuration = 'Release'
79+
def runType = isPR ? 'private' : 'rolling'
80+
def benchViewName = isPR ? 'coreclr private %BenchviewCommitName%' : 'coreclr rolling %GIT_BRANCH_WITHOUT_ORIGIN% %GIT_COMMIT%'
81+
def uploadString = isSmoketest ? '' : '-uploadToBenchview'
82+
8283
steps {
8384
// Batch
8485

@@ -104,17 +105,17 @@ def static getOSGroup(def os) {
104105

105106
batchFile("tests\\runtest.cmd ${configuration} ${architecture} GenerateLayoutOnly")
106107

107-
// Run with just stopwatch
108+
// Run with just stopwatch: Profile=Off
108109
batchFile("tests\\scripts\\run-xunit-perf.cmd -arch ${arch} -configuration ${configuration} -testBinLoc bin\\tests\\${os}.${architecture}.${configuration}\\performance\\perflab\\Perflab -library -generateBenchviewData \"%WORKSPACE%\\Microsoft.Benchview.JSONFormat\\tools\" ${uploadString} -runtype ${runType} -stabilityPrefix \"START \"CORECLR_PERF_RUN\" /B /WAIT /HIGH /AFFINITY 0x2\"")
109110
batchFile("tests\\scripts\\run-xunit-perf.cmd -arch ${arch} -configuration ${configuration} -testBinLoc bin\\tests\\${os}.${architecture}.${configuration}\\Jit\\Performance\\CodeQuality -generateBenchviewData \"%WORKSPACE%\\Microsoft.Benchview.JSONFormat\\tools\" ${uploadString} -runtype ${runType} -stabilityPrefix \"START \"CORECLR_PERF_RUN\" /B /WAIT /HIGH /AFFINITY 0x2\"")
110111
batchFile("tests\\scripts\\run-xunit-perf.cmd -arch ${arch} -configuration ${configuration} -testBinLoc bin\\tests\\${os}.${architecture}.${configuration}\\performance\\linkbench\\linkbench -generateBenchviewData \"%WORKSPACE%\\Microsoft.Benchview.JSONFormat\\tools\" ${uploadString} -nowarmup -runtype ${runType} -scenarioTest -group ILLink")
111112

112-
// Run with the full set of counters enabled
113+
// Run with the full set of counters enabled: Profile=On
113114
batchFile("tests\\scripts\\run-xunit-perf.cmd -arch ${arch} -configuration ${configuration} -testBinLoc bin\\tests\\${os}.${architecture}.${configuration}\\performance\\perflab\\Perflab -library -generateBenchviewData \"%WORKSPACE%\\Microsoft.Benchview.JSONFormat\\tools\" ${uploadString} -runtype ${runType} -collectionFlags default+BranchMispredictions+CacheMisses+InstructionRetired+gcapi -stabilityPrefix \"START \"CORECLR_PERF_RUN\" /B /WAIT /HIGH /AFFINITY 0x2\"")
114115
batchFile("tests\\scripts\\run-xunit-perf.cmd -arch ${arch} -configuration ${configuration} -testBinLoc bin\\tests\\${os}.${architecture}.${configuration}\\Jit\\Performance\\CodeQuality -generateBenchviewData \"%WORKSPACE%\\Microsoft.Benchview.JSONFormat\\tools\" ${uploadString} -runtype ${runType} -collectionFlags default+BranchMispredictions+CacheMisses+InstructionRetired+gcapi -stabilityPrefix \"START \"CORECLR_PERF_RUN\" /B /WAIT /HIGH /AFFINITY 0x2\"")
115116
}
116117
}
117-
118+
118119
if (isSmoketest)
119120
{
120121
Utilities.setMachineAffinity(newJob, "Windows_NT", '20170427-elevated')
@@ -123,18 +124,20 @@ def static getOSGroup(def os) {
123124
def archiveSettings = new ArchivalSettings()
124125
archiveSettings.addFiles('Perf-*.xml')
125126
archiveSettings.addFiles('Perf-*.etl')
127+
archiveSettings.addFiles('Perf-*.log')
128+
archiveSettings.addFiles('machinedata.json')
126129
Utilities.addArchival(newJob, archiveSettings)
127130

128131
Utilities.standardJobSetup(newJob, project, isPR, "*/${branch}")
129-
132+
130133
newJob.with {
131134
wrappers {
132135
timeout {
133136
absolute(240)
134137
}
135138
}
136139
}
137-
140+
138141
if (isPR) {
139142
TriggerBuilder builder = TriggerBuilder.triggerOnPullRequest()
140143
if (isSmoketest)
@@ -185,7 +188,7 @@ def static getOSGroup(def os) {
185188
def configuration = 'Release'
186189
def runType = isPR ? 'private' : 'rolling'
187190
def benchViewName = isPR ? 'coreclr-throughput private %BenchviewCommitName%' : 'coreclr-throughput rolling %GIT_BRANCH_WITHOUT_ORIGIN% %GIT_COMMIT%'
188-
191+
189192
steps {
190193
// Batch
191194

@@ -235,49 +238,59 @@ def static getOSGroup(def os) {
235238
[true, false].each { isPR ->
236239
['Ubuntu14.04'].each { os ->
237240
def newJob = job(Utilities.getFullJobName(project, "perf_${os}", isPR)) {
238-
241+
239242
label('linux_clr_perf')
240-
wrappers {
241-
credentialsBinding {
242-
string('BV_UPLOAD_SAS_TOKEN', 'CoreCLR Perf BenchView Sas')
243-
}
243+
wrappers {
244+
credentialsBinding {
245+
string('BV_UPLOAD_SAS_TOKEN', 'CoreCLR Perf BenchView Sas')
244246
}
245-
247+
}
248+
246249
if (isPR)
247250
{
248251
parameters
249252
{
250253
stringParam('BenchviewCommitName', '\${ghprbPullTitle}', 'The name that you will be used to build the full title of a run in Benchview. The final name will be of the form <branch> private BenchviewCommitName')
251254
}
252255
}
256+
257+
// Cap the maximum number of iterations to 21.
258+
parameters {
259+
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION', '21', 'Sets the number of iterations to twenty one. We are doing this to limit the amount of data that we upload as 20 iterations is enought to get a good sample')
260+
stringParam('XUNIT_PERFORMANCE_MAX_ITERATION_INNER_SPECIFIED', '21', 'Sets the number of iterations to twenty one. We are doing this to limit the amount of data that we upload as 20 iterations is enought to get a good sample')
261+
}
262+
253263
def osGroup = getOSGroup(os)
254264
def architecture = 'x64'
255265
def configuration = 'Release'
256266
def runType = isPR ? 'private' : 'rolling'
257267
def benchViewName = isPR ? 'coreclr private \$BenchviewCommitName' : 'coreclr rolling \$GIT_BRANCH_WITHOUT_ORIGIN \$GIT_COMMIT'
258-
268+
259269
steps {
260-
shell("bash ./tests/scripts/perf-prep.sh")
270+
shell("./tests/scripts/perf-prep.sh")
261271
shell("./init-tools.sh")
262272
shell("./build.sh ${architecture} ${configuration}")
263273
shell("GIT_BRANCH_WITHOUT_ORIGIN=\$(echo \$GIT_BRANCH | sed \"s/[^/]*\\/\\(.*\\)/\\1 /\")\n" +
264274
"python3.5 \"\${WORKSPACE}/tests/scripts/Microsoft.BenchView.JSONFormat/tools/submission-metadata.py\" --name \" ${benchViewName} \" --user \"[email protected]\"\n" +
265275
"python3.5 \"\${WORKSPACE}/tests/scripts/Microsoft.BenchView.JSONFormat/tools/build.py\" git --branch \$GIT_BRANCH_WITHOUT_ORIGIN --type ${runType}")
266-
shell("""sudo -E bash ./tests/scripts/run-xunit-perf.sh \\
276+
shell("""./tests/scripts/run-xunit-perf.sh \\
267277
--testRootDir=\"\${WORKSPACE}/bin/tests/Windows_NT.${architecture}.${configuration}\" \\
268278
--testNativeBinDir=\"\${WORKSPACE}/bin/obj/${osGroup}.${architecture}.${configuration}/tests\" \\
269279
--coreClrBinDir=\"\${WORKSPACE}/bin/Product/${osGroup}.${architecture}.${configuration}\" \\
270280
--mscorlibDir=\"\${WORKSPACE}/bin/Product/${osGroup}.${architecture}.${configuration}\" \\
271281
--coreFxBinDir=\"\${WORKSPACE}/corefx\" \\
272282
--runType=\"${runType}\" \\
273283
--benchViewOS=\"${os}\" \\
284+
--generatebenchviewdata=\"\${WORKSPACE}/tests/scripts/Microsoft.BenchView.JSONFormat/tools\" \\
285+
--stabilityPrefix=\"taskset 0x00000002 nice --adjustment=-10\" \\
274286
--uploadToBenchview""")
275287
}
276288
}
277289

278290
// Save machinedata.json to /artifact/bin/ Jenkins dir
279291
def archiveSettings = new ArchivalSettings()
280-
archiveSettings.addFiles('sandbox/perf-*.xml')
292+
archiveSettings.addFiles('Perf-*.log')
293+
archiveSettings.addFiles('Perf-*.xml')
281294
archiveSettings.addFiles('machinedata.json')
282295
Utilities.addArchival(newJob, archiveSettings)
283296

@@ -313,14 +326,14 @@ def static getOSGroup(def os) {
313326
[true, false].each { isPR ->
314327
['Ubuntu14.04'].each { os ->
315328
def newJob = job(Utilities.getFullJobName(project, "perf_throughput_${os}", isPR)) {
316-
329+
317330
label('linux_clr_perf')
318331
wrappers {
319332
credentialsBinding {
320333
string('BV_UPLOAD_SAS_TOKEN', 'CoreCLR Perf BenchView Sas')
321334
}
322335
}
323-
336+
324337
if (isPR)
325338
{
326339
parameters
@@ -333,7 +346,7 @@ def static getOSGroup(def os) {
333346
def configuration = 'Release'
334347
def runType = isPR ? 'private' : 'rolling'
335348
def benchViewName = isPR ? 'coreclr private \$BenchviewCommitName' : 'coreclr rolling \$GIT_BRANCH_WITHOUT_ORIGIN \$GIT_COMMIT'
336-
349+
337350
steps {
338351
shell("bash ./tests/scripts/perf-prep.sh --throughput")
339352
shell("./init-tools.sh")

0 commit comments

Comments
 (0)