CHANGE: switch performance tests to DynamicMeasurementCount() #2162
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Performance tests are usually using MeasurementCount(X) to run tests X times and then calculate the average. Using DynamicMeasurementCount() (see manual) it dynamically finds a suitable measurement count based on the margin of error of the samples. (Jira ISX-1496)
Testing status & QA
CI green
data getting submitted to observer
tests remain on implicit [Version="1"] as recommended by @marek-u3d
the overall runtime of the CI is a bit shorter since it needs less iterations on more stable platforms to finish the test
it is more stable in quite a few tests
it is as (un)stable in most tests
it is a bit more unstable in a handful of tests
some tests are faster
a handful of tests are a little bit slower
Overall Product Risks
low risk, we are going to minitor them for a while and adjust if needed
Complexity: low
Halo Effect: low
Checklist
Before review:
Changed,Fixed,Addedsections.Area_CanDoX,Area_CanDoX_EvenIfYIsTheCase,Area_WhenIDoX_AndYHappens_ThisIsTheResult.During merge:
NEW: ___.FIX: ___.DOCS: ___.CHANGE: ___.RELEASE: 1.1.0-preview.3.After merge: