Skip to content

Conversation

@stefanunity
Copy link
Collaborator

Description

Performance tests are usually using MeasurementCount(X) to run tests X times and then calculate the average. Using DynamicMeasurementCount() (see manual) it dynamically finds a suitable measurement count based on the margin of error of the samples. (Jira ISX-1496)

Testing status & QA

  • CI green

  • data getting submitted to observer

  • tests remain on implicit [Version="1"] as recommended by @marek-u3d

  • the overall runtime of the CI is a bit shorter since it needs less iterations on more stable platforms to finish the test

  • it is more stable in quite a few tests

  • it is as (un)stable in most tests

  • it is a bit more unstable in a handful of tests

  • some tests are faster

  • a handful of tests are a little bit slower

Overall Product Risks

  • low risk, we are going to minitor them for a while and adjust if needed

  • Complexity: low

  • Halo Effect: low

Checklist

Before review:

  • Changelog entry added.
    • Explains the change in Changed, Fixed, Added sections.
    • For API change contains an example snippet and/or migration example.
    • JIRA ticket linked, example (case %%). If it is a private issue, just add the case ID without a link.
    • Jira port for the next release set as "Resolved".
  • Tests added/changed, if applicable.
    • Functional tests Area_CanDoX, Area_CanDoX_EvenIfYIsTheCase, Area_WhenIDoX_AndYHappens_ThisIsTheResult.
    • Performance tests.
    • Integration tests.
  • Docs for new/changed API's.
    • Xmldoc cross references are set correctly.
    • Added explanation how the API works.
    • Usage code examples added.
    • The manual is updated, if needed.

During merge:

  • Commit message for squash-merge is prefixed with one of the list:
    • NEW: ___.
    • FIX: ___.
    • DOCS: ___.
    • CHANGE: ___.
    • RELEASE: 1.1.0-preview.3.

After merge:

  • Create forward/backward port if needed. If you are blocked from creating a forward port now please add a task to ISX-1444.

Copy link
Collaborator

@ekcoh ekcoh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Much better to have iteration count derived from observed statistical variance.

@stefanunity stefanunity merged commit d93d150 into develop Apr 4, 2025
110 checks passed
@stefanunity stefanunity deleted the performance-DynamicMeasurementCount branch April 4, 2025 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants