MSTest performance

[Youssef1313]

This issue tracks few ideas for improving the performance of MSTest.

• We shouldn't do any serialization for parameterized tests if appdomain is disabled. This will be v4 only where we changed the test id to not rely serialized data.

• ManagedNameHelper from VSTest is extremely slow and memory consuming (PR: Avoid ManagedNameHelper when possible #6319)

  • Note: on a single test class with 10k tests.

    

• Dictionary<string, string> used in UpdateInfoIfTestInitializeOrCleanupMethod:

  • is used to act more as of HashSet. So arguably a HashSet will be simply faster.
  • is consuming more time and memory for providing zero functionality if the test class inherits directly from object.
  • Its overall value is too low (prevents considering base initializes if they are shadowed in concrete test class). Should we get rid of it altogether?
  • Switch to HashSet and avoiding it altogether when base type is object is in PR Performance: Reduce unnecessary logic in UpdateInfoIfTestInitializeOrCleanupMethod #6318. Removing the whole thing as "low value" is still an open question.

[Youssef1313]

Quick profiling (single run, not averaging, but makes it quite obvious 3.11 is significantly faster in multiple scenarios due to #6319)

Single test class with 10k tests

3.3.1: 10s
3.4.3: 14s
3.5.2: 9s
3.6.4: 9s
3.7.3: 10s
3.8.3: 15s
3.9.3: 9s
3.10.1: 8s
3.11 pre: 3s

1k test class, each with 200 tests

3.3.1: 28s
3.4.3: 29s
3.5.2: 27s
3.6.4: 25s
3.7.3: 28s
3.8.3: 39s
3.9.3: 26s
3.10.1: 33s
3.11 pre: 20s

100 test class, each with 2000 tests

3.3.1: 43s
3.4.3: 49s
3.5.2: 33s
3.6.4: 32s
3.7.3: 44s
3.8.3: 42s
3.9.3: 43s
3.10.1: 37s
3.11 pre: 17s

[Youssef1313]

I did two extra runs. So here is the result of 3 runs of each scenario:

Single test class with 10k tests

3.3.1: 10s 10s 9s
3.4.3: 14s 9s 10s
3.5.2: 9s 9s 9s
3.6.4: 9s 10s 10s
3.7.3: 10s 9s 8s
3.8.3: 15s 8s 9s
3.9.3: 9s 9s 9s
3.10.1: 8s 8s 9s
3.11 pre: 3s 2s 3s

1k test class, each with 200 tests

3.3.1: 28s 29s 29s
3.4.3: 29s 29s 27s
3.5.2: 27s 27s 27s
3.6.4: 25s 25s 27s
3.7.3: 28s 24s 25s
3.8.3: 39s 28s 26s
3.9.3: 26s 25s 25s
3.10.1: 33s 21s 23s
3.11 pre: 20s 17s 18s

100 test class, each with 2000 tests

3.3.1: 43s 44s 42s
3.4.3: 49s 41s 42s
3.5.2: 33s 33s 33s
3.6.4: 32s 32s 32s
3.7.3: 44s 41s 40s
3.8.3: 42s 43s 43s
3.9.3: 43s 42s 42s
3.10.1: 37s 40s 39s
3.11 pre: 17s 18s 18s

[nohwnd]

graphed by copilot, since my brain does not comprehend the numbers without being in a table.