runSettings

[didier-kuttel-kollmorgen]

We just moved from NUnit 2.x to NUnit 3.x, starting to use the Adapter inside VS.
In the past, we were using TestDriven.NET, it was possible to include/exclude categories defined for our unit tests.

Looking in the doc, I found that probably the best way is to use a .runsettings file including/excluding categories upon our need.

I tried to create such a file and use it in the VS Test Explorer.

cat != MyCatToExclude

Then, in the Test Explorer, hitting Run All Tests In View - I would expect that all tests from MyCatToExclude are not run.
They are run ...

What am I missing ?

[OsirisTerje]

1. You can add categories to be included/excluded using the Traits in the search box in the Test Explorer, a Trait is the same as a Category.
   So to include: Trait:"MyCat" and exclude -Trait:"MyCat"
2. If you want to use a runsettings file, you have to place the command you have above into the NUnit.Where property. If you also enable NUnit.Verbosity=5 you can see in the Output/Test command window a list of all settings and their values in a test run. That way you can ensure you got it right. A good tip is to name the file .runsettings and it will be automatically included (if you didn't actively turn that feature off). The documentation is here https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#where, and there is also a link to a blogpost about the Where property. And here is the link to the docs for the Where syntax: https://docs.nunit.org/articles/nunit/running-tests/Test-Selection-Language.html

[OsirisTerje]

Which versions of NUnit and the adapter is you using? Please also verify with the latest https://www.nuget.org/packages/NUnit3TestAdapter/4.2.0-beta.4 Using the log you can then see the actual filter it will be using.

UPDATE: Saw you are on 4.1 in the log

[OsirisTerje]

The issue here is that the NUnit.Where statement only works on commandline, either dotnet test or vstest. Inside the test explorer you should use the Traits mechanism, as described in pt.1 above.

That could have been made more clear in the documentation. Also, one could discuss whether it should take effect also in the Test Explorer. It could be a bit confusing though, because you would still see the tests when you have the Real Time Test Discovery option in VS turned on.

Update: It is actually pretty easy to add that feature to the adapter, but it should probably be optional.

What is the reason you want to use runsettings instead of the traits feature? Is there some permanent exclusion you're looking for?

[didier-kuttel-kollmorgen]

Tried with the new version, you are correct not working.

These particular tests cannot be run on developers machine, it requires some special hardware.
I was trying to find a way not juggling with the search box, having something more permanent.
On the dev machine, we exclude 2 categories, on the build machines we include them or test only them.
On the build machine I can use the command line, all good, but in VS Test Explorer I was searching for an elegant solution.

Not an adapter issue, but the search will not update the total display at the top, in my case always showing some failed tests, was tryingto find a way to not have these failures.

[OsirisTerje]

The way you can solve it is by marking these tests as Explicit, keep also the category. You can add that at the class level so you don't need to mark all tests, if that is more convenient. They will then not be run on the developers machines.
On the build server you can run them using the category, but be aware that you will need to run them alone (meaning only explicit tests, but as many categories as you like), not together with other non-explicit tests.
If you really need to run them together with something else (non-explicit), then you can use the Where filter and add the following property to that, as a suffix to whatever other filter you may have: <prop name='Explicit'>true</prop>. It should work.

[didier-kuttel-kollmorgen]

Thx - I will give it a try and reply here.