From 21e980b24964bceed7adca906bc48c95d707a9a9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Mon, 25 Nov 2024 14:47:50 +0100 Subject: [PATCH 1/9] MSTest: document testconfig.json --- .../testing/unit-testing-mstest-configure.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index 592dcc161368b..8c20603dc1946 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -100,3 +100,97 @@ Each element of the file is optional because it has a default value. ``` + +## testconfig.json + +Starting with MSTest 3.7, when running your tests through MSTest runner (a.k.a Microsoft.Testing.Platform), you can use a `testconfig.json` file to configure the behavior of the test runner. The `testconfig.json` file is a JSON file that contains the configuration settings for the test runner. The file is used to configure the test runner and the test execution environment. For more information, refer to [Microsoft.Testing.Platform testconfig.json documentation](unit-testing-platform-config.md#testconfigjson). + +### MSTest element + +MSTest settings are grouped by functionality that are described in the sections below. + +| Entry | Default | Description | +|-------|---------|-------------| +| orderTestsByNameInClass | false | If you want to run tests by test names both in Test Explorers and on the command line, set this value to **true**. | +| enableBaseClassTestMethodsFromOtherAssemblies | true | A value indicating whether to enable discovery of test methods from base classes in a different assembly from the inheriting test class. | +| classCleanupLifecycle | EndOfAssembly | If you want the class cleanup to occur at the end of the class, set it to **EndOfClass**. | + +#### AssemblyResolution settings + +| Entry | Default | Description | +|-------|---------|-------------| +| paths | None | You can specify paths to extra assemblies when finding and running unit tests. For example, use these paths for dependency assemblies that aren't in the same directory as the test assembly. You can specify a path in the shape `{ "path": "...", "includeSubDirectories": "true/false" }`. | + +#### Deployment settings + +| Entry | Default | Description | +|-------|---------|-------------| +| deleteDeploymentDirectoryAfterTestRunIsComplete | true | To retain the deployment directory after a test run, set this value to **false**. | +| deployTestSourceDependencies | true |A value indicating whether the test source references are to be deployed. | +| enabled | true | If you set the value to **false**, deployment items that you specify in your test method aren't copied to the deployment directory. | + +#### Output settings + +| Entry | Default | Description | +|-------|---------|-------------| +| captureTrace | false | Capture text messages coming from the `Console.Write*`, `Trace.Write*`, and `Debug.Write*` APIs that will be associated to the current running test. | + +#### Parallelism settings + +| Entry | Default | Description | +|-------|---------|-------------| +| enabled | false | Enable test parallelization. | +| scope | class | The scope of parallelization. You can set it to `method`. The default, `class`, corresponds to running all tests of a given class sequentially but multiple classes in parallel. | +| workers | 0 | The number of threads/workers to be used for parallelization. The default value maps to the number of processors on the current machine. | + +#### Run settings + +| Entry | Default | Description | +|-------|---------|-------------| +| considerEmptyDataSourceAsInconclusive | false | When set to `true`, an empty data source is considered as inconclusive. | +| considerFixturesAsSpecialTests | false | To display `AssemblyInitialize`, `AssemblyCleanup`, `ClassInitialize`, `ClassCleanup` as individual entries in Visual Studio and Visual Studio Code `Test Explorer` and _.trx_ log, set this value to **true**. | +| mapInconclusiveToFailed | false | If a test completes with an inconclusive status, it's mapped to the skipped status in **Test Explorer**. If you want inconclusive tests to be shown as failed, set the value to **true**. | +| mapNotRunnableToFailed | true | A value indicating whether a not runnable result is mapped to failed test. | +| treatClassAndAssemblyCleanupWarningsAsErrors | false | To see your failures in class cleanups as errors, set this value to **true**. | +| treatDiscoveryWarningsAsErrors | false | To report test discovery warnings as errors, set this value to **true**. | + +#### Timeout settings + +| Entry | Default | Description | +|-------|---------|-------------| +| assemblyCleanup | 0 | Specify globally the timeout to apply on each instance of assembly cleanup method. | +| assemblyInitialize | 0 | Specify globally the timeout to apply on each instance of assembly initialize method. | +| classCleanup | 0 | Specify globally the timeout to apply on each instance of class cleanup method. | +| classInitialize | 0 | Specify globally the timeout to apply on each instance of class initialize method. | +| test | 0 | Specify globally the test timeout. | +| testCleanup | 0 | Specify globally the timeout to apply on each instance of test cleanup method. | +| testInitialize | 0 | Specify globally the timeout to apply on each instance of test initialize method. | +| useCooperativeCancellation | false | When set to `true`, in case of timeout, MSTest will only trigger cancellation of the `CancellationToken` but will not stop observing the method. This is more performant but relies on user to correctly flow the token through all paths. | + +> [!NOTE] +> `[Timeout]` attribute specified on a method overrides the global timeout. For example, `[Timeout(1000)]` on a method marked with [AssemblyCleanup] will override the global `assemblyCleanup` timeout. + +### Example *testconfig.json* file + +The following json shows the contents of a typical *.testconfig.json* file. Copy this code and edit it to suit your needs. + +Each element of the file is optional because it has a default value. + +```json +{ + "mstest": { + "execution": { + "mapInconclusiveToFailed" : true, + "disableAppDomain": true, + "considerFixturesAsSpecialTests" : false, + }, + "parallelism" : { + "enabled": true, + "scope": "method", + }, + "output": { + "captureTrace": false + } + } +} +``` From 88228242c6646c75e62063f346024d4bef17189e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Mon, 25 Nov 2024 15:18:54 +0100 Subject: [PATCH 2/9] Rewrite introduction --- docs/core/testing/unit-testing-mstest-configure.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index 8c20603dc1946..b66a2c01c5812 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -103,7 +103,9 @@ Each element of the file is optional because it has a default value. ## testconfig.json -Starting with MSTest 3.7, when running your tests through MSTest runner (a.k.a Microsoft.Testing.Platform), you can use a `testconfig.json` file to configure the behavior of the test runner. The `testconfig.json` file is a JSON file that contains the configuration settings for the test runner. The file is used to configure the test runner and the test execution environment. For more information, refer to [Microsoft.Testing.Platform testconfig.json documentation](unit-testing-platform-config.md#testconfigjson). +When running your tests with MSTest, you can use a `testconfig.json` file to configure the behavior of the test runner. The `testconfig.json` file is a JSON file that contains the configuration settings for the test runner. The file is used to configure the test runner and the test execution environment. For more information, refer to [Microsoft.Testing.Platform testconfig.json documentation](unit-testing-platform-config.md#testconfigjson). + +Starting with MSTest 3.7, you can also configure MSTest runs in the same configuration file. The sections below describe the settings that you can use in the `testconfig.json` file. ### MSTest element From b924b91209470f6a4ef623674af0b0da3829000e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 26 Nov 2024 09:23:14 +0100 Subject: [PATCH 3/9] Update docs/core/testing/unit-testing-mstest-configure.md --- docs/core/testing/unit-testing-mstest-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index b66a2c01c5812..353a78f5ec6a6 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -145,7 +145,7 @@ MSTest settings are grouped by functionality that are described in the sections | scope | class | The scope of parallelization. You can set it to `method`. The default, `class`, corresponds to running all tests of a given class sequentially but multiple classes in parallel. | | workers | 0 | The number of threads/workers to be used for parallelization. The default value maps to the number of processors on the current machine. | -#### Run settings +#### Execution settings | Entry | Default | Description | |-------|---------|-------------| From b9166713b74e4f8475b0a5a590a7004754839464 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 26 Nov 2024 09:23:52 +0100 Subject: [PATCH 4/9] Update docs/core/testing/unit-testing-mstest-configure.md --- docs/core/testing/unit-testing-mstest-configure.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index 353a78f5ec6a6..102be64e80ebb 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -180,6 +180,8 @@ Each element of the file is optional because it has a default value. ```json { + "platformOptions": { + }, "mstest": { "execution": { "mapInconclusiveToFailed" : true, From 1b94d4db272ffcb9c0aa332206c800f43dbfdbf6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 3 Dec 2024 07:04:28 +0100 Subject: [PATCH 5/9] Update docs/core/testing/unit-testing-mstest-configure.md Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> --- docs/core/testing/unit-testing-mstest-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index 102be64e80ebb..e3b2c583e0164 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -105,7 +105,7 @@ Each element of the file is optional because it has a default value. When running your tests with MSTest, you can use a `testconfig.json` file to configure the behavior of the test runner. The `testconfig.json` file is a JSON file that contains the configuration settings for the test runner. The file is used to configure the test runner and the test execution environment. For more information, refer to [Microsoft.Testing.Platform testconfig.json documentation](unit-testing-platform-config.md#testconfigjson). -Starting with MSTest 3.7, you can also configure MSTest runs in the same configuration file. The sections below describe the settings that you can use in the `testconfig.json` file. +Starting with MSTest 3.7, you can also configure MSTest runs in the same configuration file. The following sections describe the settings that you can use in the `testconfig.json` file. ### MSTest element From 0a0c986ce4ba0a20a9d010bcd9d2eb756f62f277 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 3 Dec 2024 07:04:50 +0100 Subject: [PATCH 6/9] Update docs/core/testing/unit-testing-mstest-configure.md Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> --- docs/core/testing/unit-testing-mstest-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index e3b2c583e0164..cb6515f97722f 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -109,7 +109,7 @@ Starting with MSTest 3.7, you can also configure MSTest runs in the same configu ### MSTest element -MSTest settings are grouped by functionality that are described in the sections below. +MSTest settings are grouped by functionality that are described in the sections that follow. | Entry | Default | Description | |-------|---------|-------------| From d517d50e8cdf990de32fb1f2b4d64e397bd20608 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 3 Dec 2024 07:05:11 +0100 Subject: [PATCH 7/9] Update docs/core/testing/unit-testing-mstest-configure.md Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> --- docs/core/testing/unit-testing-mstest-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index cb6515f97722f..4dd1e45277c28 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -128,7 +128,7 @@ MSTest settings are grouped by functionality that are described in the sections | Entry | Default | Description | |-------|---------|-------------| | deleteDeploymentDirectoryAfterTestRunIsComplete | true | To retain the deployment directory after a test run, set this value to **false**. | -| deployTestSourceDependencies | true |A value indicating whether the test source references are to be deployed. | +| deployTestSourceDependencies | true | Indicates whether the test source references are to be deployed. | | enabled | true | If you set the value to **false**, deployment items that you specify in your test method aren't copied to the deployment directory. | #### Output settings From bc2982bef1ede7383237c794b4a1da700c719436 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 3 Dec 2024 07:05:29 +0100 Subject: [PATCH 8/9] Update docs/core/testing/unit-testing-mstest-configure.md Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> --- docs/core/testing/unit-testing-mstest-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index 4dd1e45277c28..8a7cfcf01c175 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -167,7 +167,7 @@ MSTest settings are grouped by functionality that are described in the sections | test | 0 | Specify globally the test timeout. | | testCleanup | 0 | Specify globally the timeout to apply on each instance of test cleanup method. | | testInitialize | 0 | Specify globally the timeout to apply on each instance of test initialize method. | -| useCooperativeCancellation | false | When set to `true`, in case of timeout, MSTest will only trigger cancellation of the `CancellationToken` but will not stop observing the method. This is more performant but relies on user to correctly flow the token through all paths. | +| useCooperativeCancellation | false | When set to `true`, in case of timeout, MSTest will only trigger cancellation of the `CancellationToken` but will not stop observing the method. This behavior is more performant but relies on the user to correctly flow the token through all paths. | > [!NOTE] > `[Timeout]` attribute specified on a method overrides the global timeout. For example, `[Timeout(1000)]` on a method marked with [AssemblyCleanup] will override the global `assemblyCleanup` timeout. From 1bb175818475653990e5a765a29fbe2c92973c05 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 3 Dec 2024 07:05:41 +0100 Subject: [PATCH 9/9] Update docs/core/testing/unit-testing-mstest-configure.md Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> --- docs/core/testing/unit-testing-mstest-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index 8a7cfcf01c175..1a1a884c121bb 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -174,7 +174,7 @@ MSTest settings are grouped by functionality that are described in the sections ### Example *testconfig.json* file -The following json shows the contents of a typical *.testconfig.json* file. Copy this code and edit it to suit your needs. +The following JSON shows the contents of a typical *.testconfig.json* file. Copy this code and edit it to suit your needs. Each element of the file is optional because it has a default value.