You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Updated script to build TestFx environment as well as the document (#20592)
* Updated script to build TestFx environment as well as the document
* Update documentation/testing-docs/using-azure-test-framework.md
Co-authored-by: Yeming Liu <[email protected]>
Co-authored-by: Yeming Liu <[email protected]>
-[Record or Playback Tests](#record-or-playback-tests)
16
+
-[Change Test Environment settings at run-time](#change-test-environment-settings-at-run-time)
17
+
-[Once you set your connection string, you can add or update key/value settings](#once-you-set-your-connection-string-you-can-add-or-update-keyvalue-settings)
18
+
-[Note:](#note)
19
+
-[Troubleshooting](#troubleshooting)
20
+
-[Issue: exceptions in Microsoft.Azure.Test.HttpRecorder](#issue-exceptions-in-microsoftazuretesthttprecorder)
21
+
-[Supported Environments in Test Framework](#supported-environments-in-test-framework)
22
+
-[Default Environments and associated Uri](#default-environments-and-associated-uri)
23
+
-[Environment = Prod](#environment--prod)
24
+
-[Environment = Dogfood](#environment--dogfood)
25
+
-[Environment = Next](#environment--next)
26
+
-[Environment = Current](#environment--current)
27
+
-[Environment = Custom](#environment--custom)
21
28
22
29
## Getting Started
23
-
- Install the latest Az.Resources from the [PSGallery](https://www.powershellgallery.com/) into Windows PowerShell
24
-
- Run Windows PowerShell as administrator and execute following command
- Import the `TestFx-Tasks` module that helps to configure the settings
35
+
- Run the command `Import-Module ./tools/Modules/TestFx-Tasks.psd1`
36
+
37
+
## Azure PowerShell Test Framework
32
38
33
39
Azure PowerShell repo now has its own test framework located under `tools\TestFx`, which supports recording all the HTTP requests from behind Azure PowerShell cmdlets and then playing them back.
34
40
35
41
The target framework of test is .Net Core 3.1, please ensure .Net runtime Microsoft.NETCore.App 3.1 is installed. You can list all installed version via `dotnet --info`.
36
42
37
43
## Setup prior to Record or Playback of tests
38
44
39
-
In order to Record/Playback a test, you need to setup a connection string that consists of various key/value pairs that provide information to the test environment.
45
+
In order to Record/Playback a test, test framework needs to setup a connection string that consists of various key/value pairs that provide necessary information.
46
+
47
+
You can choose either option to configure the settings:
40
48
41
-
You have three options to set up the connection string:
42
-
- Run the [`New-TestCredential` cmdlet](#new-testcredential) (recommended for PowerShell development)
43
-
- Run the [`Set-TestEnvironment` cmdlet](#set-testenvironment)
44
-
-[Manually set the environment variables](#manually-set-environment-variables).
49
+
- Run the [`Set-TestFxEnvironment` cmdlet](#run-command-set-testfxenvironment-to-build-connection-string) (Recommended)
50
+
-[Manually set the environment variables](#manually-set-environment-variables-to-build-connection-string)
45
51
46
-
### New-TestCredential
52
+
### Run Command Set-TestFxEnvironment to Build Connection String (Recommended)
47
53
48
-
This cmdlet, located in the [`Repo-Tasks` module](/tools/Repo-Tasks.psd1), which pulls in the [`TestFx-Tasks` module](/tools/Modules/TestFx-Tasks.psd1) and [`Build-Tasks` module](/tools/Modules/Build-Tasks.psd1), will allow you to create a credentials file (located in `C:/Users/\<currentuser\>/.azure/testcredentials.json`) that will be used to set the environment variable when scenario tests are run. This credentials file will be used in all future sessions unless it is deleted or the environment variables are manually set. This cmdlet is not currently available for .NET SDK development.
54
+
This cmdletwill allow you to create a credentials file (located in `C:/Users/<currentuser>/.Azure/testcredentials.json`) that will be used to set the connection string when scenario tests are run. This credentials file will be used in all future sessions unless it is deleted.
49
55
50
56
#### Create New Service Principal
51
57
52
-
Using a service principal is the preferred option for recording tests because it works with both .NET Framework and .NET Core. In order to create a new service principal, run this command with an unused service principal display name:
58
+
Using a service principal is the preferred option for recording tests because it works with both .NET Framework and .NET Core.
59
+
In order to create a new service principal, run the following command with an unused service principal display name:
This command will create a new service principal, set the correct role assignment for this service principal based upon the subscription provided, and place the service principal id and automatically generated secret into the credentials file.
65
+
This command will first create a new service principal. And then set the `Contributor` role assignment for this service principal based upon the subscription provided. After that, it will place the service principal application id and automatically generated secret into the credentials file.
60
66
61
-
Alternatively, to create a service principal, follow the [Azure AD guide to create a Application Service Principal](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#create-an-active-directory-application). The application type should be `Web app / API` and the sign-on URL value is irrelevant (you can set any value).
67
+
If the display name of the service principal already exists, it will prompt if you would like to create a new one with the same name.
68
+
If the answer is "Y", the new generated application id and the secret will be saved.
69
+
70
+
Alternatively, if you prefer creating a service principal by yourself from Azure portal, follow the [Azure AD guide to create a Application Service Principal](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#create-an-active-directory-application).
62
71
63
72
#### Use Existing Service Principal
64
73
65
-
If you would like to use an existing service principal, run this command with an existing service principal display name and secret:
74
+
If you would like to use an existing service principal, run the following command with an existing service principal application id and secret:
66
75
67
76
```powershell
68
-
New-TestCredential -ServicePrincipalDisplayName "Existing Service Principal" -ServicePrincipalSecret `
### Manually Set Environment Variables to Build Connection String
86
85
87
86
#### Environment Variables
88
87
89
88
`TEST_CSM_ORGID_AUTHENTICATION`
90
-
* This is the connection string that determines how to connect to Azure. This includes both your authentication and the Azure environment to connect to.
91
89
92
-
`AZURE_TEST_MODE`
93
-
* This specifies whether the test framework will `Record` test sessions or `Playback` previously recorded test sessions.
90
+
* This determines how to connect to Azure. It includes both your authentication and the Azure environment information.
94
91
95
-
#### Playback Test
96
-
97
-
The default test mode is `Playback` mode, so setting up the connection string is not required. You can optionally set environment variables:
92
+
`AZURE_TEST_MODE`
98
93
99
-
```
100
-
TEST_CSM_ORGID_AUTHENTICATION=
101
-
AZURE_TEST_MODE=Playback
102
-
```
94
+
* This specifies whether the test framework will `Record` test sessions or `Playback` previously recorded test sessions.
103
95
104
-
#### Record Test with service principal
96
+
#### Record Tests
105
97
106
-
After the service principal is created, you will need to give it access to Azure resources. This can be done with the following PowerShell command, with the [Service Principal Application ID](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key) (this is a guid, not the display name of the service principal) substituted in for `{clientId}`.
98
+
After the service principal is created, you will need to give it access to Azure resources. This can be done with the following PowerShell command. The argument for this command is the application id (See [Service Principal Application ID](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key))
To use this option, set the following environment variable before starting Visual Studio. The following values are substituted into the below connection string:
104
+
To use this option, set the following environment variable before starting Visual Studio. The following values are substituted into the below environment variable:
105
+
106
+
`ClientId`
113
107
114
-
`clientId`
115
108
* The [Service Principal Application ID](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key)
116
109
117
-
`clientSecret`
118
-
* A [Service Principal Authentication Key](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key)
110
+
`ClientSecret`
119
111
120
-
`tenantId`
121
-
* The [AAD Tenant ID](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-tenant-id)
112
+
* The [Service Principal Authentication Key](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key)
Opting for config file is the recommended way to build connection string because any changes you make will take effect immediately without having to restart Visual Studio. However, updating the environment variables is different. It requires rebooting Visual Studio before it can read the updated values. So following is the steps how Test Framework detects the settings.
131
+
132
+
- If JSON config file exists
133
+
- It will be used to build the connection string. Anything set in the environment variables will be ignored
134
+
- If JSON config file does not exist
135
+
- Test framework will first retrieve the environment variable `TEST_CSM_ORGID_AUTHENTICATION` and use its value to build the connection string except for the test mode (Record/Playback).
136
+
- Then test framework will try to get the value of the environment variable `AZURE_TEST_MODE`
137
+
- If `AZURE_TEST_MODE` is set, its value will be used as the test mode
138
+
- Otherwise, the property named `HttpRecorderMode` configured in `TEST_CSM_ORGID_AUTHENTICATION` will be used
139
+
- If the property `HttpRecorderMode` is also not set, `Playback` will be applied as the default value
140
+
141
+
If you are not sure the settings on your machine, please run command `Get-TestFxEnvironment`. It will consolidate above steps and display the ultimate result.
142
+
128
143
## Record or Playback Tests
129
144
130
145
-[Run the tests](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/azure-powershell-developer-guide.md#recordingrunning-tests) and make sure that you got a generated `.json` file that matches the test name under the `SessionRecords` folder in the test project.
131
-
- To assure that the records work fine, delete the connection string (default mode is Playback mode) OR change HttpRecorderMode within the connection string to "Playback" and run the tests
146
+
- If you want to switch from Record to Playback or from Playback to Record, consider below steps.
147
+
- If you opt for JSON config file, update the value of the property `HttpRecorderMode` in the JSON.
148
+
- If you opt for environment variables
149
+
- If you have `AZURE_TEST_MODE` set, update the value of this variable
150
+
- Otherwise, update the value of the property `HttpRecorderMode` defined in the variable `TEST_CSM_ORGID_AUTHENTICATION`
132
151
133
152
## Change Test Environment settings at run-time
134
153
@@ -146,6 +165,7 @@ TestEnvironment.Endpoints.GraphUri = new Uri("https://newGraphUri.windows.net");
146
165
```
147
166
148
167
### Note:
168
+
149
169
Changing the above properties at run-time has the potential to hard code few things in your tests. Best practice would be to use these properties to change values at run-time from immediate window at run-time and avoid hard-coding certain values.
150
170
151
171
## Troubleshooting
@@ -160,21 +180,21 @@ Ensure that the `HttpRecorderMode` in the `TEST_CSM_ORGID_AUTHENTICATION` enviro
0 commit comments