You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: test/e2e/Tests/e2e-tests-readme.md
+91-22Lines changed: 91 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,21 +7,34 @@ This document provides instructions on how to use the end-to-end (E2E) test proj
7
7
- PowerShell
8
8
- npm/Node
9
9
- .NET SDK
10
+
- Maven
11
+
- Docker (if running against a non-AzureStorage backend)
12
+
13
+
## About the tests
14
+
15
+
The goal of this project is to run a set of standardized behavior checks against real FunctionApps using Core Tools to validate key scenarios and guard against breaking changes. This is accomplished by defining one app per Durable Functions language in the /test/e2e/apps folder, which defines a set of standardized functions. The pipeline for these tests will run against each app in sequence using a variety of supported Durable storage backends (Azure Storage, MSSQL, DTS).
10
16
11
17
## Running the E2E Tests
12
18
13
19
### Step 1: Increment the host and worker package versions (optional)
14
20
15
-
Note: This step is optional. However, if you do not perform this step, the versions of these two packages in your local NuGet cache will be replaced with the build output from the test run, which may lead to unexpected behavior debugging live versions in other apps. Be warned.
21
+
Note: This step is optional. However, if you do not perform this step, the versions of these two packages in your local NuGet cache will be replaced with the build output from the test run, which may lead to unexpected behavior debugging live versions in other apps. Be warned.
### Step 2: Configure the test project for the language/backend you are interested in
32
+
33
+
With no other changes, the test project will run against dotnet-isolated with the Azure Storage backend. If you need to run the tests against a different language or backend, you can use test.runsettings to accomplish this when debugging from Visual Studio. Edit the E2ETests.csproj file and uncomment the RunSettingsFilePath line. Next, go to the test.runsettings file found at the root of the test project and make modifications as necessary.
34
+
35
+
NOTE: If running through the commandline using `dotnet test`, you will need to define the environment variables from test.runsettings manually.
36
+
37
+
### Step 3: Build the E2E Test Project
25
38
26
39
To build the E2E test project, run the following PowerShell script:
27
40
@@ -30,30 +43,86 @@ To build the E2E test project, run the following PowerShell script:
30
43
```
31
44
32
45
This script prepares your system for running the E2E tests by performing the following steps:
33
-
1. Installing a copy of Core Tools into your system's temp directory to ensure an unmodified Core Tools. This is necessary, as the tests will not attempt to use the "func" referenced in PATH
34
-
2. Ensure the test app(s) are running the correct extension code by:
35
-
* Building the host and worker extensions from their projects within this repo
36
-
* Packing the extensions into local NuGet packages
37
-
* Copying the built packages into the test app's local nuget source folder as configured in nuget.config
38
-
* Updating the test app's .csproj files to reference the local package version
39
-
* Building the test app projects
40
-
3. Install and start azurite emulator using Node
41
46
42
-
NOTE: It should not be necessary to run start-emulators.ps1 manually, as it should be called by the build script. If you have a instance of Azurite already running, it will recognize and skip this step.
47
+
1. Installing a copy of Core Tools into your system's temp directory to ensure an unmodified Core Tools. This is necessary, as the tests will not attempt to use the "func" referenced in PATH
48
+
2. Ensure the test app(s) are running the correct extension code by:
49
+
- Building the host and worker extensions from their projects within this repo
50
+
- Packing the extensions into local NuGet packages
51
+
- Copying the built packages into the test app's local nuget source folder as configured in nuget.config
52
+
- Updating the test app's .csproj files to reference the local package version
53
+
- Building the test app projects
54
+
3. Install and start azurite emulator using npm
55
+
56
+
NOTE: It should not be necessary to run start-emulators.ps1 manually, as it should be called by the build script. If you have a instance of Azurite already running, it will automatically skip this step.
57
+
58
+
NOTE: The build script will make changes to the extensions.csproj file in all non-dotnet language apps. Do not commit these changes.
59
+
60
+
#### Other backends
61
+
62
+
build-e2e-test.ps1 includes several flags to assist setup for other storage backends.
63
+
64
+
For MSSQL, you can use `-StartMSSqlContainer` to spin up a MSSQL docker container. Define the sa password you will use either in the MSSQL_SA_PASSWORD environment variable or with the `-MSSQLpwd` script argument.
65
+
66
+
For DTS, pass `-StartDTSContainer` flag. This will create a docker container using the DTS emulator image.
67
+
68
+
### Step 4: Build the test project
69
+
70
+
At this point, you are ready to run the tests. You may start them using the Visual Studio test explorer as normal, the tests will take care of instancing Core Tools and starting the apps.
71
+
NOTE: ENSURE AZURITE IS RUNNING. If Azure is not available, the function app loaded by the test framework will 502 and the test suite will loop indefinitely waiting for it to come up. This will be addressed in future versions.
72
+
73
+
### Step 5: Attach a Debugger
74
+
75
+
To debug the extension code while running test functions, you can attach a debugger directly to Core Tools from whichever repository you are interested in:
43
76
44
-
### Step 3: Build the test project
77
+
- For debugging the WebJobs extension in this repo, attach to `func.exe`
78
+
- For debugging Durabletask.Core or DurableTask.AzureStorage, attach to `func.exe`
79
+
- For debugging the Worker extension from this repo for dotnet-isolated, or the functionapp code, attach to `dotnet.exe` that is a child process of `func.exe`
45
80
46
-
At this point, you are ready to run the tests. You may start them using the Visual Studio test explorer as normal, the tests will take care of instancing Core Tools and starting the apps.
47
-
NOTE: ENSURE AZURITE IS RUNNING. If Azure is not available, the function app loaded by the test framework will 502 and the test suite will loop indefinitely waiting for it to come up. This will be addressed in future versions.
81
+
Set a breakpoint on the first line of the test so that the test will start the Core Tools process before ataching.
48
82
49
-
### Step 4: Attach a Debugger
83
+
##Understanding the test code
50
84
51
-
To debug the extension code while running test functions, you need to attach a debugger to the `func` process before the test code runs. Follow these steps:
85
+
### Exclusions
52
86
53
-
1. Open your preferred IDE (e.g., Visual Studio or Visual Studio Code).
54
-
2. Set a breakpoint in the test.
55
-
3. Manually search for and attach the test process. For Out-Of-Process workers, attach func.exe to debug the host extension, and attach the child process representing the worker (dotnet.exe for dotnet OOProc) to debug the worker extension.
87
+
Some tests are incompatible with a certain Durable storage backend, a certain functions language, or a combination of both of these factors. These tests can be skipped using test traits:
88
+
89
+
```csharp
90
+
[Trait("MSSQL", "Skip")] // Skips the test for all languages when running with MSSQL storage backend
91
+
```
92
+
93
+
```csharp
94
+
[Trait("PowerShell", "Skip")] // Skips the test for all storage backends when language is PowerShell
95
+
```
96
+
97
+
```csharp
98
+
[Trait("PowerShell-MSSQL", "Skip")] // Skips the test only when running with both powershell and MSSQL
99
+
```
100
+
101
+
Supported values are `AzureStorage`, `MSSQL` and `DTS` for storage backends, and `Dotnet`, `PowerShell`, `Python`, `Node`, and `Java` for languages. When skipping a specific combo, the order is "\[language]-\[backend]". Multiple tags may be used to skip multiple scenarios.
102
+
103
+
IMPORTANT: Include a comment after each skip tag explaining why it is skipped. If it is due to a bug, link the GitHub issue in the comment (and ideally, note which test is failing on the GitHub issue as well)
104
+
105
+
### Language-specific behavior
106
+
107
+
Every test file will have a `FunctionAppFixture` available. This fixture allows interaction with the functions host from the test perspective, including Core Tools output logs which can be used when verifying test outputs, and an ITestLanguageLocalizer. ITestLanguageLocalizer provides two ways of defining language-specific behavior. The convention for this is as follows:
108
+
109
+
If the behavior is a difference in output formatting, and we don't expect to ever try to get parity between the languages, use GetLocalizedStringValue(). You will need to define a key-value pair in each language's ITestLanguageLocalizer implementation for your string, and then you can call GetLocalizedStringValue(key) in your test. This keeps test code clean and readable.
110
+
111
+
If the behavior is with logic or is something that we need to eventually address in the language worker to improve parity, use GetLanguageType() and if/case statements in the test code instead. This will improve visibility of these kinds of inconsistencies and (hopefully) motivate eventual changes towards parity.
112
+
113
+
### Common issues
114
+
115
+
#### GRPC max size exceeded
116
+
117
+
If you see the following exception while running the tests (specifically, DurableTaskClientWriteOutputTests and ListAllOrchestrations_ShouldSucceed):
118
+
119
+
```text
120
+
Exception: Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size."
121
+
```
56
122
57
-
## Conclusion
123
+
This is due to your TaskHub history being too large to be processed by the app. This can happen in (at least) dotnet-isolated and Java.
124
+
To resolve this, there are several steps, in order of increasing severity:
58
125
59
-
Following these steps will help you set up and run the E2E tests for the Azure Functions Durable Extension project. If you encounter any issues, refer to the project documentation or seek help from the community.
126
+
1. Run the test app manually and call PurgeOrchestrationHistory to delete all past instances.
127
+
2. Use the Azure Storage explorer to connect to your azurite instance and delete the Task Hub manually by deleting the storage queues and tables.
128
+
3. Completely remove the Azurite state and start fresh. This can be done by stopping the azurite process, navigating to the directory where it is running (for the azurite instance started by build-e2e-test.ps1 on Windows this is %LOCALAPPDATA%/Temp/DurableTaskExtensionE2ETests/azurite) and deleting all Azurite files.
0 commit comments