Skip to content

Commit b6cdb7f

Browse files
mikekistlermikekistler
authored
Add docs on running RESTler and capturing test recordings with RESTler and test-proxy. (#22460)
1 parent 9576bd7 commit b6cdb7f

File tree

2 files changed

+312
-0
lines changed

2 files changed

+312
-0
lines changed

‎documentation/restler/MakeTestProxyRecording.md

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# How to use RESTler and test-proxy to make traffic recording
2+
3+
In this section, we'll show you how to generate a traffic recording with [RESTler][] and the
4+
[Azure SDK Tools Test Proxy][] and save it to the azure-sdk-assets git repository.
5+
For information on how to run RESTler, please go to [QuickStart](./QuickStart.md).
6+
7+
The Azure SDK Tools Test Proxy is a tool that provides out-of-process record/playback capabilities compatible with any language.
8+
It also supports pushing/restoring the API test recordings from a git repository.
9+
Traffic recordings can be used to validate that the REST API definition is consistent with service behavior.
10+
11+
[RESTler]: https://github.com/microsoft/restler-fuzzer
12+
[Azure SDK Tools Test Proxy]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
13+
14+
## Install the test-proxy
15+
16+
See [how to install test-proxy](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#installation)
17+
18+
## Prepare assets.json file
19+
20+
The assets.json file is a configuration file used by the test-proxy to push/restore recording to/from a git repository.
21+
Create an assets.json file with following content:
22+
23+
```json
24+
{
25+
"AssetsRepo": "Azure/azure-sdk-assets",
26+
"AssetsRepoPrefixPath": "",
27+
"TagPrefix": "apitest/<ServiceName>/<package>"
28+
}
29+
```
30+
31+
Take the appConfiguration data-plane API as an example:
32+
- ServiceName="appconfiguration"
33+
- package="dataplane"
34+
35+
So the contents of `assets.json` should look like:
36+
```json
37+
{
38+
"AssetsRepo": "Azure/azure-sdk-assets",
39+
"AssetsRepoPrefixPath": "apitest",
40+
"TagPrefix": "apitest/appconfiguration/dataplane",
41+
}
42+
```
43+
44+
Place this file in the `scenarios/` folder under the API version being tested.
45+
46+
```
47+
specification/appconfiguration/data-plane
48+
├── Microsoft.AppConfiguration
49+
│ └── stable
50+
│ └── 1.0
51+
│ ├── appconfiguration.json
52+
│ ├── examples
53+
│ │ ├── CheckKeyValue.json
54+
│ │ ├── PutLock.json
55+
│ │ └── PutLock_IfMatch.json
56+
│ └── scenarios
57+
│ └── assets.json <----- check-in assets.json here
58+
└── readme.md
59+
```
60+
61+
## Start the test-proxy
62+
63+
Open a new terminal window, cd to root of azure-rest-api-specs, and start test-proxy in this directory
64+
65+
```bash
66+
test-proxy start
67+
```
68+
69+
## Update the RESTler configuration file to use test-proxy
70+
71+
The general instructions for running RESTler can be found [here](./QuickStart.md).
72+
There are some specific instructions for running RESTler with test-proxy.
73+
74+
### Update engineSettings.json
75+
76+
You need to modify the `engineSettings.json` file to use the test-proxy.
77+
78+
```sh
79+
jq '.host = "localhost" | .target_port = 5000 | .no_ssl = true' restlerConfig/engine_settings.json > engine_settings.json
80+
```
81+
82+
### Update the Restler custom dictionary with test-proxy headers
83+
84+
The test-proxy requires custom headers to be sent with each request that tell it how to handle the request.
85+
One of these headers specifies the "recording id", which is obtained by sending a request to the test-proxy to start a new recording session.
86+
87+
To start a recording session, send a `/Record/Start` request to the test-proxy.
88+
The request should specify the name of the recording file and the path to the `assets.json` file.
89+
The recording file will be created in the `.assets` folder in the root of the azure-rest-api-specs repo,
90+
but must have a unique name to avoid conflicts with other recordings.
91+
Here we make the name unique by prepending the same path as the `assets.json` file.
92+
The response of this request will contain an "x-recording-id" header. Save the value in `recording_id` variable.
93+
94+
```sh
95+
scenarios=specifications/appconfiguration/data-plane/Microsoft.AppConfiguration/stable/1.0/scenarios
96+
body='{
97+
"x-recording-file": "'${scenarios}'/recording.json",
98+
"x-recording-assets-file": "'${scenarios}'/assets.json"
99+
}'
100+
recording_id=$(curl -X POST -s -D - -d ${body} http://localhost:5000/Record/Start | grep 'x-recording-id' | awk '{print $2}' | sed 's/\r$//')
101+
```
102+
103+
Now you can update the custom dictionary with the recording id and [the test-proxy headers](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#run-your-tests):
104+
- `x-recording-upstream-base-uri`: the base URI of the upstream service
105+
- `x-recording-id`: the recording id obtained from the test-proxy
106+
- `x-recording-mode`: the recording mode, which should be `record` for recording
107+
108+
```sh
109+
headers='{
110+
"x-recording-upstream-base-uri": ["https://management.azure.com"],
111+
"x-recording-id": ["'${recording_id}'"],
112+
"x-recording-mode": ["record"]
113+
}'
114+
jq ".restler_custom_payload_header = $headers" dict.json > tmp.json && mv tmp.json dict.json
115+
```
116+
117+
### Recompile the configuration
118+
119+
Now recompile the configuration to incorporate these changes.
120+
121+
```sh
122+
dotnet $restler_bin/restler/Restler.dll compile config.json
123+
```
124+
125+
## Run the tests
126+
127+
Run the tests as describe in [QuickStart](./QuickStart.md#run-the-tests).
128+
129+
## Stop the recording
130+
131+
Once the tests have completed, stop the recording session by sending a `/Record/Stop` request to the test-proxy.
132+
```
133+
curl -X POST -D - -H "x-recording-id: ${recording_id}" -H 'Content-Length: 0' http://localhost:5000/Record/Stop
134+
```
135+
136+
Return to the terminal window where the test-proxy is running and press `Ctrl+C` to stop the test-proxy.
137+
138+
The recording file can be found in the `.assets` folder.
139+
140+
```sh
141+
find .assets -name recording.json
142+
```
143+
144+
## Upload the Recording
145+
146+
Check the recording file to ensure that all secrets have been removed.
147+
If any secrets are found, see the [Test Proxy documentation](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#session-and-test-level-transforms-sanitizers-and-matchers)
148+
for information on how to construct a custom sanitizer.
149+
150+
After ensuring that all secrets have been removed, push the recording to the azure-sdk-assets git repository.
151+
From the root of the azure-rest-api-specs repo, run the command:
152+
153+
```bash
154+
test-proxy push -a <path to assets.json>
155+
```
156+
157+
## Add assets.json for recording to azure-rest-api-specs
158+
159+
After pushing the recording, the test-proxy should write the latest git tag to assets.json file.
160+
But check the assets.json file and if the "Tag" property is not set, set it to the git tag from the output of the push command.
161+
Then commit the updated assets.json file and submit a PR to the azure-rest-api-specs repo.
162+
163+
```sh
164+
git checkout -b apitest-appconfiguration-assets
165+
git add <path to assets.json>
166+
git commit -m "Add assets.json file for appconfiguration data-plane test recording."
167+
git push origin HEAD
168+
```
169+
170+
## Update the recording
171+
172+
If you need to update the recording ...
173+
174+
## Playback recording
175+
176+
TBA

‎documentation/restler/QuickStart.md

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# Quick start with RESTler API testing tool
2+
3+
## Prerequisites
4+
5+
Install [Python 3.8.2][] and [.NET 6.0][] or higher, for your appropriate OS.
6+
7+
[Python 3.8.2]: https://www.python.org/downloads/
8+
[.NET 6.0]:https://dotnet.microsoft.com/download/dotnet-core?utm_source=getdotnetcorecli&utm_medium=referral
9+
10+
## Install
11+
12+
Here are the steps to build and install RESTler locally.
13+
See the [RESTler project readme][] for other installation options.
14+
15+
[RESTler project readme]: https://github.com/microsoft/restler-fuzzer/blob/main/README.md
16+
17+
```sh
18+
git clone https://github.com/microsoft/restler-fuzzer.git
19+
restler_bin=~/bin/restler
20+
mkdir -p $restler_bin
21+
# Need --python because it can’t find “python” — I use an alias that it did not understand.
22+
python ./build-restler.py --dest_dir $restler_bin --python_path /opt/homebrew/bin/python3
23+
```
24+
25+
## Create the initial Restler config and dict files
26+
27+
You need to create a Restler config file and a dict file for each API you want to test. You can use RESTler to create the initial files for you, with the `generateConfig` command. You need to specify the API spec file(s) -- here we use the Azure Databricks service as an example.
28+
29+
```sh
30+
specs=$(find /Users/mikekistler/Projects/Azure/azure-rest-api-specs/specification/databricks/resource-manager/Microsoft.Databricks/preview/2022-04-01-preview -type f -depth 1)
31+
dotnet $restler_bin/restler/Restler.dll generate_config --specs ${=specs}
32+
```
33+
34+
This creates a new directory `restlerConfig` with the following files:
35+
- `annotations.json` - an empty file where you can add annotations for the API spec
36+
- `config.json` - the Restler config file
37+
- `dict.json` - a Restler dictionary file
38+
- `engine_settings.json` - the Restler engine settings file
39+
40+
## Customize the Restler configuration files
41+
42+
You often need to specify values for variables in the API definition.
43+
You can do this by adding these values in `restler_custom_payload` property in the `dict.json` file.
44+
45+
For ARM APIs you generally need to specify the subscriptionId, resourceGroupName, and location.
46+
You may also want to specify the api-version, in case an operation does not have an example value for it.
47+
You can use the `jq` command to add these values. Here is an example:
48+
49+
```sh
50+
vars='{
51+
"subscriptionId": ["my-subscription-id"],
52+
"resourceGroupName": ["my-resource-group"],
53+
"location": ["my-location"],
54+
"api-version": ["2022-04-01-preview"]
55+
}'
56+
jq ".restler_custom_payload = $vars" restlerConfig/dict.json > dict.json
57+
```
58+
59+
No changes are needed to `config.json`, `annotations.json`, or `engine_settings.json`, so you can just copy them from the `restlerConfig` directory.
60+
61+
```sh
62+
cp restlerConfig/config.json .
63+
cp restlerConfig/annotations.json .
64+
cp restlerConfig/engine_settings.json .
65+
```
66+
67+
## Compile tne new configuration
68+
69+
```sh
70+
dotnet $restler_bin/restler/Restler.dll compile config.json
71+
```
72+
73+
This creates a new directory `Compile` with many files.
74+
75+
## Set up the authentication script
76+
77+
One last piece to put in place is the script that will supply RESTler with the authentication token.
78+
For Azure services that accept AAD tokens, you can use the following script (called `getToken.sh`),
79+
which uses the [Azure CLI][] to get the token.
80+
81+
```sh
82+
#!/bin/bash
83+
84+
find . -name 'token.json' -depth 1 -mtime -1h | grep . &> /dev/null || az account get-access-token > token.json
85+
86+
token=$(jq -r '.accessToken' token.json)
87+
88+
echo "{'user1':{}}"
89+
echo "Authorization: bearer ${token}"
90+
```
91+
92+
You might need to modify this script to specify a specific AAD scope on the get-access-token command.
93+
94+
Don't forget to make the script executable.
95+
96+
## Run the tests
97+
98+
Now you are ready to run the Test phase of Restler.
99+
100+
```sh
101+
dotnet $restler_bin/restler/Restler.dll test --dictionary_file Compile/dict.json --grammar_file Compile/grammar.py --settings Compile/engine_settings.json --token_refresh_command "bash $PWD/getToken.sh" --token_refresh_interval 60
102+
```
103+
104+
In most cases the tests will take no more than a few minutes to run and display results that look like:
105+
106+
```text
107+
Starting task Test...
108+
Using python: 'python3' (Python 3.10.9)
109+
Request coverage (successful / total): 9 / 24
110+
Attempted requests: 10 / 24
111+
No bugs were found.
112+
See 'coverage_failures_to_investigate.txt' to investigate API coverage.
113+
Task Test succeeded.
114+
Collecting logs...
115+
```
116+
117+
The time required to run the tests depends on the number and operations and complexity of the dependency graph,
118+
and for a very large service API it may be several hours.
119+
120+
## Analyze the results
121+
122+
If any tests failed or were not attempted, you will need to dig into the test results to understand why.
123+
124+
Some common reasons for failures are:
125+
- The request body may contain values that must be customized for your environment,
126+
for example a `tenantId` or a reference to another resource in your environment.
127+
128+
If there are operations that were not attempted, that happens when RESTler has determined that these operations
129+
depend on some other operation, perhaps indirectly, that failed.
130+
RESTler produces a file called `coverage_failures_to_investigate.txt` that lists the failed operations and which
131+
dependent operations were not attempted because of this.
132+
133+
## Make fixes and re-run
134+
135+
Once you have found the cause of the failures, you can make changes to the config files and re-run the tests.
136+
However, be sure to re-compile the config files before re-running the tests -- it's easy to forget this step.

0 commit comments

Comments
 (0)