Skip to content

Commit a97069d

Browse files
authored
[Core] add perf tests (#33687)
* add upload/download perf tests * add stream download/pageable perf tests * update readme + rename test files * start adding json (tables) request to core perf tests * add tables put json * add more json request tests * cleanup readme + code * add tables to perf reqs * add policies arg * add perf pipeline * address annas comments * address more comments * add results-per-page option for list entities pageable perf test * add aad auth arg * address comments * more comments * fix upload binary test * update perf pipeline * comments * test setting processes to 1 * fix resourcenotfound error * update perf yml * update regex of args * remove debug logging * add len to random stream to fix requests content length bug * add requests transport to perf tests yml * black * fix mypy error in samples * update perf timeout * fix aiohttp sync upload stream bug
1 parent 02138b6 commit a97069d

17 files changed

+1054
-4
lines changed

sdk/core/azure-core/dev_requirements.txt

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,6 @@ opencensus-ext-threading
99
-e ../../../tools/azure-sdk-tools
1010
-e ../../../tools/azure-devtools
1111
-e tests/testserver_tests/coretestserver
12-
pytest-trio
12+
pytest-trio
13+
azure-storage-blob
14+
azure-data-tables

sdk/core/azure-core/samples/example_shared_transport.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
from azure.core.pipeline.transport import RequestsTransport
2424
from azure.storage.blob import BlobServiceClient
2525

26-
connection_string = os.getenv("AZURE_STORAGE_CONNECTION_STRING")
26+
connection_string = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
2727

2828

2929
def shared_transport():

sdk/core/azure-core/samples/example_shared_transport_async.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
from azure.core.pipeline.transport import AioHttpTransport
2525
from azure.storage.blob.aio import BlobServiceClient
2626

27-
connection_string = os.getenv("AZURE_STORAGE_CONNECTION_STRING")
27+
connection_string = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
2828

2929

3030
async def shared_transport_async():
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Core Python client performance tests
2+
3+
In order to run the performance tests, the `devtools_testutils` package must be installed. This is done as part of the `dev_requirements.txt` installation. Start be creating a new virtual environment for your perf tests. This will need to be a Python 3 environment, preferably >=3.7.
4+
5+
### Setup for test resources
6+
The following environment variables will need to be set for the tests to access the live resources:
7+
8+
```
9+
AZURE_STORAGE_CONN_STR=<the connection string to the Storage account>
10+
AZURE_STORAGE_ACCOUNT_NAME=<the Storage account name>
11+
AZURE_STORAGE_ACCOUNT_KEY=<the Storage account key>
12+
13+
AZURE_STORAGE_CONTAINER_NAME=<the container name>
14+
AZURE_STORAGE_BLOBS_ENDPOINT=<The Storage Blobs endpoint in the format 'https://{storageAccountName}.blob.core.windows.net'>
15+
16+
AZURE_STORAGE_TABLE_NAME=<The name to use for the Storage Table>
17+
AZURE_STORAGE_TABLES_ENDPOINT=<The Storage Tables endpoint in the format 'https://{storageAccountName}.table.core.windows.net'>
18+
```
19+
20+
### Setup for perf test runs
21+
22+
```cmd
23+
(env) ~/azure-core> pip install -r dev_requirements.txt
24+
(env) ~/azure-core> pip install .
25+
```
26+
27+
## Test commands
28+
29+
When `devtools_testutils` is installed, you will have access to the `perfstress` command line tool, which will scan the current module for runable perf tests. Only a specific test can be run at a time (i.e. there is no "run all" feature).
30+
31+
```cmd
32+
(env) ~/azure-core> cd tests
33+
(env) ~/azure-core/tests> perfstress
34+
```
35+
36+
Using the `perfstress` command alone will list the available perf tests found.
37+
38+
### Tests
39+
40+
The tests currently available:
41+
42+
- `UploadBinaryDataTest` - Puts binary data of `size` in a Storage Blob (corresponds to the `upload_blob` Blob operation).
43+
- `DownloadBinaryDataTest` - Gets binary data of `size` from a Storage Blob (corresponds to the `download_blob` Blob operation).
44+
- `UpdateEntityJSONTest` - Puts JSON data of `size` in a Storage Table (corresponds to the `update_entity` Tables operation).
45+
- `QueryEntitiesJSONTest` - Gets JSON data of `size` from a Storage Table (corresponds to the `query_entities` Tables operation).
46+
- `ListEntitiesPageableTest` - Gets pageable data from a Storage Table (corresponds to the `list_entities` Tables operation).
47+
48+
### Common perf command line options
49+
50+
The `perfstress` framework has a series of common command line options built in. View them [here](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/perfstress_tests.md#default-command-options).
51+
52+
- `--sync` Whether to run the tests in sync or async. Default is False (async).
53+
- `-d --duration=10` Number of seconds to run as many operations (the "run" function) as possible. Default is 10.
54+
- `-i --iterations=1` Number of test iterations to run. Default is 1.
55+
- `-p --parallel=1` Number of tests to run in parallel. Default is 1.
56+
- `-w --warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
57+
58+
#### Core perf test common command line options
59+
60+
The options that are available for all Core perf tests:
61+
62+
- `--transport` - By default, uses AiohttpTransport ("aiohttp") for async. By default, uses RequestsTransport ("requests") for sync. All options:
63+
- For async:
64+
- `"aiohttp"`: AiohttpTransport (default)
65+
- `"requests"`: AsyncioRequestsTransport
66+
- For sync:
67+
- `"requests"`: RequestsTransport (default)
68+
- `--aad` - Flag to pass in to use Azure Active Directory as the authentication. By default, set to False.
69+
- `--size=10240` - Size of request content (in bytes). Defaults to 10240. (Not used by `ListEntitiesPageableTest`.)
70+
71+
#### Additional ListEntitiesPageableTest command line options
72+
73+
The options that are additionally available for `ListEntitiesPageableTest`:
74+
75+
- `--count=100` - Number of table entities to list. Defaults to 100.
76+
- `--page-size=None` - Maximum number of entities to list per page. Default is None, which will return all possible results per page.
77+
78+
## Example command
79+
80+
```cmd
81+
(env) ~/azure-core> perfstress DownloadBinaryDataTest --aad --transport requests --size=20480 --parallel=2
82+
```

sdk/core/azure-core/tests/perf_tests/__init__.py

Whitespace-only changes.
Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# --------------------------------------------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# Licensed under the MIT License. See License.txt in the project root for license information.
4+
# --------------------------------------------------------------------------------------------
5+
6+
import os
7+
import uuid
8+
import string
9+
import random
10+
11+
from devtools_testutils.perfstress_tests import PerfStressTest
12+
13+
from azure.core import PipelineClient, AsyncPipelineClient
14+
from azure.core.pipeline import Pipeline, AsyncPipeline
15+
from azure.core.pipeline.transport import (
16+
RequestsTransport,
17+
AioHttpTransport,
18+
AsyncioRequestsTransport,
19+
)
20+
from azure.core.pipeline.policies import (
21+
UserAgentPolicy,
22+
HeadersPolicy,
23+
ProxyPolicy,
24+
NetworkTraceLoggingPolicy,
25+
HttpLoggingPolicy,
26+
RetryPolicy,
27+
CustomHookPolicy,
28+
RedirectPolicy,
29+
AsyncRetryPolicy,
30+
AsyncRedirectPolicy,
31+
BearerTokenCredentialPolicy,
32+
AsyncBearerTokenCredentialPolicy,
33+
)
34+
import azure.core.pipeline.policies as policies
35+
from azure.core.credentials import AzureNamedKeyCredential
36+
from azure.core.exceptions import (
37+
ClientAuthenticationError,
38+
ResourceExistsError,
39+
ResourceNotFoundError,
40+
ResourceNotModifiedError,
41+
)
42+
from azure.identity import ClientSecretCredential
43+
from azure.identity.aio import ClientSecretCredential as AsyncClientSecretCredential
44+
from azure.data.tables.aio import TableClient
45+
46+
from azure.storage.blob._shared.authentication import SharedKeyCredentialPolicy as BlobSharedKeyCredentialPolicy
47+
from azure.data.tables._authentication import SharedKeyCredentialPolicy as TableSharedKeyCredentialPolicy
48+
49+
_LETTERS = string.ascii_letters
50+
51+
52+
class _ServiceTest(PerfStressTest):
53+
transport = None
54+
async_transport = None
55+
56+
def __init__(self, arguments):
57+
super().__init__(arguments)
58+
self.account_name = self.get_from_env("AZURE_STORAGE_ACCOUNT_NAME")
59+
self.account_key = self.get_from_env("AZURE_STORAGE_ACCOUNT_KEY")
60+
async_transport_types = {"aiohttp": AioHttpTransport, "requests": AsyncioRequestsTransport}
61+
sync_transport_types = {"requests": RequestsTransport}
62+
self.tenant_id = os.environ["CORE_TENANT_ID"]
63+
self.client_id = os.environ["CORE_CLIENT_ID"]
64+
self.client_secret = os.environ["CORE_CLIENT_SECRET"]
65+
self.storage_scope = "https://storage.azure.com/.default"
66+
67+
# defaults transports
68+
self.sync_transport = RequestsTransport
69+
self.async_transport = AioHttpTransport
70+
71+
# if transport is specified, use that
72+
if self.args.transport:
73+
# if sync, override sync default
74+
if self.args.sync:
75+
try:
76+
self.sync_transport = sync_transport_types[self.args.transport]
77+
except KeyError:
78+
raise ValueError(f"Invalid sync transport:{self.args.transport}\n Valid options are:\n- requests\n")
79+
# if async, override async default
80+
else:
81+
try:
82+
self.async_transport = async_transport_types[self.args.transport]
83+
except KeyError:
84+
raise ValueError(
85+
f"Invalid async transport:{self.args.transport}\n Valid options are:\n- aiohttp\n- requests\n"
86+
)
87+
88+
self.error_map = {
89+
401: ClientAuthenticationError,
90+
404: ResourceNotFoundError,
91+
409: ResourceExistsError,
92+
304: ResourceNotModifiedError,
93+
}
94+
95+
def _build_sync_pipeline_client(self, auth_policy):
96+
default_policies = [
97+
UserAgentPolicy,
98+
HeadersPolicy,
99+
ProxyPolicy,
100+
NetworkTraceLoggingPolicy,
101+
HttpLoggingPolicy,
102+
RetryPolicy,
103+
CustomHookPolicy,
104+
RedirectPolicy,
105+
]
106+
107+
if self.args.policies is None:
108+
# if None, only auth policy is passed in
109+
sync_pipeline = Pipeline(transport=self.sync_transport(), policies=[auth_policy])
110+
elif self.args.policies == "all":
111+
# if all, autorest default policies + auth policy
112+
sync_policies = [auth_policy]
113+
sync_policies.extend([policy(sdk_moniker=self.sdk_moniker) for policy in default_policies])
114+
sync_pipeline = Pipeline(transport=self.sync_transport(), policies=sync_policies)
115+
else:
116+
sync_policies = [auth_policy]
117+
for p in self.args.policies.split(","):
118+
try:
119+
policy = getattr(policies, p)
120+
except AttributeError as exc:
121+
raise ValueError(
122+
f"Azure Core has no policy named {exc.name}. Please use policies from the following list: {policies.__all__}"
123+
) from exc
124+
sync_policies.append(policy(sdk_moniker=self.sdk_moniker))
125+
sync_pipeline = Pipeline(transport=self.sync_transport(), policies=sync_policies)
126+
return PipelineClient(self.account_endpoint, pipeline=sync_pipeline)
127+
128+
def _build_async_pipeline_client(self, auth_policy):
129+
default_policies = [
130+
UserAgentPolicy,
131+
HeadersPolicy,
132+
ProxyPolicy,
133+
NetworkTraceLoggingPolicy,
134+
HttpLoggingPolicy,
135+
AsyncRetryPolicy,
136+
CustomHookPolicy,
137+
AsyncRedirectPolicy,
138+
]
139+
if self.args.policies is None:
140+
# if None, only auth policy is passed in
141+
async_pipeline = AsyncPipeline(transport=self.async_transport(), policies=[auth_policy])
142+
elif self.args.policies == "all":
143+
# if all, autorest default policies + auth policy
144+
async_policies = [auth_policy]
145+
async_policies.extend([policy(sdk_moniker=self.sdk_moniker) for policy in default_policies])
146+
async_pipeline = AsyncPipeline(transport=self.async_transport(), policies=async_policies)
147+
else:
148+
async_policies = [auth_policy]
149+
# if custom list of policies, pass in custom list + auth policy
150+
for p in self.args.policies.split(","):
151+
try:
152+
policy = getattr(policies, p)
153+
except AttributeError as exc:
154+
raise ValueError(
155+
f"Azure Core has no policy named {exc.name}. Please use policies from the following list: {policies.__all__}"
156+
) from exc
157+
async_policies.append(policy(sdk_moniker=self.sdk_moniker))
158+
async_pipeline = AsyncPipeline(transport=self.async_transport(), policies=async_policies)
159+
return AsyncPipelineClient(self.account_endpoint, pipeline=async_pipeline)
160+
161+
def _set_auth_policies(self):
162+
if not self.args.aad:
163+
# if tables, create table credential policy, else blob policy
164+
if "tables" in self.sdk_moniker:
165+
self.sync_auth_policy = TableSharedKeyCredentialPolicy(
166+
AzureNamedKeyCredential(self.account_name, self.account_key)
167+
)
168+
self.async_auth_policy = self.sync_auth_policy
169+
else:
170+
self.sync_auth_policy = BlobSharedKeyCredentialPolicy(self.account_name, self.account_key)
171+
self.async_auth_policy = self.sync_auth_policy
172+
else:
173+
sync_credential = ClientSecretCredential(self.tenant_id, self.client_id, self.client_secret)
174+
self.sync_auth_policy = BearerTokenCredentialPolicy(sync_credential, self.storage_scope)
175+
async_credential = AsyncClientSecretCredential(self.tenant_id, self.client_id, self.client_secret)
176+
self.async_auth_policy = AsyncBearerTokenCredentialPolicy(async_credential, self.storage_scope)
177+
178+
@staticmethod
179+
def add_arguments(parser):
180+
super(_ServiceTest, _ServiceTest).add_arguments(parser)
181+
parser.add_argument(
182+
"--transport",
183+
nargs="?",
184+
type=str,
185+
help="""Underlying HttpTransport type. Defaults to `aiohttp` if async, `requests` if sync. Other possible values for async:\n"""
186+
""" - `requests`\n""",
187+
default=None,
188+
)
189+
parser.add_argument(
190+
"-s", "--size", nargs="?", type=int, help="Size of data to transfer. Default is 10240.", default=10240
191+
)
192+
parser.add_argument(
193+
"--policies",
194+
nargs="?",
195+
type=str,
196+
help="""List of policies to pass in to the pipeline. Options:"""
197+
"""\n- None: No extra policies passed in, except for authentication policy. This is the default."""
198+
"""\n- 'all': All policies added automatically by autorest."""
199+
"""\n- 'policy1,policy2': Comma-separated list of policies, such as 'RetryPolicy,HttpLoggingPolicy'""",
200+
default=None,
201+
)
202+
parser.add_argument("--aad", action="store_true", help="Use AAD authentication instead of shared key.")
203+
204+
205+
class _BlobTest(_ServiceTest):
206+
container_name = "perfstress-" + str(uuid.uuid4())
207+
208+
def __init__(self, arguments):
209+
super().__init__(arguments)
210+
self.account_endpoint = self.get_from_env("AZURE_STORAGE_BLOBS_ENDPOINT")
211+
self.container_name = self.get_from_env("AZURE_STORAGE_CONTAINER_NAME")
212+
self.api_version = "2021-12-02"
213+
self.sdk_moniker = f"storage-blob/{self.api_version}"
214+
215+
self._set_auth_policies()
216+
self.pipeline_client = self._build_sync_pipeline_client(self.sync_auth_policy)
217+
self.async_pipeline_client = self._build_async_pipeline_client(self.async_auth_policy)
218+
219+
async def close(self):
220+
self.pipeline_client.close()
221+
await self.async_pipeline_client.close()
222+
await super().close()
223+
224+
225+
class _TableTest(_ServiceTest):
226+
table_name = "".join(random.choice(_LETTERS) for i in range(30))
227+
228+
def __init__(self, arguments):
229+
super().__init__(arguments)
230+
self.account_endpoint = self.get_from_env("AZURE_STORAGE_TABLES_ENDPOINT")
231+
self.api_version = "2019-02-02"
232+
self.data_service_version = "3.0"
233+
self.sdk_moniker = f"tables/{self.api_version}"
234+
self._set_auth_policies()
235+
236+
self.pipeline_client = self._build_sync_pipeline_client(self.sync_auth_policy)
237+
self.async_pipeline_client = self._build_async_pipeline_client(self.async_auth_policy)
238+
239+
self.connection_string = self.get_from_env("AZURE_STORAGE_CONN_STR")
240+
self.async_table_client = TableClient.from_connection_string(self.connection_string, self.table_name)
241+
242+
async def global_setup(self):
243+
await super().global_setup()
244+
await self.async_table_client.create_table()
245+
246+
async def global_cleanup(self):
247+
await self.async_table_client.delete_table()
248+
249+
def get_base_entity(self, pk, rk, size):
250+
# 227 is the length of the entity with Data of length 0
251+
base_entity_length = 227
252+
data_length = max(size - base_entity_length, 0)
253+
# size = 227 + data_length
254+
return {
255+
"PartitionKey": pk,
256+
"RowKey": rk,
257+
"Data": "a" * data_length,
258+
}
259+
260+
def get_entity(self, rk=0):
261+
return {"PartitionKey": "pk", "RowKey": str(rk), "Property1": f"a{rk}", "Property2": f"b{rk}"}
262+
263+
async def close(self):
264+
self.pipeline_client.close()
265+
await self.async_pipeline_client.close()
266+
await super().close()

0 commit comments

Comments
 (0)