Skip to content

Commit d718fc4

Browse files
authored
Clean up advanced testing guide (#36818)
1 parent 715f417 commit d718fc4

File tree

2 files changed

+113
-139
lines changed

2 files changed

+113
-139
lines changed

doc/dev/tests-advanced.md

Lines changed: 102 additions & 135 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,74 @@
1-
# Setup Python Development Environment - Advanced
2-
In this document we will provide additional information about the test environments:
1+
# Python SDK advanced testing guide
2+
This guide covers advanced testing scenarios for Azure SDK for Python libraries.
33

4-
- [Setup Python Development Environment - Advanced](#setup-python-development-environment---advanced)
5-
- [Test Mixin Classes](#test-mixin-classes)
6-
- [Preparers](#preparers)
7-
- [Examples with Preparers](#examples-with-preparers)
8-
- [Example 2: Basic Preparer Usage with Storage](#example-2-basic-preparer-usage-with-storage)
9-
- [Example 3: Cached Preparer Usage](#example-3-cached-preparer-usage)
10-
- [mgmt\_settings\_real file](#mgmt_settings_real-file)
4+
## Table of contents
115

12-
## Test Mixin Classes
13-
Many of our test suites use a mixin class to reduce re-writing code in multiple test files. For example, in the Tables test suite there is a `_shared` directory containing two of these mixin classes, a [sync one](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/testcase.py) and an [async version](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/asynctestcase.py). These classes will often have ways to create connection strings from an account name and key, formulate the account url, configure logging, or validate service responses. In order for these mixin classes to be used by both the functional and unit tests they should inherit from `object`. For example:
6+
- [Mixin classes](#test-mixin-classes)
7+
- [Pre-test setup](#pre-test-setup)
8+
- [xunit-style setup](#xunit-style-setup)
9+
- [Fixture setup](#fixture-setup)
1410

15-
```python
11+
## Mixin classes
12+
Many of our test suites use a base/mixin class to consolidate shared test logic. Mixin classes can define instance attributes to handle environment variables, make complex assertions, and more. By inheriting from these mixins, test classes can then share this logic throughout multiple files.
13+
14+
For example, in the Tables test suite there is a `_shared` directory containing two of these mixin classes: a [sync version](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/testcase.py) and an [async version](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/asynctestcase.py).
1615

17-
class TablesTestMixin(object):
18-
def connection_string(self, account, key):
19-
return "DefaultEndpointsProtocol=https;AccountName=" + account + ";AccountKey=" + str(key) + ";EndpointSuffix=core.windows.net"
16+
```python
17+
class TableTestCase(object):
2018

2119
def account_url(self, account, endpoint_type):
2220
"""Return an url of storage account.
21+
2322
:param str storage_account: Storage account name
2423
:param str storage_type: The Storage type part of the URL. Should be "table", or "cosmos", etc.
2524
"""
2625
try:
2726
if endpoint_type == "table":
2827
return account.primary_endpoints.table.rstrip("/")
2928
if endpoint_type == "cosmos":
30-
return "https://{}.table.cosmos.azure.com".format(account.name)
31-
else:
32-
raise ValueError("Unknown storage type {}".format(storage_type))
33-
except AttributeError: # Didn't find "primary_endpoints"
29+
cosmos_suffix = os.getenv("TABLES_COSMOS_ENDPOINT_SUFFIX", DEFAULT_COSMOS_ENDPOINT_SUFFIX)
30+
return f"https://{account.name}.table.{cosmos_suffix}"
31+
except AttributeError: # Didn't find "account.primary_endpoints"
3432
if endpoint_type == "table":
35-
return 'https://{}.{}.core.windows.net'.format(account, endpoint_type)
33+
storage_suffix = os.getenv("TABLES_STORAGE_ENDPOINT_SUFFIX", DEFAULT_STORAGE_ENDPOINT_SUFFIX)
34+
return f"https://{account}.table.{storage_suffix}"
3635
if endpoint_type == "cosmos":
37-
return "https://{}.table.cosmos.azure.com".format(account)
38-
39-
def enable_logging(self):
40-
handler = logging.StreamHandler()
41-
handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
42-
self.logger.handlers = [handler]
43-
self.logger.setLevel(logging.INFO)
44-
self.logger.propagate = True
45-
self.logger.disabled = False
36+
cosmos_suffix = os.getenv("TABLES_COSMOS_ENDPOINT_SUFFIX", DEFAULT_COSMOS_ENDPOINT_SUFFIX)
37+
return f"https://{account}.table.{cosmos_suffix}"
38+
39+
...
40+
41+
def _assert_delete_retention_policy_equal(self, policy1, policy2):
42+
"""Assert that two deletion retention policies are equal."""
43+
if policy1 is None or policy2 is None:
44+
assert policy1 == policy2
45+
return
46+
47+
assert policy1.enabled == policy2.enabled
48+
assert policy1.days == policy2.days
49+
50+
...
4651
```
4752

4853
In action this class can be used in functional tests:
4954

5055
```python
51-
class TestTablesFunctional(AzureTestCase, TablesTestMixin):
52-
...
53-
def test_with_mixin(self, account, key):
54-
conn_str = self.connection_string(account, key)
55-
client = TableClient.from_connection_string(conn_str)
56-
client.create_table('first')
57-
client.create_table('second')
58-
tables = 0
59-
for table in client.list_tables():
60-
tables += 1
61-
62-
assert tables == 2
56+
class TestTable(AzureRecordedTestCase, TableTestCase):
57+
@tables_decorator
58+
@recorded_by_proxy
59+
def test_create_properties(self, tables_storage_account_name, tables_primary_storage_account_key):
60+
# # Arrange
61+
account_url = self.account_url(tables_storage_account_name, "table")
62+
ts = TableServiceClient(credential=tables_primary_storage_account_key, endpoint=account_url)
63+
table_name = self._get_table_reference()
64+
# Act
65+
created = ts.create_table(table_name)
66+
...
6367
```
6468

6569
Or can be used in a unit test:
6670
```python
67-
class TestTablesUnit(TablesTestMixin):
71+
class TestTablesUnit(TableTestCase):
6872
...
6973
def test_valid_url(self):
7074
account = "fake_tables_account"
@@ -74,114 +78,77 @@ class TestTablesUnit(TablesTestMixin):
7478
client = TableClient(account_url=url, credential=credential)
7579

7680
assert client is not None
77-
assert client.account_url == "https://{}.tables.core.windows.net/".format(account)
81+
assert client.account_url == f"https://{account}.tables.core.windows.net/"
7882
```
7983

80-
## Preparers
81-
82-
The Azure SDK team has created some in house tools to help with easier testing. These additional tools are located in the `devtools_testutils` package that was installed with your `dev_requirements.txt`. In this package are the preparers that will be commonly used throughout the repository to test various resources. A preparer is a way to programmatically create fresh resources to run our tests against and then deleting them after running a test suite. These help guarantee standardized behavior by starting each test group from a fresh resource and account.
84+
## Pre-test setup
85+
Tests will often use shared resources that make sense to set up before tests execute. There are two recommended
86+
approaches for this kind of setup, with each having benefits and drawbacks.
8387

84-
If this situation is a requirement for your tests, you can opt to create a new preparer for your service from the management plane library for a service. There are already a few preparers built in the [devtools_testutils](https://github.com/Azure/azure-sdk-for-python/tree/main/tools/azure-sdk-tools/devtools_testutils). Most prepares will start with the [`ResourceGroupPreparer`](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py#L29-L99) to first create a resource group for your service.
85-
86-
To build your own preparer you will need to use the management plane library to create a service and pass the credentials you need into your tests. The two important methods for a preparer are the `create_resource` and `remove_resource` methods. In the `create_resource` method you will use the management client to create the resource and return a dictionary of key-value pairs. The keys will be matched with the test method parameters and passed in as positional arguments to the test. The `remove_resource` method will clean up and remove the resource to prevent a backlog of unused resources in your subscription. For examples of each of these methods, check out these examples:
87-
88-
| Preparer | `create_resource` | `remove_resource` |
89-
|-|-|-|
90-
| Resource Group | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py#L57-L85) | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py#L87-L99) |
91-
| Storage Account | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/storage_testcase.py#L53-L102) | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/storage_testcase.py#L104-L107) |
92-
93-
## Examples with Preparers
94-
95-
### Example 2: Basic Preparer Usage with Storage
88+
### xunit-style setup
89+
Pytest has documentation describing this setup style: https://docs.pytest.org/en/latest/how-to/xunit_setup.html. For
90+
example:
9691

9792
```python
98-
import os
99-
import pytest
100-
101-
from azure.data.tables import TableServiceClient
102-
from devtools_testutils import (
103-
AzureTestCase,
104-
ResourceGroupPreparer,
105-
StorageAccountPreparer
106-
)
93+
from devtools_testutils.azure_recorded_testcase import get_credential
94+
95+
class TestService(AzureRecordedTestCase):
96+
def setup_method(self, method):
97+
"""This method is called before each test in the class executes."""
98+
credential = self.get_credential(ServiceClient) # utility from parent class
99+
self.client = ServiceClient("...", credential)
100+
101+
@classmethod
102+
def setup_class(cls):
103+
"""This method is called only once, before any tests execute."""
104+
credential = get_credential() # only module-level and classmethod utilities are available
105+
cls.client = ServiceClient("...", credential)
106+
```
107107

108-
class ExampleStorageTestCase(AzureTestCase):
108+
The primary benefit of using `setup_method` is retaining access to the utilities provided your test class. You could
109+
use `self.get_credential`, for example, to pick up our core utility for selecting a client credential based on your
110+
environment. A drawback is that `setup_method` runs before each test method in the class, so your setup needs to be
111+
idempotent to avoid issues caused by repeated invocations.
109112

110-
@ResourceGroupPreparer()
111-
@StorageAccountPreparer()
112-
def test_create_table(self, resource_group, location, storage_account, storage_account_key):
113-
account_url = self.account_url(storage_account, "table")
114-
client = self.create_client_from_credential(TableServiceClient, storage_account_key, account_url=account_url)
113+
Alternatively, the class-level `setup_class` method runs once before all tests, but doesn't give you access to all
114+
instance attributes on the class. You can still set attributes on the test class to reference from tests, and
115+
module-level utilities can be used in place of instance attributes, as shown in the example above.
115116

116-
valid_table_name = "validtablename"
117-
table = client.create_table(valid_table_name)
117+
### Fixture setup
118+
Pytest has documentation explaining how to implement and use fixtures:
119+
https://docs.pytest.org/en/latest/how-to/fixtures.html. For example, in a library's `conftest.py`:
118120

119-
assert valid_table_name == table.table_name
121+
```python
122+
from devtools_testutils.azure_recorded_testcase import get_credential
123+
124+
@pytest.fixture(scope="session")
125+
def setup_teardown_fixture():
126+
# Note that we can't reference AzureRecordedTestCase.get_credential but can use the module-level function
127+
client = ServiceClient("...", get_credential())
128+
client.set_up_resource()
129+
yield # <-- Tests run here, and execution resumes after they finish
130+
client.tear_down_resources()
120131
```
121132

122-
This test uses preparers to create resources, then creates a table, and finally verifies the name is correct.
123-
124-
Notes:
125-
1. This test is aiming to create a new Table, which requires a storage account, which in hand requires a resource group. The first decorator (`@ResourceGroupPreparer()`) creates a new resource group, and passes the parameters of this resource group into the `@StorageAccountPreparer()` which creates the storage account. The parameters from the storage account creation is passed into the signature of `test_create_table` .
126-
2. The `create_client_from_credential` is used again but this time with `storage_account_key` instead of getting a credential from the `self.get_credential` method showed in the previous section. The storage account preparer returns the key for the account which is a valid credential.
127-
133+
We can then request the fixture from a test class:
128134

129-
### Example 3: Cached Preparer Usage
130135
```python
131-
import os
132-
import pytest
133-
134-
from azure.core.exceptions import ResourceExistsError
135-
from azure.data.tables import TableServiceClient
136-
from devtools_testutils import (
137-
AzureTestCase,
138-
CachedResourceGroupPreparer,
139-
CachedStorageAccountPreparer
140-
)
141-
142-
class ExampleStorageTestCase(AzureTestCase):
143-
144-
@CachedResourceGroupPreparer(name_prefix="storagetest")
145-
@CachedStorageAccountPreparer(name_prefix="storagetest")
146-
def test_create_table(self, resource_group, location, storage_account, storage_account_key):
147-
account_url = self.account_url(storage_account, "table")
148-
client = self.create_client_from_credential(TableServiceClient, storage_account_key, account_url=account_url)
149-
150-
valid_table_name = "validtablename"
151-
table = client.create_table(valid_table_name)
152-
153-
assert valid_table_name == table.table_name
154-
155-
@CachedResourceGroupPreparer(name_prefix="storagetest")
156-
@CachedStorageAccountPreparer(name_prefix="storagetest")
157-
def test_create_table_if_exists (self, resource_group, location, storage_account, storage_account_key):
158-
account_url = self.account_url(storage_account, "table")
159-
client = self.create_client_from_credential(TableServiceClient, storage_account_key, account_url=account_url)
160-
161-
valid_table_name = "validtablename"
162-
with pytest.raises(ResourceExistsError):
163-
table = client.create_table(valid_table_name)
136+
@pytest.mark.usefixtures("setup_teardown_fixture")
137+
class TestService(AzureRecordedTestCase):
138+
...
164139
```
165140

166-
The first test is the same as above, the second test tries to create a table that already exists and asserts that the correct type of error is raised in response. These tests use cached preparers unlike the previous example.
167-
168-
Notes:
169-
1. The cached preparers here will first look to see if an existing resource group or storage account exists with the given parameters, in this case the `name_prefix`. For more information on what parameters differentiate a new resource group or storage account look for the `self.set_cache()` method in the preparer source code [here](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/storage_testcase.py#L49). The advantage to using a cached preparer is the time saver to re-using the same resource instead of creating a new resource for each test. However, this can increase the possibility that you have to be more exact about cleaning up the entities created in between test runs.
141+
By requesting a fixture from the test class, the fixture will execute before any tests in the class do. Fixtures are the
142+
preferred solution from pytest's perspective and offer a great deal of modular functionality.
170143

171-
## mgmt_settings_real file
144+
As shown in the example above, the
145+
[`yield`](https://docs.pytest.org/latest/how-to/fixtures.html#yield-fixtures-recommended) command will defer to test
146+
execution -- after tests finish running, the fixture code after `yield` will execute. This enables the use of a fixture
147+
for both setup and teardown.
172148

173-
A `mgmt_settings_real.py` can be used in place of a `.env` file by copying `sdk/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py` to `sdk/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_real.py` and providing real credentials to it. The following changes need to be made to the `mgmt_settings_real.py` file:
149+
However, fixtures in this context have similar drawbacks to the `setup_class` method described in
150+
[xunit-style setup](#xunit-style-setup). Since their scope is outside of the test class, test class instance utilities
151+
can't be accessed and class state can't be modified.
174152

175-
1. Change the value of the `SUBSCRIPTION_ID` variable to your organizations subscription ID, which can be found in the "Overview" section of the "Subscriptions" blade in the [Azure portal](https://portal.azure.com/).
176-
2. Define `TENANT_ID`, `CLIENT_ID`, and `CLIENT_SECRET`, which are available after creating a Service Principal or can be retrieved from the Azure Portal after creating a Service Principal. Check out the [Azure docs](https://docs.microsoft.com/cli/azure/ad/sp?view=azure-cli-latest#az_ad_sp_create_for_rbac) to create a Service Principal with a simple one line command to create one. The recommended practice is to include your alias or name in the Service Principal name.
177-
3. Change the [`get_azure_core_credentials(**kwargs):`](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py#L39-L53) function in the `mgmt_settings_real.py` file to construct and return a `ClientSecretCredential` object. Pass in the `CLIENT_ID`, `CLIENT_SECRET`, and `TENANT_ID` values to the `ClientSecretCredential` object. This method should look like this:
178-
```python
179-
def get_azure_core_credentials(**kwargs):
180-
from azure.identity import ClientSecretCredential
181-
import os
182-
return ClientSecretCredential(
183-
client_id = CLIENT_ID,
184-
client_secret = CLIENT_SECRET,
185-
tenant_id = TENANT_ID
186-
)
187-
```
153+
By convention, fixtures should be defined in a library's `tests/conftest.py` file. This will provide access to the
154+
fixture across test files, and the fixture can be requested without having to manually import it.

0 commit comments

Comments
 (0)