[Docs] Add info about test proxy sanitizer opt-out (#35579)

Added some documentation updates regarding test proxy sanitization.

Signed-off-by: Paul Van Eck <[email protected]>
Co-authored-by: McCoy Patiño <[email protected]>

Author: pvaneck

doc/dev/test_proxy_troubleshooting.md

@@ -144,6 +144,43 @@ To match requests for query parameter content instead of exact ordering, you can
 `ignore_query_ordering=True`. Calling this method inside the body of a test function will update the matcher for only
 that test, which is recommended.
 
+### Sanitization impacting request URL/body/headers
+
+In some cases, a value in a response body is used in the following request as part of the URL, body, or headers. If this value is sanitized, the recorded request might differ than what is expected during playback. Common culprits include sanitization of "name", "id", and "Location" fields. To resolve this, you can either opt out of specific sanitization or add another sanitizer to align with the sanitized value.
+
+#### Opt out
+
+You can opt out of sanitization for the fields that are used for your requests by calling the `remove_batch_sanitizer` method from `devtools_testutils` with the [sanitizer IDs][test_proxy_sanitizers] to exclude. Generally, this is done in the `conftest.py` file, in the one of the session-scoped fixtures. Example:
+
+```python
+from devtools_testutils import remove_batch_sanitizers, test_proxy
+
+
+@pytest.fixture(scope="session", autouse=True)
+def add_sanitizers(test_proxy):
+    ...
+    #  Remove the following body key sanitizer: AZSDK3493: $..name
+    remove_batch_sanitizers(["AZSDK3493"])
+```
+
+Some sanitizer IDs that are often opted out of are:
+  - `AZSDK2003`: `Location` - Header regex sanitizer
+  - `AZSDK3430`: `$..id` - Body key sanitizer
+  - `AZSDK3493`: `$..name` - Body key sanitizer
+
+However, **please be mindful when opting out of a sanitizer, and ensure that no sensitive data is being exposed**.
+
+#### Add another sanitizer
+
+Alternatively, you can add another sanitizer to align the recorded request with the expected request, modifying the URL, body, or headers as needed. Example:
+
+```python
+from devtools_testutils import add_uri_regex_sanitizer
+
+
+add_uri_regex_sanitizer(regex="(?<=https://.+/foo/bar/)(?<id>[^/?\\.]+)", group_for_replace="id", value="Sanitized")
+```
+
 ## Recordings not being produced
 
 Ensure the environment variable `AZURE_SKIP_LIVE_RECORDING` **isn't** set to "true", and that `AZURE_TEST_RUN_LIVE`
@@ -245,4 +282,5 @@ chmod +x .../azure-sdk-for-python/.proxy/Azure.Sdk.Tools.TestProxy
 [pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery
 [pytest_commands]: https://docs.pytest.org/latest/usage.html
 [record_request_failure]: https://github.com/Azure/azure-sdk-for-python/blob/e23d9a6b1edcc1127ded40b9993029495b4ad08c/tools/azure-sdk-tools/devtools_testutils/proxy_testcase.py#L97
+[test_proxy_sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/57382d5dc00b10a2f9cfd597293eeee0c2dbd8fd/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/SanitizerDictionary.cs#L65
 [wrong_exception]: https://github.com/Azure/azure-sdk-tools/issues/2907

doc/dev/tests.md

@@ -24,6 +24,7 @@ testing infrastructure, and demonstrates how to write and run tests for a servic
     - [Update test recordings](#update-test-recordings)
     - [Sanitize secrets](#sanitize-secrets)
       - [Special case: SAS tokens](#special-case-sas-tokens)
+      - [Opt out of specific sanitizers](#opt-out-of-specific-sanitizers)
   - [Functional vs. unit tests](#functional-vs-unit-tests)
   - [Further reading](#further-reading)
 
@@ -163,7 +164,7 @@ the
 
 ## Write or run tests
 
-Newer SDK tests utilize the [Azure SDK Tools Test Proxy][proxy_general_docs] to record and playback HTTP interactions.
+Newer SDK tests utilize the [Azure SDK Tools Test Proxy][proxy_general_docs] to record and playback HTTP interactions, while also automatically sanitizing sensitive information from recordings.
 To migrate an existing test suite to use the test proxy, or to learn more about using the test proxy, refer to the
 [test proxy migration guide][proxy_migration_guide].
 
@@ -356,7 +357,7 @@ set for test resources and authentication. With the `AZURE_TEST_RUN_LIVE` enviro
 This will generate test recordings and enable playback testing. Set `AZURE_TEST_RUN_LIVE` to "false" and run tests with
 the same command to verify that playback tests pass.
 
-Playback test errors most frequently indicate a need for additional sanitizers and/or matchers (see
+Playback test errors most frequently indicate a need to add/remove sanitizers and/or add matchers (see
 [Sanitize secrets](#sanitize-secrets)). If you encounter any unexpected errors, refer to the
 [test proxy troubleshooting guide][troubleshooting_guide].
 
@@ -437,11 +438,14 @@ The `.json` files created from running tests in live mode can include authorizat
 access signatures, and other secrets. The recordings are included in our public GitHub repository, making it important
 for us to remove any secrets from these recordings before committing them to the repository.
 
+By default, the test proxy server sanitizes several [common patterns][test_proxy_sanitizers] of secrets, but there are additional
+steps you can take to ensure that any other sensitive information is removed from recordings.
+
 There are two primary ways to keep secrets from being written into recordings:
 
 1. The `EnvironmentVariableLoader` will automatically sanitize the values of captured environment variables with the
    provided fake values.
-2. Sanitizers can be registered via `add_*_sanitizer` methods in `devtools_testutils`. For example, the general-use
+2. Additional sanitizers can be registered via `add_*_sanitizer` methods in `devtools_testutils`. For example, the general-use
    method for sanitizing recording bodies, headers, and URIs is `add_general_string_sanitizer`. Other sanitizers are
    available for more specific scenarios and can be found at [devtools_testutils/sanitizers.py][py_sanitizers].
 
@@ -480,15 +484,29 @@ For more details about sanitizers and their options, please refer to [devtools_t
 
 #### Special case: SAS tokens
 
-Tests that use a Shared Access Signature (SAS) token to authenticate a client should use the
-[`AzureRecordedTestCase.generate_sas`][generate_sas] method to generate the token. This will automatically register a
-sanitizer to keep this token out of test recordings. An example of using this method can be found
-[here][generate_sas_example].
+In the past, it was recommended that the tests using Shared Access Signature (SAS) tokens should use the `AzureRecordedTestCase.generate_sas` method to generate the token and automatically register a sanitizer to keep this token out of test recordings. This method is now deprecated since the test proxy automatically sanitizes SAS tokens. If you have tests that use SAS tokens, you can remove the usage of the `generate_sas` method.
+
+#### Opt out of specific sanitizers
+
+Since, in some cases, the default sanitizers might be considered too aggressive and breaks tests during playback, you can opt out of certain sanitizers using the `remove_batch_sanitizers` function in your respective `conftest.py` files. For example:
+
+```python
+from devtools_testutils import remove_batch_sanitizers, test_proxy
+
+
+@pytest.fixture(scope="session", autouse=True)
+def add_sanitizers(test_proxy):
+    ...
+    #  Remove the following body key sanitizer: AZSDK3493: $..name
+    remove_batch_sanitizers(["AZSDK3493"])
+```
+
+A list of sanitizers and their IDs can be found [here][test_proxy_sanitizers]. However, **please be mindful when opting out of a sanitizer, and ensure that no sensitive data is being exposed**.
 
-`generate_sas` accepts any number of positional arguments: the first being the method that creates the SAS, and the
-remaining positional arguments being positional arguments for the SAS-generating method. Any keyword arguments given to
-`generate_sas` will be passed to the SAS-generating method as well. The generated token will be returned and its value
-will be sanitized.
+Some sanitizers IDs that are often opted out of are:
+  - `AZSDK2003`: `Location` - Header regex sanitizer
+  - `AZSDK3430`: `$..id` - Body key sanitizer
+  - `AZSDK3493`: `$..name` - Body key sanitizer
 
 ## Functional vs. unit tests
 
@@ -561,7 +579,6 @@ For information about more advanced testing scenarios, refer to the [advanced te
 [env_var_docs]: https://github.com/Azure/azure-sdk-for-python/tree/main/tools/azure-sdk-tools/devtools_testutils#use-the-environmentvariableloader
 [env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
 [generate_sas]: https://github.com/Azure/azure-sdk-for-python/blob/bf4749babb363e2dc972775f4408036e31f361b4/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L196
-[generate_sas_example]: https://github.com/Azure/azure-sdk-for-python/blob/3e3fbe818eb3c80ffdf6f9f1a86affd7e879b6ce/sdk/tables/azure-data-tables/tests/test_table_entity.py#L1691
 [get_credential]: https://github.com/Azure/azure-sdk-for-python/blob/20cf5b0bd9b87f90bd5ad4fd36358d3b257f95c5/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L96
 [git_setup]: https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup
 [kv_test_resources]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/keyvault/test-resources.json
@@ -576,6 +593,7 @@ For information about more advanced testing scenarios, refer to the [advanced te
 [pytest_logging]: https://docs.pytest.org/en/stable/logging.html
 [python-dotenv_readme]: https://github.com/theskumar/python-dotenv
 [recording_move]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/recording_migration_guide.md
+[test_proxy_sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/57382d5dc00b10a2f9cfd597293eeee0c2dbd8fd/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/SanitizerDictionary.cs#L65
 [test_proxy_startup]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_migration_guide.md#start-the-proxy-server
 [test_resources]: https://github.com/Azure/azure-sdk-for-python/tree/main/eng/common/TestResources#readme
 [troubleshooting_guide]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_troubleshooting.md