Cosmos Diagnostics Logging Filters and optimizations. (Azure#39897)

* Updates on Cosmos Diagnostics

Adds the ability to use logging.Filters on diagnostics logs based on the same filterable parameters as the cosmos diagnostics handler. Also adds some optimizations to reduce time spent on logging diagnostics. Lastly improves the formatting of the diagnostics log to improve readabilty especially when errors are logged.

* Update CHANGELOG.md

* Update CHANGELOG.md

* updates to cosmos diagnostics

* update to logging policy and tests

* pylint updates

* Update README.md

* Remove diagnostics handler

This removes the custom diagnostics handler and instead allows the use of logging filters.

* Update CHANGELOG.md

* Update _cosmos_http_logging_policy.py

* update recommended changes

* updates

Author: bambriz

sdk/cosmos/azure-cosmos/CHANGELOG.md

@@ -3,6 +3,7 @@
 ### 4.10.0b5 (Unreleased)
 
 #### Features Added
+* Added ability to use Filters from Logging module on Diagnostics Logging based on Http request/response related attributes. See [PR 39897](https://github.com/Azure/azure-sdk-for-python/pull/39897)
 
 #### Breaking Changes
 
@@ -11,8 +12,10 @@
 * Fixed health check to check the first write region when it is not specified in the preferred regions. See [PR 40588](https://github.com/Azure/azure-sdk-for-python/pull/40588).
 
 #### Other Changes
+* Optimized Diagnostics Logging by reducing time spent on logging. Logged Errors are more readable and formatted. See [PR 39897](https://github.com/Azure/azure-sdk-for-python/pull/39897)
 * Health checks are now done concurrently and for all regions for async apis. See [PR 40588](https://github.com/Azure/azure-sdk-for-python/pull/40588).
 
+
 ### 4.10.0b4 (2025-04-01)
 
 #### Bugs Fixed

sdk/cosmos/azure-cosmos/README.md

@@ -950,53 +950,39 @@ However, if you desire to use the CosmosHttpLoggingPolicy to obtain additional i
 client = CosmosClient(URL, credential=KEY, enable_diagnostics_logging=True)
 database = client.create_database(DATABASE_NAME, logger=logger)
 ```
-**NOTICE: The Following is a Preview Feature that is subject to significant change.**
-To further customize what gets logged, you can use a  **PREVIEW** diagnostics handler to filter out the logs you don't want to see.
-There are several ways to use the diagnostics handler, those include the following:
-- Using the "CosmosDiagnosticsHandler" class, which has default behaviour that can be modified.
-    **NOTE: The diagnostics handler will only be used if the `enable_diagnostics_logging` argument is passed in at the client constructor.
-      The CosmosDiagnosticsHandler is also a special type of dictionary that is callable and that has preset keys. The values it expects are functions related to it's relevant diagnostic data. (e.g. ```diagnostics_handler["duration"]``` expects a function that takes in an int and returns a boolean as it relates to the duration of an operation to complete).**
-    ```python
-    from azure.cosmos import CosmosClient, CosmosDiagnosticsHandler
-    import logging
-    # Initialize the logger
-    logger = logging.getLogger('azure.cosmos')
-    logger.setLevel(logging.INFO)
-    file_handler = logging.FileHandler('diagnostics1.output')
-    logger.addHandler(file_handler)
-    diagnostics_handler = cosmos_diagnostics_handler.CosmosDiagnosticsHandler()
-    diagnostics_handler["duration"] = lambda x: x > 2000
-    client = CosmosClient(URL, credential=KEY,logger=logger, diagnostics_handler=diagnostics_handler, enable_diagnostics_logging=True)
-    
-    ```
-- Using a dictionary with the relevant functions to filter out the logs you don't want to see.
-    ```python
-    # Initialize the logger
-    logger = logging.getLogger('azure.cosmos')
-    logger.setLevel(logging.INFO)
-    file_handler = logging.FileHandler('diagnostics2.output')
-    logger.addHandler(file_handler)
-    diagnostics_handler = {
-        "duration": lambda x: x > 2000
-    }
-    client = CosmosClient(URL, credential=KEY,logger=logger, diagnostics_handler=diagnostics_handler, enable_diagnostics_logging=True)
-    ```
-- Using a function that will replace the should_log function in the CosmosHttpLoggingPolicy which expects certain paramameters and returns a boolean. **Note: the parameters of the custom should_log must match the parameters of the original should_log function as shown in the sample.**
-  ```python
-  # Custom should_log method
-  def should_log(self, **kwargs):
-      return kwargs.get('duration') and kwargs['duration'] > 2000
-  
-  # Initialize the logger
+**NOTICE: The Following is a Preview Feature.**
+To further customize what gets logged, you can use logger filters to filter out the logs you don't want to see. You are able to filter based on the following attributes in the log record of cosmos diagnostics logs:
+- `status_code`
+- `sub_status_code`
+- `duration`
+- `verb`
+- `database_name`
+- `collection_name`
+- `operation_type`
+- `url`
+- `resource_type`
+- `is_request`
+
+You can take a look at the samples [here][cosmos_diagnostics_filter_sample] or take a quick look at this snippet:
+- Using **filters** from the **logging** library, it is possible to filter the diagnostics logs. Several filterable attributes are made available to the log record of the diagnostics logs when using logging filters.
+```python
+  import logging
+  from azure.cosmos import CosmosClient
   logger = logging.getLogger('azure.cosmos')
   logger.setLevel(logging.INFO)
-  file_handler = logging.FileHandler('diagnostics3.output')
+  file_handler = logging.FileHandler('diagnostics.output')
   logger.addHandler(file_handler)
-  
-  # Initialize the Cosmos client with custom diagnostics handler
-  client = CosmosClient(endpoint, key,logger=logger, diagnostics_handler=should_log, enable_diagnostics_logging=True)
-    ```
-
+  # Create a filter to filter out logs
+  class CustomFilter(logging.Filter):
+    def filter(self, record):
+        ret = (hasattr(record, 'status_code') and record.status_code > 400
+           and not (record.status_code in [404, 409, 412] and getattr(record, 'sub_status_code', None) in [0, None])
+           and hasattr(record, 'duration') and record.duration > 1000)
+        return ret
+  # Add the filter to the logger
+  logger.addFilter(CustomFilter())
+  client = CosmosClient(endpoint, key,logger=logger, enable_diagnostics_logging=True)
+```
 ### Telemetry
 Azure Core provides the ability for our Python SDKs to use OpenTelemetry with them. The only packages that need to be installed
 to use this functionality are the following:
@@ -1060,6 +1046,7 @@ For more extensive documentation on the Cosmos DB service, see the [Azure Cosmos
 [BM25]: https://learn.microsoft.com/azure/search/index-similarity-and-scoring
 [cosmos_fts]: https://aka.ms/cosmosfulltextsearch
 [cosmos_index_policy_change]: https://learn.microsoft.com/azure/cosmos-db/index-policy#modifying-the-indexing-policy
+[cosmos_diagnostics_filter_sample]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cosmos/azure-cosmos/samples/diagnostics_filter_sample.py
 
 ## Contributing
 

sdk/cosmos/azure-cosmos/azure/cosmos/_cosmos_client_connection.py

@@ -214,7 +214,6 @@ def __init__( # pylint: disable=too-many-statements
                 logger=kwargs.pop("logger", None),
                 enable_diagnostics_logging=kwargs.pop("enable_diagnostics_logging", False),
                 global_endpoint_manager=self._global_endpoint_manager,
-                diagnostics_handler=kwargs.pop("diagnostics_handler", None),
                 **kwargs
             ),
         ]