Skip to content

Commit 741181a

Browse files
tvaron3tvaron3
authored
Fix Environment Variable Parsing (Azure#40303)
1 parent 87b4e46 commit 741181a

10 files changed

+107
-30
lines changed

‎sdk/cosmos/azure-cosmos/CHANGELOG.md

Lines changed: 20 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
#### Breaking Changes
88

99
#### Bugs Fixed
10+
* Fixed how the environment variables in the sdk are parsed. See [PR 40303](https://github.com/Azure/azure-sdk-for-python/pull/40303).
1011

1112
#### Other Changes
1213

@@ -27,61 +28,61 @@
2728
### 4.10.0b2 (2025-03-17)
2829

2930
#### Bugs Fixed
30-
* Fixed bug preventing health check in some scenarios. See [PR 39647](https://github.com/Azure/azure-sdk-for-python/pull/39647)
31-
* Fixed `partition_key` filter for `query_items_change_feed` API. See [PR 39895](https://github.com/Azure/azure-sdk-for-python/pull/39895)
31+
* Fixed bug preventing health check in some scenarios. See [PR 39647](https://github.com/Azure/azure-sdk-for-python/pull/39647).
32+
* Fixed `partition_key` filter for `query_items_change_feed` API. See [PR 39895](https://github.com/Azure/azure-sdk-for-python/pull/39895).
3233

3334
#### Other Changes
34-
* Moved endpoint health check to the background for async APIs. See [PR 39647](https://github.com/Azure/azure-sdk-for-python/pull/39647)
35+
* Moved endpoint health check to the background for async APIs. See [PR 39647](https://github.com/Azure/azure-sdk-for-python/pull/39647).
3536

3637
### 4.10.0b1 (2025-02-13)
3738

3839
#### Features Added
39-
* Added ability to replace `computed_properties` through `replace_container` method. See [PR 39543](https://github.com/Azure/azure-sdk-for-python/pull/39543)
40+
* Added ability to replace `computed_properties` through `replace_container` method. See [PR 39543](https://github.com/Azure/azure-sdk-for-python/pull/39543).
4041

4142
#### Other Changes
42-
* Un-marked `computed_properties` keyword as **provisional**. See [PR 39543](https://github.com/Azure/azure-sdk-for-python/pull/39543)
43+
* Un-marked `computed_properties` keyword as **provisional**. See [PR 39543](https://github.com/Azure/azure-sdk-for-python/pull/39543).
4344

4445
### 4.9.1b4 (2025-02-06)
4546

4647
#### Bugs Fixed
47-
* Improved retry logic for read requests to failover on other regions in case of timeouts and any error codes >= 500. See [PR 39596](https://github.com/Azure/azure-sdk-for-python/pull/39596)
48+
* Improved retry logic for read requests to failover on other regions in case of timeouts and any error codes >= 500. See [PR 39596](https://github.com/Azure/azure-sdk-for-python/pull/39596).
4849
* Fixed a regression where read operations were not retrying on timeouts. See [PR 39596](https://github.com/Azure/azure-sdk-for-python/pull/39596)
4950
* Updated default read timeout for `getDatabaseAccount` calls to 3 seconds. See [PR 39596](https://github.com/Azure/azure-sdk-for-python/pull/39596)
5051

5152
### 4.9.1b3 (2025-02-04)
5253

5354
#### Features Added
54-
* Improved retry logic by retrying alternative endpoint for writes within a region before performing a cross region retry. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
55+
* Improved retry logic by retrying alternative endpoint for writes within a region before performing a cross region retry. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
5556
* Added endpoint health check logic during database account calls. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
5657

5758
#### Bugs Fixed
58-
* Fixed unnecessary retries on the wrong region for timout retry policy. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
59-
* All client connection errors from aiohttp will be retried. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
59+
* Fixed unnecessary retries on the wrong region for timout retry policy. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
60+
* All client connection errors from aiohttp will be retried. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
6061

6162
#### Other Changes
62-
* Changed defaults for retry delays. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
63-
* Changed default connection timeout to be 5 seconds. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
64-
* Changed default read timeout to be 65 seconds. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
65-
* On database account calls send a client id header for load balancing. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
66-
* Removed aiohttp dependency. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390)
63+
* Changed defaults for retry delays. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
64+
* Changed default connection timeout to be 5 seconds. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
65+
* Changed default read timeout to be 65 seconds. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
66+
* On database account calls send a client id header for load balancing. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
67+
* Removed aiohttp dependency. See [PR 39390](https://github.com/Azure/azure-sdk-for-python/pull/39390).
6768

6869
### 4.9.1b2 (2025-01-24)
6970

7071
#### Features Added
71-
* Added new cross-regional retry logic for `ServiceRequestError` and `ServiceResponseError` exceptions. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396)
72+
* Added new cross-regional retry logic for `ServiceRequestError` and `ServiceResponseError` exceptions. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396).
7273

7374
#### Bugs Fixed
7475
* Fixed `KeyError` being returned by location cache when most preferred location is not present in cached regions. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396).
75-
* Fixed cross-region retries on `CosmosClient` initialization. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396)
76+
* Fixed cross-region retries on `CosmosClient` initialization. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396).
7677

7778
#### Other Changes
78-
* This release requires aiohttp version 3.10.11 and above. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396)
79+
* This release requires aiohttp version 3.10.11 and above. See [PR 39396](https://github.com/Azure/azure-sdk-for-python/pull/39396).
7980

8081
### 4.9.1b1 (2024-12-13)
8182

8283
#### Features Added
83-
* Added change feed mode support in `query_items_change_feed`. See [PR 38105](https://github.com/Azure/azure-sdk-for-python/pull/38105)
84-
* Added a **Preview Feature** for adding Diagnostics Handler to filter what diagnostics get logged. This feature is subject to change significantly. See [PR 38105](https://github.com/Azure/azure-sdk-for-python/pull/38581)
84+
* Added change feed mode support in `query_items_change_feed`. See [PR 38105](https://github.com/Azure/azure-sdk-for-python/pull/38105).
85+
* Added a **Preview Feature** for adding Diagnostics Handler to filter what diagnostics get logged. This feature is subject to change significantly. See [PR 38105](https://github.com/Azure/azure-sdk-for-python/pull/38581).
8586

8687
### 4.9.0 (2024-11-18)
8788

‎sdk/cosmos/azure-cosmos/azure/cosmos/_constants.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,14 @@ class _Constants:
4444
# ServiceDocument Resource
4545
EnableMultipleWritableLocations: Literal["enableMultipleWriteLocations"] = "enableMultipleWriteLocations"
4646

47+
# Environment variables
48+
NON_STREAMING_ORDER_BY_DISABLED_CONFIG: str = "AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY"
49+
NON_STREAMING_ORDER_BY_DISABLED_CONFIG_DEFAULT: str = "False"
50+
HS_MAX_ITEMS_CONFIG: str = "AZURE_COSMOS_HYBRID_SEARCH_MAX_ITEMS"
51+
HS_MAX_ITEMS_CONFIG_DEFAULT: int = 1000
52+
MAX_ITEM_BUFFER_VS_CONFIG: str = "AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH"
53+
MAX_ITEM_BUFFER_VS_CONFIG_DEFAULT: int = 50000
54+
4755
# Error code translations
4856
ERROR_TRANSLATIONS: Dict[int, str] = {
4957
400: "BAD_REQUEST - Request being sent is invalid.",

‎sdk/cosmos/azure-cosmos/azure/cosmos/_cosmos_client_connection.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3183,7 +3183,8 @@ def _GetQueryPlanThroughGateway(self, query: str, resource_link: str, **kwargs:
31833183
documents._QueryFeature.NonStreamingOrderBy + "," +
31843184
documents._QueryFeature.HybridSearch + "," +
31853185
documents._QueryFeature.CountIf)
3186-
if os.environ.get('AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY', False):
3186+
if os.environ.get(Constants.NON_STREAMING_ORDER_BY_DISABLED_CONFIG,
3187+
Constants.NON_STREAMING_ORDER_BY_DISABLED_CONFIG_DEFAULT) == "True":
31873188
supported_query_features = (documents._QueryFeature.Aggregate + "," +
31883189
documents._QueryFeature.CompositeAggregate + "," +
31893190
documents._QueryFeature.Distinct + "," +

‎sdk/cosmos/azure-cosmos/azure/cosmos/_execution_context/aio/execution_dispatcher.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
from azure.cosmos.documents import _DistinctType
3535
from azure.cosmos.exceptions import CosmosHttpResponseError
3636
from azure.cosmos.http_constants import StatusCodes
37+
from ..._constants import _Constants as Constants
3738

3839
# pylint: disable=protected-access
3940

@@ -117,8 +118,9 @@ async def _create_pipelined_execution_context(self, query_execution_info):
117118
raise ValueError("Executing a vector search query without TOP or LIMIT can consume many" +
118119
" RUs very fast and have long runtimes. Please ensure you are using one" +
119120
" of the two filters with your vector search query.")
120-
if total_item_buffer > os.environ.get('AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH', 50000):
121-
raise ValueError("Executing a vector search query with more items than the max is not allowed." +
121+
if total_item_buffer > int(os.environ.get(Constants.MAX_ITEM_BUFFER_VS_CONFIG,
122+
Constants.MAX_ITEM_BUFFER_VS_CONFIG_DEFAULT)):
123+
raise ValueError("Executing a vector search query with more items than the max is not allowed. " +
122124
"Please ensure you are using a limit smaller than the max, or change the max.")
123125
execution_context_aggregator =\
124126
non_streaming_order_by_aggregator._NonStreamingOrderByContextAggregator(self._client,

‎sdk/cosmos/azure-cosmos/azure/cosmos/_execution_context/execution_dispatcher.py

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
from azure.cosmos._execution_context.query_execution_info import _PartitionedQueryExecutionInfo
3434
from azure.cosmos.documents import _DistinctType
3535
from azure.cosmos.http_constants import StatusCodes, SubStatusCodes
36+
from .._constants import _Constants as Constants
3637

3738
# pylint: disable=protected-access
3839

@@ -56,8 +57,9 @@ def _verify_valid_hybrid_search_query(hybrid_search_query_info):
5657
raise ValueError("Executing a hybrid search query without TOP or LIMIT can consume many" +
5758
" RUs very fast and have long runtimes. Please ensure you are using one" +
5859
" of the two filters with your hybrid search query.")
59-
if hybrid_search_query_info['take'] > os.environ.get('AZURE_COSMOS_HYBRID_SEARCH_MAX_ITEMS', 1000):
60-
raise ValueError("Executing a hybrid search query with more items than the max is not allowed." +
60+
if hybrid_search_query_info['take'] > int(os.environ.get(Constants.HS_MAX_ITEMS_CONFIG,
61+
Constants.HS_MAX_ITEMS_CONFIG_DEFAULT)):
62+
raise ValueError("Executing a hybrid search query with more items than the max is not allowed. " +
6163
"Please ensure you are using a limit smaller than the max, or change the max.")
6264

6365

@@ -149,8 +151,9 @@ def _create_pipelined_execution_context(self, query_execution_info):
149151
raise ValueError("Executing a vector search query without TOP or LIMIT can consume many" +
150152
" RUs very fast and have long runtimes. Please ensure you are using one" +
151153
" of the two filters with your vector search query.")
152-
if total_item_buffer > os.environ.get('AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH', 50000):
153-
raise ValueError("Executing a vector search query with more items than the max is not allowed." +
154+
if total_item_buffer > int(os.environ.get(Constants.MAX_ITEM_BUFFER_VS_CONFIG,
155+
Constants.MAX_ITEM_BUFFER_VS_CONFIG_DEFAULT)):
156+
raise ValueError("Executing a vector search query with more items than the max is not allowed. " +
154157
"Please ensure you are using a limit smaller than the max, or change the max.")
155158
execution_context_aggregator = \
156159
non_streaming_order_by_aggregator._NonStreamingOrderByContextAggregator(self._client,

‎sdk/cosmos/azure-cosmos/azure/cosmos/aio/_cosmos_client_connection_async.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3195,7 +3195,8 @@ async def _GetQueryPlanThroughGateway(self, query: str, resource_link: str, **kw
31953195
documents._QueryFeature.NonStreamingOrderBy + "," +
31963196
documents._QueryFeature.HybridSearch + "," +
31973197
documents._QueryFeature.CountIf)
3198-
if os.environ.get('AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY', False):
3198+
if os.environ.get(Constants.NON_STREAMING_ORDER_BY_DISABLED_CONFIG,
3199+
Constants.NON_STREAMING_ORDER_BY_DISABLED_CONFIG_DEFAULT) == "True":
31993200
supported_query_features = (documents._QueryFeature.Aggregate + "," +
32003201
documents._QueryFeature.CompositeAggregate + "," +
32013202
documents._QueryFeature.Distinct + "," +

‎sdk/cosmos/azure-cosmos/tests/test_query_hybrid_search.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
# Copyright (c) Microsoft Corporation. All rights reserved.
33

44
import json
5+
import os
56
import time
67
import unittest
78
import uuid
@@ -94,6 +95,20 @@ def test_wrong_hybrid_search_queries(self):
9495
assert ("One of the input values is invalid" in e.message or
9596
"Specifying a sort order (ASC or DESC) in the ORDER BY RANK clause is not allowed." in e.message)
9697

98+
def test_hybrid_search_env_variables(self):
99+
os.environ["AZURE_COSMOS_HYBRID_SEARCH_MAX_ITEMS"] = "0"
100+
try:
101+
query = "SELECT TOP 1 c.index, c.title FROM c WHERE FullTextContains(c.title, 'John') OR " \
102+
"FullTextContains(c.text, 'John') ORDER BY RANK FullTextScore(c.title, ['John'])"
103+
results = self.test_container.query_items(query, enable_cross_partition_query=True)
104+
[item for item in results]
105+
pytest.fail("Config was not applied properly.")
106+
except ValueError as e:
107+
assert e.args[0] == ("Executing a hybrid search query with more items than the max is not allowed. "
108+
"Please ensure you are using a limit smaller than the max, or change the max.")
109+
finally:
110+
os.environ["AZURE_COSMOS_HYBRID_SEARCH_MAX_ITEMS"] = "1000"
111+
97112
def test_hybrid_search_queries(self):
98113
query = "SELECT TOP 10 c.index, c.title FROM c WHERE FullTextContains(c.title, 'John') OR " \
99114
"FullTextContains(c.text, 'John') ORDER BY RANK FullTextScore(c.title, ['John'])"

‎sdk/cosmos/azure-cosmos/tests/test_query_hybrid_search_async.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# The MIT License (MIT)
22
# Copyright (c) Microsoft Corporation. All rights reserved.
3-
3+
import os
44
import time
55
import unittest
66
import uuid
@@ -98,6 +98,20 @@ async def test_wrong_hybrid_search_queries_async(self):
9898
assert ("One of the input values is invalid" in e.message or
9999
"Specifying a sort order (ASC or DESC) in the ORDER BY RANK clause is not allowed." in e.message)
100100

101+
async def test_hybrid_search_env_variables_async(self):
102+
os.environ["AZURE_COSMOS_HYBRID_SEARCH_MAX_ITEMS"] = "0"
103+
try:
104+
query = "SELECT TOP 1 c.index, c.title FROM c WHERE FullTextContains(c.title, 'John') OR " \
105+
"FullTextContains(c.text, 'John') ORDER BY RANK FullTextScore(c.title, ['John'])"
106+
results = self.test_container.query_items(query)
107+
[item async for item in results]
108+
pytest.fail("Config was not applied properly.")
109+
except ValueError as e:
110+
assert e.args[0] == ("Executing a hybrid search query with more items than the max is not allowed. "
111+
"Please ensure you are using a limit smaller than the max, or change the max.")
112+
finally:
113+
os.environ["AZURE_COSMOS_HYBRID_SEARCH_MAX_ITEMS"] = "1000"
114+
101115
async def test_hybrid_search_queries_async(self):
102116
query = "SELECT TOP 10 c.index, c.title FROM c WHERE FullTextContains(c.title, 'John') OR " \
103117
"FullTextContains(c.text, 'John') ORDER BY RANK FullTextScore(c.title, ['John'])"

‎sdk/cosmos/azure-cosmos/tests/test_query_vector_similarity.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# The MIT License (MIT)
22
# Copyright (c) Microsoft Corporation. All rights reserved.
3-
3+
import os
44
import unittest
55
import uuid
66

@@ -121,6 +121,22 @@ def test_wrong_vector_search_queries(self):
121121
assert ("One of the input values is invalid." in e.message
122122
or "Specifying a sorting order (ASC or DESC) with VectorDistance function is not supported." in e.message)
123123

124+
def test_vector_search_environment_variables(self):
125+
vector_string = vector_test_data.get_embedding_string("I am having a wonderful day.")
126+
query = "SELECT TOP 10 c.text, VectorDistance(c.embedding, [{}]) AS " \
127+
"SimilarityScore FROM c ORDER BY VectorDistance(c.embedding, [{}])".format(vector_string, vector_string)
128+
os.environ["AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH"] = "1"
129+
try:
130+
[item for item in self.created_large_container.query_items(query=query, enable_cross_partition_query=True)]
131+
pytest.fail("Config was not set correctly.")
132+
except ValueError as e:
133+
assert e.args[0] == ("Executing a vector search query with more items than the max is not allowed. "
134+
"Please ensure you are using a limit smaller than the max, or change the max.")
135+
136+
os.environ["AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH"] = "50000"
137+
os.environ["AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY"] = "False"
138+
[item for item in self.created_large_container.query_items(query=query, enable_cross_partition_query=True)]
139+
124140
def test_ordering_distances(self):
125141
# Besides ordering distances, we also verify that the query text properly replaces any set embedding policies
126142
# load up previously calculated embedding for the given string

‎sdk/cosmos/azure-cosmos/tests/test_query_vector_similarity_async.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# The MIT License (MIT)
22
# Copyright (c) Microsoft Corporation. All rights reserved.
3-
3+
import os
44
import unittest
55
import uuid
66

@@ -127,6 +127,22 @@ async def test_wrong_vector_search_queries_async(self):
127127
assert ("One of the input values is invalid." in e.message
128128
or "Specifying a sorting order (ASC or DESC) with VectorDistance function is not supported." in e.message)
129129

130+
async def test_vector_search_environment_variables_async(self):
131+
vector_string = vector_test_data.get_embedding_string("I am having a wonderful day.")
132+
query = "SELECT TOP 10 c.text, VectorDistance(c.embedding, [{}]) AS " \
133+
"SimilarityScore FROM c ORDER BY VectorDistance(c.embedding, [{}])".format(vector_string, vector_string)
134+
os.environ["AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH"] = "1"
135+
try:
136+
[item async for item in self.created_large_container.query_items(query=query)]
137+
pytest.fail("Config was not set correctly.")
138+
except ValueError as e:
139+
assert e.args[0] == ("Executing a vector search query with more items than the max is not allowed. "
140+
"Please ensure you are using a limit smaller than the max, or change the max.")
141+
142+
os.environ["AZURE_COSMOS_MAX_ITEM_BUFFER_VECTOR_SEARCH"] = "50000"
143+
144+
os.environ["AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY"] = "False"
145+
[item async for item in self.created_large_container.query_items(query=query)]
130146

131147
async def test_ordering_distances_async(self):
132148
# Besides ordering distances, we also verify that the query text properly replaces any set embedding policies

0 commit comments

Comments
 (0)