Cosmos Scale Testing (Azure#40611)

* background call for get database account call

* only call get database account in health check for different endpoints

* Change health check logic in sync

* Revert removing timing

* update changelog

* fix tests

* use asyncio create_task

* fix tests

* fix pylint

* Renamed variables and added effective preferred locations

* update changelog

* Add test for effective preferred regions

* Add test for effective preferred regions

* sync test for preferred regions

* Renaming and add test for health check

* fix tests

* fix tests and add more health check tests

* fix tests

* add tests

* fix tests

* Move to breaking change

* Added scale workload scripts

* Scale testing upserts

* fix cspell and tests

* fix tests

* fix initial upserts

* fix tests

* revert preferred locations

* timeout mark unavailable

* moving health check to background

* only finish health check after 2 successes

* fix tests

* add tests

* fix and add timeout tests for marking endpoints unavailable

* Reacting to comments

* Reacting to comments

* Add cleanup of background task when cosmos client closes

* React to comments

* Remove consecutive failures code

* updated changelog

* Fix test

* Fix tests that weren't awaiting properly

* Add a sync workload and fix initial setup and separate client configs to separate file

* fix imports

* fix tests

* update proxy

* change unavailable time to 7 minutes

* remove marking unavailable, health check four regions, health check primary and alternate

* fix type hints

* logs to debug

* logs to debug

* fix startup scenario

* debug logs

* fix updating cache

* adding logs

* fix operation type check unavailable

* add logic to check by endpoint and not by regional routing context

* bash sript to run workloads

* adding more workloads

* fix tests

* fix tests

* lower write workloads to use

* fix read envoy workloads

* increase read concurrency

* fix test

* more read workloads

* more read workloads

* fix tests

* fix tests

* add smaller lag for some writes and reads

* debug multimaster test

* cleanup

* mark global endpoint available

* fix tests and only use endpoint from gateway for multimaster

* balance read and write workloads

* retry multi region writes same as reads

* react to comments

* multi write fix

* debug logs

* multiwrite config

* fix multimaster workloads

* cleanup / pylint

* refactor workloads

* fix read proxy workload

* swallow exceptions and add mocking

* fix import

* fix import

* fix mocking

* fix mocking

* fix mocking

* logging improvements for timeout and failover retry policy

* debugging logs

* fix mock

* fix mock

* improve mock and adding more logs

* service request instead of response

* more logs

* cleanup logs

* remove import

* more logs and fix mock

* fix logs

* more logs for debugging

* more logs for debugging

* fix mocking

* fix mocking

* fix mocking

* add envoy

* update envoy and remove mocking

* envoy changes

* fix workloads

* lower sleep time between concurrent operations

* remove sleeps

* increase concurrency

* increase default concurrency

* remove excessive logs

* change proxy workloads

* change proxy file

* change proxy file

* change proxy file

* change proxy file

* useful logs

* change envoy file

* change envoy file

* change envoy file

* change envoy file

* change envoy file

* change envoy file

* change envoy file

* Added envoy open ai simple file

* update simple

* Updated scale testing workloads for ycsb benchmark

* Added run workload script without proxy

* Added more workloads

* Fixed the run workloads script

* Fixed one more scriptp

* Reduced the clients to below 60

* Updated file handler to rotating file handler

* Logging headers for testing

* Skipping session token for writes

* Skipping session token for writes

* commented out request / response headers

* Updated file handler to rotating file handler with 10 mb max

* Updated file handler to rotating file handler with 10 mb max

* Fixed rotating file handler for every workload script

* Updated test directory to tests

* Updated test directory to tests and added all files

* Updated test directory to tests and added workloads

* Fixed envoy proxy code

* Container and Database is configurable now

* Container and Database is configurable fix

* ru change to create container and incorrect client workload

* add None etag and match_condition edge case

* Use process id for log files and refactor common methods across workloads

* fix initial-setup.py

* fix logger naming

* fix logger naming

* fix workloads and envoy file

* improve workload scripts

* improve workload scripts

* test improvements

* cleanup

* envoy changes

* envoy changes

* revert changes

* Update sdk/cosmos/azure-cosmos/tests/workloads/w_workload.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/initial-setup.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/get-database-account-call.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/r_proxy_workload.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/r_w_q_with_incorrect_client_workload.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/r_workload.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/r_w_q_workload.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/cosmos/azure-cosmos/tests/workloads/w_proxy_workload.py

Co-authored-by: Copilot <[email protected]>

* fix analyze and add licensing

* fix analyze

* react to comments

* fix imports

* make shutdown executable

* fix import

* fix workloads

* fix imports

* fix shutdown

---------

Co-authored-by: Kushagra Thapar <[email protected]>
Co-authored-by: Tomas Varon <[email protected]>
Co-authored-by: Ubuntu <openai-test@openai-testing-vm1.q2mff55cv1bu1hrxqay0qbmh1g.cbnx.internal.cloudapp.net>
Co-authored-by: Simon Moreno <[email protected]>
Co-authored-by: Copilot <[email protected]>

Author: 6 people

sdk/cosmos/azure-cosmos/tests/workloads/dev.md

@@ -0,0 +1,41 @@
+## SDK Scale Testing
+This directory contains the scale testing workloads for the SDK. The workloads are designed to test the performance 
+and scalability of the SDK under various conditions. There are different types of workloads and each will create a log 
+file when run. These logs are named in this format `<file name>-<process id>-<date time>.log`. 
+
+### Setup VM
+1. Create a VM in Azure with the following configuration:
+   - 8 vCPUs
+   - 32 GB RAM
+   - Ubuntu
+   - Accelerated networking
+2. Fork and clone this repository
+3. Go to azure cosmos folder
+   - `cd azure-sdk-for-python/sdk/cosmos/azure-cosmos`
+4. Install the required packages and create virtual environment
+   - `sudo apt-get update`
+   - `sudo apt-get install python3-pip`
+   - `sudo apt-get install python3.12-venv`
+   - `python3 -m venv azure-cosmosdb-sdk-environment`
+   - `source azure-cosmosdb-sdk-environment/bin/activate`
+   - `pip install -r dev_requirements.txt`
+5. Checkout the branch with the changes to test. 
+6. Install azure-cosmos
+   - `pip install .`
+7. Go to workloads folder
+    - `cd tests/workloads`
+8. Fill out relevant configs in `workload_configs.py`: key, host, etc
+9. Install envoy proxy https://www.envoyproxy.io/docs/envoy/latest/start/install
+10. Update envoy_simple_config.yaml to have the correct account info. Replace <> with account name.
+11. Go to envoy folder and start envoy
+    - `cd envoy`
+    - `mkdir logs`
+    - `envoy -c <envoy_file>.yaml --log-level debug --log-path logs/debug.txt`
+12. Run the setup workload to create the database and containers and insert data
+    - `python3 initial-setup.py`
+13. Run the scale workloads
+    - `./run_workloads.sh <number of clients per workload>`
+
+### Monitor Run
+- `ps -eaf | grep "python3"` to see the running processes
+- `tail -f <log_file>` to see the logs in real time