seamless predictable upgrade pipeline and tests

[barakb]

The purpose of this issue is to determine and fix tests which have to be run twice in order to pass and try to reduce the time for long running tests.

so far:

• Increase retries when receiving the 429 too many retries http error (most frequent)
• Sentinel split brain possibility during test after start and stop causing master related errors ( second most frequent)
• redis.exceptions.ClusterError: TTL exhausted. ( I have yet not found an answer as to why this happens during failover or how to prevent it) ( second most frequent)
• Client tries to connect to the instance but never succeeds, this was previously investigated by Muhammad Qadora but could not find a resolution to the issue.
• Stopping instance times out
• Changing network from Private back to Public times out

[dudizimber]

@MuhammadQadora

DEBUG    urllib3.connectionpool:connectionpool.py:544 [https://api.omnistrate.cloud:443](https://api.omnistrate.cloud/) "PATCH /2022-09-01-00/resource-instance/sp-JvkxkPhinN/falkordb/v1/testing/enterprise-schedule-customer-hosted/enterprise-schedule-falkordb-customer-hosted-model-omnistrate-dedicated-tenancy/multi-Zone/instance-wrclzukj9 HTTP/1.1" 429 18
ERROR    root:omnistrate_fleet_api.py:31 Failed to update instance type instance-wrclzukj9: Too Many Requests

[MuhammadQadora]

indefinite connection time out from 2026-02-17 13:26:21,400 till 2026-02-17 16:42:10,663

629xxIgMtR/instance/instance-mnc035xmq HTTP/1.1" 200 None
2026-02-17 13:26:21,400 - Instance is RUNNING
2026-02-17 13:26:21,400 - Asserting data presence in graph 'test' with at least 1 rows
2026-02-17 13:26:21,400 - Checking if graph 'test' has at least 1 rows
2026-02-17 13:26:21,400 - Connecting to r-wreeimcltf.instance-mnc035xmq.hc-ln9zjpko8.us-east-2.aws.f2e0a955bb84.cloud:6379
tests/test_grouped_cluster.py::test_cluster_pack Waiting for instance to be ready. Timeout: 2600 seconds
Waiting for instance to be ready. Timeout: 2600 seconds
Waiting for instance to be ready. Timeout: 2600 seconds
Waiting for instance to be ready. Timeout: 2600 seconds
Waiting for instance to be ready. Timeout: 1200 seconds
Waiting for instance to be ready. Timeout: 2400 seconds
Waiting for instance to be ready. Timeout: 1200 seconds
Waiting for instance to be ready. Timeout: 2400 seconds
2026-02-17 16:42:10,663 - Deleting instance
2026-02-17 16:42:10,663 - Resetting dropped connection: api.omnistrate.cloud
2026-02-17 16:42:14,397 - [https://api.omnistrate.cloud:443](https://api.omnistrate.cloud/) "DELETE /2022-09-01-00/fleet/service/s-KgFDwg5vBS/environment/se-629xxIgMtR/instance/instance-mnc035xmq HTTP/1.1" 200 0
2026-02-17 16:42:15,008 - [https://api.omnistrate.cloud:443](https://api.omnistrate.cloud/) "GET /2022-09-01-00/fleet/service/s-KgFDwg5vBS/environment/se-629xxIgMtR/instance/instance-mnc035xmq HTTP/1.1" 200 None
2026-02-17 16:42:15,008 - Instance is in DELETING state