Resilient start of simulated storages (#2583)

#### Reference Issues/PRs
<!--Example: Fixes #1234. See also #3456.-->

#### What does this implement or fix?

Problem shoed here:
https://github.com/man-group/ArcticDB/actions/runs/16915115493/job/47928193692

On runners time start seems to be varying as found in a comment for moto
- it was 240 secs. For azurite and mongo it was default 20.

The fix now makes start timeout equal and adds one retry. This code will
not harm execution on laptops, but will make startup of tests for
simulated and local storages on runners resilient.

#### Any other comments?

#### Checklist

<details>
  <summary>
   Checklist for code changes...
  </summary>
 
- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

---------

Co-authored-by: Georgi Rusev <Georgi Rusev>

Author: grusev

python/arcticdb/storage_fixtures/azure.py

@@ -193,8 +193,10 @@ def _safe_enter(self):
             self.client_cert_dir = ""
         if self.http_protocol == "https":
             args += f" --key {self.key_file} --cert {self.cert_file}"
-        self._p = GracefulProcessUtils.start(args, cwd=self.working_dir)
-        wait_for_server_to_come_up(self.endpoint_root, "azurite", self._p)
+        self._p = GracefulProcessUtils.start_with_retry(url=self.endpoint_root, 
+                                                        service_name="azurite", num_retries=2, timeout=240,
+                                                        process_start_cmd=args,
+                                                        cwd=self.working_dir)            
         return self
 
     def __exit__(self, exc_type, exc_value, traceback):

python/arcticdb/storage_fixtures/mongo.py

@@ -123,9 +123,11 @@ def __init__(self, data_dir: Optional[str] = None, port=0, executable="mongod"):
 
     def _safe_enter(self):
         cmd = [self._executable, "--port", str(self._port), "--dbpath", self._data_dir]
-        self._p = GracefulProcessUtils.start(cmd)
         self.mongo_uri = f"mongodb://localhost:{self._port}"
-        wait_for_server_to_come_up(f"http://localhost:{self._port}", "mongod", self._p)
+        self._p = GracefulProcessUtils.start_with_retry(url=f"http://localhost:{self._port}", 
+                                                        service_name="mongod", num_retries=2, timeout=240,
+                                                        process_start_cmd=cmd)                
+            
         self._client = get_mongo_client(self.mongo_uri)
 
     def __exit__(self, exc_type, exc_value, traceback):

python/arcticdb/storage_fixtures/utils.py

@@ -61,6 +61,23 @@ def start(cmd, **kwargs):
         print("About to run:", cmd)
         creation_flags = subprocess.CREATE_NEW_PROCESS_GROUP if _WINDOWS else 0
         return subprocess.Popen(cmd, creationflags=creation_flags, **kwargs)
+    
+    @staticmethod
+    def start_with_retry(url: str, service_name: str, num_retries: int, timeout: int, 
+                         process_start_cmd: str, **kwargs):
+        """Attempts to start the process up to specified times.
+        
+        Each time will wait for service to be avil at specified url up to the specified timeout"""
+        for i in range(num_retries): # retry in case of connection problems
+            try:
+                p = GracefulProcessUtils.start(process_start_cmd, **kwargs)
+                wait_for_server_to_come_up(url, service_name, p, timeout=timeout)
+                return p
+            except AssertionError:
+                try:
+                    p.terminate()
+                except:
+                    pass
 
     @staticmethod
     def wait(p: ProcessUnion, timeout_sec: int):