Fix workflow problems for Persistеnce test executions and ASV AWS S3 tests execution (#2585)

#### Reference Issues/PRs
<!--Example: Fixes #1234. See also #3456.-->

#### What does this implement or fix?

ASV tests problems with AWS S3: the code wrongly assumed aws bucket from
azure env vars. This works locally as I did had Azure vars. But the ASW
workflow does not supply those vars. This is fixed and run is initiated
from Actions:
https://github.com/man-group/ArcticDB/actions/runs/16987111363

Persistance tests are executed from a separate workflow. This workflow
runc cleanup seed and verify tasks. The workflow does not install any
arcticdb test libraries. Further it works with old arctic also,
therefore it is much more closer to installation tests and must not
include any new stuff from master but stuff that has been there in 3.0.0
only. This again was invisible requirement during development of Azure.
Thus all worked fine
Run from Actions with AWS_S3:
https://github.com/man-group/ArcticDB/actions/runs/16988249102

Both could only be revealed if executed against latest changed sources
from Azure branc. As those workflows are not part pf the PR they were
forgotten ....


#### Any other comments?

#### Checklist

<details>
  <summary>
   Checklist for code changes...
  </summary>
 
- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

---------

Co-authored-by: Georgi Rusev <Georgi Rusev>

Author: grusev

.github/workflows/build.yml

@@ -21,7 +21,6 @@ on:
           - 'GCPXML'
           - 'AZURE'
         default: 'no'
-
       pypi_publish:
         type: boolean
       publish_env:
@@ -230,6 +229,7 @@ jobs:
     secrets: inherit
     with:
       job_type: cleanup
+      persistent_storage: ${{ needs.storage_type.outputs.storage }}
 
   persistent_storage_seed_linux:
     needs: [common_config, pre_seed_cleanup, storage_type]
@@ -254,6 +254,8 @@ jobs:
       arcticdb_version: ${{matrix.arcticdb_version}}
       matrix: ${{toJson(matrix.matrix_override)}}
       python_deps_ids: ${{toJson(matrix.python_deps_ids)}}
+      persistent_storage: ${{ needs.storage_type.outputs.storage }}
+
 
   persistent_storage_seed_windows:
     needs: [common_config, pre_seed_cleanup, storage_type]
@@ -278,6 +280,7 @@ jobs:
       arcticdb_version: ${{matrix.arcticdb_version}}
       matrix: ${{toJson(matrix.matrix_override)}}
       python_deps_ids: ${{toJson(matrix.python_deps_ids)}}
+      persistent_storage: ${{ needs.storage_type.outputs.storage }}
 
   cpp-test-linux:
     needs: [cibw_docker_image, common_config]
@@ -387,7 +390,7 @@ jobs:
       DEBUG_LOGGING_ENABLED: ${{ needs.storage_type.outputs.DEBUG_LOGGING_ENABLED }}            
 
   persistent_storage_verify_linux:
-    needs: [common_config, build-python-wheels-linux, build-python-wheels-windows]
+    needs: [common_config, build-python-wheels-linux, build-python-wheels-windows, storage_type]
     # This step is valid only for AWS_S3.
     if: ${{ needs.storage_type.outputs.storage == 'AWS_S3'}}
     strategy:
@@ -409,8 +412,10 @@ jobs:
       arcticdb_version: ${{matrix.arcticdb_version}}
       matrix: ${{toJson(matrix.matrix_override)}}
       python_deps_ids: ${{toJson(matrix.python_deps_ids)}}
+      persistent_storage: ${{ needs.storage_type.outputs.storage }}
 
   persistent_storage_verify_windows:
+    needs: [common_config, storage_type]
     # This step is valid only for AWS_S3.
     if: ${{ needs.storage_type.outputs.storage == 'AWS_S3'}}
     strategy:
@@ -432,6 +437,7 @@ jobs:
       arcticdb_version: ${{matrix.arcticdb_version}}
       matrix: ${{toJson(matrix.matrix_override)}}
       python_deps_ids: ${{toJson(matrix.python_deps_ids)}}
+      persistent_storage: ${{ needs.storage_type.outputs.storage }}
 
   post_verify_cleanup:
     needs: [persistent_storage_verify_windows, persistent_storage_verify_linux, storage_type]
@@ -446,6 +452,7 @@ jobs:
     secrets: inherit
     with:
       job_type: cleanup
+      persistent_storage: ${{ needs.storage_type.outputs.storage }}
 
   build-python-wheels-macos:
     needs: [common_config, storage_type]

.github/workflows/persistent_storage.yml

@@ -12,6 +12,8 @@ on:
       endpoint:          {default: 's3.eu-west-1.amazonaws.com', type: string, description: The address of the S3 endpoint}
       region:            {default: 'eu-west-1', type: string, description: The S3 region of the bucket}
       clear:             {default: 0, type: number, description: Controls wheather we will clear the libraries in the versions stores that we test against}      
+      persistent_storage: {required: true, default: "no", type: string, description: Specifies whether the python tests should tests against real storages e.g. AWS S3  }
+
 jobs:
   setup:
     if: inputs.job_type != 'cleanup'
@@ -75,6 +77,7 @@ jobs:
           aws_access_key: "${{ secrets.AWS_S3_ACCESS_KEY }}"
           aws_secret_key: "${{ secrets.AWS_S3_SECRET_KEY }}"
           strategy_branch: "${{matrix.os}}_${{env.python_impl_name}}"
+          persistent_storage: ${{ inputs.persistent_storage }}
 
       - name: Seed the storage
         if: inputs.job_type == 'seed'
@@ -109,6 +112,7 @@ jobs:
         with:
           aws_access_key: "${{ secrets.AWS_S3_ACCESS_KEY }}"
           aws_secret_key: "${{ secrets.AWS_S3_SECRET_KEY }}"
+          persistent_storage: ${{ inputs.persistent_storage }}
 
       - name: Clean the storage
         run: |

python/arcticdb/util/environment_setup.py

@@ -91,9 +91,9 @@ def __new__(cls, *args, **kwargs):
 
             # Azure variable setup
             cls._azure_factory = real_azure_from_environment_variables(shared_path=True)
-            cls._aws_default_factory.default_prefix = None
-            cls._aws_default_factory.default_bucket = AZURE_DEFAULT_CONTAINER
-            cls._aws_default_factory.clean_bucket_on_fixture_exit = False
+            cls._azure_factory.default_prefix = None
+            cls._azure_factory.default_container = AZURE_DEFAULT_CONTAINER
+            cls._azure_factory.clean_bucket_on_fixture_exit = False
 
 
     @classmethod

python/tests/util/storage_test.py

@@ -1,4 +1,5 @@
 from enum import Enum
+import sys
 import pandas as pd
 from pandas.testing import assert_frame_equal
 import os
@@ -10,7 +11,6 @@
 
 from arcticdb import Arctic
 from arcticc.pb2.s3_storage_pb2 import Config as S3Config
-from arcticdb.storage_fixtures.azure import real_azure_from_environment_variables
 try:
     # from pytest this way will work
     from tests.util.mark import PERSISTENT_STORAGE_TESTS_ENABLED
@@ -96,6 +96,19 @@ def real_gcp_credentials(shared_path: bool = True):
     return endpoint, bucket, region, access_key, secret_key, path_prefix, clear
 
 
+def real_azure_credentials(shared_path: bool = True):
+    if shared_path:
+        path_prefix = os.getenv("ARCTICDB_PERSISTENT_STORAGE_SHARED_PATH_PREFIX")
+    else:
+        path_prefix = os.getenv("ARCTICDB_PERSISTENT_STORAGE_UNIQUE_PATH_PREFIX", "")
+    constr=os.getenv("ARCTICDB_REAL_AZURE_CONNECTION_STRING"),
+    container=os.getenv("ARCTICDB_REAL_AZURE_CONTAINER"), 
+
+    clear = str(os.getenv("ARCTICDB_REALL_AZURE_CLEAR")).lower() in ("true", "1")
+
+    return container, constr, path_prefix, clear
+
+
 def get_real_s3_uri(shared_path: bool = True):
     # TODO: Remove this when the latest version that we support
     # contains the s3 fixture code as defined here:
@@ -133,10 +146,38 @@ def get_real_gcp_uri(shared_path: bool = True):
     )
     return aws_uri
 
+def find_ca_certs():
+    # Common CA certificates locations
+    default_paths = ssl.get_default_verify_paths()
+    possible_paths =  [
+        default_paths.cafile,
+        default_paths.openssl_cafile_env,
+        default_paths.openssl_cafile,
+        '/etc/ssl/certs/ca-certificates.crt',
+        '/usr/lib/ssl/certs/ca-certificates.crt',
+        '/etc/pki/tls/certs/ca-bundle.crt',
+        '/etc/ssl/cert.pem'
+    ]
+    for path in possible_paths:
+        if path and os.path.isfile(path):
+            return path
+    return None
+
+### IMPORTANT: When adding new STORAGE we must implement
+###  the whole connection logic here even if this does mean effectively duplicating the code
+###   
+### REASON: We run this file from command line on arcticdb version 3.0. 
+###  there is no way how arcticdb 3.0 could have had the functions that we are going to implement
+###  and support from now on
 def get_real_azure_uri(shared_path: bool = True):
-    return real_azure_from_environment_variables(
-        shared_path=shared_path,
-        ).get_arctic_uri()
+    container, constr, path_prefix = real_azure_credentials(shared_path)
+    ca_certs_file = find_ca_certs()
+    uri = f"azure://Container={container};Path_prefix={path_prefix}"
+    assert ca_certs_file, f"CA file: {ca_certs_file} not found!"
+    if sys.platform.lower().startswith("linux"):
+        uri += f";CA_cert_path={ca_certs_file}"
+    url += f";{constr}"
+    return url
 
 
 class PersistentTestType(Enum):
@@ -159,7 +200,8 @@ def persistent_test_type() -> PersistentTestType:
             return PersistentTestType.AZURE
         return PersistentTestType.AWS_S3
     else:
-        raise Exception("Persistence storage tests are not enabled or not configured properly")
+        raise Exception("Persistence storage tests are not enabled or not configured properly."
+                        + "ARCTICDB_PERSISTENT_STORAGE_TESTS_ENABLED environment variable is not set")
 
 
 def get_real_uri(shared_path: bool = True):