Merge pull request #579 from aperture-data/release-0.4.45

gsaluja9 · web-flow · commit 38483d7a6560 · 2025-04-04T14:22:35.000-04:00
Release 0.4.45
diff --git a/.github/workflows/develop.yml b/.github/workflows/develop.yml
@@ -61,6 +61,11 @@ jobs:
         AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
         AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
         AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        RUNNER_NAME: ${{ runner.name }}
+        ADB_REPO: aperturedata/aperturedb
+        ADB_TAG: vci_develop
+        LENZ_REPO: aperturedata/lenz
+        LENZ_TAG: vci_develop
       run: BUILD_COMPLETE=true ./ci.sh
       shell: bash
 
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -61,6 +61,11 @@ jobs:
         AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
         AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
         AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        RUNNER_NAME: ${{ runner.name }}
+        ADB_REPO: aperturedata/aperturedb
+        ADB_TAG: vci_develop
+        LENZ_REPO: aperturedata/lenz
+        LENZ_TAG: vci_develop
       run: BUILD_COMPLETE=true ./ci.sh
       shell: bash
 
diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml
@@ -35,5 +35,14 @@ jobs:
         AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
         AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
         GCP_SERVICE_ACCOUNT_KEY: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
-      run: RUN_TESTS=true NO_PUSH=true BRANCH_NAME=${{ github.event.pull_request.head.ref }} TARGET_BRANCH_NAME=${{ github.event.pull_request.base.ref }} ./ci.sh
+        RUNNER_NAME: ${{ runner.name }}
+        RUN_TESTS: true
+        NO_PUSH: true
+        BRANCH_NAME: ${{ github.event.pull_request.head.ref }}
+        TARGET_BRANCH_NAME: ${{ github.event.pull_request.base.ref }}
+        ADB_REPO: aperturedata/aperturedb
+        ADB_TAG: vci_develop
+        LENZ_REPO: aperturedata/lenz
+        LENZ_TAG: vci_develop
+      run: ./ci.sh
       shell: bash
diff --git a/.gitignore b/.gitignore
@@ -153,3 +153,4 @@ docker/pytorch-gpu/aperturedata/*
 
 .aperturedb
 test/data/
+test/aperturedb/certificate/
diff --git a/README.md b/README.md
@@ -18,7 +18,7 @@ or an installation with only the core part of the SDK
 pip install aperturedb
 ```
 
-A complete [reference](https://docs.aperturedata.io/category/aperturedb-python-sdk) of this SDK is available on the offical [ApertureDB Documentation](https://docs.aperturedata.io)
+A complete [reference](https://docs.aperturedata.io/category/aperturedb-python-sdk) of this SDK is available on the official [ApertureDB Documentation](https://docs.aperturedata.io)
 
 
 # Development setup
@@ -31,7 +31,34 @@ pip install -e .[dev]
 
 
 # Running tests
-The tests are inside the `test` dir.
+The tests are inside the `test` dir. Currently these get run in Linux container. Refer to `docker/tests` and `test/run_test_container` for details. Following explanation assumes that the current working directory is `test`.
+
+The tests bring up a set of components in an isolated network, namely:
+- aperturedb-community
+- lenz
+- nginx
+- ca (for initial provisioning of certificates)
+- webui
+
+
+To connect to this setup, the ports are exposed to the host as follows:
+- 55556 for TCP connection to aperturedb (via lenz).
+- 8087 for HTTP connection to aperturedb (via nginx).
+
+
+
+This can be done manually as:
+```bash
+docker compose up -d
+```
+
+## Changes to run the tests in development environment.
+Edit the file `test/dbinfo.py` to loook like the following.
+- DB_TCP_HOST = `localhost`
+- DB_REST_HOST = `localhost`
+- DB_TCP_PORT  = `55556`
+- DB_REST_PORT = `8087`
+
 
 All the tests can be run with:
 
@@ -43,9 +70,23 @@ bash run_test.sh
 Running specific tests can be accomplished by invoking it with pytest as follows:
 
 ```bash
-cd test && docker compose up -d && PROJECT=aperturedata KAGGLE_username=ci KAGGLE_key=dummy coverage run -m pytest test_Session.py -v --log-cli-level=DEBUG
+PROJECT=aperturedata KAGGLE_username=ci KAGGLE_key=dummy coverage run -m pytest test_Session.py -v -s --log-cli-level=DEBUG
 ```
 
+**NOTE:The running environment is assumed to be Linux x86_64. There might be certain changes required for them to be run on MacOS or Windows python environments.**
+
+## Certain Environment variables that affect the runtime beaviour of the SDK.
+
+These can be used as debugging aids.
+
+| Variable | type | Comments | Default value |
+| --- | --- | --- | --- |
+|ADB_DEBUGGABLE | boolean | allows the application to register a fault handler that dumps a trace when SIGUSR1 is sent to the process | not set |
+|LOG_FILE_LEVEL |  <a href="https://docs.python.org/3/library/logging.html#logging-levels">log levels</a> | The threshold for emitting log messages into the error<timestamp>.log file | WARN |
+|LOG_CONSOLE_LEVEL | <a href="https://docs.python.org/3/library/logging.html#logging-levels">log levels</a> | The threshold for emitting log messages into stdout | ERROR |
+|ADB_LOG_FILE | string | custom file path for the LOG file | not set|
+
+
 # Reporting bugs
 Any error in the functionality / documentation / tests maybe reported by creating a
 [github issue](https://github.com/aperture-data/aperturedb-python/issues).
diff --git a/aperturedb/CommonLibrary.py b/aperturedb/CommonLibrary.py
@@ -153,18 +153,20 @@ def _store_config(config: Configuration, name: str):
 
 def create_connector(
     name: Optional[str] = None,
+    key: Optional[str] = None,
     create_config_for_colab_secret=True,
 ) -> Connector:
     """
     **Create a connector to the database.**
 
     This function chooses a configuration in the folowing order:
-    1. The configuration named by the `name` parameter.
-    2. The configuration described in the `APERTUREDB_JSON` environment variable.
-    3. The configuration described in the `APERTUREDB_JSON` Google Colab secret.
-    4. The configuration described in the `APERTUREDB_JSON` secret in a `.env` file.
-    5. The configuration named by the `APERTUREDB_CONFIG` environment variable.
-    6. The active configuration.
+    1. The configuration named by the `name` parameter or `key` parameter
+    2. The configuration described in the `APERTUREDB_KEY` environment variable.
+    3. The configuration described in the `APERTUREDB_JSON` environment variable.
+    4. The configuration described in the `APERTUREDB_JSON` Google Colab secret.
+    5. The configuration described in the `APERTUREDB_JSON` secret in a `.env` file.
+    6. The configuration named by the `APERTUREDB_CONFIG` environment variable.
+    7. The active configuration.
 
     If there are both global and local configurations with the same name, the global configuration is preferred.
 
@@ -197,9 +199,19 @@ def lookup_config_by_name(name: str, source: str) -> Configuration:
             return configs["local"][name]
         assert False, f"Configuration '{name}' not found ({source})."
 
-    if name is not None:
+    if key is not None:
+        if name is not None:
+            raise ValueError(
+                "Specify only name or key when creating a connector")
+        logger.info(f"Using configuration from key parameter")
+        config = Configuration.reinflate(key)
+    elif name is not None:
         logger.info(f"Using configuration '{name}' explicitly")
         config = lookup_config_by_name(name, "explicit")
+    elif (data := os.environ.get("APERTUREDB_KEY")) is not None and data != "":
+        logger.info(
+            f"Using configuration from APERTUREDB_KEY environment variable")
+        config = Configuration.reinflate(data)
     elif (data := os.environ.get("APERTUREDB_JSON")) is not None and data != "":
         logger.info(
             f"Using configuration from APERTUREDB_JSON environment variable")
diff --git a/aperturedb/Configuration.py b/aperturedb/Configuration.py
@@ -1,5 +1,11 @@
 from dataclasses import dataclass
 
+import json
+from base64 import b64encode, b64decode
+
+APERTURE_CLOUD = ".cloud.aperturedata.io"
+AD_CLOUD_SCHEME = "adbc://"
+
 
 @dataclass(repr=False)
 class Configuration:
@@ -22,3 +28,35 @@ class Configuration:
     def __repr__(self) -> str:
         mode = "REST" if self.use_rest else "TCP"
         return f"[{self.host}:{self.port} as {self.username} using {mode} with SSL={self.use_ssl}]"
+
+    def deflate(self) -> list:
+        deflate_version = 1
+        host = self.host
+        if host.endswith(APERTURE_CLOUD):
+            host = "adbc://{}".format(host[:-1 * len(APERTURE_CLOUD)])
+        as_list = [deflate_version, host, self.port, self.username, self.password, self.name, int(self.use_ssl),
+                   int(self.use_rest), int(self.use_keepalive),
+                   self.retry_interval_seconds, self.retry_max_attempts]
+        simplified = json.dumps(as_list)
+        encoded_key = b64encode(simplified.encode('utf-8')).decode('utf-8')
+        return encoded_key
+
+    @classmethod
+    def reinflate(cls, encoded_key: list) -> object:
+        decoded_key = b64decode(encoded_key.encode('utf-8'))
+        as_list = json.loads(decoded_key.decode('utf-8'))
+        if as_list[0] != 1:
+            raise ValueError(f"version identifier of configuration was"
+                             "{as_list[0]}, which is not supported")
+        host, port, username, password, name, use_ssl, \
+            use_rest, use_keepalive, retry_interval_seconds, \
+            retry_max_attempts = as_list[1:]
+        if host.startswith(AD_CLOUD_SCHEME):
+            host = host[len(AD_CLOUD_SCHEME):] + APERTURE_CLOUD
+        use_ssl = bool(use_ssl)
+        use_rest = bool(use_rest)
+        use_keepaliave = bool(use_keepalive)
+        c = Configuration(
+            host, port, username, password, name, use_ssl, use_rest, use_keepalive,
+            retry_interval_seconds, retry_max_attempts)
+        return c
diff --git a/aperturedb/Connector.py b/aperturedb/Connector.py
@@ -103,6 +103,7 @@ class Connector(object):
             and it means that disconnections are detected sooner.
             Turn this off to reduce traffic on high-cost network connections.
         Configuration (config): Configuration object to use for connection.
+        str (key): Apeture Key, configuration as a deflated compressed string
     """
 
     def __init__(self, host="localhost", port=55555,
@@ -113,7 +114,8 @@ def __init__(self, host="localhost", port=55555,
                  use_keepalive=True,
                  retry_interval_seconds=1,
                  retry_max_attempts=3,
-                 config: Optional[Configuration] = None):
+                 config: Optional[Configuration] = None,
+                 key: Optional[str] = None):
         """
         Constructor for the Connector class.
         """
@@ -126,7 +128,9 @@ def __init__(self, host="localhost", port=55555,
         # after the last query
         self.query_connection_error_suppression_delta = timedelta(seconds=30)
 
-        if config is None:
+        if key is not None:
+            self.config = Configuration.reinflate(key)
+        elif config is None:
             self.host = host
             self.port = port
             self.use_ssl = use_ssl
diff --git a/aperturedb/ConnectorRest.py b/aperturedb/ConnectorRest.py
@@ -32,6 +32,7 @@
 import logging
 
 from types import SimpleNamespace
+from typing import Optional
 from aperturedb.Connector import Connector
 from aperturedb.Configuration import Configuration
 
@@ -53,12 +54,14 @@ class ConnectorRest(Connector):
         str (password): Password to specify while connecting to ApertureDB.
         str (token): Token to use while connecting to the database.
         bool (use_ssl): Use SSL to encrypt communication with the database.
+        str (key): Apeture Key, configuration as a deflated compressed string
     """
 
     def __init__(self, host="localhost", port=None,
                  user="", password="", token="",
                  use_ssl=True, shared_data=None,
-                 config: Configuration = None):
+                 config: Optional[Configuration] = None,
+                 key: Optional[str] = None):
         self.use_keepalive = False
         super().__init__(
             host=host,
@@ -68,7 +71,8 @@ def __init__(self, host="localhost", port=None,
             token=token,
             use_ssl=use_ssl,
             shared_data=shared_data,
-            config=config)
+            config=config,
+            key=key)
 
         # A Convenience feature to not require the port
         # Relies on common ports for http and https, but can be overriden
diff --git a/aperturedb/Utils.py b/aperturedb/Utils.py
@@ -910,8 +910,8 @@ def user_log_message(self, message: str, level: str = "INFO") -> None:
         self.execute(q)
 
 
-def create_connector():
+def create_connector(name=None, key=None):
     from aperturedb.CommonLibrary import create_connector, issue_deprecation_warning
     issue_deprecation_warning("Utils.create_connector",
                               "CommonLibrary.create_connector")
-    return create_connector()
+    return create_connector(name=name, key=key)
diff --git a/aperturedb/__init__.py b/aperturedb/__init__.py
@@ -10,7 +10,7 @@
 import signal
 import sys
 
-__version__ = "0.4.44"
+__version__ = "0.4.45"
 
 logger = logging.getLogger(__name__)
 
diff --git a/aperturedb/cli/configure.py b/aperturedb/cli/configure.py
@@ -1,6 +1,5 @@
 import json
 import os
-from json import JSONEncoder
 from pathlib import Path
 
 import typer
@@ -12,7 +11,7 @@
 from aperturedb.CommonLibrary import _create_configuration_from_json
 
 
-class ObjEncoder(JSONEncoder):
+class ObjEncoder(json.JSONEncoder):
     """
     A bit of boiler plate to allow us to serialize our Configuration object.
     """
@@ -124,11 +123,13 @@ def create(
             help="Interactive mode")] = True,
         overwrite: Annotated[bool, typer.Option(
             help="overwrite existing configuration")] = False,
-        from_json: Annotated[bool, typer.Option(help="create config from a JSON string")] = False):
+        from_json: Annotated[bool, typer.Option(
+            help="create config from a JSON string")] = False,
+        from_key: Annotated[bool, typer.Option(help="create config from an encoded string")] = False):
     """
     Create a new configuration for the client.
 
-    If --from-json is used, then the options --host, --port, --username, --password, --use-rest, and --use-ssl will be ignored.
+    If --from-json or --from-key is used, then the options --host, --port, --username, --password, --use-rest, and --use-ssl will be ignored.
     The user will be prompted to enter the JSON string.
     This will be treated as a password entry.
 
@@ -172,6 +173,12 @@ def check_for_overwrite(name):
             json_str, name=name, name_required=True)
         check_for_overwrite(gen_config.name)
         name = gen_config.name
+    elif from_key:
+        assert interactive, "Interactive mode must be enabled for --from-key"
+        encoded_str = typer.prompt("Enter encoded string", hide_input=True)
+        gen_config = Configuration.reinflate(encoded_str)
+        name = gen_config.name
+
     else:
         if interactive:
             if name is None:
@@ -240,3 +247,33 @@ def activate(
 
     with open(config_path.as_posix(), "w") as config_file:
         config_file.write(json.dumps(configs, indent=2, cls=ObjEncoder))
+
+
+@app.command()
+def tokenize(name: Annotated[str, typer.Argument(
+        help="Name of the configuration to tokenize")],
+        as_global: Annotated[bool, typer.Option(help="Project level vs global level")] = True):
+    """
+    Makes a token from the configuration
+    """
+    global_config_path = _config_file_path(True)
+    gc, ga = get_configurations(global_config_path)
+
+    config_path = _config_file_path(as_global)
+    configs = {}
+    to_tokenize = None
+    try:
+        configs, ac = get_configurations(config_path.as_posix())
+        if name not in configs and name not in gc:
+            console.log(f"Configuration {name} not found")
+            raise typer.Exit(code=2)
+        to_tokenize = configs[name]
+    except FileNotFoundError:
+        check_configured(as_global=False) or \
+            check_configured(as_global=True, show_error=True)
+    except json.JSONDecodeError:
+        check_configured(as_global=False) or \
+            check_configured(as_global=True, show_error=True)
+
+    print("{} Encoded: {}"
+          .format(to_tokenize.name,  to_tokenize.deflate()))
diff --git a/ci.sh b/ci.sh
@@ -99,7 +99,7 @@ echo "Repository: ${DOCKER_REPOSITORY}"
 build_tests(){
     TESTS_IMAGE=${DOCKER_REPOSITORY}/aperturedb-python-tests:latest
     mkdir -p docker/tests/aperturedata
-    sudo rm -rf test/aperturedb/db
+    sudo rm -rf test/aperturedb/{db,logs}
     cp -r aperturedb pyproject.toml README.md docker/tests/aperturedata
     mkdir -m 777 -p docker/tests/aperturedata/test/aperturedb
     cp -r test/*.py test/*.sh test/input docker/tests/aperturedata/test
diff --git a/docker/dependencies/Dockerfile b/docker/dependencies/Dockerfile
@@ -8,7 +8,7 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends \
        python3-venv
 
-ENV DEBIAN_FRONTEND noninteractive
+ENV DEBIAN_FRONTEND=noninteractive
 ENV VIRTUAL_ENV=/opt/venv
 ENV NODEJS_LTS=v20.12.2
 RUN python3 -m venv $VIRTUAL_ENV
diff --git a/pyproject.toml b/pyproject.toml
@@ -19,7 +19,7 @@ dependencies = [
     'protobuf >=3.20.3,<6.0.0',
     #Folllowing is needed parallel loaders, and basic things for
     # making the notebooks.
-    'scikit-image', 'image', 'requests', 'boto3',
+    'requests', 'boto3',
     # https://github.com/Kaggle/kaggle-api/issues/611
     'numpy<2; python_version<"3.9.0"', 'numpy', 'distributed',
     'matplotlib', 'pandas', 'kaggle!=1.6.15', 'google-cloud-storage',
diff --git a/test/.env b/test/.env
diff --git a/test/dbinfo.py b/test/dbinfo.py
diff --git a/test/docker-compose.yml b/test/docker-compose.yml
diff --git a/test/run_test.sh b/test/run_test.sh
diff --git a/test/run_test_container.sh b/test/run_test_container.sh

Original file line number	Diff line number	Diff line change
`@@ -153,3 +153,4 @@ docker/pytorch-gpu/aperturedata/*`
`153`	`153`
`154`	`154`	`.aperturedb`
`155`	`155`	`test/data/`
	`156`	`+test/aperturedb/certificate/`