Release 0.2.0

* autopep changes

* Add REST API Example

* Add error handling on Utils for descriptors

* More explicit, and up to date docstrings for loaders

* Fix Connector

* Update DataLoaders. Add Examples.

* Example 1 : Interaction with pytorch

* Example 2 : classification using alexnet.

* Added a readme with some instruction to run example scripts.

* Better data frame rendetion in sphinx.

* Example 1 and 2 after refactoring common code.

* Example 3 : Similarity Search

* Introduces Changes for ingesting images from kaggle.

* Clean up the requirements file to not include transitive deps.

* Refactored DataHelper to KaggleDataset

* Refactoring of the manner in which data is ingested into aperturedb.

* Reintroduced the Generator + Loader mechanism with depreciation warning.

* Addressed some typos in docs.

* Bugfix based on default args being a mutable object in Utils.

* Add method for counting descriptors on Utils

* Add method for counting descriptors on Utils

* Add testcase for count_descriptors_in_set on Utils

* Better logging and post mortem for data integrity.

* Upgrade protobuf generated file using protoc 3.20.0 (#91)

* Bump up version to 0.2.0 (protobuf change)

* Update requirements.txt

Co-authored-by: Drew Ogle <[email protected]>
Co-authored-by: Gautam Saluja <[email protected]>

Author: 3 people

.github/workflows/checks.yml

@@ -14,19 +14,5 @@ jobs:
       with:
         python-version: '3.8.10'
     - uses: pre-commit/[email protected]
-
     - uses: actions/checkout@v2
     - uses: luisremis/find-trailing-whitespace@master
-
-  trailing-spaces:
-
-    # The type of runner that the job will run on Ubuntu 18.04 (latest)
-    runs-on: ubuntu-latest
-
-    # Steps represent a sequence of tasks that will be
-    # executed as part of the job
-    steps:
-      # Checks-out your repository under $GITHUB_WORKSPACE,
-      # so your job can access it
-      - uses: actions/checkout@v2
-      - uses: luisremis/find-trailing-whitespace@master

.github/workflows/main.yaml

@@ -38,7 +38,6 @@ jobs:
           AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
         run: |
-          pip3 install ipython torch torchvision boto3
           cd test
           bash run_test.sh
 

.gitignore

@@ -132,6 +132,12 @@ dmypy.json
 *.adb.csv
 *.jpg
 *.npy
-env3/
 test/aperturedb/db/
 test/input/blobs/
+docs/examples/
+examples/*/coco
+examples/*/classification.txt
+kaggleds/
+examples/*/kaggleds/
+docs/*/*.svg
+test/aperturedb/logs

README.md

@@ -8,8 +8,32 @@ the open source connector for [VDMS](https://github.com/IntelLabs/vdms).
 It also implements an Object-Mapper API to interact with
 elements in ApertureDB at the object level.
 
-* Status.py provides helper methods to retrieve information about the db.
+* Utils.py provides helper methods to retrieve information about the db.
 * Images.py provides the Object-Mapper for image related objetcs (images, bounding boxes, etc)
 * NotebookHelpers.py provides helpers to show images/bounding boxes on Jupyter Notebooks
 
-For more information, visit https://aperturedata.io
+For more information, visit https://python.docs.aperturedata.io
+
+# Running tests.
+The tests are inside the test dir.
+
+All the tests can be run with:
+
+``bash run_test.sh``
+
+Running specefic tests can be accomplished by invoking it with pytest as follows:
+
+``python -m pytest test_Session.py -v --log-cli-level=DEBUG``
+
+# Reporting bugs.
+Any error in the functionality / documentation / tests maybe reported by creating a
+[github issue](https://github.com/aperture-data/aperturedb-python/issues).
+
+# Development guidelines.
+For inclusion of any features, a PR may be created with a patch,
+and a brief description of the problem and the fix.
+The CI enforces a coding style guideline with autopep8 and
+a script to detect trailing white spaces.
+
+In case a PR encounters failures, the log would describe the location of
+the offending line with a description of the problem.

aperturedb/BBoxDataCSV.py

@@ -0,0 +1,131 @@
+from matplotlib.transforms import Bbox
+from aperturedb import ParallelLoader
+from aperturedb import CSVParser
+
+HEADER_X_POS  = "x_pos"
+HEADER_Y_POS  = "y_pos"
+HEADER_WIDTH  = "width"
+HEADER_HEIGHT = "height"
+IMG_KEY_PROP  = "img_key_prop"
+IMG_KEY_VAL   = "img_key_value"
+
+
+class BBoxDataCSV(CSVParser.CSVParser):
+    """
+    **ApertureDB BBox Data.**
+
+    This class loads the Bounding Box Data which is present in a csv file,
+    and converts it into a series of aperturedb queries.
+
+    .. note::
+        Is backed by a csv file with the following columns:
+
+            ``IMG_KEY``, ``x_pos``, ``y_pos``, ``width``, ``height``, ``BBOX_PROP_NAME_1``, ... ``BBOX_PROP_NAME_N``, ``constraint_BBOX_PROP_NAME_1``
+
+    **IMG_KEY**: column has the property name of the image property that
+    the bounding box will be connected to, and each row has the value
+    that will be used for finding the image.
+
+    **x_pos, y_pos**: Specify the coordinates of top left of the bounding box.
+
+    **width, height**: Specify the dimensions of the bounding box, as integers (unit is in pixels).
+
+    **BBOX_PROP_NAME_N**: is an arbitrary name of the property of the bounding
+    box, and each row has the value for that property.
+
+    **constraint_BBOX_PROP_NAME_1**: Constraints against specific property, used for conditionally adding a Bounding Box.
+
+    Example csv file::
+
+        img_unique_id,x_pos,y_pos,width,height,type,dataset_id,constraint_dataset_id
+        d5b25253-9c1e,257,154,84,125,manual,12345,12345
+        d5b25253-9c1e,7,537,522,282,manual,12346,12346
+        ...
+
+    Example usage:
+
+    .. code-block:: python
+
+        data = BBoxDataCSV("/path/to/BoundingBoxesData.csv")
+        loader = ParallelLoader(db)
+        loader.ingest(data)
+
+
+    .. important::
+        In the above example, the constraint_dataset_id ensures that a bounding box with the specified
+        dataset_id would be only inserted if it does not already exist in the database.
+
+    """
+
+    def __init__(self, filename):
+
+        super().__init__(filename)
+
+        self.props_keys       = [x for x in self.header[5:]
+                                 if not x.startswith(CSVParser.CONTRAINTS_PREFIX)]
+        self.constraints_keys = [x for x in self.header[5:]
+                                 if x.startswith(CSVParser.CONTRAINTS_PREFIX)]
+
+        self.img_key = self.header[0]
+        self.command = "AddBoundingBox"
+
+    def getitem(self, idx):
+
+        val = self.df.loc[idx, self.img_key]
+        box_data_headers = [HEADER_X_POS,
+                            HEADER_Y_POS, HEADER_WIDTH, HEADER_HEIGHT]
+        box_data = [int(self.df.loc[idx, h]) for h in box_data_headers]
+
+        q = []
+
+        ref_counter = idx + 1
+        # TODO we could reuse image references within the batch
+        # instead of creating a new find for every image.
+        img_ref = ref_counter
+        fi = {
+            "FindImage": {
+                "_ref": img_ref,
+            }
+        }
+
+        key = self.img_key
+        val = val
+        constraints = {}
+        constraints[key] = ["==", val]
+        fi["FindImage"]["constraints"] = constraints
+        q.append(fi)
+
+        rect_attrs = ["x", "y", "width", "height"]
+        custom_fields = {
+            "image": img_ref,
+            "rectangle": {
+                attr: val for attr, val in zip(rect_attrs, box_data)
+            },
+        }
+        abb = self._basic_command(idx, custom_fields)
+
+        properties  = self.parse_properties(self.df, idx)
+        if properties:
+            props = properties
+            if "_label" in props:
+                abb[self.command]["label"] = props["_label"]
+                props.pop("_label")
+            # Check if props is not empty after removing "_label"
+            if props:
+                abb[self.command]["properties"] = props
+        q.append(abb)
+
+        return q, []
+
+    def validate(self):
+
+        self.header = list(self.df.columns.values)
+
+        if self.header[1] != HEADER_X_POS:
+            raise Exception("Error with CSV file field: " + HEADER_X_POS)
+        if self.header[2] != HEADER_Y_POS:
+            raise Exception("Error with CSV file field: " + HEADER_Y_POS)
+        if self.header[3] != HEADER_WIDTH:
+            raise Exception("Error with CSV file field: " + HEADER_WIDTH)
+        if self.header[4] != HEADER_HEIGHT:
+            raise Exception("Error with CSV file field: " + HEADER_HEIGHT)

aperturedb/BBoxLoader.py

@@ -10,28 +10,34 @@
 
 
 class BBoxGeneratorCSV(CSVParser.CSVParser):
-    """**ApertureDB BBox Data loader.**
+    """**ApertureDB Bounding Box Data generator.**
+
+    .. warning::
+        Deprecated. Use :class:`~aperturedb.BBoxDataCSV.BBoxDataCSV` instead.
 
     .. note::
-        Expects a csv file with the following columns:
+        Is backed by a csv file with the following columns:
 
-            ``IMG_KEY``, ``x_pos``, ``y_pos``, ``width``, ``height``, ``BBOX_PROP_NAME_1``, ... ``BBOX_PROP_NAME_N``
+            ``IMG_KEY``, ``x_pos``, ``y_pos``, ``width``, ``height``, ``BBOX_PROP_NAME_1``, ... ``BBOX_PROP_NAME_N``, ``constraint_BBOX_PROP_NAME_1``
 
-    IMG_KEY column has the property name of the image property that
+    **IMG_KEY**: column has the property name of the image property that
     the bounding box will be connected to, and each row has the value
     that will be used for finding the image.
 
-    x_pos,y_pos,width,height are the coordinates of the bounding boxes,
-    as integers (unit is in pixels)
+    **x_pos, y_pos**: Specify the coordinates of top left of the bounding box.
+
+    **width, height**: Specify the dimensions of the bounding box, as integers (unit is in pixels).
 
-    BBOX_PROP_NAME_N is an arbitrary name of the property of the bounding
+    **BBOX_PROP_NAME_N**: is an arbitrary name of the property of the bounding
     box, and each row has the value for that property.
 
+    **constraint_BBOX_PROP_NAME_1**: Constraints against specific property, used for conditionally adding a Bounding Box.
+
     Example csv file::
 
-        img_unique_id,x_pos,y_pos,width,height,type
-        d5b25253-9c1e,257,154,84,125,manual
-        d5b25253-9c1e,7,537,522,282,manual
+        img_unique_id,x_pos,y_pos,width,height,type,dataset_id,constraint_dataset_id
+        d5b25253-9c1e,257,154,84,125,manual,12345,12345
+        d5b25253-9c1e,7,537,522,282,manual,12346,12346
         ...
     """
 
@@ -46,7 +52,7 @@ def __init__(self, filename):
 
         self.img_key = self.header[0]
 
-    def __getitem__(self, idx):
+    def getitem(self, idx):
 
         data = {
             "x":      int(self.df.loc[idx, HEADER_X_POS]),
@@ -87,6 +93,32 @@ def validate(self):
 
 
 class BBoxLoader(ParallelLoader.ParallelLoader):
+    """
+    **A loader that helps to ingest Bounding box information against existing images**
+
+    This class facilitates the insertions of Bounding box information connecting them with
+    the images already inserted in the database.
+
+    This executes in conjunction with a **generator** object for example :class:`~aperturedb.BBoxLoader.BBoxGeneratorCSV`,
+    which is a class that implements iterable inteface and generates "bounding box" elements.
+
+    Example::
+
+        bbox_data = {
+            "x": ,
+            "y": ,
+            "width": ,
+            "height":,
+            "properties: {
+                "BBOX_PROP_NAME_1": "BBOX_PROP_VALUE_1",
+                .
+                .
+                .
+                "BBOX_PROP_NAME_N": "BBOX_PROP_VALUE_N"
+            }
+        }
+
+    """
 
     def __init__(self, db, dry_run=False):
 

aperturedb/BlobDataCSV.py

@@ -0,0 +1,91 @@
+import logging
+from aperturedb import CSVParser
+
+PROPERTIES  = "properties"
+CONSTRAINTS = "constraints"
+BLOB_PATH   = "filename"
+
+logger = logging.getLogger(__name__)
+
+
+class BlobDataCSV(CSVParser.CSVParser):
+    """**ApertureDB Blob Data.**
+
+    This class loads the Blob Data which is present in a csv file,
+    and converts it into a series of aperturedb queries.
+
+    .. note::
+        Is backed by a csv file with the following columns:
+
+            ``FILENAME``, ``PROP_NAME_1``, ... ``PROP_NAME_N``, ``constraint_PROP_NAME_1``
+
+    **FILENAME**: The path of the blob object on the file system.
+
+    **PROP_NAME_1 ... PROP_NAME_N**: Arbitrary property names associated with this blob.
+
+    **constraint_PROP_NAME_1**: Constraints against specific property, used for conditionally adding a Blob.
+
+    Example csv file::
+
+        filename,name,lastname,age,id,constraint_id
+        /mnt/blob1,John,Salchi,69,321423532,321423532
+        /mnt/blob2,Johna,Salchi,63,42342522,42342522
+        ...
+
+    Example usage:
+
+    .. code-block:: python
+
+        data = BlobDataCSV("/path/to/BlobData.csv")
+        loader = ParallelLoader(db)
+        loader.ingest(data)
+
+
+
+    .. important::
+        In the above example, the constraint_id ensures that a blob with the specified
+        id would be only inserted if it does not already exist in the database.
+    """
+
+    def __init__(self, filename):
+
+        super().__init__(filename)
+
+        self.props_keys       = [x for x in self.header[1:]
+                                 if not x.startswith(CSVParser.CONTRAINTS_PREFIX) and x != BLOB_PATH]
+        self.constraints_keys = [x for x in self.header[1:]
+                                 if x.startswith(CSVParser.CONTRAINTS_PREFIX)]
+        self.command = "AddBlob"
+
+    def getitem(self, idx):
+        filename = self.df.loc[idx, BLOB_PATH]
+
+        blob_ok, blob = self.load_blob(filename)
+        if not blob_ok:
+            logger.error("Error loading blob: " + filename)
+            raise Exception("Error loading blob: " + filename)
+
+        q = []
+        ab = self._basic_command(idx)
+        q.append(ab)
+
+        return q, [blob]
+
+    def load_blob(self, filename):
+
+        try:
+            fd = open(filename, "rb")
+            buff = fd.read()
+            fd.close()
+            return True, buff
+        except Exception as e:
+            logger.exception(e)
+
+        return False, None
+
+    def validate(self):
+
+        self.header = list(self.df.columns.values)
+
+        if self.header[0] != BLOB_PATH:
+            raise Exception("Error with CSV file field: " + BLOB_PATH)