Merge branch 'datajoint:master' into r013-docs

Author: dimitri-yatsenko

.coveragerc

@@ -3,3 +3,4 @@ branch = False
 source = datajoint
 
 [report]
+show_missing = True

.github/workflows/development.yaml

@@ -15,7 +15,7 @@ jobs:
     strategy:
       matrix:
         py_ver: ["3.8"]
-        mysql_ver: ["8.0", "5.7", "5.6"]
+        mysql_ver: ["8.0", "5.7"]
         include:
           - py_ver: "3.7"
             mysql_ver: "5.7"
@@ -36,11 +36,11 @@ jobs:
       - name: Run primary tests
         env:
           UID: "1001"
-          GID: "116"
+          GID: "121"
           PY_VER: ${{matrix.py_ver}}
           MYSQL_VER: ${{matrix.mysql_ver}}
           ALPINE_VER: "3.10"
-          MINIO_VER: RELEASE.2019-09-26T19-42-35Z
+          MINIO_VER: RELEASE.2021-09-03T03-56-13Z
           COMPOSE_HTTP_TIMEOUT: "120"
           COVERALLS_SERVICE_NAME: travis-ci
           COVERALLS_REPO_TOKEN: fd0BoXG46TPReEem0uMy7BJO5j0w1MQiY

.gitignore

@@ -24,4 +24,5 @@ notebook
 .vscode
 __main__.py
 jupyter_custom.js
-apk_requirements.txt
+apk_requirements.txt
+.eggs

CHANGELOG.md

@@ -1,29 +1,59 @@
 ## Release notes
 
-### 0.13.0 -- Feb 15, 2021
-* Re-implement query transpilation into SQL, fixing issues (#386, #449, #450, #484). PR #754
-* Re-implement cascading deletes for better performance. PR #839.
-* Add table method `.update1` to update a row in the table with new values PR #763
-* Python datatypes are now enabled by default in blobs (#761). PR #785
+### 0.13.3 -- TBD
+* Bugfix - Dependencies not properly loaded on populate. (#902) PR #919
+* Bugfix - Replace use of numpy aliases of built-in types with built-in type. (#938) PR #939
+* Bugfix - `ExternalTable.delete` should not remove row on error (#953) PR #956
+* Bugfix - Fix error handling of remove_object function in `s3.py` (#952) PR #955
+
+### 0.13.2 -- May 7, 2021
+* Update `setuptools_certificate` dependency to new name `otumat`
+* Bugfix - Explicit calls to `dj.Connection` throw error due to missing `host_input` (#895) PR #907
+* Bugfix - Correct count of deleted items. (#897) PR #912
+
+### 0.13.1 -- Apr 16, 2021
+* Add `None` as an alias for `IS NULL` comparison in `dict` restrictions (#824) PR #893
+* Drop support for MySQL 5.6 since it has reached EOL PR #893
+* Bugfix - `schema.list_tables()` is not topologically sorted (#838) PR #893
+* Bugfix - Diagram part tables do not show proper class name (#882) PR #893
+* Bugfix - Error in complex restrictions (#892) PR #893
+* Bugfix - WHERE and GROUP BY clases are dropped on joins with aggregation (#898, #899) PR #893
+
+### 0.13.0 -- Mar 24, 2021
+* Re-implement query transpilation into SQL, fixing issues (#386, #449, #450, #484, #558). PR #754
+* Re-implement cascading deletes for better performance. PR #839
+* Add support for deferred schema activation to allow for greater modularity. (#834) PR #839
+* Add query caching mechanism for offline development (#550) PR #839
+* Add table method `.update1` to update a row in the table with new values (#867) PR #763, #889
+* Python datatypes are now enabled by default in blobs (#761). PR #859
 * Added permissive join and restriction operators `@` and `^` (#785) PR #754
 * Support DataJoint datatype and connection plugins (#715, #729) PR 730, #735
-* add `dj.key_hash` alias to `dj.hash.key_hash`
-* default enable_python_native_blobs to True
-* Drop support for Python 3.5
+* Add `dj.key_hash` alias to `dj.hash.key_hash` (#804) PR #862
+* Default enable_python_native_blobs to True
+* Bugfix - Regression error on joins with same attribute name (#857) PR #878
+* Bugfix - Error when `fetch1('KEY')` when `dj.config['fetch_format']='frame'` set (#876) PR #880, #878
+* Bugfix - Error when cascading deletes in tables with many, complex keys (#883, #886) PR #839
+* Add deprecation warning for `_update`. PR #889
+* Add `purge_query_cache` utility. PR #889
+* Add tests for query caching and permissive join and restriction. PR #889
+* Drop support for Python 3.5 (#829) PR #861
+
+### 0.12.9 -- Mar 12, 2021
+* Fix bug with fetch1 with `dj.config['fetch_format']="frame"`. (#876) PR #880
 
 ### 0.12.8 -- Jan 12, 2021
 * table.children, .parents, .descendents, and ancestors can return queryable objects. PR #833
 * Load dependencies before querying dependencies. (#179) PR #833
 * Fix display of part tables in `schema.save`. (#821) PR #833
 * Add `schema.list_tables`. (#838) PR #844
 * Fix minio new version regression.  PR #847
-* Add more S3 logging for debugging. (#831) PR #832 
+* Add more S3 logging for debugging. (#831) PR #832
 * Convert testing framework from TravisCI to GitHub Actions (#841) PR #840
- 
+
 ### 0.12.7 -- Oct 27, 2020
 * Fix case sensitivity issues to adapt to MySQL 8+.  PR #819
 * Fix pymysql regression bug (#814) PR #816
-* Adapted attribute types now have dtype=object in all recarray results. PR #811 
+* Adapted attribute types now have dtype=object in all recarray results. PR #811
 
 ### 0.12.6 -- May 15, 2020
 * Add `order_by` to `dj.kill` (#668, #779) PR #775, #783
@@ -115,9 +145,9 @@
 * Bugfix in restriction of the form (A & B) * B (#463)
 * Improved error messages (#466)
 
-### 0.10.0 -- Jan 10, 2018 
+### 0.10.0 -- Jan 10, 2018
 * Deletes are more efficient (#424)
-* ERD shows table definition on tooltip hover in Jupyter (#422) 
+* ERD shows table definition on tooltip hover in Jupyter (#422)
 * S3 external storage
 * Garbage collection for external sorage
 * Most operators and methods of tables can be invoked as class methods rather than instance methods (#407)
@@ -131,7 +161,7 @@
 * Implement union operator +
 * Implement file-based external storage
 
-### 0.8.0 -- Jul 26, 2017 
+### 0.8.0 -- Jul 26, 2017
 Documentation and tutorials available at https://docs.datajoint.io and https://tutorials.datajoint.io
 * improved the ERD graphics and features using the graphviz libraries (#207, #333)
 * improved password handling logic (#322, #321)
@@ -150,11 +180,11 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 * Added `dj.create_virtual_module`
 
 ### 0.4.10 (#286) -- Feb 6, 2017
-* Removed Vagrant and Readthedocs support 
+* Removed Vagrant and Readthedocs support
 * Explicit saving of configuration (issue #284)
 
 ### 0.4.9 (#285) -- Feb 2, 2017
-* Fixed setup.py for pip install 
+* Fixed setup.py for pip install
 
 ### 0.4.7 (#281) -- Jan 24, 2017
 * Fixed issues related to order of attributes in projection.
@@ -183,10 +213,10 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 
 ### 0.3.8  -- Aug 2, 2016
 * added the `_update` method in `base_relation`. It allows updating values in existing tuples.
-* bugfix in reading values of type double.  Previously it was cast as float32. 
+* bugfix in reading values of type double.  Previously it was cast as float32.
 
 ### 0.3.7  -- Jul 31, 2016
-* added parameter `ignore_extra_fields` in `insert` 
+* added parameter `ignore_extra_fields` in `insert`
 * `insert(..., skip_duplicates=True)` now relies on `SELECT IGNORE`.  Previously it explicitly checked if tuple already exists.
 * table previews now include blob attributes displaying the string <BLOB>
 

LNX-docker-compose.yml

@@ -1,4 +1,4 @@
-# docker-compose -f LNX-docker-compose.yml --env-file LNX.env up --build --exit-code-from app
+# docker-compose -f LNX-docker-compose.yml --env-file LNX.env up --exit-code-from app --build
 version: '2.2'
 x-net: &net
   networks:
@@ -32,7 +32,7 @@ services:
       interval: 1s
   fakeservices.datajoint.io:
     <<: *net
-    image: raphaelguzman/nginx:v0.0.13
+    image: datajoint/nginx:v0.0.18
     environment:
     - ADD_db_TYPE=DATABASE
     - ADD_db_ENDPOINT=db:3306
@@ -72,15 +72,17 @@ services:
       - COVERALLS_SERVICE_NAME
       - COVERALLS_REPO_TOKEN
     working_dir: /src
-    command: >
-      /bin/sh -c
-       "
-        pip install --user -r test_requirements.txt;
-        pip install --user .;
-        pip freeze | grep datajoint;
-        nosetests -vsw tests --with-coverage --cover-package=datajoint && coveralls;
-        # jupyter notebook;
-       "
+    command:
+      - sh
+      - -c
+      - |
+        set -e
+        pip install --user -r test_requirements.txt
+        pip install -e .
+        pip freeze | grep datajoint
+        nosetests -vsw tests --with-coverage --cover-package=datajoint
+        coveralls
+        # jupyter notebook
     # ports:
     #   - "8888:8888"
     user: ${UID}:${GID}

README.md

@@ -108,7 +108,7 @@ A number of labs are currently adopting DataJoint and we are quickly getting the
 PY_VER=3.7
 ALPINE_VER=3.10
 MYSQL_VER=5.7
-MINIO_VER=RELEASE.2019-09-26T19-42-35Z
+MINIO_VER=RELEASE.2021-09-03T03-56-13Z
 UID=1000
 GID=1000
 ```
@@ -136,6 +136,8 @@ GID=1000
 * Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
 
 
+
+
 ### Launch Jupyter Notebook for Interactive Use
 * Navigate to `localhost:8888`
 * Input Jupyter password

datajoint/autopopulate.py

@@ -39,8 +39,8 @@ def _rename_attributes(table, props):
         if self._key_source is None:
             parents = self.target.parents(primary=True, as_objects=True, foreign_key_info=True)
             if not parents:
-                raise DataJointError(
-                    'A relation must have primary dependencies for auto-populate to work')
+                raise DataJointError('A table must have dependencies '
+                                     'from its primary key for auto-populate to work')
             self._key_source = _rename_attributes(*parents[0])
             for q in parents[1:]:
                 self._key_source *= _rename_attributes(*q)

datajoint/blob.py

@@ -164,7 +164,7 @@ def pack_blob(self, obj):
             return self.pack_recarray(np.array(obj))
         if isinstance(obj, np.number):
             return self.pack_array(np.array(obj))
-        if isinstance(obj, (np.bool, np.bool_)):
+        if isinstance(obj, (bool, np.bool_)):
             return self.pack_array(np.array(obj))
         if isinstance(obj, (datetime.datetime, datetime.date, datetime.time)):
             return self.pack_datetime(obj)
@@ -365,7 +365,7 @@ def read_struct(self):
         raw_data = [
             tuple(self.read_blob(n_bytes=int(self.read_value('uint64'))) for _ in range(n_fields))
             for __ in range(n_elem)]
-        data = np.array(raw_data, dtype=list(zip(field_names, repeat(np.object))))
+        data = np.array(raw_data, dtype=list(zip(field_names, repeat(object))))
         return self.squeeze(data.reshape(shape, order="F"), convert_to_scalar=False).view(MatStruct)
 
     def pack_struct(self, array):

datajoint/condition.py

@@ -21,8 +21,9 @@ def __init__(self, operand):
 
 class AndList(list):
     """
-    A list of conditions to by applied to a query expression by logical conjunction: the conditions are AND-ed.
-    All other collections (lists, sets, other entity sets, etc) are applied by logical disjunction (OR).
+    A list of conditions to by applied to a query expression by logical conjunction: the
+    conditions are AND-ed. All other collections (lists, sets, other entity sets, etc) are
+    applied by logical disjunction (OR).
 
     Example:
     expr2 = expr & dj.AndList((cond1, cond2, cond3))
@@ -49,14 +50,16 @@ def assert_join_compatibility(expr1, expr2):
     the matching attributes in the two expressions must be in the primary key of one or the
     other expression.
     Raises an exception if not compatible.
+
     :param expr1: A QueryExpression object
     :param expr2: A QueryExpression object
     """
     from .expression import QueryExpression, U
 
     for rel in (expr1, expr2):
         if not isinstance(rel, (U, QueryExpression)):
-            raise DataJointError('Object %r is not a QueryExpression and cannot be joined.' % rel)
+            raise DataJointError(
+                'Object %r is not a QueryExpression and cannot be joined.' % rel)
     if not isinstance(expr1, U) and not isinstance(expr2, U):  # dj.U is always compatible
         try:
             raise DataJointError(
@@ -70,9 +73,11 @@ def assert_join_compatibility(expr1, expr2):
 def make_condition(query_expression, condition, columns):
     """
     Translate the input condition into the equivalent SQL condition (a string)
+
     :param query_expression: a dj.QueryExpression object to apply condition
     :param condition: any valid restriction object.
-    :param columns: a set passed by reference to collect all column names used in the condition.
+    :param columns: a set passed by reference to collect all column names used in the
+        condition.
     :return: an SQL condition string or a boolean value.
     """
     from .expression import QueryExpression, Aggregation, U
@@ -84,7 +89,8 @@ def prep_value(k, v):
                 try:
                     v = uuid.UUID(v)
                 except (AttributeError, ValueError):
-                    raise DataJointError('Badly formed UUID {v} in restriction by `{k}`'.format(k=k, v=v)) from None
+                    raise DataJointError(
+                        'Badly formed UUID {v} in restriction by `{k}`'.format(k=k, v=v))
             return "X'%s'" % v.bytes.hex()
         if isinstance(v, (datetime.date, datetime.datetime, datetime.time, decimal.Decimal)):
             return '"%s"' % v
@@ -101,12 +107,13 @@ def prep_value(k, v):
     # restrict by string
     if isinstance(condition, str):
         columns.update(extract_column_names(condition))
-        return template % condition.strip().replace("%", "%%")  # escape % in strings, see issue #376
+        return template % condition.strip().replace("%", "%%")  # escape %, see issue #376
 
     # restrict by AndList
     if isinstance(condition, AndList):
         # omit all conditions that evaluate to True
-        items = [item for item in (make_condition(query_expression, cond, columns) for cond in condition)
+        items = [item for item in (make_condition(query_expression, cond, columns)
+                 for cond in condition)
                  if item is not True]
         if any(item is False for item in items):
             return negate  # if any item is False, the whole thing is False
@@ -122,18 +129,21 @@ def prep_value(k, v):
     if isinstance(condition, bool):
         return negate != condition
 
-    # restrict by a mapping such as a dict -- convert to an AndList of string equality conditions
+    # restrict by a mapping/dict -- convert to an AndList of string equality conditions
     if isinstance(condition, collections.abc.Mapping):
         common_attributes = set(condition).intersection(query_expression.heading.names)
         if not common_attributes:
             return not negate  # no matching attributes -> evaluates to True
         columns.update(common_attributes)
         return template % ('(' + ') AND ('.join(
-            '`%s`=%s' % (k, prep_value(k, condition[k])) for k in common_attributes) + ')')
+            '`%s`%s' % (k, ' IS NULL' if condition[k] is None
+                        else f'={prep_value(k, condition[k])}')
+            for k in common_attributes) + ')')
 
     # restrict by a numpy record -- convert to an AndList of string equality conditions
     if isinstance(condition, numpy.void):
-        common_attributes = set(condition.dtype.fields).intersection(query_expression.heading.names)
+        common_attributes = set(condition.dtype.fields).intersection(
+            query_expression.heading.names)
         if not common_attributes:
             return not negate   # no matching attributes -> evaluate to True
         columns.update(common_attributes)
@@ -153,7 +163,8 @@ def prep_value(k, v):
     if isinstance(condition, QueryExpression):
         if check_compatibility:
             assert_join_compatibility(query_expression, condition)
-        common_attributes = [q for q in condition.heading.names if q in query_expression.heading.names]
+        common_attributes = [q for q in condition.heading.names
+                             if q in query_expression.heading.names]
         columns.update(common_attributes)
         if isinstance(condition, Aggregation):
             condition = condition.make_subquery()
@@ -175,15 +186,17 @@ def prep_value(k, v):
     except TypeError:
         raise DataJointError('Invalid restriction type %r' % condition)
     else:
-        or_list = [item for item in or_list if item is not False]  # ignore all False conditions
-        if any(item is True for item in or_list):  # if any item is True, the whole thing is True
+        or_list = [item for item in or_list if item is not False]  # ignore False conditions
+        if any(item is True for item in or_list):  # if any item is True, entirely True
             return not negate
-        return template % ('(%s)' % ' OR '.join(or_list)) if or_list else negate  # an empty or list is False
+        return template % ('(%s)' % ' OR '.join(or_list)) if or_list else negate
 
 
 def extract_column_names(sql_expression):
     """
-    extract all presumed column names from an sql expression such as the WHERE clause, for example.
+    extract all presumed column names from an sql expression such as the WHERE clause,
+    for example.
+
     :param sql_expression: a string containing an SQL expression
     :return: set of extracted column names
     This may be MySQL-specific for now.
@@ -205,5 +218,8 @@ def extract_column_names(sql_expression):
     s = re.sub(r"(\b[a-z][a-z_0-9]*)\(", "(", s)
     remaining_tokens = set(re.findall(r"\b[a-z][a-z_0-9]*\b", s))
     # update result removing reserved words
-    result.update(remaining_tokens - {"is", "in", "between", "like", "and", "or", "null", "not"})
+    result.update(remaining_tokens - {"is", "in", "between", "like", "and", "or", "null",
+                                      "not", "interval", "second", "minute", "hour", "day",
+                                      "month", "week", "year"
+                                      })
     return result