Merge branch 'master' of https://github.com/datajoint/datajoint-python into switch-to-djtest

Author: A-Baji

.github/workflows/development.yaml

@@ -9,6 +9,23 @@ on:
       - '**' # every branch
       - '!stage*' # exclude branches beginning with stage
 jobs:
+  build-docs:
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLIENT_TIMEOUT: "120"
+      COMPOSE_HTTP_TIMEOUT: "120"
+    steps:
+      - uses: actions/checkout@v2
+      - name: Compile docs static artifacts
+        run: |
+          export HOST_UID=$(id -u)
+          docker-compose -f ./docs-api/docker-compose.yaml up --exit-code-from docs-builder --build 
+      - name: Add docs static artifacts
+        uses: actions/upload-artifact@v2
+        with:
+          name: docs-api-static
+          path: docs-api/build/html
+          retention-days: 1
   test:
     if: github.event_name == 'push' || github.event_name == 'pull_request'
     runs-on: ubuntu-latest
@@ -52,3 +69,42 @@ jobs:
                  --count --max-complexity=62 --max-line-length=127 --statistics
           black datajoint --check -v
           black tests --check -v
+  publish-docs:
+    if: |
+      github.event_name == 'push' &&
+      (
+        github.repository_owner == 'datajoint' ||
+        github.repository_owner == 'datajoint-company' ||
+        github.repository_owner == 'dj-sciops'
+      )
+    needs: build-docs
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Fetch docs static artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: docs-api-static
+          path: docs-api/build/html
+      - name: Commit documentation changes
+        run: |
+          git clone https://github.com/${GITHUB_REPOSITORY}.git \
+              --branch gh-pages --single-branch gh-pages
+          rm -R gh-pages/*
+          cp -r docs-api/build/html/* gh-pages/
+          cp .gitignore gh-pages/
+          touch gh-pages/.nojekyll
+          echo "docs-api.datajoint.org" > gh-pages/CNAME
+          cd gh-pages
+          git config --local user.email "[email protected]"
+          git config --local user.name "GitHub Action"
+          git add . --all
+          git commit -m "Update documentation" -a || true
+          # The above command will fail if no changes were present, so we ignore
+          # the return code.
+      - name: Push changes
+        uses: ad-m/github-push-action@master
+        with:
+          branch: gh-pages
+          directory: gh-pages
+          github_token: ${{secrets.GITHUB_TOKEN}}

README.md

@@ -9,7 +9,7 @@
 DataJoint for Python is a framework for scientific workflow management based on relational principles. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, computing, and querying data.
 
 DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
-Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com). Related resources are listed at https://datajoint.org
+Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com). Related resources are listed at https://datajoint.org.
 
 ## Installation
 ```
@@ -29,6 +29,13 @@ pip3 install --upgrade datajoint
 * https://elements.datajoint.org -- catalog of example pipelines
 * https://codebook.datajoint.io -- interactive online tutorials
 
+## Citation
++ If your work uses DataJoint for Python, please cite the following Research Resource Identifier (RRID) and manuscript.
+
++ DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for Python (version `<Enter version number>`)
+
++ Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
+
 ## Python Native Blobs
 <details>
 <summary>Click to expand details</summary>
@@ -103,6 +110,18 @@ important DataJoint schema or records.
 
 </details>
 
+### API docs
+
+The API documentation can be built using sphinx by running
+
+``` bash
+pip install sphinx sphinx_rtd_theme 
+(cd docs-api/sphinx && make html)
+```
+
+Generated docs are written to `docs-api/docs/html/index.html`.
+More details in [docs-api/README.md](docs-api/README.md).
+
 ## Running Tests Locally
 <details>
 <summary>Click to expand details</summary>

datajoint/__init__.py

@@ -10,8 +10,9 @@
 that any use of DataJoint leading to a publication be acknowledged in the publication.
 
 Please cite:
-    http://biorxiv.org/content/early/2015/11/14/031658
-    http://dx.doi.org/10.1101/031658
+
+  - http://biorxiv.org/content/early/2015/11/14/031658
+  - http://dx.doi.org/10.1101/031658
 """
 
 __author__ = "DataJoint Contributors"

datajoint/admin.py

@@ -28,6 +28,7 @@ def set_password(
 def kill(restriction=None, connection=None, order_by=None):  # pragma: no cover
     """
     view and kill database connections.
+
     :param restriction: restriction to be applied to processlist
     :param connection: a datajoint.Connection object. Default calls datajoint.conn()
     :param order_by: order by a single attribute or the list of attributes. defaults to 'id'.
@@ -86,6 +87,7 @@ def kill(restriction=None, connection=None, order_by=None):  # pragma: no cover
 def kill_quick(restriction=None, connection=None):
     """
     Kill database connections without prompting. Returns number of terminated connections.
+
     :param restriction: restriction to be applied to processlist
     :param connection: a datajoint.Connection object. Default calls datajoint.conn()
 

datajoint/attribute_adapter.py

@@ -18,14 +18,17 @@ def attribute_type(self):
     def get(self, value):
         """
         convert value retrieved from the the attribute in a table into the adapted type
+
         :param value: value from the database
+
         :return: object of the adapted type
         """
         raise NotImplementedError("Undefined attribute adapter")
 
     def put(self, obj):
         """
         convert an object of the adapted type into a value that DataJoint can store in a table attribute
+
         :param obj: an object of the adapted type
         :return: value to store in the database
         """

datajoint/connection.py

@@ -1,6 +1,6 @@
 """
-This module contains the Connection class that manages the connection to the database,
- and the `conn` function that provides access to a persistent connection in datajoint.
+This module contains the Connection class that manages the connection to the database, and
+the ``conn`` function that provides access to a persistent connection in datajoint.
 """
 import warnings
 from contextlib import contextmanager
@@ -52,6 +52,7 @@ def connect_host_hook(connection_obj):
 def translate_query_error(client_error, query):
     """
     Take client error and original query and return the corresponding DataJoint exception.
+
     :param client_error: the exception raised by the client interface
     :param query: sql query with placeholders
     :return: an instance of the corresponding subclass of datajoint.errors.DataJointError
@@ -108,11 +109,9 @@ def conn(
     :param password: mysql password
     :param init_fun: initialization function
     :param reset: whether the connection should be reset or not
-    :param use_tls: TLS encryption option. Valid options are: True (required),
-                    False (required no TLS), None (TLS prefered, default),
-                    dict (Manually specify values per
-                    https://dev.mysql.com/doc/refman/5.7/en/connection-options.html
-                        #encrypted-connection-options).
+    :param use_tls: TLS encryption option. Valid options are: True (required), False
+        (required no TLS), None (TLS prefered, default), dict (Manually specify values per
+        https://dev.mysql.com/doc/refman/5.7/en/connection-options.html#encrypted-connection-options).
     """
     if not hasattr(conn, "connection") or reset:
         host = host if host is not None else config["database.host"]
@@ -247,6 +246,7 @@ def set_query_cache(self, query_cache=None):
         1. Only SELECT queries are allowed.
         2. The results of queries are cached under the path indicated by dj.config['query_cache']
         3. query_cache is a string that differentiates different cache states.
+
         :param query_cache: a string to initialize the hash for query results
         """
         self._query_cache = query_cache
@@ -298,6 +298,7 @@ def query(
     ):
         """
         Execute the specified query and return the tuple generator (cursor).
+
         :param query: SQL query
         :param args: additional arguments for the client.cursor
         :param as_dict: If as_dict is set to True, the returned cursor objects returns

datajoint/declare.py

@@ -153,6 +153,7 @@ def build_index_parser():
 
 def is_foreign_key(line):
     """
+
     :param line: a line from the table definition
     :return: true if the line appears to be a foreign key definition
     """
@@ -366,6 +367,7 @@ def prepare_declare(definition, context):
 def declare(full_table_name, definition, context):
     """
     Parse declaration and generate the SQL CREATE TABLE code
+
     :param full_table_name: full name of the table
     :param definition: DataJoint table definition
     :param context: dictionary of objects that might be referred to in the table
@@ -566,6 +568,7 @@ def substitute_special_type(match, category, foreign_key_sql, context):
 def compile_attribute(line, in_key, foreign_key_sql, context):
     """
     Convert attribute definition from DataJoint format to SQL
+
     :param line: attribution line
     :param in_key: set to True if attribute is in primary key set
     :param foreign_key_sql: the list of foreign key declarations to add to

datajoint/diagram.py

@@ -146,6 +146,7 @@ def __init__(self, source, context=None):
         def from_sequence(cls, sequence):
             """
             The join Diagram for all objects in sequence
+
             :param sequence: a sequence (e.g. list, tuple)
             :return: Diagram(arg1) + ... + Diagram(argn)
             """

datajoint/errors.py

@@ -34,6 +34,7 @@ def __init__(self, *args):
     def suggest(self, *args):
         """
         regenerate the exception with additional arguments
+
         :param args: addition arguments
         :return: a new exception of the same type with the additional arguments
         """

datajoint/expression.py

@@ -121,6 +121,7 @@ def where_clause(self):
     def make_sql(self, fields=None):
         """
         Make the SQL SELECT statement.
+
         :param fields: used to explicitly set the select attributes
         """
         return "SELECT {distinct}{fields} FROM {from_}{where}".format(
@@ -323,6 +324,7 @@ def __add__(self, other):
     def proj(self, *attributes, **named_attributes):
         """
         Projection operator.
+
         :param attributes:  attributes to be included in the result. (The primary key is already included).
         :param named_attributes: new attributes computed or renamed from existing attributes.
         :return: the projected expression.
@@ -481,6 +483,7 @@ def aggr(self, group, *attributes, keep_all_rows=False, **named_attributes):
         """
         Aggregation of the type U('attr1','attr2').aggr(group, computation="QueryExpression")
         has the primary key ('attr1','attr2') and performs aggregation computations for all matching elements of `group`.
+
         :param group:  The query expression to be aggregated.
         :param keep_all_rows: True=keep all the rows from self. False=keep only rows that match entries in group.
         :param named_attributes: computations of the form new_attribute="sql expression on attributes of group"
@@ -510,6 +513,7 @@ def head(self, limit=25, **fetch_kwargs):
         """
         shortcut to fetch the first few entries from query expression.
         Equivalent to fetch(order_by="KEY", limit=25)
+
         :param limit:  number of entries
         :param fetch_kwargs: kwargs for fetch
         :return: query result
@@ -520,6 +524,7 @@ def tail(self, limit=25, **fetch_kwargs):
         """
         shortcut to fetch the last few entries from query expression.
         Equivalent to fetch(order_by="KEY DESC", limit=25)[::-1]
+
         :param limit:  number of entries
         :param fetch_kwargs: kwargs for fetch
         :return: query result
@@ -561,6 +566,7 @@ def __contains__(self, item):
         """
         returns True if the restriction in item matches any entries in self
             e.g. ``restriction in q1``.
+
         :param item: any restriction
         (item in query_expression) is equivalent to bool(query_expression & item) but may be
         executed more efficiently.
@@ -907,6 +913,7 @@ def aggr(self, group, **named_attributes):
         """
         Aggregation of the type U('attr1','attr2').aggr(group, computation="QueryExpression")
         has the primary key ('attr1','attr2') and performs aggregation computations for all matching elements of `group`.
+
         :param group:  The query expression to be aggregated.
         :param named_attributes: computations of the form new_attribute="sql expression on attributes of group"
         :return: The derived query expression