Merge pull request #514 from jgrewe/docs

rewrite of the docs

Author: mpsonntag

.gitignore

@@ -46,3 +46,6 @@ Makefile
 .cache
 .eggs
 .mypy_cache
+
+# vscode
+.vscode

README.rst

@@ -27,13 +27,6 @@ The simplest way to install *NIXPY* is from PyPI using pip. The name of the pack
 
     pip install nixio
 
-Bindings for C++ NIX
---------------------
-
-The setup script will automatically build the bindings for *NIX* if it is detected on the system, otherwise only the pure Python version will be installed.
-
-For instructions on building *NIX*, see the `NIX README <https://github.com/G-Node/nix/blob/master/README.md>`_ file.
-
 
 To check if installed properly
 ------------------------------

docs/source/annotations.rst

@@ -0,0 +1,74 @@
+.. toctree::
+    :maxdepth: 1
+
+.. _metadata:
+
+Annotations with arbitrary metadata
+===================================
+
+The entities of the data model that were discussed so far carry just enough information to get a basic understanding of the stored data. Often much more information than that is required. Storing additional metadata is a central part of the NIX concept. We use a slightly modified version of the *odML* data model for metadata to store additional information. In brief: the model consists of *Sections* that contain *Properties* which in turn carry a list of values. Again, *Sections* can be nested to represent logical dependencies in the hierarchy of a tree. While all data entities discussed above are children of *Block* entities, the metadata lives parallel to the *Blocks*. The idea behind this is that several blocks may refer to the same metadata, or, the other way round the metadata applies to data entities in several blocks. The *types* used for the *Sections* in the following example are defined
+in the `odml terminologies <https://github.com/G-Node/odml-terminologies>`_
+
+Most of the data entities can link to metadata sections.
+
+.. literalinclude:: examples/annotations.py
+    :lines: 5-21
+    :caption: We can add arbitrary metadata in trees of *Sections* and *Properties* (:download:`example code <examples/annotations.py>`).
+
+For a quick view of the metadata tree pretty-print it:
+
+.. literalinclude:: examples/annotations.py
+    :lines: 24
+
+which leads to an output like this. The argument ``max_depth=-1`` notes that the full depth of the tree should be displayed. In the default case (``max_depth=1``) the display will be more compact and will not recursively traverse the whole tree. 
+
+.. code-block:: text
+
+    recording session [odml.recording]
+        |- experimenter: ('John Doe',)
+        |- recording date: ('2014-01-01',)
+    subject [odml.subject]
+        |- id: ('mouse xyz',)
+        cell [odml.cell]
+            |- resting potential: (-64.5,)mV
+
+The *Sections* add much like dictionaries. To access e.g. the "resting potential" of the cell we may call:
+
+.. literalinclude:: examples/annotations.py
+    :lines: 26
+
+Extending Properties
+--------------------
+
+Properties can carry multiple values and additional information such as a definition. 
+
+.. literalinclude:: examples/annotations.py
+    :lines: 33-35
+
+Reading the values will return a tuple. This has the background that one cannot change a tuple. Changing the values stored in a *Property* can be done e.g. by the ``extend_values`` method
+
+.. literalinclude:: examples/annotations.py
+    :lines: 37-38
+
+One can extend it by single values or by lists of values. 
+
+The data type of all values must, however, be the same. Adding a number to the list of strings will fail with a ``TypeError``:
+
+.. literalinclude:: examples/annotations.py
+    :lines: 40-43
+
+.. code-block:: text
+
+    New data type '<class 'numpy.int64'>' is inconsistent with the Property's data type '<class 'numpy.str_'>'
+
+**Note:** Once the property has been created, the data type can't be changed. One would need to create a replacement with the desired data type.
+
+Finding annotations
+-------------------
+
+If we do not know the exact path of the *Section* we are looking for, we need to search it by passing a function (in this case a lambda function) to the ``find_section`` method. The following code shows two examples in which we look first for a section with a given name or second a section which contains a property with a certain name. 
+
+.. literalinclude:: examples/annotations.py
+    :lines: 28 - 31
+
+The result of the ``find_sections`` will always be a list which may be empty if no match was found. Therefore, the call in the last line is to some extent risky and would lead to an ``OutOfBounds`` exception if the search failed.

docs/source/api/modules.rst

@@ -1,5 +1,5 @@
-nixio package
-=============
+Internal API
+============
 
 Subpackages
 -----------

docs/source/api/nixio.rst

@@ -0,0 +1,17 @@
+Idea
+====
+
+The basic idea of the *NIX* project is to come up with a **generic**
+data model that defines as few structures/entities as possible while
+being able to represent a multitude of different data structures, allows
+for in-depth annotation and supports standardization.
+
+Designing a **generic** data model implies that the defined entities are
+named in a way that may seem uncommon but are more general than the
+*domain-specific* terms used in any given field or community.
+
+
+Please follow this link to the `C++ nix library documentation <https://nixio.readthedocs.io/en/master/data_model.html>`_ to lear more about the data model. 
+
+The idea of the *NIX* data model has been implemented using the
+`HDF5 <https://www.hdfgroup.org>`__ file format.

docs/source/basic_idea.rst

@@ -1,4 +1,4 @@
-# Copyright © 2014, German Neuroinformatics Node (G-Node)
+# Copyright © 2014-2021, German Neuroinformatics Node (G-Node)
 #
 # All rights reserved.
 #
@@ -9,40 +9,31 @@
 from nixio.info import RELEASE, COPYRIGHT, BRIEF
 
 # general config
-extensions              = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx']
-source_suffix           = '.rst'
-master_doc              = 'index'
-project                 = BRIEF
-copyright               = COPYRIGHT
-release                 = RELEASE
-exclude_patterns        = []
-pygments_style          = 'sphinx'
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.autosectionlabel']
+source_suffix = '.rst'
+master_doc = 'index'
+project = BRIEF
+copyright = COPYRIGHT
+release = RELEASE
+exclude_patterns = []
+pygments_style = 'sphinx'
 
 # html options
-htmlhelp_basename       = 'nixio'
+htmlhelp_basename = 'nixio'
 try:
-    html_theme          = 'sphinx_rtd_theme'
-    html_sidebars       = {
-        '**': [
-                'about.html', 'navigation.html', 'searchbox.html',
-            ]
-    }
+    html_theme = 'sphinx_rtd_theme'
+    html_sidebars = {'**': ['about.html', 'navigation.html', 'searchbox.html']}
 
-    html_theme_options  = {
-        'logo'          : 'logo.png',
-        'github_user'   : 'G-Node',
-        'github_repo'   : 'nixpy',
-        'github_button' : True,
-        'github_count'  : False,
-        'travis_button' : True,
-        'link'          : '#456BA8'
-    }
+    html_theme_options = {'logo_only': True,
+                          'display_version': True,
+                          'style_external_links': True,
+                          'prev_next_buttons_location': "both"}
+
+    html_logo = "nix_logo.png"
 
 except ImportError:
-    html_theme          = 'default'
+    html_theme = 'default'
 
 # intersphinx configuration
-intersphinx_mapping     = {
-    'http://docs.python.org/' : None,
-    'http://docs.scipy.org/doc/numpy': None
-}
+intersphinx_mapping = {'http://docs.python.org/': None,
+                       'http://docs.scipy.org/doc/numpy': None}

docs/source/conf.py

@@ -0,0 +1,20 @@
+.. :toctree::
+         :maxdepth: 2
+
+Getting support
+===============
+
+If you experience problems using *NIX* feel free to join our IRC
+channel `#gnode at FreeNode <irc://irc.freenode.net/gnode>`__ or write
+an email to [email protected]. If you find a bug or have a feature
+request please report it using the `project issue tracker
+<https://github.com/G-Node/nixpy/issues>`__.
+
+
+Contact
+-------
+
+The project is maintained by the `German Neuroinformatics Node,
+G-Node <http://www.g-node.org>`__. `G-Node at
+GitHub <https://github.com/g-node>`__,
+`email <mailto:[email protected]>`__.

docs/source/contact.rst

@@ -0,0 +1,100 @@
+.. toctree::
+   :maxdepth: 1
+
+Working with data
+=================
+
+Storing data is one thing, but we want to work with it. The following examples illustrate reading of data from *DataArray*, *Tag* and *MultiTag* entities. We will use the dummy dataset already used in the :doc:`tagging <./tagging>` example. The figure below shows what is stored in the dataset. 
+
+.. figure:: ./images/tag1.png
+   :alt: a system's response to a stimulus
+
+At some instance in time a system was exposed to a stimulus that leads to the system's response. The response has been recorded and stored in a *DataArray*, a *Tag* is used to highlight the "stimulus-on" segment of the data.
+
+.. literalinclude:: examples/tagging_example.py
+    :caption: :download:`example code <examples/tagging_example.py>`
+    :lines: 78 - 90 
+
+In this example we know the interesting entities by name, i.e. the
+*DataArray* is called **response** and the *Tag* is called **stimulus**.
+In cases in which we have no clue about the names, we just have to
+browse the file or `search <./finding_things.md>`__ by name or type.
+
+Reading data
+------------
+
+The first and maybe most common problem is to read the data stored in a
+*DataArray*.
+
+Reading all data
+~~~~~~~~~~~~~~~~
+
+In *NIX* when you open a *DataArray* the stored data is **not** automatically read from file. This keeps the object lightweight and easy to create. To read the data you can simply access it in a numpy style:
+
+.. literalinclude:: examples/tagging_example.py
+    :caption: :download:`example code <examples/tagging_example.py>`
+    :lines: 71 - 73 
+
+There are a few noteworthy things:
+
+-  We use some previous knowledge here. For one, we know the names of the entities. Further, we know that the data is 1-D and the single dimension is a ``SampledDimension``. If these things are not known, the NIX library offers the necessary functions to get this information.
+-  ``DataArray.shape`` returns a tuple with the data shape.
+-  ``DataArray.dtype`` returns the data type.
+-  To find out the ``DimensionType``, we need to access the dimension:
+
+.. code-block:: python
+
+    for dim in data_array.dimensions:
+        print(dim.dimension_type)
+
+
+Reading partial data
+~~~~~~~~~~~~~~~~~~~~
+
+In other instances it might be wanted to read only parts of the data.
+Reading slices of the data is straight forward using the numpy style.
+
+.. literalinclude:: examples/tagging_example.py
+    :caption: :download:`example code <examples/tagging_example.py>`
+    :lines: 98-101
+
+An alternative approach is to use the ``DataArray.get_slice`` method which by default works with indices but can also work in data coordinates. E.g. we know that the data is 1-D and covers a span of 3.5s and we want to have the data in the interval 0.5s through 1.75s. The method returns a ``nixio.DataView`` object. The actual reading is done by accessing the data.
+
+.. literalinclude:: examples/tagging_example.py
+    :caption: :download:`example code <examples/tagging_example.py>`
+    :lines: 103-106
+    :emphasize-lines: 3
+
+The arguments ``positions`` and ``extents`` are passed as lists. There must be one entry for each dimension of the data. In this case, since the data is 1-D, positions and extents are 1-element lists.
+Note: the slice is defined by the starting point(s) and the *extent(s)*, not with start and end points.
+
+Reading tagged data
+~~~~~~~~~~~~~~~~~~~
+
+*Tag* and *MultiTag* tag single or multiple points or regions in data stored in the referenced *DataArrays* (see `tagging <./tagging.md>`__ for more information and the example data created in the `example <#mtag_regions>`__  will be used in the following code snippets).
+
+.. figure:: ./images/multiple_regions.png
+    :alt: tagging multiple segments
+
+    Plot of the data created by this :download:`example <examples/multiple_regions.py>` on tagging multiple regions in data. 
+
+In order to read the data that belongs to the highlighted region(s) *Tag* and *MultiTag* define the ``tagged_data`` methods which return ``nixio.DataView`` objects from which the data is read as shown above. The following code snippet shows how to use these function:
+
+.. literalinclude:: examples/multiple_regions.py
+   :lines: 86 - 97
+   :emphasize-lines: 10
+   :caption: Reading data segments tagged by the *Tag* or *MultiTag* can be done using the ``tagged_data`` method (:download:`example code <examples/multiple_regions.py>`).
+
+The *MultiTag* version of the ``tagged_data`` method takes two arguments. The first is the index of the tagged region (0 for the first), the second argument is the name of the referenced *DataArray* (you can also use the index or the id). Since the *Tag* tags only a single region, it takes only one argument, i.e. the name (id, index) of the referenced *DataArray*.
+
+.. figure:: ./images/reading_tagged_data.png
+   :alt: reading tagged data
+   
+
+Analogously, the feature data attached to the *Tag* or *MultiTag* can be
+obtained using the ``feature_data`` methods.
+
+.. literalinclude:: examples/multiple_regions.py
+   :lines: 69 - 76
+   :emphasize-lines: 6, 7
+   :caption: ``feature_data`` works analogously, the first argument is the index of the tagged region, the second the name (or id or index) of the feature. Here the feature stores a single number, i.e. the frequency of the stimulus for each tagged region which is plotted below the highlighted regions in the figure above (:download:`example code <examples/multiple_regions.py>`). 

docs/source/data_handling.rst

@@ -0,0 +1,48 @@
+import nixio
+
+def main():
+
+    nixfile = nixio.File.open("annotations.nix", mode=nixio.FileMode.Overwrite)
+    block = nixfile.create_block("recording 1", "nix.session")
+
+    session = nixfile.create_section('recording session', 'odml.recording')
+    session['experimenter'] = 'John Doe'
+    session['recording date'] = '2014-01-01'
+
+    subject = session.create_section('subject', 'odml.subject')
+    subject['id'] = 'mouse xyz'
+
+    cell = subject.create_section('cell', 'odml.cell')
+    p = cell.create_property('resting potential', -64.5)
+    p.uncertainty = 2.25
+    p.unit = 'mV'
+
+    # set the recording block metadata
+    block.metadata = session
+
+    # query the metadata
+    block.metadata.pprint(max_depth=-1)
+
+    print(block.metadata["subject"]["cell"]["resting potential"])
+
+    print(session.find_sections(lambda s: s.name.lower() == "cell"))
+
+    print(session.find_sections(lambda s: "resting potential" in s.props))
+    print(session.find_sections(lambda s: "resting potential" in s.props)[0]["resting potential"])
+
+    observations = subject.create_property("observations", ["p10", "p12", "p14"])
+    observations.definition = "Some behavioral observations I did on several days"
+    print(observations.values)
+
+    observations.extend_values("p16")
+    observations.extend_values(["p18", "p20"])
+
+    try:
+        observations.extend_values(22)
+    except TypeError as e:
+        print(e)
+
+    nixfile.close()
+
+if __name__ == "__main__":
+    main()