Merge pull request #290 from mpsonntag/order

Sort and order

Author: jgrewe

CHANGELOG.md

@@ -0,0 +1,236 @@
+# Changelog
+
+Used to document all changes from previous releases and collect changes 
+until the next release.
+
+# Version 1.4.0
+## Breaking changes
+
+The switch from odML version 1.3 to 1.4 contains many cool updates which should make work more comfortable, but also includes some breaking changes.
+
+### Update of the odML file format version
+- The odML format version number in odML files has changed from "1" to "1.1".
+
+### Changes in odML classes
+- The odML class hierarchy has been flattened:
+  - removing `base._baseobj` class, leaving `BaseObject` as the root odML class.
+  - removing `doc.Document` class, leaving `BaseDocument` as the only odML Document class.
+  - removing `section.Section` class, leaving `BaseSection` as the only odML Section class.
+  - removing `property.Property` class leaving `BaseProperty` as the only odML Property class.
+- `baseobject` and `sectionable` are renamed to `BaseObject` and `Sectionable` respectively.
+- `base.SafeList` and `base.SmartList` have been merged, `base.SafeList` has been removed.
+- `base.SmartList` can now only contain Sections or Properties. See #272.
+- The `reorder` method is moved from the `base` to the `Section` class. See #267.
+
+### Changes in Value handling: 
+- The `Value` class has been removed.
+- `Property.value` now always holds a list of uniform values. `Property.value` always 
+    returns a copy of the actual value list. See #227.
+- Values can only be changed directly via the `__setitem__` method of a `Property`
+- `Value` attributes `uncertainty`, `unit`, `dtype` and `reference` have been moved to 
+    `Property` and now apply to all values of the `Property.value` list.
+- The `Value` attributes `filename`, `encoder` and `checksum` have been removed.
+
+### DType changes:
+- The `binary` dtype has been removed. Providing binary content via odML files is 
+    discouraged in favor of providing a reference to the origin files using the `URL` 
+    dtype instead.
+
+### Mapping
+- Any `mapping` functionality has been removed.
+
+### Minor breaking changes
+- `XMLReader.fromFile()` and `.fromString()` have been renamed to `.from_file()` and `.from_string()` respectively.
+
+
+## Features and changes
+
+### Required odML entity attributes handling
+- Required attributes of odML entities in `odml.format` where changed: `Section.name`, 
+    `Section.type` and `Property.name` are the only attributes set to be required for 
+    their respective odML entities. See #240.
+- `Section.name` and `Property.name` can now be `None` on init. If this is the case, the 
+    entities' `id` value is used as `name` value.
+- Hardcoded checks for existing `name` attributes in the XML Parser are removed. Only 
+    attributes set as required in `format` are now used to check for missing required odML 
+    entity attributes. See #241.
+- The `name` attribute of a `Section` or a `Property` can now only be rewritten if there 
+    is no sibling with the same name on the same hierarchical level. See #283.
+
+### Addition of the 'id' attribute
+- `Document`, `Section` and `Property` now have an `id` attribute to uniquely identify any 
+    entity. If no valid id is provided when an entity is initialized, an id is 
+    automatically generated and assigned.
+- Adding the `new_id()` method to `Document`, `Section` and `Property` which generates 
+    and sets a new valid id for any entity. See #262.
+
+### Changes in DType handling
+- Setting a dtype now also supports odML style tuple types. See #254.
+- DTypes now always return the defined default values if a value is `None` or `""`.
+- Any boolean dtype value other than `"false", "f", 0, False, "true", "t", 1` or `True` 
+    will now raise a `ValueError`. See #224
+
+### 'base.Sectionable' (Document and Section) changes
+- Adds a `base.Sectionable.extend` method for child Sections and Properties. See #237.
+- Refactors the `base.Sectionable.insert` and `.append` methods. Only proper 
+    `BaseSections` with a unique name can be added to the Section child list of a 
+    `Sectionable`.
+- Appending multiple Sections or Properties has been removed from the `append` method to 
+    mirror Property `append` functionality and since `extend` now serves this need.
+
+### 'Section' and 'Property' merge 
+- `Property` now provides a `merge` method to merge two properties. This will sync all but 
+    the dependency and dependencyValue attributes. ValueErrors are raised, if information 
+    is set in both properties but are in conflict. See #221.
+- Adds a `Section.merge_check()` method which validates whether a Section including all 
+    its sub-sections and sub-properties can properly be merged. A `ValueError` is raised 
+    if any potential merge problem arises. This is necessary since a recursive Section 
+    merge cannot be easily rolled back once begun.
+- A Section merge imports `reference` and `definition` from the "source" Section if they 
+    were `None` in the "destination" Section. See #273.
+- Adds a `strict` flag to any `merge` method. Now all `Section` and `Property` attribute 
+    checks during a merge will only be done, if `strict=True`. On `strict=False` a 
+    `Section` or `Property` attribute will only be replaced with the "source" value, if 
+    the "destination" value is `None`. Otherwise the "destination" value will be kept and 
+    the "source" value lost. See #270.
+
+### Changes of 'Section' and 'Property' clone
+- When a `Section` or a `Property` is cloned, a new id is set for the clone and of any 
+    cloned children. See #259.
+
+### 'Document' changes
+- Tuples of Sections can now no longer be used with `Document.append` since 
+    `Document.extend` should be used to add multiple new Sections to a Document.
+
+### 'Section' changes
+- Adds a `Section.extend` method.
+
+### 'Property' changes
+- `Property` has the new attribute `value_origin` which may contain the origin of the 
+    property's value e.g. a filename.
+- `Property` init now supports setting all attributes as well as its parent.
+- `Property` now provides `append`, `extend` and `remove` methods to change the actual 
+    value list. This approach is required to ensure DType checks when adding new values 
+    to an existing list. See #223. 
+- Only valid dtypes can now be set on `Property` init. See #253.
+
+### Terminology changes
+- The default odML terminology repository is set to `http://portal.g-node.org/odml/terminologies/v1.1/terminologies.xml`.
+
+### Changes in Tools and IO
+- The `XMLParser` can now be run in warning mode: any errors encountered during parsing 
+    will just print a warning, but will not stop and exit during the parsing process.
+- An odML document can now only be saved, if the validation does not show any errors. 
+    Saving an invalid document will stop the process before saving and print all 
+    encountered errors.
+- All parsers are now more relaxed when encountering unsupported or missing tags and only 
+    print warnings instead of ending with an exception. Warnings are collected and can be 
+    accessed via the parser object.
+- When trying to open a file with any of the odML parsers, the document format version 
+    number is checked. If the version number does not match the supported one, file 
+    loading will fail with an exception. 
+
+## New tools
+- Added the `tools.RDFWriter` and `toosl.RDFReader` classes, which enable the export of 
+    odML documents to RDF and also provides the used ontology OWL file at `doc/odml_terminology/`.
+- Added the `tools.ODMLWriter` and `tools.ODMLReader` classes which serve as an easy 
+    entry point to saving and loading for all the supported file formats `XML`, `YAML`, 
+    `JSON` and `RDF`.
+- Added the `tools.DictWriter` and `tools.DictReader` classes which convert Python 
+    dictionary data to odML data and vice versa, which in turn is required for both YAML 
+    and JSON format loading and saving.
+- Removed the `tools.jsonparser` file which is no longer required due to the classes in 
+    `tools.odmlparser` and `tools.dict_parser`. 
+- Added the `tools.FormatConverter` class which enables batch conversion of one odML 
+    format into another.
+- Added the `tools.VersionConverter` class which enables conversion of pre-v1.4 odML files 
+    into valid v1.4 odML.
+  - The `VersionConverter` converts `XML`, `JSON` and `YAML` based odML files of odML file 
+        version 1.0 to odML file version 1.1.
+  - Only attributes supported by `Document`, `Section` and `Property` are exported. Any 
+        non supported attribute will produce a warning message, the content will be 
+        discarded.
+  - The value content is moved from a `Value` object to its parent `Property` value list.
+  - The first encountered `unit` or `uncertainty` of values of a `Property` will be moved 
+        to its parent `Property`. Any differing subsequent `unit` or `uncertainty` of 
+        values of the same `Property` will produce a warning message, the content will be 
+        discarded.
+  - The first `filename` attribute content of a `Value` is moved to the `value_origin` 
+        attribute of its parent `Property`.
+  - Any g-node terminology URL in `repository` or `link` is updated from v1.0 to their 
+        v1.1 counterparts if available. 
+  - A `VersionConverter` object provides a `.conversion_log` list attribute to access all 
+        info and warning messages after a conversion has taken place. See #234.
+
+## Fixes
+- Various installation issues have been resolved for Linux and MacOS.
+- `False` as well as `F` are now properly converted to bool values in both 
+    Python 2 and 3. See #222.
+- Fixes saving datetime related values to JSON. See #248.
+- odML style custom tuples can now properly be saved using the `XMLParser`.
+- `Document` now properly uses the dtypes date setter on init. See #249.
+- Fixes load errors on Empty and `None` boolean and datetime related values. See #245.
+- Excludes `id` when comparing odML entities for equality. See #260.
+- When a `Property` is cloned, the parent of the clone is now properly set to `None`.
+- Avoids an `AttributeError` on `get_path()` when a `Property` has no parent. See #256.
+- Avoids an `AttributeError` on `get_merged_equivalent()` when a `Property` 
+    has no parent. See #257.
+- Avoids an error on `Property.append()`, if the dtype was not set. See #266.
+- Makes sure that `Property.append()` exits on empty values but accepts `0` and `False`.
+- Sets `Property.uncertainty` to `None` if an empty string is passed to it.
+- Changes the `Property.__init__` set attributes order: In the previous set attribute 
+    order, the repository attribute was overwritten with `None` by the `super.__init__` 
+    after it had been set.
+- Fixes set `Property.parent = None` bugs in `remove()` and `insert()` methods.
+- Consistently use relative imports to address circular imports and remove code that 
+    circumvents previous circular import errors in the `ODMLParser` class. See #199.
+- Consistently uses `BaseSection` or `BaseDocument` for isinstance checks throughout 
+    `base` instead of a mixture of `BaseSection` and `Section`.
+
+
+# Version 1.3.4
+
+## Fixes
+- Potential installation issues due to import from `info.py`.
+
+
+# Version 1.3.3
+## Features
+
+- Terminology caching and loading update.
+- Terminology section access and type listing functions.
+- Define and use common format version number for all parsers.
+- Supported format version check: When trying to open a file with any of the odml parsers, 
+    first the document format version number is checked. If the found version number does 
+    not match the supported one, file loading will fail an exception, since this is the 
+    oldest format version. If anyone tries to open a newer format, they should first 
+    update their odML package and not use this one.
+- Document saving: An odML document can now only be saved, if the validation does not show 
+    any errors. Saving an invalid document will exit while printing all encountered 
+    errors.
+- Parser: All parsers are now more relaxed when encountering unsupported tags or missing 
+    tags and only print warnings instead of ending with an exception. Warnings are 
+    collected and can be accessed via the parser object (required for display in 
+    [odml-ui](https://github.com/G-Node/odml-ui) to avoid potential loss of information).
+- Package and format information added or updated: `Version`, `Format version`, `Contact`, 
+    `Homepage`, `Author`, PyPI `Classifiers`, `Copyright`.
+- Removes the license text from `setup.py`. The license text interfered with the PyPI 
+    process in a way, that the description was not displayed on PyPI.
+- Removes the image folder from the project, since they are exclusively used in the 
+    outsourced [odml-ui](https://github.com/G-Node/odml-ui) project.
+
+## Fixes
+- Fixes a bug that prohibits the parsing of `json` or `yaml` files; #191.
+- Fixes a bug that fails parsing of `json` or `yaml` files when `Section.repository`, `Section.link` or `Section.include` are present; #194.
+
+
+# Version 1.3.2
+- Expose load, save, and display functions to top level module
+    - These functions accept a `backend` argument that specifies the parser or writer. 
+        Can be one of `XML`, `JSON`, or `YAML`.
+
+
+# Version 1.3.1
+- move ui to a separate repository https://github.com/g-node/odml-ui
+- python3 compatibility
+- add json and yaml storage backends

README.rst

@@ -94,10 +94,11 @@ To install the Python-odML library, enter the corresponding directory and run::
 **Note** The master branch is our current development branch, not all features might be
 working as expected. Use the release tags instead.
 
-Documentation
--------------
+odML Project page
+-----------------
 
-`Documentation <https://g-node.github.io/python-odml>`_
+More information about the project including related projects as well as tutorials and
+examples can be found at our `odML project page <https://g-node.github.io/python-odml>`_
 
 Bugs & Questions
 ----------------

doc/conf.py

@@ -69,9 +69,9 @@ def __new__(meta, name, bases, clsdict):
 # built documents.
 #
 # The short X.Y version.
-version = '0.1'
+version = '1.4'
 # The full version, including alpha/beta/rc tags.
-release = '0.1'
+release = '1.4.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -115,7 +115,7 @@ def __new__(meta, name, bases, clsdict):
 
 # The theme to use for HTML and HTML Help pages.  Major themes that come with
 # Sphinx are currently 'default' and 'sphinxdoc'.
-html_theme = 'default'
+html_theme = 'sphinxdoc'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

odml/base.py

@@ -131,6 +131,28 @@ def __contains__(self, key):
             if (hasattr(obj, "name") and obj.name == key) or key == obj:
                 return True
 
+    def __eq__(self, obj):
+        """
+        SmartList attributes of 'sections' and 'properties' are
+        handled specially: We want to make sure that the lists'
+        objects are properly compared without changing the order
+        of the individual lists.
+        """
+        # This special case was introduced only due to the fact
+        # that RDF files will be loaded with randomized list
+        # order. With any other file format the list order
+        # remains unchanged.
+        if sorted(self, key=lambda x: x.name) != sorted(obj, key=lambda x: x.name):
+            return False
+
+        return True
+
+    def __ne__(self, obj):
+        """
+        Use the __eq__ function to determine if both objects are equal
+        """
+        return not self == obj
+
     def index(self, obj):
         """
         Find obj in list
@@ -158,6 +180,13 @@ def append(self, *obj_tuple):
 
             super(SmartList, self).append(obj)
 
+    def sort(self, key=lambda x: x.name, reverse=False):
+        """
+        If not otherwise defined, sort by the *name* attribute
+        of the lists *_content_type* object.
+        """
+        super(SmartList, self).sort(key=key, reverse=reverse)
+
 
 @allow_inherit_docstring
 class Sectionable(BaseObject):

setup.py

@@ -27,7 +27,7 @@
 with open('README.rst') as f:
     description_text = f.read()
 
-install_req = ["lxml", "pyyaml", "rdflib"]
+install_req = ["lxml", "pyyaml==3.12", "rdflib"]
 
 if sys.version_info < (3, 4):
     install_req += ["enum34"]