Merge pull request #424 from eclipse-basyx/develop

Prepare Release v2.0.0

# Release Notes

Version 2.0.0 of the BaSyx-Python SDK comes with a major refactoring of the server and a renewed concept for data persistence.

Previously, the server code was split between the `sdk` and `server` packages (due to historic development of the code). 
Now all the code relevant just for the server is located in `server`, where it belongs. Since this means, some code that was previously in `sdk` is not there anymore, this is a breaking change and warranted the new major release.

> [!note]
> This release does not have any changes in implemented AAS specification versions. It is the preparotory release in order to get ready for the new versions of the specifications, as well as new features for the SDK, such as Registry and Discovery server.

> [!warning]
> Due to these major refactorings, there were some backward incompatible changes. Please check the documentation, if you encounter any issues.

# Changelog

**Notable:**
- Backward Incompatible: Refactor server functionality from `sdk` to `server` (See: #388)
- Backward Incompatible: Refactor `backend` concept for data persistence (See: #370)
- Backward Incompatible: Refactor server `start-up` options (See: #418)
- Remove support for Python 3.9 (as it is EoL) (See #433)

**Improvements:**
- Clarify documentation of running the server with Docker (See: #398)
- Document running the server without Docker (See: #403)
- Improve XML serialization (See: #406)
- Improve server reading of JSON and XML files (See: #408)
- Add more utility methods for `Referable` and `Key` handling (See: #410)

**Bugfixes:**
- Fix type issues found with a new version of `mypy` (See: #399)
- Fix parsing of `ConceptDescription`s in the server (See: #420)
- Update `pyecma376-2` and `lxml` dependencies (See: #419)

Author: s-heppner

.github/workflows/ci.yml

@@ -3,7 +3,7 @@ name: ci
 on: [push, pull_request]
 
 env:
-  X_PYTHON_MIN_VERSION: "3.9"
+  X_PYTHON_MIN_VERSION: "3.10"
   X_PYTHON_MAX_VERSION: "3.12"
 
 jobs:
@@ -45,7 +45,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.9", "3.12"]
+        python-version: ["3.10", "3.12"]
     env:
       COUCHDB_ADMIN_PASSWORD: "yo0Quai3"
       # (2024-10-11, s-heppner)
@@ -154,15 +154,14 @@ jobs:
         working-directory: ./sdk
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python ${{ env.X_PYTHON_MIN_VERSION }}
+    - name: Set up Python ${{ env.X_PYTHON_MAX_VERSION }}
       uses: actions/setup-python@v5
       with:
-        python-version: ${{ env.X_PYTHON_MIN_VERSION }}
+        python-version: ${{ env.X_PYTHON_MAX_VERSION }}
     - name: Install Python dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install .
-        pip install -r docs/add-requirements.txt
+        pip install .[docs]
     - name: Check documentation for errors
       run: |
         SPHINXOPTS="-a -E -n -W --keep-going" make -C docs html
@@ -209,7 +208,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.9", "3.12"]
+        python-version: ["3.10", "3.12"]
     defaults:
       run:
         working-directory: ./compliance_tool
@@ -255,13 +254,13 @@ jobs:
         pip install .[dev]
     - name: Check typing with MyPy
       run: |
-        mypy ./aas_compliance_tool test
+        mypy aas_compliance_tool test
     - name: Check code style with PyCodestyle
       run: |
-        pycodestyle --count --max-line-length 120 ./aas_compliance_tool test
+        pycodestyle --count --max-line-length 120 aas_compliance_tool test
 
-  compliance-tool-readme-codeblocks:
-    # This job runs the same static code analysis (mypy and pycodestyle) on the codeblocks in our docstrings.
+  compliance-tool-package:
+    # This job checks if we can build our compliance_tool package
     runs-on: ubuntu-latest
 
     defaults:
@@ -273,42 +272,43 @@ jobs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ env.X_PYTHON_MIN_VERSION }}
-    - name: Install Python dependencies
-      # install the local sdk in editable mode so it does not get overwritten
+    - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -e ../sdk[dev]
-        pip install .[dev]
-    - name: Check typing with MyPy
-      run: |
-        mypy <(codeblocks python README.md)
-    - name: Check code style with PyCodestyle
-      run: |
-        codeblocks --wrap python README.md | pycodestyle --count --max-line-length 120 -
-    - name: Run readme codeblocks with Python
+        pip install build
+    - name: Create source and wheel dist
       run: |
-        codeblocks python README.md | python
+        python -m build
 
-  compliance-tool-package:
-    # This job checks if we can build our compliance_tool package
+ #server-test:
+    # TODO: This job runs the unittests on the python versions specified down at the matrix
+    #   and aas-test-engines on the server
+
+
+  server-static-analysis:
+    # This job runs static code analysis, namely pycodestyle and mypy
     runs-on: ubuntu-latest
 
     defaults:
       run:
-        working-directory: ./compliance_tool
+        working-directory: ./server/app
     steps:
     - uses: actions/checkout@v4
     - name: Set up Python ${{ env.X_PYTHON_MIN_VERSION }}
       uses: actions/setup-python@v5
       with:
         python-version: ${{ env.X_PYTHON_MIN_VERSION }}
-    - name: Install dependencies
+    - name: Install Python dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install build
-    - name: Create source and wheel dist
+        pip install ../../sdk
+        pip install .[dev]
+    - name: Check typing with MyPy
       run: |
-        python -m build
+        mypy .
+    - name: Check code style with PyCodestyle
+      run: |
+        pycodestyle --count --max-line-length 120 .
 
   server-package:
     # This job checks if we can build our server package

.gitignore

@@ -21,14 +21,16 @@ htmlcov/
 docs/build/
 .hypothesis/
 
-# customized config files
+# Customized config files
 sdk/test/test_config.ini
 # Schema files needed for testing
 sdk/test/adapter/schemas
 
 # Ignore dynamically generated version file
 sdk/basyx/version.py
 compliance_tool/aas_compliance_tool/version.py
+server/app/version.py
 
-# ignore the content of the server storage
+# Ignore the content of the server storage
+server/input/
 server/storage/

compliance_tool/pyproject.toml

@@ -34,7 +34,7 @@ classifiers = [
     "Operating System :: OS Independent",
     "Development Status :: 5 - Production/Stable"
 ]
-requires-python = ">=3.9"
+requires-python = ">=3.10"
 dependencies = [
     "pyecma376-2>=0.2.4",
     "jsonschema>=4.21.1",

sdk/.readthedocs.yaml

@@ -6,7 +6,7 @@ version: 2
 build:
   os: ubuntu-20.04
   tools:
-    python: "3.9"
+    python: "3.10"
 
 sphinx:
    configuration: docs/source/conf.py
@@ -15,4 +15,5 @@ python:
   install:
     - method: pip
       path: .
-    - requirements: docs/add-requirements.txt
+      extra_requirements:
+        - docs

sdk/README.md

@@ -42,8 +42,7 @@ The BaSyx Python SDK requires the following Python packages to be installed for
 * `lxml` (BSD 3-clause License, using `libxml2` under MIT License)
 * `python-dateutil` (BSD 3-clause License)
 * `pyecma376-2` (Apache License v2.0)
-* `urllib3` (MIT License)
-* `Werkzeug` (BSD 3-clause License)
+
 
 Development/testing/documentation/example dependencies:
 * `mypy` (MIT License)
@@ -128,7 +127,7 @@ For further examples and tutorials, check out the `basyx.aas.examples`-package.
 * [`tutorial_storage`](./basyx/aas/examples/tutorial_storage.py): Manage a larger number of Asset Administration Shells in an ObjectStore and resolve references
 * [`tutorial_serialization_deserialization`](./basyx/aas/examples/tutorial_serialization_deserialization.py): Use the JSON and XML serialization/deserialization for single objects or full standard-compliant files
 * [`tutorial_aasx`](./basyx/aas/examples/tutorial_aasx.py): Export Asset Administration Shells with related objects and auxiliary files to AASX package files
-* [`tutorial_backend_couchdb`](./basyx/aas/examples/tutorial_backend_couchdb.py): Use the *Backends* interface (`update()/commit()` methods) to manage and retrieve AAS objects in a CouchDB document database
+* [`tutorial_backend_couchdb`](./basyx/aas/examples/tutorial_backend_couchdb.py): Use the *CouchDBObjectStore* to manage and retrieve AAS objects in a CouchDB document database
 
 
 ### Documentation

sdk/basyx/aas/adapter/__init__.py

@@ -7,3 +7,45 @@
   Python SDK objects to/from XML.
 * :ref:`aasx <adapter.aasx>`: This package offers functions for reading and writing AASX-files.
 """
+
+from basyx.aas.adapter.aasx import AASXReader, DictSupplementaryFileContainer
+from basyx.aas.adapter.json import read_aas_json_file_into
+from basyx.aas.adapter.xml import read_aas_xml_file_into
+from basyx.aas.model.provider import DictObjectStore
+from pathlib import Path
+from typing import Union
+
+
+def load_directory(directory: Union[Path, str]) -> tuple[DictObjectStore, DictSupplementaryFileContainer]:
+    """
+    Create a new :class:`~basyx.aas.model.provider.DictObjectStore` and use it to load Asset Administration Shell and
+    Submodel files in ``AASX``, ``JSON`` and ``XML`` format from a given directory into memory. Additionally, load all
+    embedded supplementary files into a new :class:`~basyx.aas.adapter.aasx.DictSupplementaryFileContainer`.
+
+    :param directory: :class:`~pathlib.Path` or ``str`` pointing to the directory containing all Asset Administration
+        Shell and Submodel files to load
+    :return: Tuple consisting of a :class:`~basyx.aas.model.provider.DictObjectStore` and a
+        :class:`~basyx.aas.adapter.aasx.DictSupplementaryFileContainer` containing all loaded data
+    """
+
+    dict_object_store: DictObjectStore = DictObjectStore()
+    file_container: DictSupplementaryFileContainer = DictSupplementaryFileContainer()
+
+    directory = Path(directory)
+
+    for file in directory.iterdir():
+        if not file.is_file():
+            continue
+
+        suffix = file.suffix.lower()
+        if suffix == ".json":
+            with open(file) as f:
+                read_aas_json_file_into(dict_object_store, f)
+        elif suffix == ".xml":
+            with open(file) as f:
+                read_aas_xml_file_into(dict_object_store, f)
+        elif suffix == ".aasx":
+            with AASXReader(file) as reader:
+                reader.read_into(object_store=dict_object_store, file_store=file_container)
+
+    return dict_object_store, file_container

sdk/basyx/aas/adapter/_generic.py

@@ -19,6 +19,13 @@
 PathOrBinaryIO = Union[Path, BinaryIO]
 PathOrIO = Union[Path, IO]  # IO is TextIO or BinaryIO
 
+# JSON top-level keys and their corresponding model classes
+JSON_AAS_TOP_LEVEL_KEYS_TO_TYPES = (
+    ('assetAdministrationShells', model.AssetAdministrationShell),
+    ('submodels', model.Submodel),
+    ('conceptDescriptions', model.ConceptDescription),
+)
+
 # XML Namespace definition
 XML_NS_MAP = {"aas": "https://admin-shell.io/aas/3/0"}
 XML_NS_AAS = "{" + XML_NS_MAP["aas"] + "}"