Refactor server start-up options (#418)

Previously, the repository server used one and the same folder for
loading data during start-up and storing it persistently. This resulted
in a mixture of input AAS/Submodel files (AASX, JSON and XML) and
persistently stored AAS/Submodel files from the `LocalFileObjectStore`
(JSON).

This separates the server's input and storage into two different
directories to prevent their files being mixed. Moreover, the option to
overwrite existing AAS/Submodels in the storage got added and the option
to persistently store data got adapted. In accordance with the new
changes, the `README` and `Dockerfile` were adapted to present the
changes to the end users.

Fixes #404

---------

Co-authored-by: s-heppner <[email protected]>
Co-authored-by: Igor Garmaev <[email protected]>

Author: 3 people

.gitignore

@@ -21,7 +21,7 @@ htmlcov/
 docs/build/
 .hypothesis/
 
-# customized config files
+# Customized config files
 sdk/test/test_config.ini
 # Schema files needed for testing
 sdk/test/adapter/schemas
@@ -31,5 +31,6 @@ sdk/basyx/version.py
 compliance_tool/aas_compliance_tool/version.py
 server/app/version.py
 
-# ignore the content of the server storage
+# Ignore the content of the server storage
+server/input/
 server/storage/

sdk/basyx/aas/adapter/__init__.py

@@ -7,3 +7,45 @@
   Python SDK objects to/from XML.
 * :ref:`aasx <adapter.aasx>`: This package offers functions for reading and writing AASX-files.
 """
+
+from basyx.aas.adapter.aasx import AASXReader, DictSupplementaryFileContainer
+from basyx.aas.adapter.json import read_aas_json_file_into
+from basyx.aas.adapter.xml import read_aas_xml_file_into
+from basyx.aas.model.provider import DictObjectStore
+from pathlib import Path
+from typing import Union
+
+
+def load_directory(directory: Union[Path, str]) -> tuple[DictObjectStore, DictSupplementaryFileContainer]:
+    """
+    Create a new :class:`~basyx.aas.model.provider.DictObjectStore` and use it to load Asset Administration Shell and
+    Submodel files in ``AASX``, ``JSON`` and ``XML`` format from a given directory into memory. Additionally, load all
+    embedded supplementary files into a new :class:`~basyx.aas.adapter.aasx.DictSupplementaryFileContainer`.
+
+    :param directory: :class:`~pathlib.Path` or ``str`` pointing to the directory containing all Asset Administration
+        Shell and Submodel files to load
+    :return: Tuple consisting of a :class:`~basyx.aas.model.provider.DictObjectStore` and a
+        :class:`~basyx.aas.adapter.aasx.DictSupplementaryFileContainer` containing all loaded data
+    """
+
+    dict_object_store: DictObjectStore = DictObjectStore()
+    file_container: DictSupplementaryFileContainer = DictSupplementaryFileContainer()
+
+    directory = Path(directory)
+
+    for file in directory.iterdir():
+        if not file.is_file():
+            continue
+
+        suffix = file.suffix.lower()
+        if suffix == ".json":
+            with open(file) as f:
+                read_aas_json_file_into(dict_object_store, f)
+        elif suffix == ".xml":
+            with open(file) as f:
+                read_aas_xml_file_into(dict_object_store, f)
+        elif suffix == ".aasx":
+            with AASXReader(file) as reader:
+                reader.read_into(object_store=dict_object_store, file_store=file_container)
+
+    return dict_object_store, file_container

sdk/basyx/aas/adapter/aasx.py

@@ -871,3 +871,6 @@ def __contains__(self, item: object) -> bool:
 
     def __iter__(self) -> Iterator[str]:
         return iter(self._name_map)
+
+    def __len__(self) -> int:
+        return len(self._name_map)

sdk/basyx/aas/model/provider.py

@@ -11,7 +11,7 @@
 """
 
 import abc
-from typing import MutableSet, Iterator, Generic, TypeVar, Dict, List, Optional, Iterable, Set
+from typing import MutableSet, Iterator, Generic, TypeVar, Dict, List, Optional, Iterable, Set, Tuple, cast
 
 from .base import Identifier, Identifiable
 
@@ -67,7 +67,7 @@ class AbstractObjectStore(AbstractObjectProvider, MutableSet[_IT], Generic[_IT],
     :class:`~basyx.aas.model.base.Identifier` – allow to add and delete objects (i.e. behave like a Python set).
     This includes local object stores (like :class:`~.DictObjectStore`) and specific object stores
     (like :class:`~basyx.aas.backend.couchdb.CouchDBObjectStore` and
-    :class `~basyx.aas.backend.local_file.LocalFileObjectStore`).
+    :class:`~basyx.aas.backend.local_file.LocalFileObjectStore`).
 
     The AbstractObjectStore inherits from the :class:`~collections.abc.MutableSet` abstract collections class and
     therefore implements all the functions of this class.
@@ -80,6 +80,36 @@ def update(self, other: Iterable[_IT]) -> None:
         for x in other:
             self.add(x)
 
+    def sync(self, other: Iterable[_IT], overwrite: bool) -> Tuple[int, int, int]:
+        """
+        Merge :class:`Identifiables <basyx.aas.model.base.Identifiable>` from an
+        :class:`~collections.abc.Iterable` into this :class:`~basyx.aas.model.provider.AbstractObjectStore`.
+
+        :param other: :class:`~collections.abc.Iterable` to sync with
+        :param overwrite: Flag to overwrite existing :class:`Identifiables <basyx.aas.model.base.Identifiable>` in this
+            :class:`~basyx.aas.model.provider.AbstractObjectStore` with updated versions from ``other``,
+            :class:`Identifiables <basyx.aas.model.base.Identifiable>` unique to this
+            :class:`~basyx.aas.model.provider.AbstractObjectStore` are always preserved
+        :return: Counts of processed :class:`Identifiables <basyx.aas.model.base.Identifiable>` as
+            ``(added, overwritten, skipped)``
+        """
+
+        added, overwritten, skipped = 0, 0, 0
+        for identifiable in other:
+            identifiable_id = identifiable.id
+            if identifiable_id in self:
+                if overwrite:
+                    existing = self.get_identifiable(identifiable_id)
+                    self.discard(cast(_IT, existing))
+                    self.add(identifiable)
+                    overwritten += 1
+                else:
+                    skipped += 1
+            else:
+                self.add(identifiable)
+                added += 1
+        return added, overwritten, skipped
+
 
 class DictObjectStore(AbstractObjectStore[_IT], Generic[_IT]):
     """

server/Dockerfile

@@ -23,14 +23,22 @@ RUN chmod +x /etc/supervisor/stop-supervisor.sh
 
 # Makes it possible to use a different configuration
 ENV UWSGI_INI=/etc/uwsgi/uwsgi.ini
-# object stores aren't thread-safe yet
+# Object stores aren't thread-safe yet
 # https://github.com/eclipse-basyx/basyx-python-sdk/issues/205
 ENV UWSGI_CHEAPER=0
 ENV UWSGI_PROCESSES=1
 ENV NGINX_MAX_UPLOAD=1M
 ENV NGINX_WORKER_PROCESSES=1
 ENV LISTEN_PORT=80
 ENV CLIENT_BODY_BUFFER_SIZE=1M
+ENV API_BASE_PATH=/api/v3.0/
+
+# Default values for the storage envs
+ENV INPUT=/input
+ENV STORAGE=/storage
+ENV STORAGE_PERSISTENCY=False
+ENV STORAGE_OVERWRITE=False
+VOLUME ["/input", "/storage"]
 
 # Copy the entrypoint that will generate Nginx additional configs
 COPY server/entrypoint.sh /entrypoint.sh

server/README.md

@@ -6,10 +6,10 @@ The server currently implements the following interfaces:
 - [Asset Administration Shell Repository Service][4]
 - [Submodel Repository Service][5]
 
-It uses the [HTTP API][1] and the [AASX][7], [JSON][8], and [XML][9] Adapters of the [BaSyx Python SDK][3], to serve regarding files from a given directory.
+It uses the [HTTP API][1] and the [*AASX*][7], [*JSON*][8], and [*XML*][9] Adapters of the [BaSyx Python SDK][3], to serve regarding files from a given directory.
 The files are only read, changes won't persist.
 
-Alternatively, the container can also be told to use the [Local-File Backend][2] instead, which stores AAS and Submodels as individual JSON files and allows for persistent changes (except supplementary files, i.e. files referenced by `File` submodel elements).
+Alternatively, the container can also be told to use the [Local-File Backend][2] instead, which stores Asset Administration Shells (AAS) and Submodels as individual *JSON* files and allows for persistent changes (except supplementary files, i.e. files referenced by `File` SubmodelElements).
 See [below](#options) on how to configure this.
 
 ## Building
@@ -19,17 +19,20 @@ The container image can be built via:
 $ docker build -t basyx-python-server -f Dockerfile ..
 ```
 
-Note that when cloning this repository on Windows, Git may convert the line separators to CRLF. This breaks `entrypoint.sh` and `stop-supervisor.sh`. Ensure both files use LF line separators before building. 
+Note that when cloning this repository on Windows, Git may convert the line separators to CRLF. This breaks [`entrypoint.sh`](entrypoint.sh) and [`stop-supervisor.sh`](stop-supervisor.sh). Ensure both files use LF line separators (`\n`) before building. 
 
 ## Running
 
 ### Storage
 
-The container needs to be provided with the directory `/storage` to store AAS and Submodel files: AASX, JSON, XML or JSON files of Local-File Backend.
+The server makes use of two directories:
 
-This directory can be mapped via the `-v` option from another image or a local directory.
-To map the directory `storage` inside the container, `-v ./storage:/storage` can be used.
-The directory `storage` will be created in the current working directory, if it doesn't already exist.
+- **`/input`** - *start-up data*: Directory from which the server loads AAS and Submodel files in *AASX*, *JSON* or *XML* format during start-up. The server will not modify these files.
+- **`/storage`** - *persistent store*: Directory where all AAS and Submodels are stored as individual *JSON* files if the server is [configured](#options) for persistence. The server will modify these files.
+
+The directories can be mapped via the `-v` option from another image or a local directory.
+To mount the host directories into the container, `-v ./input:/input -v ./storage:/storage` can be used.
+Both local directories `./input` and `./storage` will be created in the current working directory, if they don't already exist.
 
 ### Port
 
@@ -38,31 +41,40 @@ To expose it on the host on port 8080, use the option `-p 8080:80` when running
 
 ### Options
 
-The container can be configured via environment variables:
-- `API_BASE_PATH` determines the base path under which all other API paths are made available.
-  Default: `/api/v3.0`
-- `STORAGE_TYPE` can be one of `LOCAL_FILE_READ_ONLY` or `LOCAL_FILE_BACKEND`:
-  - When set to `LOCAL_FILE_READ_ONLY` (the default), the server will read and serve AASX, JSON, XML files from the storage directory.
-    The files are not modified, all changes done via the API are only stored in memory.
-  - When instead set to `LOCAL_FILE`, the server makes use of the [LocalFileBackend][2], where AAS and Submodels are persistently stored as JSON files.
-    Supplementary files, i.e. files referenced by `File` submodel elements, are not stored in this case.
-- `STORAGE_PATH` sets the directory to read the files from *within the container*. If you bind your files to a directory different from the default `/storage`, you can use this variable to adjust the server accordingly.
+The container can be configured via environment variables. The most important ones are summarised below:
+
+| Variable              | Description                                                                                                                                                                                                                                                                                                  | Default      |
+|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
+| `API_BASE_PATH`       | Base path under which the API is served.                                                                                                                                                                                                                                                                     | `/api/v3.0/` |
+| `INPUT`               | Path inside the container pointing to the directory from which the server takes its start-up data (*AASX*, *JSON*, *XML*).                                                                                                                                                                                   | `/input`     |
+| `STORAGE`             | Path inside the container pointing to the directory used by the server to persistently store data (*JSON*).                                                                                                                                                                                                  | `/storage`   |
+| `STORAGE_PERSISTENCY` | Flag to enable data persistence via the [LocalFileBackend][2]. AAS/Submodels are stored as *JSON* files in the directory specified by `STORAGE`. Supplementary files, i.e. files referenced by `File` SubmodelElements, are not stored. If disabled, any changes made via the API are only stored in memory. | `False`      |
+| `STORAGE_OVERWRITE`   | Flag to enable storage overwrite if `STORAGE_PERSISTENCY` is enabled. Any AAS/Submodel from the `INPUT` directory already present in the LocalFileBackend replaces its existing version. If disabled, the existing version is kept.                                                                          | `False`      |
+
+
+This implies the following start-up behaviour:
+
+- Any AAS/Submodel found in `INPUT` is loaded during start-up.
+- If `STORAGE_PERSISTENCY = True`:
+  - Any AAS/Submodel *not* present in the LocalFileBackend is added to it.
+  - Any AAS/Submodel *already present* is skipped, unless `STORAGE_OVERWRITE = True`, in which case it is replaced.
+- Supplementary files (e.g., `File` SubmodelElements) are never persisted by the LocalFileBackend.
 
 ### Running Examples
 
 Putting it all together, the container can be started via the following command:
 ```
-$ docker run -p 8080:80 -v ./storage:/storage basyx-python-server
+$ docker run -p 8080:80 -v ./input:/input -v ./storage:/storage basyx-python-server
 ```
 
 Since Windows uses backslashes instead of forward slashes in paths, you'll have to adjust the path to the storage directory there:
 ```
-> docker run -p 8080:80 -v .\storage:/storage basyx-python-server
+> docker run -p 8080:80 -v .\input:/input -v .\storage:/storage basyx-python-server
 ```
 
-Per default, the server will use the `LOCAL_FILE_READ_ONLY` storage type and serve the API under `/api/v3.0` and read files from `/storage`. If you want to change this, you can do so like this:
+By default, the server will use the standard settings described [above](#options). Those settings can be adapted in the following way:
 ```
-$ docker run -p 8080:80 -v ./storage2:/storage2 -e API_BASE_PATH=/api/v3.1 -e STORAGE_TYPE=LOCAL_FILE_BACKEND -e STORAGE_PATH=/storage2 basyx-python-server
+$ docker run -p 8080:80 -v ./input:/input2 -v ./storage:/storage2 -e API_BASE_PATH=/api/v3.1/ -e INPUT=/input2 -e STORAGE=/storage2 -e STORAGE_PERSISTENCY=True -e STORAGE_OVERWRITE=True basyx-python-server
 ```
 
 ## Building and Running the Image with Docker Compose
@@ -72,8 +84,9 @@ The container image can also be built and run via:
 $ docker compose up
 ```
 
-This is the exemplary `compose.yml` file for the server:
+An exemplary [`compose.yml`](compose.yml) file for the server is given [here](compose.yml):
 ```yaml
+name: basyx-python-server
 services:
   app:
     build:
@@ -82,13 +95,16 @@ services:
     ports:
     - "8080:80"
     volumes:
+      - ./input:/input
       - ./storage:/storage
+    environment:
+      STORAGE_PERSISTENCY: True
 ```
 
-Here files are read from `/storage` and the server can be accessed at http://localhost:8080/api/v3.0/ from your host system. 
-To get a different setup this compose.yaml file can be adapted and expanded.
+Input files are read from `./input` and stored persistently under `./storage` on your host system. The server can be accessed at http://localhost:8080/api/v3.0/ from your host system. 
+To get a different setup, the [`compose.yml`](compose.yml) file can be adapted using the options described [above](#options), similar to the third [running example](#running-examples).
 
-Note that the `Dockerfile` has to be specified explicitly, as the build context must be set to the parent directory of `/server` to allow access to the local `/sdk`.
+Note that the `Dockerfile` has to be specified explicitly via `dockerfile: server/Dockerfile`, as the build context must be set to the parent directory of `/server` to allow access to the local `/sdk`.
 
 ## Running without Docker (Debugging Only)
 
@@ -103,7 +119,7 @@ The server can also be run directly on the host system without Docker, NGINX and
    $ pip install ./app
    ```
 
-2. Run the server by executing the main function in [`./app/interfaces/repository.py`](./app/interfaces/repository.py) from within the current folder.
+2. Run the server by executing the main function in [`./app/interfaces/repository.py`](./app/interfaces/repository.py).
    ```bash
    $ python -m app.interfaces.repository
    ```
@@ -119,7 +135,7 @@ This Dockerfile is inspired by the [tiangolo/uwsgi-nginx-docker][10] repository.
 [3]: https://github.com/eclipse-basyx/basyx-python-sdk
 [4]: https://app.swaggerhub.com/apis/Plattform_i40/AssetAdministrationShellRepositoryServiceSpecification/V3.0.1_SSP-001
 [5]: https://app.swaggerhub.com/apis/Plattform_i40/SubmodelRepositoryServiceSpecification/V3.0.1_SSP-001
-[6]: https://industrialdigitaltwin.org/content-hub/aasspecifications/idta_01002-3-0_application_programming_interfaces
+[6]: https://industrialdigitaltwin.io/aas-specifications/IDTA-01002/v3.0/index.html
 [7]: https://basyx-python-sdk.readthedocs.io/en/latest/adapter/aasx.html#adapter-aasx
 [8]: https://basyx-python-sdk.readthedocs.io/en/latest/adapter/json.html
 [9]: https://basyx-python-sdk.readthedocs.io/en/latest/adapter/xml.html