|
| 1 | +# Architecture and design[^1] |
| 2 | + |
| 3 | +We drew our inspiration from the microkernel architecture[^2] for the overall |
| 4 | +architecture principle[^3] and layered architecture for architecture partitioning. There was no strong rationale for |
| 5 | +choosing a microkernel architecture over alternatives such as a microservice architecture, other than the desire to |
| 6 | +start with something simple that still allows for extensibility. We wanted elAPI to serve as the foundation for all |
| 7 | +eLabFTW automation solutions. In other words, eLabFTW users should be able to package their automation scripts as |
| 8 | +plugins, thereby extending elAPI’s functionality. We refer to this architecture as the "simple microkernel |
| 9 | +architecture (SMA)". In the very beginning, SMA was designed for elAPI only. Later we have improved and generalized it |
| 10 | +enough to be useful for any software design. |
| 11 | + |
| 12 | +<img src="https://heibox.uni-heidelberg.de/f/32f04718692e4adaa1ff/?dl=1" alt="elAPI simple microkernel architecture" /> |
| 13 | + |
| 14 | +## SMA principles |
| 15 | + |
| 16 | +SMA follows most of the principles found in microkernel architecture guidelines with a few extra additions. |
| 17 | + |
| 18 | +**1. A core system must provide basic functionalities:** The core system principle is adopted straight from the |
| 19 | +textbook microkernel architecture principles[^3]. In SMA, the core system must provide the basic features. Here, |
| 20 | +features that would be considered as basic would depend on the software domain. At a minimum, the core system must |
| 21 | +enable the |
| 22 | +plugin system. The core system in elAPI offers interface style guides, logging functionality, helper functions, OS path |
| 23 | +handlers, basic validation abstractions, configuration manager, extended HTTPX API interfaces to communicate with |
| 24 | +eLabFTW endpoints, and plugin handlers that enable the entire plugin system. elAPI's core system can be extracted out |
| 25 | +and used as the foundation for the SMA implementation for any standalone program. |
| 26 | + |
| 27 | +**2. Interfaces can be flexible across internal plugins:** Typically, a microkernel architecture separates plugins |
| 28 | +from the core system entirely. SMA modifies this approach by introducing internal plugins, which reside inside the core |
| 29 | +codebase, and external plugins (a.k.a. "third-party plugins"), which can exist anywhere. Both types of plugins can share |
| 30 | +the same plugin _interface_. This is also the default with elAPI's SMA implementation. Depending on flexibility |
| 31 | +preference, |
| 32 | +however, having a unique interface or distinct interfaces for internal plugins is also allowed where a unique interface |
| 33 | +is enforced upon external plugins. |
| 34 | + |
| 35 | +**3. Internal plugins can be part of the core system:** In SMA, internal plugins can be part of the core system |
| 36 | +such that they are meant to be used by external plugins in the same way core system functionalities are used by all |
| 37 | +plugins in a typical microkernel architecture. In elAPI, public APIs offered by the internal plugin |
| 38 | +_“experiments”_ are meant to be leveraged by external plugins to make updates or advanced modifications to |
| 39 | +eLabFTW experiments, among other features. Note, if external plugins are disabled for a SMA implementation, then |
| 40 | +internal plugins should not be part of the core system. |
| 41 | + |
| 42 | +**4. Hybrid partitioning is allowed:** SMA recommends technical partitioning for the basic functionality layers. If |
| 43 | +domain partitioning is desired, then it is recommended to apply domain partitioning to internal and external plugins. |
| 44 | +The combined partitioning helps enforce a design based on strict separation of concerns. |
| 45 | + |
| 46 | +**5. A strict unidirectional dependency must be maintained between technically partitioned layers:** SMA imposes a |
| 47 | +dependency invariance between technically partitioned layers. In figure 1, we see a top layer always “depends on” a |
| 48 | +bottom layer, or in other words, a bottom layer’s APIs are always exposed to a top layer. This is denoted with an upward |
| 49 | +arrow. A layer that sits at the most bottom is an independent layer, as it doesn’t depend on any other layer. The |
| 50 | +third-party libraries (typically whose code do not reside in the codebase) do no take part in the invariance, and are |
| 51 | +considered floating layers. In that sense, an independent layer is also a floating layer. In elAPI, the `styles` layer |
| 52 | +is an independent layer. `styles` layer designate user interface definitions and methods, e.g., the color scheme used by |
| 53 | +elAPI CLI. Styles being the independent layer indicates that all dependent layers must adhere to the same UI guidelines. |
| 54 | + |
| 55 | +**6. Plugins need not be independent:** This principle is in line with the third principle. Microkernel |
| 56 | +architecture imposes that plugins must be independent, and for good reasons, i.e., to avoid “Big Ball of Mud”[^4]. SMA |
| 57 | +still lets the developer decide the rules internal and external plugins should follow. |
| 58 | +For example, external plugins can depend on internal plugins (as they are part of the core system). An external plugin |
| 59 | +vendor is free to decide whether their plugin would depend on another external plugin. |
| 60 | + |
| 61 | +## elAPI SMA layers |
| 62 | + |
| 63 | +We have previously briefly touched on the basic functionality layers elAPI has to offer. In this section, we will take |
| 64 | +a look at the responsibilities of each layer. |
| 65 | + |
| 66 | +**`styles`**: The `styles` layer, an independent layer, is created out of the philosophy that elAPI's CLI UI should be |
| 67 | +user-customizable. At the moment, though, customization is not available as a feature, but the layer offers UI |
| 68 | +guidelines to all layers above. E.g., how a JSON output is formatted and syntax highlighted is defined in this layer. |
| 69 | +All internal |
| 70 | +plugins inherit the same formatting style for JSON output. |
| 71 | + |
| 72 | +**`names`**: Almost every critical constant is defined in this layer. This includes the name of the app itself |
| 73 | +_elAPI/elapi_, configuration keys, log file path, etc. Having a single source of truth of constants makes it |
| 74 | +quite easy to fork and re-use elAPI's SMA implementation for any other software. |
| 75 | + |
| 76 | +**`_core_init`**: `_core_init` is a private layer where the most basic methods of elAPI are defined. Plugins should not |
| 77 | +import from this layer, but from the layers above it. These basic methods help initialize certain functionalities for |
| 78 | +some upper layers. E.g., the layer above `loggers` must also be able to log its own errors, so a simple logger class |
| 79 | +`STDERRLogger` is placed in `_core_init`. In fact, for the most part, elAPI's plugin developers don't even need to know |
| 80 | +`_core_init` exists, as its methods are also imported in upper layers. |
| 81 | + |
| 82 | +**`loggers`**: The `loggers` layer provides custom methods for logging to all other layers and plugins; hence it sits |
| 83 | +quite at the bottom. This custom method doesn't reinvent logging APIs, but rather simply extends upon Python's built-in |
| 84 | +logging APIs via composition. E.g., the `STDERR` handler is a [`RichHandler`](https://ludwig.guru/) that enables colored |
| 85 | +log output on the CLI. The method by default also logs to a file. |
| 86 | + |
| 87 | +```python |
| 88 | +from elapi.loggers import Logger |
| 89 | + |
| 90 | +logger = Logger() |
| 91 | +# Logger() returns a pre-configured Python built-in Logger object |
| 92 | +logger.error("An error") |
| 93 | +``` |
| 94 | + |
| 95 | +elAPI `Logger` object is a singleton object. |
| 96 | + |
| 97 | +**`utils`**: As the layer name suggests, this layer mainly hosts utility/helper methods. A few custom exceptions, |
| 98 | +special data containers like `MessagesList`, monkey-patched fixes for third-party library bugs, etc. are placed here. |
| 99 | +`utils` mainly import and expose core utility methods from `core_init/` layer: `get_app_version`, and app-wide callback |
| 100 | +methods. |
| 101 | + |
| 102 | +**`path`**: Python comes with the powerful `pathlib` library for handling OS paths. Just like the |
| 103 | +`Logger` object, elAPI again extends upon `pathlib.Path` object, and enables some additional path |
| 104 | +management features that are quintessential for elAPI. The extended path object is named `ProperPath`, and |
| 105 | +its most notable instance methods are: `kind`, `create`, `remove` and `PathException`. `kind` reveals if the |
| 106 | +instantiated OS path is a file or a directory. There are some tricky |
| 107 | +cases with special files like `/dev/null` which `kind` treats carefully. Based on the file |
| 108 | +`kind`, instance methods `create` and `remove` can effectively create and remove any file |
| 109 | +or directory, respectively. This avoids boilerplate code needed with `pathlib.Path` to first determine if a |
| 110 | +path is a file or a directory. If a `ProperPath` path encounters an exception during an I/O operation (e.g., |
| 111 | +insufficient permission for file creation), the exception is first stored in `PathException` attribute before |
| 112 | +being raised. Why? `PathException` offers a fallback mechanism for I/O failures. If writing to a file fails, |
| 113 | +it doesn't necessarily mean the entire operation has failed, but it is possible that only the file path is problematic, |
| 114 | +and an alternative file path should be tried instead. Plugins like `bill-teams` can iterate over a list of desired paths |
| 115 | +for writing data, and if the first path fails, the exception can be caught with the path's `PathException`, and |
| 116 | +the plugin can check if stored exception in `PathException` is fatal or non-fatal (e.g., insufficient |
| 117 | +permission). If non-fatal, it can move on to the next iteration of file paths and retry writing data. This fallback |
| 118 | +approach rescues elAPI from losing data over avoidable I/O issues. In some ways, `PathException` follows the "errors as |
| 119 | +values" pattern[^5] from Go. |
| 120 | + |
| 121 | +**`core-validators`**: `core-validators` mainly provides abstract classes and exception for validations that internal |
| 122 | +and external plugins are meant to make use of. Despite its name "validators", elAPI's internal plugins actually make use |
| 123 | +of validations in the form of parsing—thus adhering to the "parse, don't validate" principle. E.g., The |
| 124 | +`PathValidator` provided by `core-validators` converts any valid string to `pathlib.Path` |
| 125 | +object. `core-validators` also provides a powerful `Exit` and its subclass `CriticalValidationError` that are meant to |
| 126 | +be raised when a critical irrecoverable error is encountered. `Exit` will raise Python |
| 127 | +`SystemExit` (quit the program cleanly without showing Python traceback) if it detects elAPI is being run on |
| 128 | +the CLI. It will show proper Python traceback if it detects elAPI is being used in a Python script. `Exit` will also |
| 129 | +call result callbacks (if any) before exiting. |
| 130 | + |
| 131 | +**`configuration`**: `configuration` layer manages user configuration for elAPI. Under the hood, it mainly utilizes |
| 132 | +third-party library [Dynaconf](https://dynaconf.com/) to search and parse user configuration from YAML files. |
| 133 | +`configuration` layer also provides |
| 134 | +validators for certain configuration key-value pairs found within the configuration file. `configuration` layer also |
| 135 | +offers configuration overloading — with which any configuration value defined in a file can be overwritten on the CLI |
| 136 | +itself (this CLI option is called `--OC` or `--override-config`). To make this possible, `configuration` layer |
| 137 | +implements a `ConfigHistory` class to store Dynaconf-parsed configuration, a `InspectConfigHistory` class to |
| 138 | +allow history inspection, and finally a special configuration value container `MinimalActiveConfiguration` |
| 139 | +that stores original and overridden configuration values. |
| 140 | + |
| 141 | +**`api`**: elAPI adopted and extended HTTPX APIs to simplify HTTP requests to eLabFTW. This layer is the bread and |
| 142 | +butter of elAPI as a client. It offers a power abstract class `APIRequest` that all other main HTTP requests classes are |
| 143 | +made out of. `APIRequest` defines and simplifies connection opening/closing, HTTP client sharing across |
| 144 | +multiple calls, and separating asynchronous and synchronous methods. |
| 145 | + |
| 146 | + |
| 147 | +> [!NOTE] |
| 148 | +> This section is yet to be completed. |
| 149 | +
|
| 150 | +[^1]: This page has been adapted from an [E-Science-Tage 2025](https://e-science-tage.de/en/downloads) conference paper. |
| 151 | +[^2]: https://csse6400.uqcloud.net/handouts/microkernel.pdf |
| 152 | +[^3]: https://www.oreilly.com/videos/fundamentals-of-software/9781663728357/ |
| 153 | +[^4]: http://www.laputan.org/mud/ |
| 154 | +[^5]: https://jessewarden.com/2021/04/errors-as-values.html |
0 commit comments