Merge pull request #53 from DrMarkusVoss/52-add-requirements-in-docs-as-code-format-with-traceability-to-architecture-elements

52 add requirements in docs as code format with traceability to architecture elements

Author: DrMarkusVoss

.github/workflows/test.yml

@@ -15,7 +15,7 @@ jobs:
         python-version: 3.8
     - name: install dependencies
       run: |
-        pip install flake8 pytest requests
+        pip install flake8 pytest requests pyyaml
         pip install .
     - name: run tests
       run:

Examples_Reqs.md

@@ -0,0 +1,99 @@
+# `pumla` WeatherStation examples for requirements in docs-as-code style
+These are examples from the `./test/examples/WeatherStation` section. You find the
+source code of the .puml and .yaml files that form the example model repository
+there. In order to get the example running on your computer, you need
+to run first `pumla init`, then `pumla update`" in the examples directory, 
+because the paths in the model repo file (./test/examples/WeatherStation/reqsrepo_json.puml)
+need to be updated to the directory structure on your computer. Currently
+the model repo file has the paths from the structure as I have it on my
+machine. You can't run the pumla update from the main path of pumla, as
+it is setup to update the architecture model repo of the pumla tool itself.
+The examples section will be ignored from the top level. So you have to go
+down to the examples and call the pumla update from that directory.
+
+If you are changing the contents of the examples to play around, in order for a
+change to become alive, you need to call `pumla update` again. cd 
+
+## E#0: Creating Re-usable Requirements
+Other than with architecture elements, in `pumla` you write the requirements in YAML and not
+with PlantUML-conformant PUMLA-macros. Reason for that is, that requirements typically have
+more textual relevant information and a plain table structure with potentially quite a lot
+attributes that you want to have easy access on. Table-like data structures can be handled by PlantUML
+with JSON representations on diagrams. But when it comes to writing the text files, YAML is easier
+to use for requirements documentation than JSON. Therefore, `pumla` parses the YAML files and
+puts them into the requirements repository in JSON format, so that you can put the requirements 
+easily onto diagrams.
+
+See here an example for the definition of a re-usable requirement using by describing
+it in YAML:
+[./test/examples/WeatherStation/req.yaml](test/examples/WeatherStation/req.yaml)
+
+```
+#PUMLARR
+- type: Requirement
+  alias: REQ_WS1
+  status: decided
+  derived_from:
+  taggedvalues:
+    - tag: "Vendor"
+      values:
+      - A Inc.
+      - C Ltd.
+    - tag: "Variant"
+      values: [SysA, SysB]
+  content:
+    This is a requirement towards my Weather Station. The Weather Station shall
+    be able to measure the temperature.
+
+- type: Requirement
+  alias: REQ_WS2
+  status: new
+  derived_from:
+  content:
+    This is another requirement. The Weather Station housing shall be blue.
+
+...
+```
+
+
+## E#1: Show all Requirements
+### E#1.1: All Requirements, full attributes, no trace
+See, how all requirements are put onto a diagram using the "`PUMLAPutAllReqs()`" macro for that
+purpose.
+
+[./test/examples/WeatherStation/exampleAllReqs.puml](test/examples/WeatherStation/exampleAllReqs.puml)
+
+![](test/examples/WeatherStation/pics/exampleAllReqs.png)
+
+### E#1.2: All requirements, brief subset of attributes, showing the trace
+In this example, all requirements are layed out on the diagram with a brief description. Furthermore,
+the trace between the requirements is exposed. All with just one `pumla` macro call.
+
+[./test/examples/WeatherStation/exampleAllReqsBrief.puml](test/examples/WeatherStation/exampleAllReqsBrief.puml)
+
+![](test/examples/WeatherStation/pics/exampleAllReqsBrief.png)
+### E#1.3: Overview of all requirements (Cheat Sheet)
+See here, how with one `pumla` macro `PUMLAPutAllReqsTable()`  and overview table is put onto
+the diagram, counting the requirements and showing in which files they are implemented.
+
+[./test/examples/WeatherStation/exampleReqCheatSheet.puml](test/examples/WeatherStation/exampleReqCheatSheet.puml)
+
+![](test/examples/WeatherStation/pics/exampleReqCheatSheet.png)
+
+## E#2: Put single requirements onto a diagram
+Here is an example where two requirements are put onto a diagram, one with a reduced subset of
+attributes. 
+
+[./test/examples/WeatherStation/exampleSingleReqs.puml](test/examples/WeatherStation/exampleSingleReqs.puml)
+
+![](test/examples/WeatherStation/pics/exampleSingleReqs.png)
+
+## E#3: Show the requirements trace 
+This example shows how for a given requirement the requirements trace is created as a diagram with just one
+`pumla` macro call. You can
+argue that the example might be bad for a requirements breakdown, but it is here to show the principle
+and possibilities of pumla, not to show examples of perfect requirements engineering.
+
+[./test/examples/WeatherStation/exampleReqTrace.puml](test/examples/WeatherStation/exampleReqTrace.puml)
+
+![](test/examples/WeatherStation/pics/exampleReqTrace.png)

README.md

@@ -24,6 +24,12 @@ Therefore, in order to enable systematic re-use for architecture models with Pla
 `pumla` is intended to be an extension around PlantUML to organize and enable
 this systematic re-use.
 
+Furthermore, `pumla` now also supports requirements management in a docs-as-code manner. You can
+store your requirements in YAML text files in your git repo along with code and architecture, and
+also re-use the requirements definitions on diagrams, mixing it with architecture elements to show
+which element implements which requirement, but also create PlantUML diagrams showing a traceability
+graph.
+
 ### Goals, Use Cases & Principles
 <details>
 <summary>click to expand!</summary>
@@ -40,6 +46,8 @@ this systematic re-use.
   sequence diagrams with deployment diagrams.
 - Create an arc42 architecture documentation based on common PlantUML description patterns
   with almost no effort.
+- requirements in docs-as-code style that can be used in diagram explaining how they are implemented
+  in the architecture
 
 ---
 
@@ -209,6 +217,12 @@ to an overview and description an that example:
 
 [follow this link to the C4 example](./test/examples/C4example/pumlaC4Example.md)
 
+### Examples for managing and using requirements 
+This is a set of examples showing how to use `pumla` for requirements management.
+
+[follow this link to the requirements-as-code examples](./Examples_Reqs.md)
+
+
 ### Working with the examples
 In order to play around with the previously mentioned examples, you need to initialise the examples
 repository to work on your system. To do that, take the following steps starting in the pumla

arch/pumla_architecture.md

@@ -56,4 +56,42 @@ See the following image to get an idea of the big picture:
 
 ![](./00_pics/overview_internalstructure.png)
 
+## `pumla` Requirements as Code approach
+In systems engineering, architecture typically is highly connected to requirements engineering. The requirements
+are the base to tell you what is expected from the system. Tests are written against requirements (or are a means
+to describe a requirement). The traceability from architecture elements to requirements is crucial for knowing which
+part of the implementation is intended to do what and why at all. When product variants come into play, the
+start point of the variance is typically already on the requirements. So, not every product variant needs to have
+each requirement fulfilled.
 
+Therefore, bringing requirements engineering and architecture together into one docs-as-code/models-as-code approach
+has the potential to ease the implementation of traceability and therefore simplify systems engineering in software intense
+systems.
+
+### Design Principles & Ideas
+- [ ] requirements as YAML text: YAML better editable and readable for req. documentation as text than JSON
+  - transform internally to JSON to be able to use contents with PlantUML Preprocessing
+- [ ] traceability: in docs only one-directional trace from the lower requirement to the higher requirement (where it has been derived from)
+- [ ] generate bi-directional traceability in „pumla update“ step. create a reqrepo_json.puml file with all collected req from whole project
+    - [ ] have for each req „derived to“ and „derived from“ as keys. „derived to“ is created during the update step
+    - [ ] „derived from“ is documented as key in the YAML files
+- [ ] traceability to architecture artifacts. which arch element realizes a certain requirement.
+    - [ ] realizes is a part of the architecture model, not of the requirement.
+    - [ ] requirements are independent of their implementation
+    - [ ] a requirement belongs to a scope; within the scope it gets broken down
+- [ ] requirements can be structured in features.
+    - [ ] req that belong to a feature can have a loose or high coupling
+    - [ ] loose coupling will be realized with the „child injection“ mechanism
+    - [ ] high coupling by a feature table that directly names reqs that belong to the feature
+    - [ ] feature structure is kind of an architectural step with a composite breakdown view
+- [ ] automatic scope alignment if the req doc is next a single RUA arch spec? or manual scope connection? do we need the scope at all?
+- [ ] PUMLA macros to create diagrams that show all requirements fulfilled by a certain architecture element
+- [ ] PUMLA macros to create feature structure & breakdown views
+- [ ] PUMLA macros to create traceability views
+- [ ] status „aligned“, „new“, „decided“
+    - [ ] whether it is „realized/implemented“ will be decided with the respective test passing
+    - [ ] „rejected“ can be a commit message for deleting the req
+    - [ ] new: indicated a new unreviewed requirement which is there but not yet considered somewhere
+    - [ ] aligned: requirement is reviewed and aligned to be usable, but not yet decided how to be realized
+    - [ ] decided: there is a design decision done on to where and how the requirement shall be realized.
+  

pumla_UsersGuide.md

@@ -88,22 +88,26 @@ you can name these in a `pumla_blacklist.txt` file. You can also have comments i
 blacklist file using `#` as first character of the line. But comments must be
 standalone lines, you cannot put a comment behind a path or filename. 
 The .puml files found will be parsed
-and if they are pumla files, their relevant content will be extracted and stored in the 
+and if they are pumla architecture files, their relevant content will be extracted and stored in the 
 `modelrepo_json.puml` file or the given
-filename. Each call will overwrite the existing `modelrepo_json.puml`
+filename. For docs-as-code requirements files, the relevant content will be extracted and stored in the
+`reqsrepo_json.puml` file.
+Each call will overwrite the existing `modelrepo_json.puml` and `reqsrepo_json.puml`
 file, therefore they should not be modified by hand,
 because changes will get lost. These repository files are the basis for the
 PlantUML extension macros. The macros help to get data out of these repositories
 and thereby re-use the once defined model elements and diagrams in a structured 
 way.
 
-The generated `modelrepo_json.puml` should also not be checked-in into a version
+The generated `modelrepo_json.puml` as well as the `reqsrepo_json.puml`
+should also not be checked-in into a version
 control system, as it contains absolute paths that are different on every computer
 where the source code is checked out. Therefore, after a fresh checkout/clone of 
 a source code repository, first thing to do is call `pumla init` to create a link
 to the pumla macros to access the model repository JSON database, the second
 thing to do is call `pumla update` to scan the source code repository for pumla
-files and create the `modelrepo_json.puml` file. After that you can use the
+files and create the `modelrepo_json.puml` and `reqsrepo_json.puml` file.
+After that you can use the
 model repository to create diagrams and are able to use all the macro commands 
 explained in the following chapters.
 
@@ -127,11 +131,19 @@ is put to the command line output.
 Called without any other argument, pumla writes out to the console a
 list of all `.puml` files that are `pumla` files (marked with either `'PUMLAMR`
 (PUMLA-ModelRepo) or `'PUMLADR` (PUMLA-DiagramRepo) in the first line of the file).
+Requirements are described in YAML files (`.yaml`) and will therefore not
+be shown by this command, as they are not standalone PlantUML-compliant files.
 
 ### `pumla elements`
 Writes out to the console all `pumla` model elements and diagrams of 
 the model repository and diagram repository, with all their relevant 
-attributes.
+attributes. This does not include requirements, for that we have the
+call described next.
+
+### `pumla reqs` - not yet implemented
+Writes out to the console all `pumla` requirements elements and diagrams of 
+the model repository and diagram repository, with all their relevant 
+attributes. 
 
 ### `pumla getjson <subcommand>`
 Prints out a JSON data structure containing the request elements (by subcommand).
@@ -183,14 +195,19 @@ This comment marks the file as a re-usable asset of a dynamic behaviour
 element description. In a file marked like that, only the dynamic aspect
 should be modelled, by e.g. a sequence, state or activity diagram.
 
+### `'PUMLARR`
+This comment put into the first line of a `.yaml` file marks the file
+as `pumla` requirement element description that shall be part of the re-usable
+requirements repository.
+
 ---
 
 ## Extension Macros
 The following macros (that are PlantUML unquoted functions and procedures) can be
 used within each PlantUML file by including `pumla_macros.puml`. They are the
 user "API" of `pumla`.
 
-## Creating Re-usable Elements 
+## Creating Re-usable Model Elements 
 
 ### `PUMLAReUsableAsset($name : string, $alias : string, $type : string, $stereotypes : string (optinal)) {...}`
 Creates a re-usable asset (RUA). The asset can be of any PlantUML type, like class, component, node, rectangle, ...
@@ -313,7 +330,7 @@ you manage your architecture relations and artefacts.
 Similar to previous, but instead of information about file path and file name you
 get the list of tagged values that are attached to each of the connections.
 
-## Putting Re-usable Elements onto Diagrams
+## Putting Re-usable Model Elements onto Diagrams
 
 ### `PUMLAPutElement( elem_alias )`
 This macro puts the model element with the given alias `elem_alias` onto the diagram.
@@ -504,6 +521,11 @@ as the table cannot be shown within the port element (and would not
 make sense). This is the only way to show the tagged value table
 of a port.
 
+### `PUMLAPutReqsBreakdownTraceFor($alias)`
+Puts the requirement with the given alias onto the diagram and follows
+its "derived" breakdown and puts further requirements onto the diagram
+that the alias requirement is derived to. It connects the requirements
+with a "realization" relation.
 
 ## Diagram Filters
 
@@ -572,6 +594,108 @@ global variable `$PUMVarShowBodyInternals` is true. With this macro, the given a
 of which not to show the internals. In order to show internals for that element within the scope of the same diagram
 again, the "filter list" has to be resetted with the next macro.
 
+## Creating Re-usable Requirements Elements 
+
+### Creating a single re-usable docs-as-code requirement
+Requirements are stored in YAML files (`.yaml`) with a file marking `#PUMLARR`in the first line.
+Then, within the file, one or more requirements can be described. The requirements are elements
+of a YAML list with at least the following attributes:
+- alias: the unique short-ID 
+- type: describing the kind of requirement. Can be set according to needs. Could be "Requirement", "Constraint", ...
+- status: also here, the status model is not defined by PUMLA; put status values as you need and like.
+- derived_from: the alias of the (parent) requirement where this requirement is derived from.
+- content: The text content of the requirement itself.
+
+The following attributes are optional:
+- taggedvalues: a list of tag/values pairs, where values is list by itself.
+
+Requirement files can be distributed across the folders in a git repository and therefore be
+stored close to the code if wanted. The `pumla update`
+command will find each of them, considering the blacklist.
+
+Example:
+```
+File: req.yaml
+
+#PUMLARR
+- type: Requirement
+  alias: REQ_WS1
+  status: decided
+  derived_from:
+  taggedvalues:
+    - tag: "Vendor"
+      values:
+      - A Inc.
+      - C Ltd.
+    - tag: "Variant"
+      values: [SysA, SysB]
+  content:
+    This is a requirement towards my Weather Station. The Weather Station shall
+    be able to measure the temperature.
+
+- type: Requirement
+  alias: REQ_WS2
+  status: new
+  derived_from:
+  content:
+    This is another requirement. The Weather Station housing shall be blue.
+```
+
+## Overview on the Requirements Repository of Re-usable Elements
+### `PUMLAPutAllReqsTable()`
+This macro puts all requirements of the `reqsrepo_json.puml` into a table on the diagram,
+with a count of the requirements as well as the files where the requirements are stored in. It
+is similar to the `CheatSheets` of the architecture model elements.
+
+### `PUMLAPutReqByCount($count)`
+This macro is solely meant for overview purposes, allowing you to quickly step through each
+single requirement in the requirements repository by addressing it by its number (`$count`). 
+
+Hint: The `$count` shall not be used to address a requirement a certain requirement or put
+it onto a real diagram with a real purpose. The numbering in the requirements repository can 
+change by calling `pumla update`, so that the `$count` may point to another requirement. 
+In real diagram, requirements shall always be addressed by their unique short-ID, the `alias`.
+
+## Putting Re-usable Requirements Elements onto Diagrams
+### `PUMLAPutReq($alias)`
+Puts the requirement with the given alias in form of a PlantUML JSON table with all its
+attributes onto the diagram.
+
+### `PUMLAPutAllReqs()`
+Puts all the requirements of the requirements repository (`reqsrepo_json.puml`) onto the 
+diagram.
+
+### `PUMLAPutAllReqsWithStatusTable($value)`
+Puts all requirements of the requirements repository, where the status has the given `$value`,
+onto the diagram.
+
+### `PUMLAPutAllReqsWithTypeTable($value)`
+Puts all requirements of the requirements repository, where the type has the given `$value`,
+onto the diagram.
+
+### `PUMLAPutReqBrief($alias)`
+Puts the requirement with the given alias onto the diagram, focussing on a limited amount of
+attributes:
+- type
+- content
+- status
+
+### `PUMLAPutReqAsNote($alias)`
+Puts the brief requirement with the given alias wrapped into a PlantUML note onto the diagram.
+
+Hint: This macro can be useful to put the requirement onto diagrams along with classes, components or
+dynamic behaviour descriptions.
+
+### `PUMLAPutAllReqsBrief()`
+Puts all the requirements of the requirements repository (`reqsrepo_json.puml`) onto the 
+diagram, but only with the brief set of attributes (see `PUMLAPutReqBrief($alias)`).
+
+## Working with Requirements Traceability
+### `PUMLAPutReqsBreakdownTraceFor($alias)`
+Puts the trace for a requirement with given alias down on the diagram. Each requirement along the
+trace is put with its brief description. The trace is modelled with a "realizes" relation, so pointing
+from a derived requirement to a parent requirement where it was derived from.
+
 ## Working around some PlantUML limitations
 
 ### `PUMLASetAsComponentDiagram()`

pumla_global_cfg.puml

@@ -33,3 +33,6 @@
 ' interaction description).
 !$PUMC4UseDynamicExt = %false()
 
+' max character width of requirements
+' conent attribute.
+!$PUMReqContentWidth = 15