docs: Significant level of module documentation

Author: Alan Christie

tests/message_dispatcher.py

@@ -1,17 +1,17 @@
 """The UnitTest Message Dispatcher.
 
-A very simple object that relies on an underlying message queue.
+A very simple object that relies on an underlying message queue and is designed
+to emulate the behaviour of the message queue used in the Data Manager.
 Here we offer a minimal implementation that simply sends a (protocol buffer) message
 to the queue.
 """
 
 from google.protobuf.message import Message
 
 from tests.message_queue import UnitTestMessageQueue
-from workflow.workflow_abc import MessageDispatcher
 
 
-class UnitTestMessageDispatcher(MessageDispatcher):
+class UnitTestMessageDispatcher:
     """A minimal Message dispatcher to support testing."""
 
     def __init__(self, msg_queue: UnitTestMessageQueue):

workflow/decoder.py

@@ -1,6 +1,19 @@
-"""A module to validate and decode workflow definitions.
-
-This is typically used by the Data Manager's Workflow Engine.
+"""A module to validate and decode workflow (and step) definitions.
+
+Module philosophy
+-----------------
+The _main_ purpose of this module is to provide a 'validate_schema()' function
+to check that a workflow definition (a dictionary) that is expected to comply with
+the 'workflow-schema,yaml' schema. This function returns a string (an error) if there's
+a problem with the defintion.
+
+The decoder module also provides a number of additional functions based on the needs
+of the engine. As a developer you are 'encouraged' to place any logic that is expected
+to navigate the scheme content in a function in this module. Any code that
+is supposed to know _where_ to get content should be encoded as a function here.
+For example, rather than external code navigating the 'plumbing' blocks
+we have a function 'get_workflow_variable_names()' that returns the names of workflow
+variables.
 """
 
 import os

workflow/workflow_abc.py

@@ -1,13 +1,61 @@
 """Workflow abstract base classes.
-Interface definitions of class instances that must be made available to the Engine.
+
+A module that provides interface definitions for classes that must be made available
+to the engine.
+
+Before go any further it is important to understand that a Workflow 'Step' is realised
+by the execution of a Data Manager 'Job'. A 'Step' is simnply the definition of
+a Job's execution withion the context of a 'Workflow'. We also talk about 'Instances'.
+Instances are a Data Manger concept. They are an object (and database Table)
+represening the running state of a Job.
+
+When steps 'Steps' are run the are represented by 'Jobs' that run as an 'Instance'.
+
+To this end the workflow engine relies on a two broad external services, encapsulated
+by abstract class definitions we define here: -
+
+- An 'Instance Laucncher' to facilitate the execution of Jobs
+- An API 'wrapper' providing access to an underling database that stores
+  Workflows, RunningWorkflows, RunningWorkflowSteps, and Instances.
+
+Module philosophy
+-----------------
+The engine is responsible for orchestrating Step exection (executing Jobs) but does not
+contain the logic that is able to run them. This is because a) job execution
+(in Kubernetes) is a complex affair and b) the Data Manager already provides this
+logic. Instead the engine defines an ABC for an 'InstanceLaucnher' and the
+DM provides the implementation. The engine simply has to create a 'LaunchParameter'
+object describign the Job to be laucnhed (including variables etc.) and then
+relies on the Instance Launcher to effect the execution.
+
+The engine also does not consist of any persistence capability and instead relies on the
+Data Manager's database to host suitable 'Workflow', 'RunningWorkflow',
+and 'RunningWorkflowStep' tables. The 'WorkflowAPIAdapter' defined here provides an
+interface that a concrete implementation uses to allow access to and modification
+of records withing these tables.
+
+The engine does not create or remove records directly, they are created either by the
+Data Manager via its API or the Instance laucnher when as it starts Jobs (Steps).
+The DM API creates a Workflow record when the user creates a Workflow.
+It also creates RunnignWorkflow records (while also validating them) when the
+user 'runs' a workflow. It also creates RunningWorkflowStep records to track the
+execution state of each step when the Instance lancher is called upon
+to start a Step.
+
+The instance launcher is controlled by a complex set of 'parameters' (a
+'LaunchPrameters' dataclass object) that comprehensively descibe the Job -
+it's variables, and inputs and outputs. The instance launcher provies just one method:
+'launch()'. It takes a paramters object, and in return the yields a 'LaunchResult'
+dataclass object that contains the record IDs of the instance created, and the
+corresponding RunningWorkflowStep. The result also describes any launch error.
+If there is a laucnh error the tep can assume to have not started. if there is
+no error the step will (probably) start.
 """
 
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 from typing import Any
 
-from google.protobuf.message import Message
-
 
 @dataclass
 class LaunchParameters:
@@ -358,11 +406,3 @@ def get_running_workflow_step_output_values_for_output(
         # {
         #   "output": ["dir/file1.sdf", "dir/file2.sdf"]
         # }
-
-
-class MessageDispatcher(ABC):
-    """The class handling the sending of messages (on the Data Manager message bus)."""
-
-    @abstractmethod
-    def send(self, message: Message) -> None:
-        """Send a message"""

workflow/workflow_engine.py

@@ -1,8 +1,12 @@
 """The WorkflowEngine execution logic.
 
-It responds to Pod and Workflow protocol buffer messages received by its
-'handle_message()' function, messages delivered by the message handler in the PBC Pod.
-There are no other methods in this class.
+Module philosophy
+-----------------
+The module implements the workflow execution logic, which is driven by
+Pod and Workflow protocol buffer messages received by its 'handle_message()' function.
+Messages are delivered by the message handler in the PBC Pod.
+There are no other publci methods in this class - it's _entry point_ is
+'handle_message()'.
 
 Its role is to translate a pre-validated workflow definition into the ordered execution
 of step "Jobs" that manifest as Pod "Instances" that run in a project directory in the

workflow/workflow_validator.py

@@ -1,4 +1,40 @@
-"""The WorkflowEngine validation logic."""
+"""The WorkflowEngine validation logic.
+
+A module that provides workflow validation at levels beyond the schema. A workflow
+definition is a complex structure, and not all of its content can be checked using
+a JSON/YAML schema alone. This module provides various 'levels' of workflow
+inspection of the workflow in increasing levels of validation: -
+
+- CREATE
+- RUN
+- TAG
+
+CREATE level validation simply checks that the workflow complies with the schema.
+Workflows are permitted in the DM that do not comply with the schema. This is becuase
+the DM is also used as a persistent store for Wwrfklows while editing - this
+allows a user to 'save' a workflow that is incomplete with the intention of adjusting
+it at a later date prior to execution.
+
+TAG level validation takes things a little further. In 'production' mode
+tagging is required prior to exeution. TAG level validatioin ensures that a workflow
+_should_ run if it is run - for examplke variable names are all correctly defined
+and there are no duplicates.
+
+RUN level extends TAG level validation by ensuring, for example, all the
+workflow variables are defined.
+
+Validation is designed to allow a more relaxed engine implementation, negating the
+need for the engine to 'check', for example, that variables exist - the validator
+ensures they do.
+
+Module philosophy
+-----------------
+Here we define the 'ValidationLevel' enumeration and 'ValidationResult' dataclass
+used as a return object by the validation function. The module defines the
+'WorkflowValidator' class with one (class-level) function ... 'validate()'.
+The 'validate()' function ensures that the checks based on the validation level
+are executed and any breach is returned via a 'ValidationResult' instance.
+"""
 
 from dataclasses import dataclass
 from enum import Enum