Concept

The Workflow Engine relates to the logic responsible for the interpretation and execution of YAML-based workflow definitions. The engine's repository has two distinct responsibilities: the definition of the workflow schema (a decoder) and the implementation of the workflow state machine (a workflow engine).

Original design requirements: -

A little like our Job Testing framework (Jote) it was considered desirable if...

1. The code would be external to the Data Manager (DM)
2. Testable without the need of kubernetes

These requirements, although appearing to be of great benefit to developers of workflows, introduce significant design considerations.

Running Jobs outside of the DM is a relatively simple task - Jote creates a docker-compose file and launches a container image. But Jobs (Instances) do not have a complex operational state that workflows would undoubtably have. Jobs simply result in one instance, easily handled by one docker compose file, whereas it was clear from the outset that a Workflow would result in (potentially) a large number of instances, some running in parallel ... introducing a complex state model that would needed to be tracked in a separate (persistent) object (one that we would call a Running Workflow). Each running workflow could result in a number of Jobs (that we'd call Steps) that would also have to be tracked.