Concept

The Workflow Engine is the logic responsible for the interpretation and execution of Squonk2 YAML-based workflow definitions. The engine has two distinct responsibilities: the definition and validation of workflows using a schema, and the code to execute workflows (a workflow engine). The engine is essentially a glorified (and complex) finite state machine (FSM) driven by events dispatched by the DM.

The schema and its validation are located in a decoder module (just as the DM Job's schema validation and access methods are). The engine is located in the workflow_engine module.

Original design requirements: -

A little like our Job Testing framework (Jote) it is considered desirable...

1. To locate the code in its own repository
2. To allow the testing of the engine without the need of kubernetes

These requirements, although appearing to be of great benefit to developers of workflows, introduce significant design considerations.

Running Jobs outside of the DM is a relatively simple task - Jote creates a docker-compose file and launches a container image. But Jobs (Instances) do not have a complex operational state that workflows have. Jobs result in one instance, easily handled by one docker compose file. The workflow engine needs to execute and interpret multiple jobs concurrently in an event-driven framework needing the ability to launch jobs and get and manipulate state in a database. It was clear from the outset that Workflows introduce a complex state model that would need to be tracked in a separate database table (a Running Workflow). Each running workflow could result in a number of Steps (Job instances) that would need to be tracked in a Running Workflow Step table.

External to the DM

As the code is to be independent of the DM repository, in order to follow sensible loosely coupled highly cohesive design paradigm, the ability to launch (run) Job Instances and access to the Data Manager database would be encapsulated in two interfaces, two abstract base classes: -

• The InstanceLauncher would provide an interface that would allow the engine to run Jobs
• The WorkflowAPI would provide an interface that would allow access to and modification of workflow-related database records

When imported into the DM, the DM will provide its own concrete implementations of these interfaces as it initialises the WorkflowEngine it creates. Under test the workflow engine repository provides its own (mock) implementations.

Testing without Kubernetes

This is a significant challenge - the ability to run workflows without a real launcher or database. To do this we need functional implementations of both. This is achieved with classes that implement the above interface declarations located in the tests directory - along with a messaging framework that allows us to run the code asynchronously, generating events, as it is expected to.

A more detailed explanation of the approach to testing can be found on the Test framework page.