Package modules

The engine consists of the following modules: -

decoder.py

Logic surrounding the workflow schema. It primary method is validate_schema() used to validate a workflow file against the schema. It also offers a number of utility methods to access the workflow definition.

The module has two roles. Firstly, provision of validation functions that allow a client to provide a definition YAML file (already loaded as a Python dictionary) and check that it if properly formed. It does this by exposing a validate_schema() method (just as the Job decoder does). Secondly, access methods to allow querying of the workflow content without the client having to understand the workflow file structure.

This second design criteria is very important: we have decided to place any code that needs to inspect the content of a workflow YAML file here. If the engine needs the step's inputs variable names, for example, it calls a convenience method in the decoder module (e.g. get_step_input_variable_names()). Usign this approach all of the logic that needs to understand the workflow YAML structure is located in one module, and if we decide to alter the YAML schema then the impact on the engine code is minimised, ideally only requiring changes to the decoder.

workflow_abc.py

Here abstract base classes are defined for external services the engine depends upon. It is here we find the definition of the API and instance launcher the engine needs.

workflow_engine.py

The engine logic

workflow_validator.py

The logic that is responsible for validating a workflow for the various levels of execution. A WorkflowValidator class provides validation for CREATE, TAG, and RUN levels of execution (see the Workflow validation page).