Launching steps

The workflow engine launches Steps (Jobs Instances) in response to Events (messages) sent to its WorkflowEngine.handle_message() method. These messages are either a WorkflowMessage START message or a PodMessage that is sent when an existing step has finished (successfully or not).

Starting a workflow

Workflows are started when a client (user) invokes the /workflow/{id}/run method in the DM API. The DM will respond my creating a RunningWorkflow record and then send a WorkflowMessage message to the engine's handle_message() method with action="SATRT".

The message will provide the UUID of a RunningWorkflow record that has been created by the DM. The engine must use this message property to obtain the database record and that of the Workflow. The engine does this using the Workflow API given to the class initialiser. The engine will also use the InstanceLauncher, also given to the engine's class initialiser.

For the simple linear engine implementation the engine can assume that the first step declared in the workflow YAML definition is the first step to execute.

As well as identifying the first step, one of the most important parts of the engine's responsibilities is the formation of a dictionary of variables that it needs to pass to the InstanceLauncher in order to actually start the step. The engine must provide names and values for all the variables for every Step Job that it executes. Each step declares these in its inputs, options, and outputs declarations.

As this is a complex process, and a process common to the actions required when continuing a workflow (see below), it is documented separately in Setting step variables.

When starting a workflow's first step the engine must do the following: -

1. Use the API Adapter to create a RunningWorkflowStep record
2. Determine the steps' variables and values (see Setting step variables)
3. Optionally use the DM Job Decoder to verify that the chosen variables and values satisfy the Step's Job command
4. Save the step's variables in the RunningWorkflowStep. This is required to allow the use of a steps's variables in future steps.
5. Create and populate a LaunchParameters object
6. Call the InstanceLauncher launch() method, providing the LaunchParameters object.

Continuing a workflow

The engine decides whether a workflow is to continue (more steps are the be launched) when a PodMessage message is sent to the engine's handle_message() method.

1. The PodMessage contains an instance reference that the engine MUST use to obtain the Instance record from the DM (using the workflow DM API)
2. The Instance record will contain a running_workflow_step_id, a UUID for the RunningWorkflowStep record that is the cause of the instance. The engine MUST retrieve the RunningWorkflowStep (and then RunningWorkflow and Workflow records).
3. Armed with the suite of workflow database records the engine should be able to determine which Step* should be launched next and then orchestrate the launch of that step.
4. If there are no more steps the engine should apply the actions described in Blah

Stopping a workflow

Workflows that are running can be stopped before they finish when a client (user) invokes the /workflow/{id}/stop method in the DM API. The DM will respond my sending a WorkflowMessage message to the engine's handle_message() method with action="STOP".

In response to this message the engine MUST: -

1. Call the API adapter's set_running_workflow_done() method setting success=False, provide a non-zero value to error_num and an error_msg value (typically User stopped).

Additionally, once the message has been handled the engine must handle the termination of any existing running Steps, knowing that the user has asked the workflow to stop. When steps for the workflow end (with the receoption of a corresponding PodMessage) the engine MUST mark the step's RunningWorkflowStep record (using the DM workflow API) as finished (successfully or otherwise).

The engine MUST NOT launch any new Steps for a Workflow that has been stopped.