Setting step variables

Setting the variable map (a Python dictionary of Step variable names and values) is probably the most complex part of workflow execution. Steps (Jobs) are launched using the InstanceLauncher class object that is provided when the engine is initialised. The launch() method of this object takes a LaunchParameters object where variables can either be provided in the specification or separately using the launch parameters specification_variables member: -

# Provide variables in the launch parameters specification
launch_parameters.specificaion["variables"] = step_variables
# ...or specification_variables.
# These variables "replace" any specification variables.
launch_parameters.specification_variables = step_variables
# Provide 'inputs' to the step (files in the Project directory).
# These are typically step variables that are fulfilled by files
# in the Project directory. Thew launcher will copy these
# into the instance directory prior to launching the step Job.
launch_parameters.running_workflow_step_inputs = step_inputs

With some examples, we'll now explore the process of collecting together a step's variables, which involves inspecting the RunningWorkflow record, the step's input, options and outputs, and the workflow's variable-mapping.

Imortantly: --

• All top-level variables defined by the workflow are provided by the User when they run the workflow
• Workflow variables are available in the RunningWorkflow record's variables property (a map of names and values)
• The engine can assume that the workflow has been validated and all the required variables have been provided
• The engine must call set_running_workflow_step_variables() prior to launching the step with the variables that have been compiled for the step. This preserves the variables, which the engine may need to retrieve durign the processing of future steps in the workflow.

In the following superfluous properties are omitted from the workflow for brevity. The following are essentially excerpts from otherwise valid workflows.

Example 1

In this example, step variables are provisioned from workflow variables (present in the RunningWorkflow record).

If we have the following Workflow: -

steps:
- name: step-1
  variable-mapping:
  - variable: inputFile
    from-workflow:
      variable: candidateMolecules
  - variable: outputFile
    from-workflow:
      variable: clusteredMolecules
  - variable: name
    from-workflow:
      variable: rdkitPropertyName
  - variable: value
    from-workflow:
      variable: rdkitPropertyValue

And the following RunningWorkflow variables (note that all values are strings): -

{
  "candidateMolecules": "C=1CCCCC1",
  "clusteredMolecules": "results.sdf",
  "rdkitPropertyName": "x",
  "rdkitPropertyValue": "17"
}

Then the step-1's variables will be: -

{
  "inputFile": "C=1CCCCC1",
  "outputFile": "results.sdf",
  "name": "x",
  "value": "17"
}

• step-1's inputFile is derived from the workflow's input candidateMolecules
• The variable mapping also declares two more variables for step-1 (name and value)

Example 2

In this example, step variables are provisioned from a prior step's variables.

If we have the following Workflow: -

steps:
- name: step2
  description: Add column 2
  specification:
    collection: workflow-engine-unit-test-jobs
    job: cluster-butina
    version: "1.0.0"
    variables:
      name: "col2"
      value: "999"
  variable-mapping:
  - variable: inputFile
    from-step:
      name: step1
      variable: outputFile

Variables used in a prior step can be retrieved using its RunningWorkflowStep record. If the RunningWorkflowStep variables for step-1 are: -

{
  "inputFile": "C=1CCCCC1",
  "outputFile": "first-step.out.smi",
  "name": "x",
  "value": "17"
}

Then step-2's variables will be: -

{
  "inputFile": "first-step.out.smi",
  "col2": "999"
}

• step-2's inputFile is set to the value of step-1's outputFile