Setting step variables

Setting the variable map (a Python dictionary of Step variable names and values) is probably the most complex part of workflow execution. Steps (Jobs) are launched using the InstanceLauncher class object that is provided when the engine is initialised. The launch() method of this object takes a LaunchParameters object where variables are provided using the specification_variables member: -

With some examples, we'll now explore the process of collecting together a step's variables, which involves inspecting the RunningWorkflow record, and the step's "plumbing".

Imortantly: --

• All workflow variables are provided by the User when they run the workflow
• Workflow variables are available in the RunningWorkflow record's variables property (a map of names and values)
• The engine can assume that the workflow has been validated and all the specified workflow variables have been provided

In the following, superfluous properties are omitted from the workflow for brevity. The following are excerpts from otherwise valid workflows.

Example 1

In this example, step variables are provisioned from workflow variables (present in the RunningWorkflow record).

If we have the following Workflow: -

steps:
- name: step-1
  variable-mapping:
  - variable: inputFile
    from-workflow:
      variable: candidateMolecules
  - variable: outputFile
    from-workflow:
      variable: clusteredMolecules
  - variable: name
    from-workflow:
      variable: rdkitPropertyName
  - variable: value
    from-workflow:
      variable: rdkitPropertyValue

And the following RunningWorkflow variables (note that all values are strings): -

{
  "candidateMolecules": "C=1CCCCC1",
  "clusteredMolecules": "results.sdf",
  "rdkitPropertyName": "x",
  "rdkitPropertyValue": "17"
}

Then step-1's variable map will be: -

{
  "inputFile": "C=1CCCCC1",
  "outputFile": "results.sdf",
  "name": "x",
  "value": "17"
}

• step-1's inputFile is derived from the workflow's input candidateMolecules
• The variable mapping also declares two more variables for step-1 (name and value)

Example 2

In this example, step variables are provisioned from a prior step's variables.

If we have the following Workflow: -

steps:
- name: step2
  description: Add column 2
  specification:
    collection: workflow-engine-unit-test-jobs
    job: cluster-butina
    version: "1.0.0"
    variables:
      name: "col2"
      value: "999"
  variable-mapping:
  - variable: inputFile
    from-step:
      name: step1
      variable: outputFile

Variables used in a prior step can be retrieved using its RunningWorkflowStep record. If the RunningWorkflowStep variables for step-1 are: -

{
  "inputFile": "C=1CCCCC1",
  "outputFile": "first-step.out.smi",
  "name": "x",
  "value": "17"
}

Then step-2's variables will be: -

{
  "inputFile": "first-step.out.smi",
  "col2": "999"
}

• step-2's inputFile is set to the value of step-1's outputFile