CWL workflow `expressionLib` functions are overwritten by tool `expressionLib` when evaluating step valueFrom expressions

[ThomasHickman]

If you have a CWL workflow where the workflow and the tool both uses an expressionLib, for example:

cwlVersion: v1.2
class: Workflow
id: test-workflow

requirements:
  - class: StepInputExpressionRequirement
  - class: InlineJavascriptRequirement
    expressionLib:
      - |
        function workflowFunction(inputs) {
          return "workflow_value"
        }

inputs: {}
outputs: {}
steps:
  - id: a_step
    in:
      a_input:
        valueFrom: "$(workflowFunction())"
    out: []
    run:
      class: CommandLineTool
      id: tool
      requirements:
        - class: InlineJavascriptRequirement
          expressionLib:
          - |
            function toolFunction(inputs) {
              return "tool_value"
            }
      baseCommand: echo
      inputs:
        a_input:
          type: string
          inputBinding:
            position: 1
      outputs: {}

the function workflowFunction will not be evaluated, and the tool will get passed a_input: null

When looking into the code, it looks like this is due to the CwlBlockContext getting called with block.targetRequirements, block.targetHints here:

dxCompiler/executorCwl/src/main/scala/dx/executor/cwl/CwlWorkflowExecutor.scala

Line 352 in b74816e

private lazy val eval = Evaluator.create(block.targetRequirements, block.targetHints)

.

The targetRequirements are defined to be the inherited requirements, followed by the tool that is being run, followed by requirements of the tool that is being run, defined here:

dxCompiler/core/src/main/scala/dx/core/languages/cwl/CwlBlock.scala

Line 284 in b74816e

inheritedRequirements ++ target.requirements ++ target.run.requirements

In cwlScala's Evaluator, only the latest InlineJavascriptRequirement is considered when evaluating requirements https://github.com/dnanexus/cwlScala/blob/44b7c4ca194d1862daaa981f19e40495449b5fd0/src/main/scala/dx/cwl/Evaluator.scala#L1028-L1032, meaning the workflows expressionLib is being overwritten expressionLib from the tool that it is calling.

I'm just reporting this here, to raise awareness for others who may be using the CWL mode of dxCompiler, but I acknowledge that it may take a while for you to prioritize this issue (as noted here: #455 (comment))