Combining

Combining, or "fan in", is used to refer to a step Job that expects multiple input files and creates a single output file. Simple concatenation of the files is one example. "Combiner" Jobs typically join a large number of files (produced by a prior parallel step) in a single step instance.

The Workflow Engine determines that a step is combining multiple files by inspecting the plumbing that refers to a prior steps's output. If a step input variable is (according to the Job Definition) of type files then the step is assumed to be a combiner of files generated by multiple instances of a prior step.

Here's an example workflow excerpt: -

- name: parallel
  description: Add some params
  specification:
    collection: demo
    job: append-col
    version: "1.0.0"

- name: combine
  description: Combine the parallel files
  specification:
    collection: demo
    job: concatenate
    version: "1.0.0"
  plumbing:
  - variable: inputFile
    from-step:
      name: parallel
      variable: outputFile
  - variable: inputDirPrefix
    from-predefined:
      variable: link-glob