Merge branch 'master' into feature/optional-secondary

Author: Peter Amstutz

CommandLineTool.yml

@@ -50,6 +50,7 @@ $graph:
 
       ## Changelog
 
+        * Clarify documentation around `valueFrom` and `null` inputs.
         * Default values for some fields are now expressed in the schema.
         * When defining record types with `CommandInputRecordSchema`, fields of
           type `File` may now include `format`, `loadContents`,
@@ -69,9 +70,11 @@ $graph:
         * Clarify semantics of `shellQuote`
         * Expressions are now allowed to evaluate to `null` or `Dirent` in
           [InitialWorkDirRequirement.listing](#InitialWorkDirRequirement).
+        * Clarify behavior of secondaryFiles on output.
         * [Addition](#Requirements_and_hints) of `cwl:requirements` field to
-          input object documents
+          input object documents.
         * Clarify behavior of `glob` for absolute paths and symlinks.
+        * Clarify behavior of `glob` to include directories.
 
       See also the [CWL Workflow Description, v1.1.0-dev1 changelog](Workflow.html#Changelog).
 
@@ -202,6 +205,9 @@ $graph:
         `InputParameter.default` field) must be applied before evaluating the
         expression.
 
+        If the value of the associated input parameter is `null`, `valueFrom` is 
+        not evaluated and nothing is added to the command line.
+
         When a binding is part of the `CommandLineTool.arguments` field,
         the `valueFrom` field is required.
 
@@ -240,12 +246,12 @@ $graph:
         - type: array
           items: string
       doc: |
-        Find files relative to the output directory, using POSIX glob(3)
-        pathname matching.  If an array is provided, find files that match any
-        pattern in the array.  If an expression is provided, the expression must
-        return a string or an array of strings, which will then be evaluated as
-        one or more glob patterns.  Must only match and return files which
-        actually exist.
+        Find files or directories relative to the output directory, using POSIX
+        glob(3) pathname matching.  If an array is provided, find files or 
+        directories that match any pattern in the array.  If an expression is
+        provided, the expression must return a string or an array of strings,
+        which will then be evaluated as one or more glob patterns.  Must only
+        match and return files/directories which actually exist.
 
         If the value of glob is a relative path pattern (does not
         begin with a slash '/') then it is resolved relative to the
@@ -259,14 +265,14 @@ $graph:
 
         A glob may match a path within the output directory which is
         actually a symlink to another file.  In this case, the
-        expected behavior is for the resulting File object to take the
+        expected behavior is for the resulting File/Directory object to take the
         `basename` (and corresponding `nameroot` and `nameext`) of the
-        symlink.  The `location` of the File is implementation
-        dependent, but logically the File should have the same content
-        as the symlink target.  Platforms may stage output files to
+        symlink.  The `location` of the File/Directory is implementation
+        dependent, but logically the File/Directory should have the same content
+        as the symlink target.  Platforms may stage output files/directories to
         cloud storage that lack the concept of a symlink.  In
-        this case file content may be duplicated, or (to avoid
-        duplication) the File `location` may refer to the symlink
+        this case file content and directories may be duplicated, or (to avoid
+        duplication) the File/Directory `location` may refer to the symlink
         target.
 
         It is an error if a symlink in the output directory (or any

Process.yml

@@ -467,10 +467,16 @@ $graph:
       doc: |
         Only valid when `type: File` or is an array of `items: File`.
 
-        Provides a pattern or expression specifying files or directories that
-        should be included alongside the primary file.
-        Secondary file is required for inputs and not required for outputs
-        unless specified otherwise.
+        Provides a pattern or expression specifying files or
+        directories that should be included alongside the primary
+        file.  Secondary files may be required or optional.  When not
+        explicitly specified, secondary files specified for `inputs`
+        are required and `outputs` are optional.  An implementation
+        must include matching Files and Directories in the
+        `secondaryFiles` property of the primary file.  These Files
+        and Directories must be transferred and staged alongside the
+        primary file.  An implementation may fail workflow execution
+        if a required secondary file does not exist.
 
         If the value is an expression, the value of `self` in the expression
         must be the primary input or output File object to which this binding
@@ -481,7 +487,7 @@ $graph:
         `path` or `location` and `basename` fields set, or an array consisting
         of strings or File or Directory objects.  It is legal to reference an
         unchanged File or Directory object taken from input as a secondaryFile.
-        The expression may return "null" in which case there is no secondaryFile 
+        The expression may return "null" in which case there is no secondaryFile
         from that expression.
 
         To work on non-filename-preserving storage systems, portable tool
@@ -876,7 +882,7 @@ $graph:
         `path` or `location` and `basename` fields set, or an array consisting
         of strings or File or Directory objects.  It is legal to reference an
         unchanged File or Directory object taken from input as a secondaryFile.
-        The expression may return "null" in which case there is no secondaryFile 
+        The expression may return "null" in which case there is no secondaryFile
         from that expression.
 
         To work on non-filename-preserving storage systems, portable tool
@@ -898,8 +904,7 @@ $graph:
     - name: required
       type: boolean?
       doc: |
-        An implementation must not fail workflow execution if `required` is 
+        An implementation must not fail workflow execution if `required` is
         set to `false` and the expected secondary file does not exist.
         Default value for `required` field is `true` for secondary files on
         input and `false` for secondary files on output.
-

README.md

@@ -1,71 +1,7 @@
-# Common Workflow Language Specifications, v1.1.0-dev1
+Hello!
 
-<!-- remove this before v1.1 is final! -->
-WARNING, this is a draft and in progress! Development of CWL v1.1 is coordinated at https://github.com/common-workflow-language/common-workflow-language/milestone/6
+This repo represent the current working proposal for v1.1 of the Common Workflow Language standard.
 
-The CWL specifications are divided up into several documents.
+You can render this using https://github.com/common-workflow-language/cwl-website/blob/master/website.sh
 
-The [User Guide](http://www.commonwl.org/user_guide/) provides a gentle
-introduction to writing CWL command line tools and workflows.
-
-The [Command Line Tool Description Specification](CommandLineTool.html)
-specifies the document schema and execution semantics for wrapping and
-executing command line tools.
-
-The [Workflow Description Specification](Workflow.html) specifies the document
-schema and execution semantics for composing workflows from components such as
-command line tools and other workflows.
-
-The
-[Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html)
-specifies the preprocessing steps that must be applied when loading CWL
-documents and the schema language used to write the above specifications.
-
-Also available are inheritance graphs (as SVG images) for the [Schema Salad object model](salad.svg) and the [CWL object model](cwl.svg).
-
-# Running the CWL conformance tests
-
-1. Install a CWL runner of your choice. The reference runner can be installed as
-the default runner by doing:
-```
-pip install cwlref-runner
-```
-
-2. Install the CWL test parser:
-
-```
-pip install cwltest
-```
-You may need to activate a virtualenv first, or do a local install by adding `--user` after `install` above.
-
-3. From within a copy of [this repository](https://github.com/common-workflow-language/cwl-v1.1) (e.g. cwl-v1.1) execute the main test script
-```
-./run_test.sh
-```
-
-If the CWL runner isn't installed as `cwl-runner` then you can specify the name:
-
-```
-./run_test.sh RUNNER=cwltool
-```
-
-You can also specify additional options that are specific for the particular CWL runner you are using.
-For example, with CWL reference runner you can turn on parallel execution mode:
-
-```
-./run_test.sh RUNNER=cwltool EXTRA=--parallel
-```
-
-This can be combined with launching more than one CWL conformance test at once with `-j`:
-
-```
-./run_test.sh -j4 RUNNER=cwltool EXTRA=--parallel
-```
-
-
-For details of options you can pass to the test script, do:
-```
-./run_test.sh --help
-```
-
-The full test suite takes about 10 minutes to run
+The current rendering is at https://www.commonwl.org/v1.1.0-dev1/

index.md

@@ -0,0 +1,71 @@
+# Common Workflow Language Specifications, v1.1.0-dev1
+
+<!-- remove this before v1.1 is final! -->
+WARNING, this is a draft and in progress! Development of CWL v1.1 is coordinated at https://github.com/common-workflow-language/common-workflow-language/milestone/6
+
+The CWL specifications are divided up into several documents.
+
+The [User Guide](http://www.commonwl.org/user_guide/) provides a gentle
+introduction to writing CWL command line tools and workflows.
+
+The [Command Line Tool Description Specification](CommandLineTool.html)
+specifies the document schema and execution semantics for wrapping and
+executing command line tools.
+
+The [Workflow Description Specification](Workflow.html) specifies the document
+schema and execution semantics for composing workflows from components such as
+command line tools and other workflows.
+
+The
+[Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html)
+specifies the preprocessing steps that must be applied when loading CWL
+documents and the schema language used to write the above specifications.
+
+Also available are inheritance graphs (as SVG images) for the [Schema Salad object model](salad.svg) and the [CWL object model](cwl.svg).
+
+# Running the CWL conformance tests
+
+1. Install a CWL runner of your choice. The reference runner can be installed as
+the default runner by doing:
+```
+pip install cwlref-runner
+```
+
+2. Install the CWL test parser:
+
+```
+pip install cwltest
+```
+You may need to activate a virtualenv first, or do a local install by adding `--user` after `install` above.
+
+3. From within a copy of [this repository](https://github.com/common-workflow-language/cwl-v1.1) (e.g. cwl-v1.1) execute the main test script
+```
+./run_test.sh
+```
+
+If the CWL runner isn't installed as `cwl-runner` then you can specify the name:
+
+```
+./run_test.sh RUNNER=cwltool
+```
+
+You can also specify additional options that are specific for the particular CWL runner you are using.
+For example, with CWL reference runner you can turn on parallel execution mode:
+
+```
+./run_test.sh RUNNER=cwltool EXTRA=--parallel
+```
+
+This can be combined with launching more than one CWL conformance test at once with `-j`:
+
+```
+./run_test.sh -j4 RUNNER=cwltool EXTRA=--parallel
+```
+
+
+For details of options you can pass to the test script, do:
+```
+./run_test.sh --help
+```
+
+The full test suite takes about 10 minutes to run