refresh v1.1.0-dev1

mr-c · mr-c · commit 2d7cde374ed2 · 2019-02-26T16:00:34.000+02:00
diff --git a/cwltool/schemas/v1.1.0-dev1/CommandLineTool.yml b/cwltool/schemas/v1.1.0-dev1/CommandLineTool.yml
@@ -56,8 +56,8 @@ $graph:
         * When defining record types with `CommandInputRecordSchema`, fields of
           type `File` may now include `format`, `loadContents`,
           `secondaryFiles` and `streamable`.
-        * `CommandInputRecordSchema`, `CommandOutputRecordSchema`,
-          `CommandInputEnumSchema and `CommandInputArraySchema` now have an optional
+        * `CommandInputRecordSchema`, `CommandOutputRecordSchema`, 
+          `CommandInputEnumSchema and `CommandInputArraySchema` now have an optional 
           `doc` field.
         * `inputBinding` has been added as an optional field for
           `CommandInputRecordSchema` (was previously in CWL `draft-3` but
@@ -88,8 +88,8 @@ $graph:
         * Added `InputParameter.loadContents` field. Use of `loadContents` in
           `InputBinding` is deprecated; it is preserved for v1.0 backwards
            compatability and will be removed in CWL v2.0.
-        * [Added](#ToolTimeLimit) `ToolTimeLimit` feature, allowing to set an upper limit on the
-          execution time of a CommandLineTool.
+        * [Added](#ToolTimeLimit) `ToolTimeLimit` feature, allows setting
+          an upper limit on the execution time of a CommandLineTool.
         * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse
           behavior for a particular tool or step for implementations that
           support reusing output from past work.
@@ -197,8 +197,14 @@ $graph:
 
   fields:
     - name: position
-      type: int?
-      doc: "The sorting key.  Default position is 0."
+      type: [ int, Expression, string, "null" ]
+      doc: |
+        The sorting key.  Default position is 0. If the inputBinding is
+        associated with an input parameter, then the value of `self` in the
+        expression will be the value of the input parameter.  Input parameter
+        defaults (as specified by the `InputParameter.default` field) must be
+        applied before evaluating the expression. Expressions must return a
+        single value of type int or a null.
     - name: prefix
       type: string?
       doc: "Command line prefix to add before the value."
@@ -1097,36 +1103,6 @@ $graph:
       doc: Maximum reserved filesystem based storage for the designated output directory, in mebibytes (2**20)
 
 
-- type: record
-  name: ToolTimeLimit
-  extends: ProcessRequirement
-  doc: |
-    Set an upper limit on the execution time of a CommandLineTool.
-    A tool execution which exceeds the time limit may
-    be preemptively terminated and considered failed.  May also be
-    used by batch systems to make scheduling decisions.
-
-    Limit applies only to the command execution time. Time for
-    additional steps, such as file download/upload, docker pull or
-    expression evaluation is not considered for exceeding time limit.
-
-    If ToolTimeLimit is set on a workflow level, it propagates down to
-    individual processes, in line with requirement inheritance rules.
-    ToolTimeLimit is not used to limit the execution time of a Workflow.
-  fields:
-    - name: class
-      type: string
-      doc: "Always 'ToolTimeLimit'"
-      jsonldPredicate:
-        "_id": "@type"
-        "_type": "@vocab"
-    - name: timelimit
-      type: [long, string, Expression]
-      doc: |
-        The time limit, in seconds.  A time limit of zero means no
-        time limit.  Negative time limits are an error.
-
-
 - type: record
   name: WorkReuse
   extends: ProcessRequirement
@@ -1226,3 +1202,27 @@ $graph:
         "_type": "@vocab"
     inplaceUpdate:
       type: boolean
+
+- type: record
+  name: ToolTimeLimit
+  extends: ProcessRequirement
+  doc: |
+    Set an upper limit on the execution time of a CommandLineTool.
+    A CommandLineTool whose execution duration exceeds the time
+    limit may be preemptively terminated and considered failed.
+    May also be used by batch systems to make scheduling decisions.
+    The execution duration excludes external operations, such as
+    staging of files, pulling a docker image etc, and only counts
+    wall-time for the execution of the command line itself.
+  fields:
+    - name: class
+      type: string
+      doc: "Always 'ToolTimeLimit'"
+      jsonldPredicate:
+        "_id": "@type"
+        "_type": "@vocab"
+    - name: timelimit
+      type: [long, string, Expression]
+      doc: |
+        The time limit, in seconds.  A time limit of zero means no
+        time limit.  Negative time limits are an error.
diff --git a/cwltool/schemas/v1.1.0-dev1/README.md b/cwltool/schemas/v1.1.0-dev1/README.md
@@ -1,71 +1,9 @@
-# Common Workflow Language Specifications, v1.1.0-dev1
+Hello!
 
-<!-- remove this before v1.1 is final! -->
-WARNING, this is a draft and in progress! Development of CWL v1.1 is coordinated at https://github.com/common-workflow-language/common-workflow-language/milestone/6
+This repo represent the current working proposal for v1.1 of the Common Workflow Language standard.
 
-The CWL specifications are divided up into several documents.
+Proposed changes that aren't yet implemented are tracked at https://github.com/common-workflow-language/common-workflow-language/milestone/6
 
-The [User Guide](http://www.commonwl.org/user_guide/) provides a gentle
-introduction to writing CWL command line tools and workflows.
+You can render this using https://github.com/common-workflow-language/cwl-website/blob/master/website.sh
 
-The [Command Line Tool Description Specification](CommandLineTool.html)
-specifies the document schema and execution semantics for wrapping and
-executing command line tools.
-
-The [Workflow Description Specification](Workflow.html) specifies the document
-schema and execution semantics for composing workflows from components such as
-command line tools and other workflows.
-
-The
-[Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html)
-specifies the preprocessing steps that must be applied when loading CWL
-documents and the schema language used to write the above specifications.
-
-Also available are inheritance graphs (as SVG images) for the [Schema Salad object model](salad.svg) and the [CWL object model](cwl.svg).
-
-# Running the CWL conformance tests
-
-1. Install a CWL runner of your choice. The reference runner can be installed as
-the default runner by doing:
-```
-pip install cwlref-runner
-```
-
-2. Install the CWL test parser:
-
-```
-pip install cwltest
-```
-You may need to activate a virtualenv first, or do a local install by adding `--user` after `install` above.
-
-3. From within a copy of [this repository](https://github.com/common-workflow-language/cwl-v1.1) (e.g. cwl-v1.1) execute the main test script
-```
-./run_test.sh
-```
-
-If the CWL runner isn't installed as `cwl-runner` then you can specify the name:
-
-```
-./run_test.sh RUNNER=cwltool
-```
-
-You can also specify additional options that are specific for the particular CWL runner you are using.
-For example, with CWL reference runner you can turn on parallel execution mode:
-
-```
-./run_test.sh RUNNER=cwltool EXTRA=--parallel
-```
-
-This can be combined with launching more than one CWL conformance test at once with `-j`:
-
-```
-./run_test.sh -j4 RUNNER=cwltool EXTRA=--parallel
-```
-
-
-For details of options you can pass to the test script, do:
-```
-./run_test.sh --help
-```
-
-The full test suite takes about 10 minutes to run
+The current rendering is at https://www.commonwl.org/v1.1.0-dev1/
diff --git a/cwltool/schemas/v1.1.0-dev1/Workflow.yml b/cwltool/schemas/v1.1.0-dev1/Workflow.yml
@@ -48,10 +48,16 @@ $graph:
 
       ## Changelog
 
-        * Default values for some fields are now expressed in the schema
+        * Clarify intent of ExpressionTool and clarify that they can be run standalone. 
+        * Clarify how Requirements are inherited by step processes.
+        * Clarify behavior around step inputs not consumed by the target
+          run process.
+        * Document explicitly that CWL allows pass-through of workflow inputs to outputs.
+        * Added optional name and doc fields to `OutputRecordSchema`.
+        * Default values for some fields are now expressed in the schema.
         * When defining record types, fields of type `File` may now include
           `format`, `secondaryFiles`, and `streamable`.
-        * The Input schemas (Record, Array, Enum),  now permits a `doc` field
+        * The `Input` schemas (`Record`, `Array`, `Enum`),  now permits a `doc` field.
         * Any `doc` field may be an array of strings in addition to the
           previously allowed single string.
         * The `run` field now introduces a subscope so that embedded
@@ -65,10 +71,22 @@ $graph:
           `WorkflowInputParameter.loadContents` instead.
         * The presence of `OutputRecordField.outputBinding` in v1.0 was due to
           a technical error and as it makes no semantic sense it was removed.
-        * `WorkflowStepInput` now has a `label` field
-        * [Addition](#Requirements_and_hints) of `cwl:requirments` field to
-          input object documents
+        * `WorkflowStepInput` now has a `label` field.
+        * [Addition](#Requirements_and_hints) of `cwl:requirements` field to
+          input object documents.
         * `secondaryFiles` can now be explicitly marked as `required` or not.
+        * Add note that undefined details related to execution are left open to
+          the platform.
+        * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse
+          behavior for a particular tool or step for implementations that
+          support reusing output from past work.
+        * [Added](#NetworkAccess) `NetworkAccess` feature, allowing to indicate whether a
+          process requires outgoing network access.
+        * [Added](#InplaceUpdateRequirement) `InplaceUpdateRequirement` feature, allowing tools to directly
+          update files with `writable: true` in `InitialWorkDirRequirement`.
+        * [Added](#LoadListingRequirement) `LoadListingRequirement`
+          and [loadListing](#LoadContents) to control whether and how
+          `Directory` listings should be loaded for use in expressions.
 
       See also the [CWL Command Line Tool Description, v1.1.0-dev1 changelog](CommandLineTool.html#Changelog).
 
@@ -79,7 +97,9 @@ $graph:
       Physics, and Astronomy.  This specification is intended to define a data
       and execution model for Workflows that can be implemented on top of a
       variety of computing platforms, ranging from an individual workstation to
-      cluster, grid, cloud, and high performance computing systems.
+      cluster, grid, cloud, and high performance computing systems. Details related
+      to execution of these workflow not laid out in this specification are open to
+      interpretation by the computing platform implementing this specification.
 
     - {$include: concepts.md}
 
@@ -156,7 +176,13 @@ $graph:
       specializeTo: ExpressionToolOutputParameter
   documentRoot: true
   doc: |
-    Execute an expression as a Workflow step.
+    An ExpressionTool is a type of Process object that can be run by itself
+    or as a Workflow step. It executes a pure Javascript expression that has
+    access to the same input parameters as a workflow. It is meant to be used
+    sparingly as a way to isolate complex Javascript expressions that need to
+    operate on input data and produce some result; perhaps just a
+    rearrangement of the inputs. No Docker software container is required
+    or allowed.
   fields:
     - name: class
       jsonldPredicate:
@@ -184,8 +210,9 @@ $graph:
   docParent: "#Workflow"
   doc: |
     Describe an output parameter of a workflow.  The parameter must be
-    connected to one or more parameters defined in the workflow that will
-    provide the value of the output parameter.
+    connected to one or more parameters defined in the workflow that
+    will provide the value of the output parameter. It is legal to
+    connect a WorkflowInputParameter to a WorkflowOutputParameter.
   fields:
     - name: outputSource
       doc: |
@@ -260,7 +287,11 @@ $graph:
   doc: |
     The input of a workflow step connects an upstream parameter (from the
     workflow inputs, or the outputs of other workflows steps) with the input
-    parameters of the underlying step.
+    parameters of the process specified by the `run` field. Only input parameters
+    declared by the target process will be passed through at runtime to the process
+    though additonal parameters may be specified (for use within `valueFrom`
+    expressions for instance) - unconnected or unused parameters do not represent an
+    error condition.
 
     ## Input object
 
diff --git a/cwltool/schemas/v1.1.0-dev1/concepts.md b/cwltool/schemas/v1.1.0-dev1/concepts.md
@@ -215,9 +215,14 @@ object document as an array of requirements under the field name
 should be combined with any requirements present in the corresponding Process
 as if they were specified there.
 
-Requirements are inherited.  A requirement specified in a Workflow applies
-to all workflow steps; a requirement specified on a workflow step will
-apply to the process implementation of that step and any of its substeps.
+Requirements specified in a parent Workflow are inherited by step processes
+if they are valid for that step. If the substep is a CommandLineTool
+only the `InlineJavascriptRequirement`, `SchemaDefRequirement`, `DockerRequirement`,
+`SoftwareRequirement`, `InitialWorkDirRequirement`, `EnvVarRequirement`, 
+`ShellCommandRequirement`, `ResourceRequirement` are valid.
+
+*As good practice, it is best to have process requirements be self-contained,
+such that each process can run successfully by itself.*
 
 If the same process requirement appears at different levels of the
 workflow, the most specific instance of the requirement is used, that is,
