Merge pull request #59 from common-workflow-language/rc1_feedback

Rc1 feedback

Author: mr-c

CommandLineTool.yml

@@ -312,9 +312,8 @@ $graph:
         path pattern (it does begin with a slash '/') then it must
         refer to a path within the output directory.  It is an error
         if any glob resolves to a path outside the output directory.
-        Specifically this means globs with relative paths containing
-        '..' or absolute paths that refer outside the output directory
-        are illegal.
+        Specifically this means globs that resolve to paths outside the output
+        directory are illegal.
 
         A glob may match a path within the output directory which is
         actually a symlink to another file.  In this case, the
@@ -814,7 +813,10 @@ $graph:
         "_type": "@vocab"
     - name: dockerPull
       type: string?
-      doc: "Specify a Docker image to retrieve using `docker pull`."
+      doc: |
+        Specify a Docker image to retrieve using `docker pull`. Can contain the
+        immutable digest to ensure an exact container is used:
+        `dockerPull: ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2`
     - name: dockerLoad
       type: string?
       doc: "Specify a HTTP URL from which to download a Docker image using `docker load`."

concepts.md

@@ -17,6 +17,9 @@ https://tools.ietf.org/html/rfc3987
 
 **Resource Description Framework (RDF)**: http://www.w3.org/RDF/
 
+**XDG Base Directory Specification**: https://specifications.freedesktop.org/basedir-spec/basedir-spec-0.6.html
+
+
 ## Scope
 
 This document describes CWL syntax, execution, and object model.  It
@@ -88,8 +91,7 @@ JSON or YAML syntax.  Upon loading, a CWL implementation must apply the
 preprocessing steps described in the
 [Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html).
 An implementation may formally validate the structure of a CWL document using
-SALAD schemas located at
-https://github.com/common-workflow-language/common-workflow-language/tree/master/v1.1
+SALAD schemas located at https://github.com/common-workflow-language/cwl-v1.1/
 
 ### map
 
@@ -242,9 +244,6 @@ inputs:
   workflow_input02:     # we use the longer way
     type: File          # because we want to specify the "format" too
     format: http://edamontology.org/format_2572
-  workflow_input03: {}  # back to the short form as this entry
-                        # uses the default of no "type" just like the prior
-                        # examples
 ```
 
 Mixed option 2 and 3 specific example using [SoftwareRequirement](#SoftwareRequirement).[packages](#SoftwarePackage):
@@ -263,9 +262,9 @@ hints:
       python: {}
 ```
 
-Note: The `map<…>` (compact) versions are optional, the verbose option #1 is
+Note: The `map<…>` (compact) versions are optional for users, the verbose option #1 is
 always allowed, but for presentation reasons option 3 and 2 may be preferred
-by human readers.
+by human readers. Consumers of CWL must support all three options.
 
 The normative explanation for these variations, aimed at implementors, is in the
 [Schema Salad specification](SchemaSalad.html#Identifier_maps).
@@ -293,7 +292,7 @@ Type `<T>` ending with `[]` should be transformed to `{"type": "array", "items":
 
 ## Extensions and metadata
 
-Input metadata (for example, a lab sample identifier) may be represented within
+Input metadata (for example, a sample identifier) may be represented within
 a tool or workflow using input parameters which are explicitly propagated to
 output.  Future versions of this specification may define additional facilities
 for working with input/output metadata.
@@ -305,6 +304,9 @@ additional fields on any object.  Such extensions fields must use a namespace
 prefix listed in the `$namespaces` section of the document as described in the
 [Schema Salad specification](SchemaSalad.html#Explicit_context).
 
+It is recommended that concepts from schema.org are used whenever possible.
+For the `$schema` field we recommend their RDF encoding: http://schema.org/version/latest/schema.rdf
+
 Implementation extensions which modify execution semantics must be [listed in
 the `requirements` field](#Requirements_and_hints).
 
@@ -353,8 +355,6 @@ platform include:
 (except as described in [DockerRequirement](CommandLineTool.html#DockerRequirement)).
 * Using remote or distributed file systems to manage input and output files.
 * Transforming file paths.
-* Determining if a process has previously been executed, and if so skipping it
-and reusing previous results.
 * Pausing, resuming or checkpointing processes or workflows.
 
 Conforming CWL processes must not assume anything about the runtime
@@ -499,7 +499,7 @@ must be treated as a string interpolation.  After interpolating the first
 parameter reference, interpolation must be recursively applied to the
 trailing characters to yield the final string value.
 
-## Expressions
+## Expressions (Optional)
 
 An expression is a fragment of [Javascript/ECMAScript
 5.1](http://www.ecma-international.org/ecma-262/5.1/) code evaluated by the
@@ -510,6 +510,12 @@ and/or step 7 (capture output).  Expressions are distinct from regular
 processes in that they are intended to modify the behavior of the workflow
 itself rather than perform the primary work of the workflow.
 
+Expressions in CWL are an optional feature and are not required to be
+implemented by all consumers of CWL documents. They should be used sparingly,
+when there is no other way to achieve the desired outcome. Excessive use of
+expressions may be a signal that other refactoring of the tools or workflows
+would benefit the author, runtime, and users of the CWL document in question.
+
 To declare the use of expressions, the document must include the process
 requirement `InlineJavascriptRequirement`.  Expressions may be used in any
 field permitting the pseudo-type `Expression`, as specified by this
@@ -554,8 +560,8 @@ Implementations may apply other limits, such as process isolation, timeouts,
 and operating system containers/jails to minimize the security risks associated
 with running untrusted code embedded in a CWL document.
 
-Exceptions thrown from an exception must result in a `permanentFailure` of the
-process.
+Javascript exceptions thrown from a CWL expression must result in a
+`permanentFailure` of the CWL process.
 
 ## Executing CWL documents as scripts
 
@@ -578,11 +584,16 @@ documented in [cwl-runner.cwl](cwl-runner.cwl).
 
 To discover CWL documents look in the following locations:
 
-`/usr/share/commonwl/`
+For each value in the `XDG_DATA_DIRS` environment variable (which is a `:` colon
+separated list), check the `./commonwl` subdirectory. If `XDG_DATA_DIRS` is
+unset or empty, then check using the default value for `XDG_DATA_DIRS`:
+`/usr/local/share/:/usr/share/` (That is to say, check `/usr/share/commonwl/`
+and `/usr/local/share/commonwl/`)
 
-`/usr/local/share/commonwl/`
+Then check `$XDG_DATA_HOME/commonwl/`.
 
-`$XDG_DATA_HOME/commonwl/` (usually `$HOME/.local/share/commonwl`)
+If the `XDG_DATA_HOME` environment variable is unset, its default value is
+`$HOME/.local/share` (That is to say, check `$HOME/.local/share/commonwl`)
 
-`$XDG_DATA_HOME` is from the [XDG Base Directory
+`$XDG_DATA_HOME` and `$XDG_DATA_DIRS` are from the [XDG Base Directory
 Specification](http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html)

index.md

@@ -1,9 +1,9 @@
-# Common Workflow Language Specifications, v1.1
+# Common Workflow Language Specifications, v1.1 "release candidate 1"
 
 The CWL specifications are divided up into several documents.
 
 The [User Guide](http://www.commonwl.org/user_guide/) provides a gentle
-introduction to writing CWL command line tools and workflows.
+introduction to writing CWL command line tools and workflows. Currently it is for CWL v1.0 only.
 
 The [Command Line Tool Description Specification](CommandLineTool.html)
 specifies the document schema and execution semantics for wrapping and
@@ -66,15 +66,13 @@ You can run a particular test
 ```
 
 
-If you are running tests for an unreleased CWL version use the `--enable-dev` flag:
+If you are running tests using the CWL reference runner "cwltool" for an unreleased version of the CWL standards use the `--enable-dev` flag:
 ```
-./run_test.sh EXTRA=--enable-dev
+./run_test.sh RUNNER=cwltool EXTRA=--enable-dev
 ```
 
 
 For details of options you can pass to the test script, do:
 ```
 ./run_test.sh --help
 ```
-
-The full test suite takes about 10 minutes to run

intro.md

@@ -1,11 +1,13 @@
 # Status of this document
 
 This document is the product of the [Common Workflow Language working
-group](https://groups.google.com/forum/#!forum/common-workflow-language).  The
-latest stable version of this document is available in the "v1.0" directory at
+group](https://www.commonwl.org/).  The
+source for the latest stable version of this document is available in the "v1.0" directory at
 
 https://github.com/common-workflow-language/common-workflow-language
 
+The current development version is at https://github.com/common-workflow-language/cwl-v1.1/
+
 The products of the CWL working group (including this document) are made available
 under the terms of the Apache License, version 2.0.
 

invocation.md

@@ -96,9 +96,10 @@ output directory may each reside on different mount points on different file
 systems.
 
 An implementation may forbid the tool from directly accessing network
-resources.  Correct tools must not assume any network access.  Future versions
-of the specification may incorporate optional process requirements that
-describe the networking needs of a tool.
+resources.  Correct tools must not assume any network access unless they have
+the 'networkAccess' field of a ['NetworkAccess'](#NetworkAccess) requirement set
+to `true` but even then this does not imply a publically routable IP address or
+the ability to accept inbound connections.
 
 The `runtime` section available in [parameter references](#Parameter_references)
 and [expressions](#Expressions) contains the following fields.  As noted
@@ -115,23 +116,23 @@ not perform computation on the contents.
   * `runtime.tmpdirSize`: reserved storage space available in the designated temporary directory
 
 For `cores`, `ram`, `outdirSize` and `tmpdirSize`, if an implementation can't
-provide the actual number of reserved cores during the expression evaluation time,
+provide the actual number of reserved resources during the expression evaluation time,
 it should report back the minimal requested amount.
 
 See [ResourceRequirement](#ResourceRequirement) for details on how to
 describe the hardware resources required by a tool.
 
-The standard input stream and standard output stream may be redirected as
-described in the `stdin` and `stdout` fields.
+The standard input stream, the standard output stream, and/or the standard error
+stream may be redirected as described in the [`stdin`](#stdin),
+[`stdout`](#stdout), and [`stderr`](#stderr) fields.
 
 ## Execution
 
 Once the command line is built and the runtime environment is created, the
 actual tool is executed.
 
-The standard error stream and standard output stream (unless redirected by
-setting `stdout` or `stderr`) may be captured by platform logging facilities
-for storage and reporting.
+The standard error stream and standard output stream may be captured by
+platform logging facilities for storage and reporting.
 
 Tools may be multithreaded or spawn child processes; however, when the
 parent process exits, the tool is considered finished regardless of whether