Merge branch 'gh-pages' of github.com:common-workflow-language/user_guide into shebang

Author: Toby Hodges

_episodes/01-introduction.md

@@ -28,5 +28,3 @@ technologies such as Docker, be used with CWL implementations from different
 vendors, and is well suited for describing large-scale workflows in cluster,
 cloud and high performance computing environments where tasks are scheduled in
 parallel across many nodes.
-
-{% include links.md %}

_episodes/02-1st-example.md

@@ -17,20 +17,28 @@ The simplest "hello world" program.  This accepts one input parameter, writes a
 
 
 *1st-tool.cwl*
+```
 {% include cwl/1st-tool.cwl %}
+```
 
 Use a YAML object in a separate file to describe the input of a run:
 
 *echo-job.yml*
+```
 {% include cwl/echo-job.yml %}
+```
 
 Now invoke `cwl-runner` with the tool wrapper and the input object on the command line:
 
 ```
 $ cwl-runner 1st-tool.cwl echo-job.yml
-[job 140199012414352] $ echo 'Hello world!'
+[job 1st-tool.cwl] /tmp/tmpmM5S_1$ echo \
+    'Hello world!'
 Hello world!
+[job 1st-tool.cwl] completed success
+{}
 Final process status is success
+
 ```
 
 What's going on here?  Let's break it down:

_episodes/03-input.md

@@ -28,9 +28,9 @@ types and appearing on the command line in different ways:
 
 *inp.cwl*
 
-```
+~~~
 {% include cwl/inp.cwl %}
-```
+~~~
 
 *inp-job.yml*
 
@@ -47,10 +47,24 @@ input object on the command line:
 ```
 $ touch whale.txt
 $ cwl-runner inp.cwl inp-job.yml
-[job 140020149614160] /home/example$ echo -f -i42 --example-string hello --file=/home/example/whale.txt
--f -i42 --example-string hello --file=/home/example/whale.txt
+[job inp.cwl] /tmp/tmpzrSnfX$ echo \
+    -f \
+    -i42 \
+    --example-string \
+    hello \
+    --file=/tmp/tmpRBSHIG/stg979b6d24-d50a-47e3-9e9e-90097eed2cbc/whale.txt
+-f -i42 --example-string hello --file=/tmp/tmpRBSHIG/stg979b6d24-d50a-47e3-9e9e-90097eed2cbc/whale.txt
+[job inp.cwl] completed success
+{}
 Final process status is success
 ```
+> ## Where did those `/tmp` paths come from?
+>
+> The CWL reference runner (cwltool) and other runners create temporary
+> directories with symbolic ("soft") links to your input files to ensure that
+> the tools aren't accidently accessing files that were not explicitly
+> specified
+{: .callout}
 
 The field `inputBinding` is optional and indicates whether and how the
 input parameter should be appear on the tool's command line.  If
@@ -109,19 +123,19 @@ example_file:
 File types appear on the command line as the path to the file.  When the
 parameter type ends with a question mark `?` it indicates that the
 parameter is optional.  In the example above, this is rendered as
-`--file=/home/example/whale.txt`.  However, if the "example_file"
+`--file=/tmp/random/path/whale.txt`.  However, if the "example_file"
 parameter were not provided in the input, nothing would appear on the
 command line.
 
 Input files are read-only.  If you wish to update an input file, you must
-first copy it to the output directory.
+[first copy it to the output directory]({{ page.root }}/15-staging/).
 
 The value of `position` is used to determine where parameter should
 appear on the command line.  Positions are relative to one another, not
 absolute.  As a result, positions do not have to be sequential, three
-parameters with positions `[1, 3, 5]` will result in the same command
-line as `[1, 2, 3]`.  More than one parameter can have the same position
+parameters with positions 1, 3, 5 will result in the same command
+line as 1, 2, 3.  More than one parameter can have the same position
 (ties are broken using the parameter name), and the position field itself
-is optional.  the default position is 0.
+is optional.  The default position is 0.
 
-The `baseCommand` field always comes before parameters.
+The `baseCommand` field will always appear in the final command line before the parameters.

_episodes/04-output.md

@@ -41,16 +41,23 @@ wrapper and the input object on the command line:
 ```
 $ touch hello.txt && tar -cvf hello.tar hello.txt
 $ cwl-runner tar.cwl tar-job.yml
-[job 139868145165200] $ tar xf /home/example/hello.tar
-Final process status is success
+[job tar.cwl] /tmp/tmpqOeawQ$ tar \
+    xf \
+    /tmp/tmpGDk8Y1/stg80bbad20-494d-47af-8075-dffc32df03a3/hello.tar
+[job tar.cwl] completed success
 {
-"example_out": {
-  "location": "hello.txt",
-  "size": 13,
-  "class": "File",
-  "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b"
-  }
+    "example_out": {
+        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709", 
+        "basename": "hello.txt", 
+        "nameroot": "hello", 
+        "nameext": ".txt", 
+        "location": "file:///home/me/cwl/user_guide/hello.txt", 
+        "path": "/home/me/cwl/user_guide/hello.txt", 
+        "class": "File", 
+        "size": 0
+    }
 }
+Final process status is success
 ```
 
 The field `outputBinding` describes how to to set the value of each
@@ -61,8 +68,8 @@ outputs:
   example_out:
     type: File
     outputBinding:
-    glob: hello.txt
+      glob: hello.txt
 ```
 
 The `glob` field consists of the name of a file in the output directory.
-If you don't know name of the file in advance, you can use a wildcard pattern.
+If you don't know name of the file in advance, you can use a wildcard pattern like `*.txt`.

_episodes/05-stdout.md

@@ -31,16 +31,21 @@ on the command line:
 
 ```
 $ cwl-runner stdout.cwl echo-job.yml
-[job 140199012414352] $ echo 'Hello world!' > output.txt
-Final process status is success
+[job stdout.cwl] /tmp/tmpE0gTz7$ echo \
+    'Hello world!' > /tmp/tmpE0gTz7/output.txt
+[job stdout.cwl] completed success
 {
-"output": {
-  "location": "output.txt",
-  "size": 13,
-  "class": "File",
-  "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b"
-  }
+    "output": {
+        "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
+        "basename": "output.txt",
+        "nameroot": "output",
+        "nameext": ".txt",
+        "location": "file:///home/me/cwl/user_guide/output.txt",
+        "path": "/home/me/cwl/user_guide/output.txt",
+        "class": "File",
+        "size": 13
+    }
 }
-$ cat output.txt
-Hello world!
+Final process status is success
+
 ```

_episodes/06-params.md

@@ -34,16 +34,24 @@ input object on the command line:
 ```
 $ rm hello.tar || true && touch goodbye.txt && tar -cvf hello.tar goodbye.txt
 $ cwl-runner tar-param.cwl tar-param-job.yml
-[job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
-Final process status is success
+[job tar-param.cwl] /tmp/tmpwH4ouT$ tar \
+    xf \
+    /tmp/tmpREYiEt/stgd7764383-99c9-4848-af51-7c2d6e5527d9/hello.tar \
+    goodbye.txt
+[job tar-param.cwl] completed success
 {
-"example_out": {
-  "location": "goodbye.txt",
-  "size": 24,
-  "class": "File",
-  "checksum": "sha1$dd0a4c4c49ba43004d6611771972b6cf969c1c01"
-  }
+    "example_out": {
+        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709", 
+        "basename": "goodbye.txt", 
+        "nameroot": "goodbye", 
+        "nameext": ".txt", 
+        "location": "file:///home/me/cwl/user_guide/goodbye.txt", 
+        "path": "/home/me/cwl/user_guide/goodbye.txt", 
+        "class": "File", 
+        "size": 0
+    }
 }
+Final process status is success
 ```
 
 Certain fields permit parameter references which are enclosed in `$(...)`.

_episodes/07-containers.md

@@ -17,7 +17,7 @@ complete known-good runtime for software and its dependencies.  However,
 containers are also purposefully isolated from the host system, so in
 order to run a tool inside a Docker container there is additional work to
 ensure that input files are available inside the container and output
-files can be recovered from the container.  CWL can perform this work
+files can be recovered from the container.  A CWL runner can perform this work
 automatically, allowing you to use Docker to simplify your software
 management while avoiding the complexity of invoking and managing Docker
 containers.
@@ -42,15 +42,31 @@ input object on the command line:
 ```
 $ echo "console.log(\"Hello World\");" > hello.js
 $ cwl-runner docker.cwl docker-job.yml
-[job 140259721854416] /home/example$ docker run -i --volume=/home/example/hello.js:/var/lib/cwl/job369354770_examples/hello.js:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpDLs5hm:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp node:slim node /var/lib/cwl/job369354770_examples/hello.js
-Hello world!
+[job docker.cwl] /tmp/tmpgugLND$ docker \
+    run \
+    -i \
+    --volume=/tmp/tmpgugLND:/var/spool/cwl:rw \
+    --volume=/tmp/tmpSs5JoN:/tmp:rw \
+    --volume=/home/me/cwl/user_guide/hello.js:/var/lib/cwl/stg16848d97-e6ba-4b35-b666-4546d9965a2d/hello.js:ro \
+    --workdir=/var/spool/cwl \
+    --read-only=true \
+    --user=1000 \
+    --rm \
+    --env=TMPDIR=/tmp \
+    --env=HOME=/var/spool/cwl \
+    node:slim \
+    node \
+    /var/lib/cwl/stg16848d97-e6ba-4b35-b666-4546d9965a2d/hello.js
+Hello World
+[job docker.cwl] completed success
+{}
 Final process status is success
 ```
 
 Notice the CWL runner has constructed a Docker command line to run the
 script.  One of the responsibilities of the CWL runner is to adjust the paths of
 input files to reflect the location where they appear inside the container.  
-In this example, the path to the script `hello.js` is `/home/example/hello.js`
+In this example, the path to the script `hello.js` is `/home/me/cwl/user_guide/hello.js`
 outside the container but `/var/lib/cwl/job369354770_examples/hello.js` inside
 the container, as reflected in the invocation of the `node` command.