Merge branch 'master' into feature-js-console-output

Author: mr-c

README.rst

@@ -1,45 +1,57 @@
 ==================================================================
-Common workflow language tool description reference implementation
+Common Workflow Language tool description reference implementation
 ==================================================================
 
-CWL Conformance test: |Build Status|
-
-Travis: |Unix Build Status|
+CWL conformance tests: |Build Status| Travis CI: |Unix Build Status|
 
 .. |Unix Build Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=unix%20build
    :target: https://travis-ci.org/common-workflow-language/cwltool
 
 This is the reference implementation of the Common Workflow Language.  It is
-intended to be feature complete and provide comprehensive validation of CWL
+intended to feature complete and provide comprehensive validation of CWL
 files as well as provide other tools related to working with CWL.
 
-This is written and tested for Python 2.7.
+This is written and tested for Python ``2.7 and 3.x {x = 3, 4, 5, 6}``
 
-The reference implementation consists of two packages.  The "cwltool" package
+The reference implementation consists of two packages.  The ``cwltool`` package
 is the primary Python module containing the reference implementation in the
-"cwltool" module and console executable by the same name.
+``cwltool`` module and console executable by the same name.
 
-The "cwlref-runner" package is optional and provides an additional entry point
-under the alias "cwl-runner", which is the implementation-agnostic name for the
+The ``cwlref-runner`` package is optional and provides an additional entry point
+under the alias ``cwl-runner``, which is the implementation-agnostic name for the
 default CWL interpreter installed on a host.
 
 Install
 -------
 
-Installing the official package from PyPi (will install "cwltool" package as
-well)::
+It is highly recommended to setup virtual environment before installing `cwltool`:
+
+.. code:: bash
+
+  virtualenv -p python2 venv   # Create a virtual environment, can use `python3` as well
+  source venv/bin/activate     # Activate environment before installing `cwltool`
+
+1. Installing the official package from PyPi (will install "cwltool" package as
+well)
+
+.. code:: bash
 
   pip install cwlref-runner
 
-If installing alongside another CWL implementation then::
+If installing alongside another CWL implementation then
+
+.. code:: bash
 
   pip install cwltool
 
-To install from source::
+2. To install from source
+
+.. code:: bash
 
-  git clone https://github.com/common-workflow-language/cwltool.git
-  cd cwltool && python setup.py install
-  cd cwlref-runner && python setup.py install  # co-installing? skip this
+  git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo
+  cd cwltool         # Switch to source directory
+  pip install .      # Install `cwltool` from source
+  cwltool --version  # Check if the installation works correctly
 
 Remember, if co-installing multiple CWL implementations then you need to
 maintain which implementation ``cwl-runner`` points to via a symbolic file
@@ -50,9 +62,15 @@ Running tests locally
 
 -  Running basic tests ``(/tests)``:
 
-.. code:: bash
+We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
+to run various tests in all supported Python environments.
+You can run the test suite by simply running the following in the terminal:
+``pip install tox; tox``
 
-    python setup.py test
+List of all environment can be seen using:
+``tox --listenvs``
+and running a specfic test env using:
+``tox -e <env name>``
 
 -  Running the entire suite of CWL conformance tests:
 
@@ -123,13 +141,17 @@ For this example, grab the test.json (and input file) from https://github.com/Ca
 Import as a module
 ------------------
 
-Add::
+Add
+
+.. code:: python
 
   import cwltool
 
 to your script.
 
-The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory::
+The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
+
+.. code:: python
 
   import cwltool.factory
   fac = cwltool.factory.Factory()
@@ -188,10 +210,11 @@ for the plugin above, this is ``type`` and defines the plugin type. This paramet
 is required for all plugins. The available plugins and the parameters
 available for each are documented (incompletely) `here
 <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
-Unfortunately, this documentation is in the context of Galaxy tool ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
+Unfortunately, this documentation is in the context of Galaxy tool
+``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
 
 cwltool is distributed with an example of such seqtk tool and sample corresponding
-job. It could executed from the cwltool root using a dependency resolvers 
+job. It could executed from the cwltool root using a dependency resolvers
 configuration file such as the above one using the command::
 
   cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
@@ -208,8 +231,8 @@ the same concepts - the resolver plugin type ``galaxy_packages`` can be used.
 "Galaxy packages" are a lighter weight alternative to Environment Modules that are
 really just defined by a way to lay out directories into packages and versions
 to find little scripts that are sourced to modify the environment. They have
-been used for years in Galaxy community to adapt Galaxy tools to cluster 
-environments but require neither knowledge of Galaxy nor any special tools to 
+been used for years in Galaxy community to adapt Galaxy tools to cluster
+environments but require neither knowledge of Galaxy nor any special tools to
 setup. These should work just fine for CWL tools.
 
 The cwltool source code repository's test directory is setup with a very simple
@@ -238,7 +261,7 @@ Then cwltool will simply find that ``env.sh`` file and source it before executin
 the corresponding tool. That ``env.sh`` script is only responsible for modifying
 the job's ``PATH`` to add the required binaries.
 
-This is a full example that works since resolving "Galaxy packages" has no 
+This is a full example that works since resolving "Galaxy packages" has no
 external requirements. Try it out by executing the following command from cwltool's
 root directory::
 
@@ -302,7 +325,7 @@ user, install its own Conda environment and manage multiple versions of Conda pa
 on both Linux and Mac OS X.
 
 The Conda plugin can be endlessly configured, but a sensible set of defaults
-that has proven a powerful stack for dependency management within the Galaxy tool 
+that has proven a powerful stack for dependency management within the Galaxy tool
 development ecosystem can be enabled by simply passing cwltool the
 ``--beta-conda-dependencies`` flag.
 
@@ -346,52 +369,52 @@ at the following links:
 - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
 - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
 
-Cwltool control flow
---------------------
+CWL Tool Control Flow
+---------------------
 
 Technical outline of how cwltool works internally, for maintainers.
 
-#. Use CWL `load_tool()` to load document.
+#. Use CWL ``load_tool()`` to load document.
 
    #. Fetches the document from file or URL
    #. Applies preprocessing (syntax/identifier expansion and normalization)
    #. Validates the document based on cwlVersion
    #. If necessary, updates the document to latest spec
-   #. Constructs a Process object using `make_tool()` callback.  This yields a
+   #. Constructs a Process object using ``make_tool()``` callback.  This yields a
       CommandLineTool, Workflow, or ExpressionTool.  For workflows, this
       recursively constructs each workflow step.
    #. To construct custom types for CommandLineTool, Workflow, or
-      ExpressionTool, provide a custom `make_tool()`
+      ExpressionTool, provide a custom ``make_tool()``
 
-#. Iterate on the `job()` method of the Process object to get back runnable jobs.
+#. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
 
-   #. `job()` is a generator method (uses the Python iterator protocol)
-   #. Each time the `job()` method is invoked in an iteration, it returns one
-      of: a runnable item (an object with a `run()` method), `None` (indicating
+   #. ``job()`` is a generator method (uses the Python iterator protocol)
+   #. Each time the ``job()`` method is invoked in an iteration, it returns one
+      of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
       there is currently no work ready to run) or end of iteration (indicating
       the process is complete.)
-   #. Invoke the runnable item by calling `run()`.  This runs the tool and gets output.
+   #. Invoke the runnable item by calling ``run()``.  This runs the tool and gets output.
    #. Output of a process is reported by an output callback.
-   #. `job()` may be iterated over multiple times.  It will yield all the work
+   #. ``job()`` may be iterated over multiple times.  It will yield all the work
       that is currently ready to run and then yield None.
 
-#. "Workflow" objects create a corresponding "WorkflowJob" and "WorkflowJobStep" objects to hold the workflow state for the duration of the job invocation.
+#. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
 
    #. The WorkflowJob iterates over each WorkflowJobStep and determines if the
       inputs the step are ready.
    #. When a step is ready, it constructs an input object for that step and
-      iterates on the `job()` method of the workflow job step.
+      iterates on the ``job()`` method of the workflow job step.
    #. Each runnable item is yielded back up to top level run loop
    #. When a step job completes and receives an output callback, the
       job outputs are assigned to the output of the workflow step.
    #. When all steps are complete, the intermediate files are moved to a final
       workflow output, intermediate directories are deleted, and the output
       callback for the workflow is called.
 
-#. "CommandLineTool" job() objects yield a single runnable object.
+#. ``CommandLineTool`` job() objects yield a single runnable object.
 
-   #. The CommandLineTool `job()` method calls `makeJobRunner()` to create a
-      `CommandLineJob` object
+   #. The CommandLineTool ``job()`` method calls ``makeJobRunner()`` to create a
+      ``CommandLineJob`` object
    #. The job method configures the CommandLineJob object by setting public
       attributes
    #. The job method iterates over file and directories inputs to the
@@ -402,7 +425,7 @@ Technical outline of how cwltool works internally, for maintainers.
    #. Files are staged to targets paths using either Docker volume binds (when
       using containers) or symlinks (if not).  This staging step enables files
       to be logically rearranged or renamed independent of their source layout.
-   #. The run() method of CommandLineJob executes the command line tool or
+   #. The ``run()`` method of CommandLineJob executes the command line tool or
       Docker container, waits for it to complete, collects output, and makes
       the output callback.
 

appveyor.yml

@@ -0,0 +1,43 @@
+version: .{build}-{branch}
+
+environment:
+  SYSTEMROOT: "C:\\WINDOWS"
+
+  matrix:
+    - PYTHON: "C:\\Python27"
+      PYTHON_VERSION: "2.7.x"
+      PYTHON_ARCH: "32"
+
+    - PYTHON: "C:\\Python27-x64"
+      PYTHON_VERSION: "2.7.x"
+      PYTHON_ARCH: "64"
+
+    - PYTHON: "C:\\Python33-x64"
+      PYTHON_VERSION: "3.3.x"
+      PYTHON_ARCH: "64"
+
+    - PYTHON: "C:\\Python34-x64"
+      PYTHON_VERSION: "3.4.x"
+      PYTHON_ARCH: "64"
+
+    - PYTHON: "C:\\Python35-x64"
+      PYTHON_VERSION: "3.5.x"
+      PYTHON_ARCH: "64"
+
+
+install:
+  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+  - "python --version"
+  - "python -c \"import struct; print(struct.calcsize('P') * 8)\""
+
+build_script:
+  - "%CMD_IN_ENV% pip install ."
+
+
+test_script:
+
+  - "%CMD_IN_ENV% python setup.py test"
+
+branches:
+  only:
+    - master

cwltool/builder.py

@@ -1,5 +1,7 @@
 from __future__ import absolute_import
 import copy
+import os
+import logging
 from typing import Any, Callable, Dict, List, Text, Type, Union
 
 import six
@@ -15,11 +17,10 @@
 from .pathmapper import (PathMapper, get_listing, normalizeFilesDirs,
                          visit_class)
 from .stdfsaccess import StdFsAccess
-from .utils import aslist
+from .utils import aslist, get_feature, docker_windows_path_adjust, onWindows
+
+_logger = logging.getLogger("cwltool")
 
-# if six.PY3:
-# AvroSchemaFromJSONData = avro.schema.SchemaFromJSONData
-# else:
 AvroSchemaFromJSONData = avro.schema.make_avsc_object
 
 CONTENT_LIMIT = 64 * 1024
@@ -52,6 +53,7 @@ def __init__(self):  # type: () -> None
         self.debug = False  # type: bool
         self.js_console = False  # type: bool
         self.mutation_manager = None  # type: MutationManager
+        self.force_docker_pull = False  # type: bool
 
         # One of "no_listing", "shallow_listing", "deep_listing"
         # Will be default "no_listing" for CWL v1.1
@@ -61,8 +63,8 @@ def __init__(self):  # type: () -> None
         self.job_script_provider = None  # type: Any
 
     def build_job_script(self, commands):
-        # type: (List[bytes]) -> Text
-        build_job_script_method = getattr(self.job_script_provider, "build_job_script", None)  # type: Callable[[Builder, List[bytes]], Text]
+        # type: (List[Text]) -> Text
+        build_job_script_method = getattr(self.job_script_provider, "build_job_script", None)  # type: Callable[[Builder, Union[List[str],List[Text]]], Text]
         if build_job_script_method:
             return build_job_script_method(self, commands)
         else:
@@ -148,18 +150,25 @@ def bind_input(self, schema, datum, lead_pos=None, tail_pos=None):
                         datum["secondaryFiles"] = []
                     for sf in aslist(schema["secondaryFiles"]):
                         if isinstance(sf, dict) or "$(" in sf or "${" in sf:
-                            secondary_eval = self.do_eval(sf, context=datum)
-                            if isinstance(secondary_eval, string_types):
-                                sfpath = {"location": secondary_eval,
-                                          "class": "File"}
-                            else:
-                                sfpath = secondary_eval
-                        else:
-                            sfpath = {"location": substitute(datum["location"], sf), "class": "File"}
-                        if isinstance(sfpath, list):
-                            datum["secondaryFiles"].extend(sfpath)
+                            sfpath = self.do_eval(sf, context=datum)
                         else:
-                            datum["secondaryFiles"].append(sfpath)
+                            sfpath = substitute(datum["basename"], sf)
+                        for sfname in aslist(sfpath):
+                            found = False
+                            for d in datum["secondaryFiles"]:
+                                if not d.get("basename"):
+                                    d["basename"] = d["location"][d["location"].rindex("/")+1:]
+                                if d["basename"] == sfname:
+                                    found = True
+                            if not found:
+                                if isinstance(sfname, dict):
+                                    datum["secondaryFiles"].append(sfname)
+                                else:
+                                    datum["secondaryFiles"].append({
+                                        "location": datum["location"][0:datum["location"].rindex("/")+1]+sfname,
+                                        "basename": sfname,
+                                        "class": "File"})
+
                     normalizeFilesDirs(datum["secondaryFiles"])
 
                 def _capture_files(f):
@@ -186,6 +195,11 @@ def tostr(self, value):  # type: (Any) -> Text
         if isinstance(value, dict) and value.get("class") in ("File", "Directory"):
             if "path" not in value:
                 raise WorkflowException(u"%s object missing \"path\": %s" % (value["class"], value))
+
+            # Path adjust for windows file path when passing to docker, docker accepts unix like path only
+            (docker_req, docker_is_req) = get_feature(self, "DockerRequirement")
+            if onWindows() and docker_req is not None:  # docker_req is none only when there is no dockerRequirement mentioned in hints and Requirement
+                return docker_windows_path_adjust(value["path"])
             return value["path"]
         else:
             return Text(value)
@@ -244,4 +258,5 @@ def do_eval(self, ex, context=None, pull_image=True, recursive=False):
                                   context=context, pull_image=pull_image,
                                   timeout=self.timeout,
                                   debug=self.debug,
-                                  js_console=self.js_console)
+                                  js_console=self.js_console,
+                                  force_docker_pull=self.force_docker_pull)

cwltool/cwlrdf.py

@@ -20,9 +20,9 @@ def visitor(t):
     return g
 
 
-def printrdf(wf, ctx, sr, stdout):
-    # type: (Process, ContextType, Text, IO[Any]) -> None
-    stdout.write(gather(wf, ctx).serialize(format=sr))
+def printrdf(wf, ctx, sr):
+    # type: (Process, ContextType, Text) -> Text
+    return gather(wf, ctx).serialize(format=sr).decode('utf-8')
 
 
 def lastpart(uri):  # type: (Any) -> Text