Merge branch 'master' into add_exec_line

Author: mr-c

CODE_OF_CONDUCT.md

@@ -0,0 +1,108 @@
+CWL Code of Conduct
+===================
+
+The CWL Project is dedicated to providing a harassment-free experience for
+everyone. We do not tolerate harassment of participants in any form.
+
+This code of conduct applies to all CWL Project spaces both online and off: the
+Google Group, the Gitter chat room, the Google Hangouts chats, and any other
+CWL spaces. Anyone who violates this code of conduct may be sanctioned or
+expelled from these spaces at the discretion of the CWL Leadership Team.
+
+Some CWL Project spaces may have additional rules in place, which will be
+made clearly available to participants. Participants are responsible for
+knowing and abiding by these rules.
+
+Harassment includes, but is not limited to:
+
+ - Offensive comments related to gender, gender identity and expression, sexual
+orientation, disability, mental illness, neuro(a)typicality, physical
+appearance, body size, age, race, or religion.
+ - Unwelcome comments regarding a person’s lifestyle choices and practices,
+including those related to food, health, parenting, drugs, and employment.
+ - Deliberate misgendering or use of [dead](https://www.quora.com/What-is-deadnaming/answer/Nancy-C-Walker)
+or rejected names.
+ - Gratuitous or off-topic sexual images or behaviour  in spaces where they’re not
+appropriate.
+ - Physical contact and simulated physical contact (eg, textual descriptions like
+“\*hug\*” or “\*backrub\*”) without consent or after a request to stop.
+ - Threats of violence.
+ - Incitement of violence towards any individual, including encouraging a person
+to commit suicide or to engage in self-harm.
+ - Deliberate intimidation.
+ - Stalking or following.
+ - Harassing photography or recording, including logging online activity for
+harassment purposes.
+ - Sustained disruption of discussion.
+ - Unwelcome sexual attention.
+ - Pattern of inappropriate social contact, such as requesting/assuming
+inappropriate levels of intimacy with others
+ - Continued one-on-one communication after requests to cease.
+ - Deliberate “outing” of any aspect of a person’s identity without their consent
+except as necessary to protect vulnerable people from intentional abuse.
+ - Publication of non-harassing private communication.
+
+The CWL Project prioritizes marginalized people’s safety over privileged
+people’s comfort. The CWL Leadeship Team will not act on complaints regarding:
+
+ - ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’
+ - Reasonable communication of boundaries, such as “leave me alone,” “go away,” or
+“I’m not discussing this with you.”
+ - Communicating in a [tone](http://geekfeminism.wikia.com/wiki/Tone_argument)
+you don’t find congenial
+
+Reporting
+---------
+
+If you are being harassed by a member of the CWL Project, notice that someone
+else is being harassed, or have any other concerns, please contact the CWL
+Leadership Team at [email protected]. If person who is harassing
+you is on the team, they will recuse themselves from handling your incident. We
+will respond as promptly as we can.
+
+This code of conduct applies to CWL Project spaces, but if you are being
+harassed by a member of CWL Project outside our spaces, we still want to
+know about it. We will take all good-faith reports of harassment by CWL Project
+members, especially the CWL Leadership Team, seriously. This includes harassment
+outside our spaces and harassment that took place at any point in time. The
+abuse team reserves the right to exclude people from the CWL Project based on
+their past behavior, including behavior outside CWL Project spaces and
+behavior towards people who are not in the CWL Project.
+
+In order to protect volunteers from abuse and burnout, we reserve the right to
+reject any report we believe to have been made in bad faith. Reports intended
+to silence legitimate criticism may be deleted without response.
+
+We will respect confidentiality requests for the purpose of protecting victims
+of abuse. At our discretion, we may publicly name a person about whom we’ve
+received harassment complaints, or privately warn third parties about them, if
+we believe that doing so will increase the safety of CWL Project members or
+the general public. We will not name harassment victims without their
+affirmative consent.
+
+Consequences
+------------
+
+Participants asked to stop any harassing behavior are expected to comply
+immediately.
+
+If a participant engages in harassing behavior, the CWL Leadership Team may
+take any action they deem appropriate, up to and including expulsion from all
+CWL Project spaces and identification of the participant as a harasser to other
+CWL Project members or the general public.
+
+This anti-harassment policy is based on the [example policy from the Geek
+Feminism wiki](http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy),
+created by the Geek Feminism community.
+
+CWL Leadership Team
+-------------------
+
+As a stop gap measure until a more formal governance structure is adopted, the
+following individuals make up the leadership of the CWL Project: Peter Amstutz,
+John Chilton, Michael R. Crusoe, and Nebojša Tijanić.
+
+To report an issue with anyone on the team you can escalate to Ward Vandewege
+ (Curoverse) [email protected], Anton Nekrutenko (Galaxy)
+anton AT bx DOT psu DOT edu, C. Titus Brown (UC Davis) [email protected], or
+Brandi Davis-Dusenbery (Seven Bridges Genomics) [email protected].

MANIFEST.in

@@ -9,6 +9,7 @@ include tests/wf/*
 include tests/override/*
 include tests/checker_wf/*
 include tests/subgraph/*
+include tests/trs/*
 include cwltool/schemas/v1.0/*.yml
 include cwltool/schemas/v1.0/*.yml
 include cwltool/schemas/v1.0/*.md

Makefile

@@ -27,11 +27,11 @@ MODULE=cwltool
 # `[[` conditional expressions.
 PYSOURCES=$(wildcard ${MODULE}/**.py tests/*.py) setup.py
 DEVPKGS=pycodestyle diff_cover autopep8 pylint coverage pydocstyle flake8 \
-	pytest pytest-xdist isort
+	pytest-xdist==1.27.0 isort wheel -rtest-requirements.txt
 DEBDEVPKGS=pep8 python-autopep8 pylint python-coverage pydocstyle sloccount \
 	   python-flake8 python-mock shellcheck
-VERSION=1.0.$(shell date +%Y%m%d%H%M%S --utc --date=`git log --first-parent \
-	--max-count=1 --format=format:%cI`)
+VERSION=1.0.$(shell TZ=UTC git log --first-parent --max-count=1 \
+	--format=format:%cd --date=format-local:%Y%m%d%H%M%S)
 mkfile_dir := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 UNAME_S=$(shell uname -s)
 ifeq ($(UNAME_S),Linux)
@@ -51,7 +51,7 @@ help: Makefile
 
 ## install-dep : install most of the development dependencies via pip
 install-dep:
-	pip install --upgrade $(DEVPKGS) -rtest-requirements.txt
+	pip install --upgrade $(DEVPKGS)
 
 ## install-deb-dep: install most of the dev dependencies via apt-get
 install-deb-dep:

README.rst

@@ -2,7 +2,7 @@
 Common Workflow Language tool description reference implementation
 ==================================================================
 
-CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Coverage Status|
+CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Coverage Status| |Downloads|
 
 
 .. |Conformance Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
@@ -15,7 +15,10 @@ CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Cov
    :target: https://ci.appveyor.com/project/mr-c/cwltool
 
 .. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg
-  :target: https://codecov.io/gh/common-workflow-language/cwltool
+   :target: https://codecov.io/gh/common-workflow-language/cwltool
+
+.. |Downloads| image:: https://pepy.tech/badge/cwltool/month
+   :target: https://pepy.tech/project/cwltool
 
 This is the reference implementation of the Common Workflow Language.  It is
 intended to be feature complete and provide comprehensive validation of CWL
@@ -495,7 +498,7 @@ Running tests locally
 
 -  Running basic tests ``(/tests)``:
 
-To run the basis tests after installing `cwltool` execute the following:
+To run the basic tests after installing `cwltool` execute the following:
 
 .. code:: bash
 

appveyor.yml

@@ -32,12 +32,12 @@ install:
   - ps: 'Install-Product node 0.12 x64'
   - "set PATH=%PYTHON%\\Scripts;%PATH%"
   - "%PYTHON%\\python.exe -m pip install -U pip setuptools^>=20.3 wheel"
-  - "%PYTHON%\\python.exe -m pip install -U codecov pytest-xdist pytest-cov galaxy-lib -rtest-requirements.txt"
+  - "%PYTHON%\\python.exe -m pip install -U codecov -rtest-requirements.txt pytest-xdist==1.27 "
     # Note the use of a `^` to escape the `>`
 
 build_script:
   - "%PYTHON%\\python.exe -m pip install -rrequirements.txt"
-  - "%PYTHON%\\python.exe -m pip install -e ."
+  - "%PYTHON%\\python.exe -m pip install -e .[deps]"
 
 test_script:
   - |

cwl-docker.sh

@@ -1,2 +1,3 @@
 #!/bin/sh
+# Changing the line below? Update https://cloud.docker.com/u/commonworkflowlanguage/repository/docker/commonworkflowlanguage/cwltool
 exec docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp -v "$PWD":"$PWD" -w="$PWD" commonworkflowlanguage/cwltool "$@"

cwltool/argparser.py

@@ -57,7 +57,7 @@ def arg_parser():  # type: () -> argparse.ArgumentParser
 
     cidgroup.add_argument(
         "--cidfile-dir", type=Text, help="Store the Docker "
-        "container ID into a file in the specifed directory.",
+        "container ID into a file in the specified directory.",
         default=None, dest="cidfile_dir")
 
     cidgroup.add_argument(
@@ -105,10 +105,10 @@ def arg_parser():  # type: () -> argparse.ArgumentParser
 
     exgroup = parser.add_mutually_exclusive_group()
     exgroup.add_argument("--enable-pull", default=True, action="store_true",
-                         help="Try to pull Docker images", dest="enable_pull")
+                         help="Try to pull Docker images", dest="pull_image")
 
     exgroup.add_argument("--disable-pull", default=True, action="store_false",
-                         help="Do not try to pull Docker images", dest="enable_pull")
+                         help="Do not try to pull Docker images", dest="pull_image")
 
     parser.add_argument("--rdf-serializer",
                         help="Output RDF serialization format used by --print-rdf (one of turtle (default), n3, nt, xml)",

cwltool/builder.py

@@ -1,18 +1,24 @@
 from __future__ import absolute_import
 
 import copy
+import os
 import logging
 from typing import (Any, Callable, Dict, List, MutableMapping, MutableSequence,
                     Optional, Set, Tuple, Union)
 
+from typing_extensions import Text, Type, TYPE_CHECKING  # pylint: disable=unused-import
+# move to a regular typing import when Python 3.3-3.6 is no longer supported
+
 from rdflib import Graph, URIRef  # pylint: disable=unused-import
 from rdflib.namespace import OWL, RDFS
 from ruamel.yaml.comments import CommentedMap
 from schema_salad import validate
 from schema_salad.schema import Names, convert_to_dict
 from schema_salad.avro.schema import make_avsc_object, Schema
 from schema_salad.sourceline import SourceLine
+from schema_salad.ref_resolver import uri_file_path
 from six import iteritems, string_types
+from typing import IO
 from typing_extensions import (TYPE_CHECKING,  # pylint: disable=unused-import
                                Text, Type)
 # move to a regular typing import when Python 3.3-3.6 is no longer supported
@@ -22,15 +28,25 @@
 from .loghandler import _logger
 from .mutation import MutationManager  # pylint: disable=unused-import
 from .pathmapper import PathMapper  # pylint: disable=unused-import
-from .pathmapper import get_listing, normalizeFilesDirs, visit_class
+from .pathmapper import CONTENT_LIMIT, get_listing, normalizeFilesDirs, visit_class
 from .stdfsaccess import StdFsAccess  # pylint: disable=unused-import
 from .utils import aslist, docker_windows_path_adjust, json_dumps, onWindows
 
 
 
 if TYPE_CHECKING:
     from .provenance import ProvenanceProfile  # pylint: disable=unused-import
-CONTENT_LIMIT = 64 * 1024
+
+
+def content_limit_respected_read_bytes(f):  # type: (IO) -> bytes
+    contents = f.read(CONTENT_LIMIT + 1)
+    if len(contents) > CONTENT_LIMIT:
+        raise WorkflowException("loadContents handling encountered buffer that is exceeds maximum lenght of %d bytes" % CONTENT_LIMIT)
+    return contents
+
+
+def content_limit_respected_read(f):  # type: (IO) -> Text
+    return content_limit_respected_read_bytes(f).decode("utf-8")
 
 
 def substitute(value, replace):  # type: (Text, Text) -> Text
@@ -130,7 +146,7 @@ def __init__(self,
                  loadListing,          # type: Text
                  outdir,               # type: Text
                  tmpdir,               # type: Text
-                 stagedir,             # type: Text
+                 stagedir             # type: Text
                 ):  # type: (...) -> None
 
         self.job = job
@@ -191,11 +207,16 @@ def bind_input(self,
         value_from_expression = False
         if "inputBinding" in schema and isinstance(schema["inputBinding"], MutableMapping):
             binding = CommentedMap(schema["inputBinding"].items())
-            assert binding is not None
 
             bp = list(aslist(lead_pos))
             if "position" in binding:
-                bp.extend(aslist(binding["position"]))
+                position = binding["position"]
+                if isinstance(position, str):   # no need to test the CWL Version
+                                                # the schema for v1.0 only allow ints
+                    binding['position'] = self.do_eval(position, context=datum)
+                    bp.append(binding['position'])
+                else:
+                    bp.extend(aslist(binding['position']))
             else:
                 bp.append(0)
             bp.extend(aslist(tail_pos))
@@ -216,7 +237,6 @@ def bind_input(self,
                     avsc = self.names.get_name(t["name"], "")
                 if not avsc:
                     avsc = make_avsc_object(convert_to_dict(t), self.names)
-                assert avsc is not None
                 if validate.validate(avsc, datum):
                     schema = copy.deepcopy(schema)
                     schema["type"] = t
@@ -278,32 +298,41 @@ def _capture_files(f):
                 self.files.append(datum)
                 if (binding and binding.get("loadContents")) or schema.get("loadContents"):
                     with self.fs_access.open(datum["location"], "rb") as f:
-                        datum["contents"] = f.read(CONTENT_LIMIT).decode("utf-8")
+                        datum["contents"] = content_limit_respected_read(f)
 
                 if "secondaryFiles" in schema:
                     if "secondaryFiles" not in datum:
                         datum["secondaryFiles"] = []
                     for sf in aslist(schema["secondaryFiles"]):
-                        if isinstance(sf, MutableMapping) or "$(" in sf or "${" in sf:
-                            sfpath = self.do_eval(sf, context=datum)
+                        if 'required' in sf:
+                            sf_required = self.do_eval(sf['required'], context=datum)
+                        else:
+                            sf_required = True
+
+                        if "$(" in sf["pattern"] or "${" in sf["pattern"]:
+                            sfpath = self.do_eval(sf["pattern"], context=datum)
                         else:
-                            sfpath = substitute(datum["basename"], sf)
+                            sfpath = substitute(datum["basename"], sf["pattern"])
+
                         for sfname in aslist(sfpath):
+                            if not sfname:
+                                continue
                             found = False
                             for d in datum["secondaryFiles"]:
                                 if not d.get("basename"):
                                     d["basename"] = d["location"][d["location"].rindex("/")+1:]
                                 if d["basename"] == sfname:
                                     found = True
                             if not found:
+                                sf_location = datum["location"][0:datum["location"].rindex("/")+1]+sfname
                                 if isinstance(sfname, MutableMapping):
                                     datum["secondaryFiles"].append(sfname)
-                                elif discover_secondaryFiles:
+                                elif discover_secondaryFiles and self.fs_access.exists(sf_location):
                                     datum["secondaryFiles"].append({
-                                        "location": datum["location"][0:datum["location"].rindex("/")+1]+sfname,
+                                        "location": sf_location,
                                         "basename": sfname,
                                         "class": "File"})
-                                else:
+                                elif sf_required:
                                     raise WorkflowException("Missing required secondary file '%s' from file object: %s" % (
                                         sfname, json_dumps(datum, indent=4)))
 
@@ -321,7 +350,7 @@ def _capture_files(f):
                 visit_class(datum.get("secondaryFiles", []), ("File", "Directory"), _capture_files)
 
             if schema["type"] == "Directory":
-                ll = self.loadListing or (binding and binding.get("loadListing"))
+                ll = schema.get("loadListing") or self.loadListing
                 if ll and ll != "no_listing":
                     get_listing(self.fs_access, datum, (ll == "deep_listing"))
                 self.files.append(datum)
@@ -348,7 +377,6 @@ def tostr(self, value):  # type: (Any) -> Text
                 # docker_req is none only when there is no dockerRequirement
                 # mentioned in hints and Requirement
                 path = docker_windows_path_adjust(value["path"])
-                assert path is not None
                 return path
             return value["path"]
         else:
@@ -393,8 +421,7 @@ def generate_arg(self, binding):  # type: (Dict[Text, Any]) -> List[Text]
             if sep:
                 args.extend([prefix, self.tostr(j)])
             else:
-                assert prefix is not None
-                args.append(prefix + self.tostr(j))
+                args.append("" if not prefix else prefix + self.tostr(j))
 
         return [a for a in args if a is not None]