Merge pull request #37 from common-workflow-language/loadListing

Add LoadListingRequirement and parameter-level loadListing option.

Author: tetron

CommandLineTool.yml

@@ -83,18 +83,21 @@ $graph:
           directories.
         * Clarify that unspecified details related to execution are left open to
           the platform.
-        * Added `InputParameter.loadContents` field. Use of `loadContents` in 
-          `InputBinding` is deprecated; it is preserved for v1.0 backwards 
+        * Added `InputParameter.loadContents` field. Use of `loadContents` in
+          `InputBinding` is deprecated; it is preserved for v1.0 backwards
            compatability and will be removed in CWL v2.0.
-        * [Added](#TimeLimit) `TimeLimit` feature, allowing to set an upper limit on the 
+        * [Added](#TimeLimit) `TimeLimit` feature, allowing to set an upper limit on the
           execution time of a CommandLineTool or ExpressionTool.
-        * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse 
+        * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse
           behavior for a particular tool or step for implementations that
           support reusing output from past work.
-        * [Added](#NetworkAccess) `NetworkAccess` feature, allowing to indicate whether a 
+        * [Added](#NetworkAccess) `NetworkAccess` feature, allowing to indicate whether a
           process requires outgoing network access.
-        * [Added](#InplaceUpdateRequirement) `InplaceUpdateRequirement` feature, allowing tools to directly 
+        * [Added](#InplaceUpdateRequirement) `InplaceUpdateRequirement` feature, allowing tools to directly
           update files with `writable: true` in `InitialWorkDirRequirement`.
+        * [Added](#LoadListingRequirement) `LoadListingRequirement`
+          and [loadListing](#LoadContents) to control whether and how
+          `Directory` listings should be loaded for use in expressions.
 
       See also the [CWL Workflow Description, v1.1.0-dev1 changelog](Workflow.html#Changelog).
 
@@ -764,9 +767,9 @@ $graph:
     produces an empty command line).
 
     Use of implicit ENTRYPOINT or CMD are discouraged due to reproducibility
-    concerns of the implicit hidden execution point (For further discussion, see 
-    [https://doi.org/10.12688/f1000research.15140.1](https://doi.org/10.12688/f1000research.15140.1)). Portable 
-    CommandLineTool wrappers in which use of a container is optional must not rely on ENTRYPOINT or CMD.  
+    concerns of the implicit hidden execution point (For further discussion, see
+    [https://doi.org/10.12688/f1000research.15140.1](https://doi.org/10.12688/f1000research.15140.1)). Portable
+    CommandLineTool wrappers in which use of a container is optional must not rely on ENTRYPOINT or CMD.
     CommandLineTools which do rely on ENTRYPOINT or CMD must list `DockerRequirement` in the
     `requirements` section.
 

Process.yml

@@ -436,21 +436,44 @@ $graph:
       doc: "The unique identifier for this object."
 
 
+- name: LoadListingEnum
+  type: enum
+  symbols: [no_listing, shallow_listing, deep_listing]
+  doc:
+    - |
+      Specify the desired behavior for loading the `listing` field of
+      a Directory object for use by expressions.
+    - "no_listing: Do not load the directory listing."
+    - "shallow_listing: Only load the top level listing, do not recurse into subdirectories."
+    - "deep_listing: Load the directory listing and recursively load all subdirectories as well."
+
+
 - name: LoadContents
   type: record
   abstract: true
   fields:
     - name: loadContents
-      type:
-        - "null"
-        - boolean
+      type: boolean?
       jsonldPredicate: "cwl:loadContents"
       doc: |
         Only valid when `type: File` or is an array of `items: File`.
 
         Read up to the first 64 KiB of text from the file and place it in the
         "contents" field of the file object for use by expressions.
+    - name: loadListing
+      type: LoadListingEnum?
+      jsonldPredicate: "cwl:loadListing"
+      doc: |
+        Only valid when `type: Directory` or is an array of `items: Directory`.
+
+        Specify the desired behavior for loading the `listing` field of
+        a Directory object for use by expressions.
 
+        The order of precedence for loadListing is:
+
+          1. `loadListing` on an individual parameter
+          2. Inherited from `LoadListingRequirement`
+          3. By default: `no_listing`
 
 - name: FieldBase
   type: record
@@ -914,3 +937,20 @@ $graph:
         set to `false` and the expected secondary file does not exist.
         Default value for `required` field is `true` for secondary files on
         input and `false` for secondary files on output.
+
+- name: LoadListingRequirement
+  type: record
+  extends: ProcessRequirement
+  doc: |
+    Specify the desired behavior for loading the `listing` field of
+    a Directory object for use by expressions.
+  fields:
+    class:
+      type: string
+      doc: "Always 'LoadListingRequirement'"
+      jsonldPredicate:
+        "_id": "@type"
+        "_type": "@vocab"
+    loadListing:
+      type: LoadListingEnum?
+      jsonldPredicate: "cwl:loadListing"

Workflow.yml

@@ -76,15 +76,18 @@ $graph:
         * `secondaryFiles` can now be explicitly marked as `required` or not.
         * Add note that undefined details related to execution are left open to
           the platform.
-        * [Added](#TimeLimit) `TimeLimit` feature, allowing to set an upper limit on the 
+        * [Added](#TimeLimit) `TimeLimit` feature, allowing to set an upper limit on the
           execution time of a CommandLineTool or ExpressionTool.
-        * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse 
+        * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse
           behavior for a particular tool or step for implementations that
           support reusing output from past work.
-        * [Added](#NetworkAccess) `NetworkAccess` feature, allowing to indicate whether a 
+        * [Added](#NetworkAccess) `NetworkAccess` feature, allowing to indicate whether a
           process requires outgoing network access.
-        * [Added](#InplaceUpdateRequirement) `InplaceUpdateRequirement` feature, allowing tools to directly 
+        * [Added](#InplaceUpdateRequirement) `InplaceUpdateRequirement` feature, allowing tools to directly
           update files with `writable: true` in `InitialWorkDirRequirement`.
+        * [Added](#LoadListingRequirement) `LoadListingRequirement`
+          and [loadListing](#LoadContents) to control whether and how
+          `Directory` listings should be loaded for use in expressions.
 
       See also the [CWL Command Line Tool Description, v1.1.0-dev1 changelog](CommandLineTool.html#Changelog).
 
@@ -202,7 +205,7 @@ $graph:
   docParent: "#Workflow"
   doc: |
     Describe an output parameter of a workflow.  The parameter must be
-    connected to one or more parameters defined in the workflow that 
+    connected to one or more parameters defined in the workflow that
     will provide the value of the output parameter. It is legal to
     connect a WorkflowInputParameter to a WorkflowOutputParameter.
   fields:

tests/dir5.cwl

@@ -5,7 +5,9 @@ requirements:
   - class: InitialWorkDirRequirement
     listing: $(inputs.indir.listing)
 inputs:
-  indir: Directory
+  indir:
+    type: Directory
+    loadListing: shallow_listing
 outputs:
   outlist:
     type: File
@@ -14,4 +16,4 @@ outputs:
 arguments: ["find", "-L", ".", "!", "-path", "*.txt",
   {shellQuote: false, valueFrom: "|"},
   "sort"]
-stdout: output.txt
+stdout: output.txt

tests/inpdir_update_wf.cwl

@@ -32,12 +32,10 @@ steps:
       inputs:
         d1:
           type: Directory
-#          inputBinding:
-#            loadListing: shallow_listing
+          loadListing: shallow_listing
         d2:
           type: Directory
-#          inputBinding:
-#            loadListing: shallow_listing
+          loadListing: shallow_listing
       outputs:
         d1out: File[]
         d2out: File[]