Merge pull request #62 from common-workflow-language/update-salad

Update copy of schema-salad spec

Author: Peter Amstutz

salad/schema_salad/metaschema/ident_res.yml

@@ -5,16 +5,33 @@
   specific objects.  Processing must resolve relative identifiers to absolute
   identifiers using the following rules:
 
-    * If an identifier URI is prefixed with `#` it is a URI relative
-      fragment identifier.  It is resolved relative to the base URI by setting
-      or replacing the fragment portion of the base URI.
-
-    * If an identifier URI does not contain a scheme and is not prefixed `#` it
-      is a parent relative fragment identifier.  It is resolved relative to the
-      base URI by the following rule: if the base URI does not contain a
-      document fragment, set the fragment portion of the base URI.  If the base
-      URI does contain a document fragment, append a slash `/` followed by the
-      identifier field to the fragment portion of the base URI.
+    * If an identifier URI begins with `#` it is a current document
+      fragment identifier.  It is resolved relative to the base URI by
+      setting or replacing the fragment portion of the base URI.
+
+    * If an identifier URI contains `#` in some other position is a
+      relative URI with fragment identifier.  It is resolved relative
+      to the base URI by stripping the last path segment from the base
+      URI and adding the identifier followed by the fragment.
+
+    * If an identifier URI does not contain a scheme and does not
+      contain `#` it is a parent relative fragment identifier.
+
+    * If an identifier URI is a parent relative fragment identifier
+      and the base URI does not contain a document fragment, set the
+      document fragment on the base URI.
+
+    * If an identifier URI is a parent relative fragment identifier
+      and the object containing this identifier is assigned to a
+      parent object field defined with `subscope` in
+      `jsonldPredicate`, append a slash `/` to the base URI fragment
+      followed by the value of the parent field `subscope`.  Then
+      append the identifier as described in the next rule.
+
+    * If an identifier URI is a parent relative fragment identifier
+      and the base URI contains a document fragment, append a slash
+      `/` to the fragment followed by the identifier field to the
+      fragment portion of the base URI.
 
     * If an identifier URI begins with a namespace prefix declared in
       `$namespaces` followed by a colon `:`, the prefix and colon must be

salad/schema_salad/metaschema/ident_res_proc.yml

@@ -14,7 +14,12 @@
       },
       {
         "id": "http://example.com/acid#six",
+      },
+      {
+        "subscopeField": {
+          "id": "http://example.com/base#one/thisIsASubscope/seven"
+        }
       }
-    ]
+    ],
   }
 }

salad/schema_salad/metaschema/ident_res_schema.yml

@@ -9,6 +9,15 @@
       "name": "id",
       "type": "string",
       "jsonldPredicate": "@id"
+    }]}, {
+    "name": "SubscopeType",
+    "type": "record",
+    "fields": [{
+      "name": "subscopeField",
+      "type": "ExampleType",
+      "jsonldPredicate": {
+         "subscope": "thisIsASubscope"
+      }
     }]
   }]
 }

salad/schema_salad/metaschema/ident_res_src.yml

@@ -14,7 +14,12 @@
           },
           {
             "id": "acid:six",
+          },
+          {
+            "subscopeField": {
+              "id": "seven"
+            }
           }
-        ]
+        ],
       }
     }

salad/schema_salad/metaschema/import_include.md

@@ -110,67 +110,3 @@ This becomes:
   }
 }
 ```
-
-
-## Mixin
-
-During preprocessing traversal, an implementation must resolve `$mixin`
-directives.  An `$mixin` directive is an object consisting of the field
-`$mixin` specifying resource by URI string.  If there are additional fields in
-the `$mixin` object, these fields override fields in the object which is loaded
-from the `$mixin` URI.
-
-The URI string must be resolved to an absolute URI using the link resolution
-rules described previously.  Implementations must support loading from `file`,
-`http` and `https` resources.  The URI referenced by `$mixin` must be loaded
-and recursively preprocessed as a Salad document.  The external imported
-document must inherit the context of the importing document, however the file
-URI for processing the imported document must be the URI used to retrieve the
-imported document.  The `$mixin` URI must not include a document fragment.
-
-Once loaded and processed, the `$mixin` node is replaced in the document
-structure by the object or array yielded from the import operation.
-
-URIs may reference document fragments which refer to specific an object in
-the target document.  This indicates that the `$mixin` node must be
-replaced by only the object with the appropriate fragment identifier.
-
-It is a fatal error if an import directive refers to an external resource
-or resource fragment which does not exist or is not accessible.
-
-### Mixin example
-
-mixin.yml:
-```
-{
-  "hello": "world",
-  "carrot": "orange"
-}
-
-```
-
-parent.yml:
-```
-{
-  "form": {
-    "bar": {
-      "$mixin": "mixin.yml"
-      "carrot": "cake"
-      }
-  }
-}
-
-```
-
-This becomes:
-
-```
-{
-  "form": {
-    "bar": {
-      "hello": "world",
-      "carrot": "cake"
-    }
-  }
-}
-```

salad/schema_salad/metaschema/metaschema.yml

@@ -20,6 +20,7 @@ $graph:
     - $include: import_include.md
     - $import: map_res.yml
     - $import: typedsl_res.yml
+    - $import: sfdsl_res.yml
 
 - name: "Link_Validation"
   type: documentation
@@ -35,10 +36,40 @@ $graph:
     `noLinkCheck` in the `jsonldPredicate` section of the field schema.
 
 
-- name: "Schema_validation"
+- name: "Schema_Validation"
   type: documentation
-  doc: ""
-
+  doc: |
+    # Validating a document against a schema
+
+    To validate a document against the schema, first [apply
+    preprocessing](#Document_preprocessing), then, use the following
+    algorithm.
+
+    1. The document root must be an object or a list.  If the document root is an
+       object containing the field `$graph` (which must be a list of
+       objects), then validation applies to each object in the list.
+    2. For each object, attempt to validate as one of the record types
+       flagged with `documentRoot: true`.
+    3. To validate a record, go through `fields` and recursively
+       validate each field of the object.
+    4. For fields with a list of types (type union), go through each
+       type in the list and recursively validate the type.  For the
+       field to be valid, at least one type in the union must be valid.
+    5. Missing fields are considered `null`.  To validate, the allowed types
+       for the field must include `null`
+    6. Primitive types are null, boolean, int, long, float, double,
+       string.  To validate, the value in the document must have one
+       of these type.  For numerics, the value appearing in the
+       document must fit into the specified type.
+    7. To validate an array, the value in the document must be a list,
+       and each item in the list must recursively validate as a type
+       in `items`.
+    8. To validate an enum, the value in the document be a string, and
+       the value must be equal to the short name of one of the values
+       listed in `symbols`.
+    9. As a special case, a field with the `Expression` type validates string values
+      which contain a CWL parameter reference or expression in the form
+      `$(...)` or `${...}`
 
 # - name: "JSON_LD_Context"
 #   type: documentation
@@ -131,6 +162,10 @@ $graph:
       type: boolean?
       doc: |
         Field must be expanded based on the the Schema Salad type DSL.
+    - name: secondaryFilesDSL
+      type: boolean?
+      doc: |
+        Field must be expanded based on the the Schema Salad secondary file DSL.
     - name: subscope
       type: string?
       doc: |
@@ -168,10 +203,18 @@ $graph:
       doc: "The identifier for this type"
     - name: inVocab
       type: boolean?
+      default: true
       doc: |
-        By default or if "true", include the short name of this type in the
-        vocabulary (the keys of the JSON-LD context).  If false, do not include
-        the short name in the vocabulary.
+        If "true" (the default), include the short name of this type
+        in the vocabulary.  The vocabulary are all the symbols (field
+        names and other identifiers, such as classes and enum values)
+        which can be used in the document without a namespace prefix.
+        These are the keys of the JSON-LD context.  If false, do not
+        include the short name in the vocabulary.
+
+        This is useful for specifying schema extensions that will be
+        included in validation without introducing ambiguity by
+        introducing non-standard terms into the vocabulary.
 
 
 - name: DocType

salad/schema_salad/metaschema/salad.md

@@ -2,7 +2,7 @@
 
 Author:
 
-* Peter Amstutz <peter.amstutz@curoverse.com>, Curoverse
+* Peter Amstutz <pamstutz@veritasgenetics.com>, Veritas Genetics
 
 Contributors:
 
@@ -70,25 +70,29 @@ and RDF schema, and production of RDF triples by applying the JSON-LD
 context.  The schema language also provides for robust support of inline
 documentation.
 
-## Introduction to v1.0
+## Introduction to v1.1
 
-This is the second version of of the Schema Salad specification.  It is
-developed concurrently with v1.0 of the Common Workflow Language for use in
+This is the third version of of the Schema Salad specification.  It is
+developed concurrently with v1.1 of the Common Workflow Language for use in
 specifying the Common Workflow Language, however Schema Salad is intended to be
-useful to a broader audience.  Compared to the draft-1 schema salad
+useful to a broader audience.  Compared to the v1.0 schema salad
 specification, the following changes have been made:
 
-* Use of [mapSubject and mapPredicate](#Identifier_maps) to transform maps to lists of records.
-* Resolution of the [domain Specific Language for types](#Domain_Specific_Language_for_types)
-* Consolidation of the formal [schema into section 5](#Schema).
+* Support for `default` values on record fields to specify default values
+* Add subscoped fields (fields which introduce a new inner scope for identifiers)
+* Add the *inVocab* flag (default true) to indicate if a type is added to the vocabulary of well known terms or must be prefixed
+* Add *secondaryFilesDSL* micro DSL (domain specific language) to convert text strings to a secondaryFiles record type used in CWL
+* The `$mixin` feature has been removed from the specification, as it
+  is poorly documented, not included in conformance testing,
+  and not widely supported.
 
 ## References to Other Specifications
 
 **Javascript Object Notation (JSON)**: http://json.org
 
 **JSON Linked Data (JSON-LD)**: http://json-ld.org
 
-**YAML**: http://yaml.org
+**YAML**: https://yaml.org/spec/1.2/spec.html
 
 **Avro**: https://avro.apache.org/docs/current/spec.html
 
@@ -157,11 +161,19 @@ by a document schema, where each term maps to absolute URI.
 
 ## Syntax
 
-Conforming Salad documents are serialized and loaded using YAML syntax and
-UTF-8 text encoding.  Salad documents are written using the JSON-compatible
-subset of YAML.  Features of YAML such as headers and type tags that are
-not found in the standard JSON data model must not be used in conforming
-Salad documents.  It is a fatal error if the document is not valid YAML.
+Conforming Salad v1.1 documents are serialized and loaded using a
+subset of YAML 1.2 syntax and UTF-8 text encoding.  Salad documents
+are written using the [JSON-compatible subset of YAML described in
+section 10.2](https://yaml.org/spec/1.2/spec.html#id2803231).  The
+following features of YAML must not be used in conforming Salad
+documents:
+
+* Use of explicit node tags with leading `!` or `!!`
+* Use of anchors with leading `&` and aliases with leading `*`
+* %YAML directives
+* %TAG directives
+
+It is a fatal error if the document is not valid YAML.
 
 A Salad document must consist only of either a single root object or an
 array of objects.
@@ -223,9 +235,9 @@ document schema.  A schema may consist of:
   * Any number of documentation objects which allow in-line documentation of the schema.
 
 The schema for defining a salad schema (the metaschema) is described in
-detail in "Schema validation".
+detail in the [Schema](#Schema) section.
 
-### Record field annotations
+## Record field annotations
 
 In a document schema, record field definitions may include the field
 `jsonldPredicate`, which may be either a string or object.  Implementations
@@ -237,11 +249,11 @@ rules:
 
   * If the value of `jsonldPredicate` is an object, and contains that
   object contains the field `_type` with the value `@id`, the field is a
-  link field.
+  link field subject to [link validation](#Link_validation).
 
-  * If the value of `jsonldPredicate` is an object, and contains that
-  object contains the field `_type` with the value `@vocab`, the field is a
-  vocabulary field, which is a subtype of link field.
+  * If the value of `jsonldPredicate` is an object which contains the
+  field `_type` with the value `@vocab`, the field value is subject to
+  [vocabulary resolution](#Vocabulary_resolution).
 
 ## Document traversal
 
@@ -250,6 +262,39 @@ validation, the document must be traversed starting from the fields or
 array items of the root object or array and recursively visiting each child
 item which contains an object or arrays.
 
+## Short names
+
+The "short name" of an fully qualified identifier is the portion of
+the identifier following the final slash `/` of either the fragment
+identifier following `#` or the path portion, if there is no fragment.
+Some examples:
+
+* the short name of `http://example.com/foo` is `foo`
+* the short name of `http://example.com/#bar` is `bar`
+* the short name of `http://example.com/foo/bar` is `bar`
+* the short name of `http://example.com/foo#bar` is `bar`
+* the short name of `http://example.com/#foo/bar` is `bar`
+* the short name of `http://example.com/foo#bar/baz` is `baz`
+
+## Inheritance and specialization
+
+A record definition may inherit from one or more record definitions
+with the `extends` field.  This copies the fields defined in the
+parent record(s) as the base for the new record.  A record definition
+may `specialize` type declarations of the fields inherited from the
+base record.  For each field inherited from the base record, any
+instance of the type in `specializeFrom` is replaced with the type in
+`specializeTo`.  The type in `specializeTo` should extend from the
+type in `specializeFrom`.
+
+A record definition may be `abstract`.  This means the record
+definition is not used for validation on its own, but may be extended
+by other definitions.  If an abstract type appears in a field
+definition, it is logically replaced with a union of all concrete
+subtypes of the abstract type.  In other words, the field value does
+not validate as the abstract type, but must validate as some concrete
+type that inherits from the abstract type.
+
 # Document preprocessing
 
 After processing the explicit context (if any), document preprocessing