SHACL Shapes

This repo contains the SHACL shapes for RESCS (Research Commons).

SHACL Shapes Library Source Files

The JSON-LD files contained in the shapes directory are the SHACL shape source files. They are optimised for use with Nexus Forge.

For instructions on how to build the documentation, see README in the docs directory.

Configuration For Use With Nexus Forge

For a complete overview of Forge configuration options, see the official Forge docs and a sample configuration.

Authentication

For authentication a token has to be obtained from https://nexus.switch.ch/. The token and other configuration options have to be set in a file called .env in the project root:

TOKEN="mytoken"
ORG="SW"
PROJECT="my_test"
NEXUS="https://nexus.switch.ch/v1"
VERIFY_SSL=1 # set to 0 if SSL certificate should not be verified, i.e. for local dev

ORG defines the organisation setting for Nexus. PORJECT sets the project. NEXUS defines the server address (API).

Using SHACL Shapes From Local File System

To use these shapes with Nexus Forge, set up a YAML configuration file:

Model:
  name: RdfModel
  origin: directory
  source: "shapes" # --> path to the shapes directory (paths are relative to execution context of config file)
  context:
    iri: "ontology/jsonld-context.json" # --> path to the JSON-LD context object (paths are relative to execution context of config file)

Store:
  name: BlueBrainNexus
  searchendpoints:
    sparql:
      endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
  versioned_id_template: "{x.id}?rev={x._store_metadata._rev}"
  file_resource_mapping: ./configurations/nexus-store/file-to-resource-mapping.hjson

Formatters:
  identifier: https://kg.example.ch/{}/{}

Alternatively use configuration.yml in your setup.

Using Shapes Registered in Nexus

Once the SHACL shapes are registered in Nexus, you can use them for validation instead of using those from your local files system.

Model:
  name: RdfModel
  origin: store # --> change this from "directory" to "store".
  source: BlueBrainNexus # --> change this from the directory on your local file system, e.g. "store" to "BlueBrainNexus".
  context:
    iri: "./jsonld-context.json" # --> Note that the JSON-LD context object is stilled required on your local file system.

Registering SHACL Shapes in Nexus

To register the SHACL shapes located on your local file system in Nexus as schemas, run scripts/register_schemas.py. The script will create the SHACL shapes in Nexus in a predefined order. If you add new shapes, note that you have to add them here:

# order in which schemas are created (dependency)
order: List[str] = ['thing', 'person', 'organization', 'place', 'creativework', 'intangible', 'structuredvalue', 'contactpoint',
         'monetaryamount', 'article', 'dataset', 'mediaobject', 'scholarlyarticle', 'datadownload', 'grant',
         'monetarygrant', 'project', 'researchproject']

Note that the schemas that are referred to from other schemas have to be created first.

If you attempt to register a schema with an existing name, it will be rejected.

Registering and Updating Composite Views in Nexus

To register a composite view in Nexus or update an existing one, run scripts/register_or_update_composite_view.py <composite_view_name> (no file extension required/allowed, i.e "dataset""). Predefined composite views are kept in the folder ./compositeviews, e.g. "dataset".`

Demo

Requirements

It is recommended to set up a virtual environment in the project root (by convention, the virtual environment directory env is contained in .gitignore and is not tracked). Install the dependencies listed in requirements.txt: pip3 install -r requirements.txt.

Please note that you have to install nexus-forge from master: pip3 install git+https://github.com/BlueBrain/nexus-forge. A new release (> 0.6.3) should make this unnecessary.

Run Demo

For a demonstration of the SHACL shapes, you can use test.py. Make sure to create a file .env first containing configuration settings (see above).

Use with Standard Tools

Run scripts/generate_shapes_graph.py to build a graph of all SHACL shapes contained in the library. Two files are generated:

• ontology/shapes_graph.json: the graph of all combined SHACL shape definitions.
• ontology/shapes_ontology_graph.json: the graph of all SHACL shape definitions and the ontology (classes and properties).

The resulting collections contain all shapes in one JSON-LD graph and only contain standard SHACL statements (no Nexus Forge specifics).

Then use them for validation, e.g., pyshacl -sf json-ld -s ontology/shapes_graph.json -df json-ld test/thing/thing.json (use the debug mode -d for more detailed error messages).

By design, SHACL shapes for Nexus are combined using sh:and, see official docs. If the validation fails, it reports that the AndConstraintComponent has been violated which is not very helpful for debugging. Run scripts/transform_shapes_graph.py to generate a transformed graph which does not use sh:and. Instead, make use of inheritance by including the ontology file when validating, e.g., pyshacl -sf json-ld -s ontology/shapes_graph_transformed.json -ef json-ld -e ontology/ontology.json -df json-ld test/person/person.json. The error message directly points out what went wrong.

The validation will only produce correct results with the inclusion of ontology/ontology.json.

Tests

Run scripts/test_all.sh directly from within the directory scripts to check if the test data files contained in the directory test conform to the shapes.

Note that relative paths won't work when you do not run this script directly from within scripts.

Architecture

Source Files

The JSON-LD files contained in the shapes directory are the source files of the SHACL shapes library. Running scripts/generate_shapes_graph.py will generate a file called ontology/shapes_ontology_graph.json containing all SHACL shapes.

Please note that you have to run generate_shapes_graph.py each time you changed something in the JSON-LD shape files.

Structure of a SHACL Shape File

By convention, each source file goes in a separate folder in the shapes directory and has the name schema.json, e.g., shapes/monetarygrant/schema.json, shapes/monetaryamount/schema.json.
A SHACL source file is a JSON-LD document with the following structure:

{
  "@context": [
    "https://incf.github.io/neuroshapes/contexts/schema.json", --> reference to an external context object simplifying properties such as targetClass etc.
    {
      "this": "http://rescs.org/dash/monetarygrant/" --> base reference path for each shape defined in this schema file
    }
  ],
  "@type": "nxv:Schema", --> Nexus vocabulary schema type (<https://bluebrain.github.io/nexus/vocabulary/>)
  "@id": "http://rescs.org/dash/monetarygrant", --> id of this schema file (used in import paths of other schema files)
  "imports": [
     "http://rescs.org/dash/grant",
     "http://rescs.org/dash/monetaryamount" --> import of other schema files by their id
  ],
  "shapes": [
    ... --> SHACL shape definitions (one or more shapes can be defined in one schema file)
  ]
}

Each file contains one node shape. Each node shape is identified by its unique id. A shape definition looks like this:

{
    ... --> main file structure, see above
    "shapes": [
    {
      "@id": "this:MonetaryGrantShape", --> unique name of this shape
      "@type": "sh:NodeShape", --> SHACL node shape
      "label": "A monetary grant",
      "targetClass":"schema:MonetaryGrant", --> the shape targets schema:MonetaryGrant
      "and": [
          {
          "node": "http://rescs.org/dash/grant/GrantShape" --> "inherits" from schema:Grant
        },
        {
          "property": [
            {
              "path": "schema:amount",
              "name": "amount",
              "description": "The amount of money.",
              "nodeKind": "sh:BlankNode",
              "class": "schema:MonetaryAmount",
              "node": "http://rescs.org/dash/monetaryamount/MonetaryAmountShape", --> reference to another shape (has to be imported)
              "minCount": 1,
              "maxCount": 1
            },
            {
              "path": "schema:funder",
              "name": "funder",
              "description": "A person or organization that supports a thing through a pledge, promise, or financial contribution. e.g. a sponsor of a Medical Study or a corporate sponsor of an event.",
              "or": [
                {
                  "class": "schema:Person",
                  "nodeKind": "sh:IRI"
                },
                {
                  "class": "schema:Organization",
                  "nodeKind": "sh:IRI"
                }
              ],
              "minCount": 1
            }
          ]
        }
      ]
    }
  ]
}

For more details about SHACL shape definitions, see the official docs.

JSON-LD Context Object

The JSON-LD context object located in ontology/jsonld-context.json serves as a kind of registry file for Forge, translating shorthands like MonetaryGrant to a full IRI. Each class and property for which shapes and constraints are defined has to be listed here.

Example:

  { 
    ...
    "MonetaryGrant": {
      "@id": "http://schema.org/MonetaryGrant"
    },
    "name": {
      "@id": "http://schema.org/name"
    },
    "amount": {
      "@id": "http://schema.org/amount"
    }
  ...

forge.types() and forge.template() will only list classes defined in the JSON-LD context object.

Adding a New Schema File

1. Add a new directory to the shapes directory, e.g., person.
2. Within the new subdirectory, add a file schema.json.
3. Add the schema file structure as described above,
4. Make sure to use unique ids. (Re-use the subfolder's name as the last part of the id, e.g., http://rescs.org/dash/person and adapt this in the context object accordingly.)
5. Add NodeShapes with property constraints as described above.
6. Add class and property names (sh:targetClass, sh:name) to the JSON-LD context object, e.g., Person, givenName etc., see above.
7. Add the new to shape in scripts/register_schemas.py, see above.
8. Add the class and its properties to ontology/ontology.json (needed to generate the HTML docs).

License and Additional Terms

License

This software is licensed under the terms of the GNU AFFERO GENERAL PUBLIC LICENSE Version 3 or any later version, see license document. Software dependencies are explicitly mentioned in the dependencies document.

The contents of the shapes and ontology folders are licensed under the terms of the Creative Commons Attribution-ShareAlike License (version 3.0) since they are derived from schema.org.

COPYRIGHT © 2022 SWITCH