Minimum Viable Dataspace with CKAN importer

The Minimum Viable Dataspace (MVD) is a sample implementation of a dataspace that leverages the Eclipse Dataspace Components (EDC).

The default implementation of the MVD and its documentation can be found here: Minimum Viable Dataspace (MVD).

This project will focus on explaining the CKAN importer extension in the KupferDigital context and how it can be executed, used and customized to reach its full potential.

Documentation

Developers can find different documentation sources based on their current needs. Here is a collection of important links that can help to get you started:

• Eclipse Dataspace Components repository: EDC
• Minimum Viable Dataspace repository: MVD
• Developer documentation: docs/developer
• Decision records: Decision records
• Connector API (management-api) specification: Connector API

Local Development Setup

Prerequisites

1. Install Docker.

Initial Setup

To run this setup of the MVD with the CKAN importer extension and the MVD Dashboard (UI) follow these steps:

1. Clone this repository:

git clone https://gitlab.cc-asp.fraunhofer.de/becker4/kupferdigital-ckan-importer.git

2. Clone the DataDashboard repository:

git clone https://github.com/eclipse-dataspaceconnector/DataDashboard.git

3. Build MVD:

./gradlew build -x test

4. Configure FS Vault (in MVD root) for connector:

./gradlew -DuseFsVault="true" :launchers:connector:shadowJar

5. Configure FS Vault (in MVD root) for registration service:

./gradlew -DuseFsVault="true" :launchers:registrationservice:shadowJar

Run MVD + CKAN importer

1. Set path to the DataDashboard project (remember where you cloned to repository to):

export MVD_UI_PATH="/path/to/mvd-datadashboard"

2. Run MVD (including dashboards and CKAN importer):

docker-compose --profile ui -f system-tests/docker-compose.yml up --build

Observations when running

When the MVD + CKAN importer are setup properly some things can be observed which are explained here.

The DataDashboards of company1 and company2 can be looked at by going to the corresponding websites:

1. Company1: localhost:7080
2. Company2: localhost:7081

When running multiple tests, the build-tag can be left out in order to have shorter loading times:

docker-compose --profile ui -f system-tests/docker-compose.yml up

The logs of single containers can be looked at for deeper analysis:

docker logs ckan-importer1

Possible container names can be seen in the system-tests/docker-compose.yml file.

Config

The CKAN importer extension has its own configuration file (ckan-importer/config.py). The parameters are all explained and need to be setup according to the setup the dataspace is running on.

KupferDigital specific next steps

In the KupferDigital project the demonstrator with the CKAN importer extension needs to be implemented. It is important to get the access permissions working correctly instead of only using the "no-restriction-policy". Therefore a few more steps are necessary to implement this system. The following steps are suggestions on how the integration into the existing structures could be possible.

1. Firstly it needs to be clear which restrictions should be possible.
2. The restrictions defined in step 1 need to be formulated and implemented as policies. The best place to start writing own policies is to read the documentation and take a look at existing examples: Policy Documentation
3. Policies need to be added to the dataspace. The file ckan-importer/ckan_catalog_requester.py contains the method create_no_restriction_policy() which is invoked when executing the main()-method. Similar methods or other means of adding policies via the Connector API should be implemented.
4. Existing datasets need to be annotated with the policy that should be used.
5. Going forward, newly created datasets should require the user to also add a policy. A dropdown menu when adding a dataset would be a good option. To implement this, the endpoint /v2/policydefinitions/request could be requested in order to get all available policies. The policies need to be defined in the dataspace first for this, so step 3 should be fulfilled first.
6. At the moment no policies are defined and the importer ckan-importer/ckan_catalog_requester.py only uses the basic "no-restriction-policy". When the datasets contain a policy tag, the program needs to be updated in order to attach the correct policies when creating or updating datasets.