Guidance on how to plug-in backend services in DSC v5

[thebe14]

Hello,

I am working on adapting the IDS Dataspace Connector to work with EGI DataHub resources (backed by OneData). I (re)started from v5.2.0. What I find so far is that there is difficult to understand where I should modify/plugin our backend specific code. In v4.x there was a conditional loading of our own OfferedRepository implementation, and from that we were directly fetching/manipulating a resource. Also in v4.x there was a class ResourceMetadata used by classes OfferedResource and RequestedResource. In v5.x all these classes are replaced with a set of classes for Artifact, ArtifactImpl, ArtifactDesc, ArtifactFactory (takes an ArtifactDesc and returns an Artifact), then ArtifactView and ArtifactViewAssembler (takes an Artifact and returns an ArtifactView). The same factory patterns are used for most of the entities in this project.

Can you offer some insight on how the authors envisioned using/extending this connector to work with a specific resource backend? Am I expected to modify the classes provided instead of extending them with additional classes for the specific behavior we need? E.g. trying to provide an alternative OfferedResource implementation as in v4.x causes clashes with the compiler not being able to decide which bean to use/inject, as this class is referred to in multiple places, particularly from template parameters like in this method declaration:

public <S extends OfferedResource> List<S> saveAll(Iterable<S> iterable) {…}

Am I really expected to go though all these places and manually specify which bean to use?

Any guidance would be greatly appreciated.

[HeinrichPet]

Hi,

in IDS we using IDS Apps to bring in new functionalities like analytics, transformation, administration, or as in this case connection to different specific systems. For connecting a system like the EGI DataHub, we would purpose to generate a container-based Adapter Application.

Copy from the ThinkTank Repository:

Adapter App

Naming variants: System Adapter App

Type: system-specific usage

The Adapter App is a re-usable application that can be downloaded and fully managed by the IDS
Connector. When defining the route, matching inputs and outputs of Adapter App, IDS Connector, and
backend system (as data source or sink) have to be put together to define a data flow on consumer or
provider side.

1. Start: IDS Connector endpoint or Adapter App (Data Source) endpoint of type OUTPUT
2. (optional) Data App endpoint of type INPUT
3. (optional) Data App endpoint of type OUTPUT
4. End: Adapter App (Data Sink) endpoint of type INPUT or IDS Connector endpoint

The Data App is only for illustration and not required by the Adapter App or Connector.
The Adapter App can be used in various use cases and provides the IDS Connector with the
functionality to connect to custom backends. This is important when the Routing Framework is not
natively able to support the provided endpoints or protocols. It can be executed connector- but not
system-independent, as it only works with pre-defined systems. In general, it is not interested in
the context in which it is used.

With a trusted profile, the Adapter App may be able to enforce usage control. For this, it
possibly has to interact with more than one endpoint of the backend system.

Basic Profile:

• Possibly n input and m output endpoints with an unspecified protocol; the route definition uses
  one specific input endpoint to integrate the app into the data route
• IDS Infomodel description of app endpoints and protocol bindings
• No interaction with the IDS Connector
• Defined interaction with backend system
• As generic endpoints interacting with the backend system is not sufficient for actually
  e.g. storing data, the app needs to be configured.
• The IDS App needs a defined endpoint for output configuration. It needs to be specified where
  exactly to route the data into the backend system.

Advanced Profile: trusted requirements, e.g. usage control or secure protocol

• Possibly n input and m output endpoints with an unspecified protocol; the route definition uses
  one specific input endpoint to integrate the app into the data route
• IDS Infomodel description of app endpoints, protocol bindings, and supported usage control
  patterns
• No interaction with the IDS Connector
• Defined interaction with backend system
• As generic endpoints interacting with the backend system is not sufficient for actually
  e.g. storing data, the app needs to be configured.
• The IDS App needs a defined endpoint for output configuration. It needs to be specified where
  exactly to route the data into the backend system. Does the IDS Connector need a standardized API?
  The IDS App gets the artifact and contract agreement id as input. For fetching the agreement, it
  needs to request the IDS Connector.

With this approach the customer is more flexible to load various adapters to the connector.

[HeinrichPet]

So its good that you asked for it ;-)
The Repo is not public available because it is still a discussion inside the IDS Association and not approved or published yet. The AppStore is not available yet and the integration of IDS Apps is therefore not implemented into the DSC.

But you are right, we will add this in the documentation, as soon as this approach is public available and supported by the connector. For the time being the simplest way for you is to develop a separate application and use the API of the DSC to generate or get data from the IDS.

[thebe14]

Let me see if I understand correctly what you are saying: we should implement an application on our own to talk directly to the connector via its REST API? But why would anyone replicate the effort you are supposedly already doing internally? Just to throw it away in days/weeks/months when you finally come out with the DataApps and ApdatorApps? Then if someone does this (considerable) effort, why would they later care about DataApps and AdaptorApps?

[HeinrichPet]

I don't understand what you are getting at....
You wanted to add backend specific code, and I am now saying it makes more sense to do that as separate code/app. We don't include backend specific code in the DSC. So what internal effort do you mean?

It's rather the other way around, if the development of the DSC continues no one can guarantee that your inserted code will be compatible. With the REST API there is a clear defined interface and you can see the compatibility by the DSC version number and changelog.

[thebe14]

I understand and thank you for the clarification. What I would like to avoid is to code something that will be made much easier by an Adaptor App.

[riedel]

Is there a public timeline for upcoming changes?