Adding IDs to files

[janpmeyer]

The background for this idea is to track data and map it when it leaves the Connector (relates in part to previous Discussion #1381 ).

The idea would be to assign a UID as meta information to a file before it is transferred to the provider. How exactly this meta information can be safely added to the file and it can no longer be manipulated would have to be considered in more detail. First I would suggest as prototype procedure a simple addition of a META data field, like you can add for example your geo location at pictures or the time and date.

Why you could use an ID for files

If you take a look beyond the connector, the data will be used for a specific purpose after a successful transfer through the connector. The data can be read, manipulated, deleted or forwarded.

How exactly a control over the exact use should be executed after the transfer I do not want to put here for discussion. But an example could be a certified software which uses the files. In any case the data flow and the data usage should be documented/logged ("audit logging").

The Connector can be seen as an instance where software outside the Connector can write its logs about the use of these various files as an interface. To be able to logically assign the various logs to the files within the connector, an ID for files would be very helpful, if not necessary.

If at this time such a discussion is inappropriate, I would be happy to take note of that and consider the issue further for myself and address it again at a later time. In general, I would be happy to have a discussion about adding an ID to a file.

[jimmarino]

Adding metadata to transferred data can either be done during provisioning or at transfer time. This can be handled by a provisioner extension or data plane/data flow controller extension, respectively. The core connector should not be involved in this processing.

[juliapampus]

Still, a data space standard would be characterized by unifying this (e.g. format), no matter where exactly this is implemented, right? This is what e.g. IDS aims for. So independent from the implementation or used technology, it would be necessary to specify that a data flow has to cover at least e.g. a unique identitfier of a data asset and a contract agreement id. So every deployed service would have the same knowledge?

[jimmarino]

I'm not sure I follow the use case or what needs to be standardized. The contract id should be a GUID. Other than that, I don't think it is possible to standardize how to associate metadata with transferred data. It will vary by transfer type. For example, an HTTP transfer could encode the associated contract id as part of a POST URL or header. Streaming data could use header properties if the messaging infrastructure supports it. Regardless of the mechanism used, it should be handled at the extension level.

Is the use case aiming at something different?

[janpmeyer]

My consideration is the general use case of Data Spaces and the role of the Connector within that Data Space. At the moment we lose all connection between the transferred data and your policy, as there is no way to reconnect data transferred to a device with their policy in the policy store.

Therefore, I think a way needs to be found to match an identification of data transmitted with a connector with its policy in the policy store for all transmission paths as an standardized way.

[jimmarino]

One of the mechanisms I outlined will solve the issue you outlined. This is not something that can be solved across all data transfer protocols and should not be handled in the core runtime.

If the provider is pushing data, correlation can be handled in one of two ways:

1. The consumer can specify an endpoint address that can be correlated back to the original request (and hence contract id)
2. The provider extension handling the data transfer can send the DataRequest id back as a header or other metadata for the consumer to correlate.

If the consumer pulls data, it already has knowledge of the contract id.

[janpmeyer]

Ok. Thanks for your input :)