Configuring EDC to transfer files from filesystem to azure blob storage

[lenasauermann]

Hi everyone,

I'm working with the EDC in my Master's thesis. My use cases is that the provider has a file in it's filesystem and the consumer wants to save it in an Azure Blob Storage.

I used the code provided in the Samples (transsfer01) and modified the consumer with a look at the mvd to be able to save the file in the blob storage.

Unfortunatly I get the error
"TransferProcessManager: Fatal error initiating data transfer: 01a5f6a1-6c7e-4247-9382-efaadb42e5b6. Error details: Failed to delegate data transfer to Data Plane: Cannot find a transfer Service that can handle File source and Azure Blob Storage destination"
on the provider side and I don't know what I need to do to make it work and don't find any helpful documentation.

Can anyone help me with providing additional documentation or has a solution for my problem?

Regards,
Lena Sauermann

[florianrusch-zf]

Hi @lenasauermann,

As far as I know, the current implementations that handle data transfers in the EDC can only transfer data between the same "storage types". So FS -> FS, Azure Blob -> Azure Blob, S3 -> S3 and so on. But I'm not very familiar with the corresponding source code.

If you want to have a look into it I could point you to this class: https://github.com/eclipse-edc/Connector/blob/main/core/data-plane/data-plane-framework/src/main/java/org/eclipse/edc/connector/dataplane/framework/pipeline/PipelineServiceImpl.java

Hope I could help you at least a little bit!

Regrads,
Florian

[lenasauermann]

Hi Florian,

thanks for the answer! In the Sample transfer-05 (https://github.com/eclipse-edc/Samples/tree/main/transfer/transfer-05-file-transfer-cloud) they are transferring data from Azure to AWS so I thought different storage types would be possible...

My provider connector is exatlcy the one from the sample transfer-01 with the extensions:

• control plane core
• api observability
• configuration filesystem
• iam mock
• auth tokenbased
• management api
• ids
• and the local file transfer extension from the samples (which is probably the wrong one but I don't know which one I should use)

My consumer has the following extensions included:

• control plane core
• api observability
• configuration filesystem
• iam mock
• auth tokenbased
• management api
• ids
• azure blob core
• provision blob

I think I have the wrong extensions or they are wrongly configured, but I'm having trouble identifies and configuring the right extensions as there is little to no documentation about them.

[ndr-brt]

As far as I know, the current implementations that handle data transfers in the EDC can only transfer data between the same "storage types". So FS -> FS, Azure Blob -> Azure Blob, S3 -> S3 and so on. But I'm not very familiar with the corresponding source code.

This is not true, sources and sinks are completely decoupled.

[florianrusch-zf]

@ndr-brt You are right, it is decoupled. But there would have to be implemented either on the source or on the sink side the handling of another counterpart? So there must be an implementation somewhere that has implemented e.g. a switch case like in the following example.

Pseudocode for handling different source types on sink side:

switch(dataflowRequest.destinationDataAddress.type){
	case "HTTP":
	  // handel http source
	case "S3":
	  // handle s3 source
}

Or how is this handled?

[jimmarino]

No switch statement is needed. The design is based on an end-to-end streaming architecture that allows for n-way communication, backpressure, and optimizations such as copy-in-place.

See :

#463

Connector/core/data-plane/data-plane-framework/src/main/java/org/eclipse/edc/connector/dataplane/framework/pipeline/PipelineServiceImpl.java

Line 101 in c2bc86f

public CompletableFuture<StatusResult<Void>> transfer(DataSource source, DataFlowRequest request) {

Connector/core/data-plane/data-plane-util/src/main/java/org/eclipse/edc/connector/dataplane/util/sink/OutputStreamDataSink.java

Line 64 in c2bc86f

private Result<Void> transferData(DataSource.Part part) {

[lenasauermann]

Thanks for the answer! What do I need to do then to make my file transfer from filesystem to azure data storage work?

[reisman234]

Hi
I did a very similar test a while ago with the edc between a file system and an on-premise s3 storage.

I'm not familiar with the current implementation (milestone 8) of the samples, but I think they still use the "provider-push" method. Therefore, only the provider is doing the transfer from A -> B, which in your case means the provider needs the required azure extensions.

[lenasauermann]

Hi,

i added the extensions azure blob core and provision blob to the provider connector but I still get the same error...

[reisman234]

I think there are still extensions missing, maybe the data plane part of azure? The later one could be superfluous

https://github.com/eclipse-edc/Connector/tree/main/extensions/data-plane/data-plane-azure-storage
https://github.com/eclipse-edc/Connector/tree/main/extensions/data-plane/data-plane-azure-data-factory

[ndr-brt]

@reisman234 you need to register the data plane to the control plane, pointing the supported source and sink types.
details: https://app.swaggerhub.com/apis/eclipse-edc-bot/management-api/0.0.1-SNAPSHOT#/Dataplane%20Selector/addEntry

[lenasauermann]

I tried registering the data plane with the following body of the POST Call /instances:

{
"allowedDestTypes": [
"Azure Blob Storage"
],
"allowedSourceTypes": [
"File"
],
"id": "file2azure",
"lastActive": 0,
"properties": {
"additionalProp1": {},
"additionalProp2": {},
"additionalProp3": {}
},
"turnCount": 0,
"url": ""
}

But I still get the same error

[ndr-brt]

the type for azure blob storage is spelled this way: AzureStorage
(ref: https://github.com/eclipse-edc/Connector/blob/main/extensions/common/azure/azure-blob-core/src/main/java/org/eclipse/edc/azure/blob/AzureBlobStoreSchema.java)

For the file system one it depends from the DataSourceFactory you registered on data-plane, take a look at the canHandle method on it.

[AnnamariaGalante]

Hi,
any update?