Issue With File Transfer Between Different AzureStorage Accounts

[zalborzi]

Hello,

I am facing an issue during file transfer between different Azure Storage accounts (tested on both Azurite and Azure, resulting in the same error). The process fails when attempting to transfer files from an src-container in azurite1 to azurite2.

When initiating a file transfer(text-dcoument.txt) by clicking on the Transfer button, the following error is triggered in the producer container, saying that "Error creating blob" in consumer account:

DEBUG 2024-03-18T15:28:22.822214193 TransferProcess e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state INITIAL
DEBUG 2024-03-18T15:28:23.636382111 [TransferProcessManagerImpl] TransferProcess e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state PROVISIONING
DEBUG 2024-03-18T15:28:23.63654538 TransferProcess: ID e3f2fdd1-a40c-48ea-bf6e-66262c91ec15. Provisioning
DEBUG 2024-03-18T15:28:23.636689003 [TransferProcessManagerImpl] TransferProcess e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state PROVISIONED
DEBUG 2024-03-18T15:28:23.63682964 [TransferProcessManagerImpl] TransferProcess e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state STARTING
DEBUG 2024-03-18T15:28:23.636909584 TransferProcess: ID e3f2fdd1-a40c-48ea-bf6e-66262c91ec15. Initiate data flow
DEBUG 2024-03-18T15:28:23.637261999 [DataPlaneManagerImpl] DataFlow e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state RECEIVED
DEBUG 2024-03-18T15:28:23.637612146 TransferProcess: ID e3f2fdd1-a40c-48ea-bf6e-66262c91ec15. Send TransferStartMessage to http://192.168.70.102:8282/api/dsp
DEBUG 2024-03-18T15:28:23.678324942 TransferProcess: ID e3f2fdd1-a40c-48ea-bf6e-66262c91ec15. Send TransferStartMessage to http://192.168.70.102:8282/api/dsp
DEBUG 2024-03-18T15:28:23.678444165 [TransferProcessManagerImpl] TransferProcess e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state STARTED
DEBUG 2024-03-18T15:28:24.591003504 DataFlow: ID e3f2fdd1-a40c-48ea-bf6e-66262c91ec15. start data flow
DEBUG 2024-03-18T15:28:24.59179856 Transferring from AzureStorage to AzureStorage.
SEVERE 2024-03-18T15:28:24.627880039 Error creating blob for text-document.txt on account templateassets
java.io.IOException: com.azure.storage.blob.models.BlobStorageException: Status code 404, "<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
  <Code>ContainerNotFound</Code>
  <Message>The specified container does not exist.
RequestId:49be7ef7-08f6-4553-9889-044a7919c1fa
Time:2024-03-18T15:28:24.624Z</Message>
</Error>"
	at com.azure.storage.blob.specialized.BlobOutputStream$BlockBlobOutputStream.lambda$new$0(BlobOutputStream.java:241)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2065)
	at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)
	at reactor.core.publisher.FluxSwitchOnFirst$SwitchOnFirstControlSubscriber.onError(FluxSwitchOnFirst.java:955)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:142)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:142)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2196)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2070)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
	at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2666)
	at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:85)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)
	at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:220)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)
	at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onComplete(FluxHandleFuseable.java:236)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1840)
	at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:129)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
	at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:424)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:478)
	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:712)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at com.azure.core.http.netty.implementation.AzureSdkHandler.channelRead(AzureSdkHandler.java:222)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.azure.storage.blob.models.BlobStorageException: Status code 404, "<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
  <Code>ContainerNotFound</Code>
  <Message>The specified container does not exist.
RequestId:49be7ef7-08f6-4553-9889-044a7919c1fa
Time:2024-03-18T15:28:24.624Z</Message>
</Error>"
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
	at com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:56)
	at com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:356)
	at com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:128)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	... 70 more
DEBUG 2024-03-18T15:28:24.638390993 [DataPlaneManagerImpl] DataFlow e3f2fdd1-a40c-48ea-bf6e-66262c91ec15 is now in state FAILED

Observations:

• The container blob with an uuid is created in the consumer container blobs but there is no file in it.
• The error arises specifically when each connector uses its distinct AzureStorage.
• Using the same Azurite for both connectors does not result in this issue.
• The error points towards a missing resource, although it's confirmed to exist in the online blob storage.

Looking forward to any insights or solutions for this scenario.

[brunopacheco1]

This is not a bug, but an architectural limitation on how the component data-plane-azure-storage:0.3.1 was developed.

Following EDC framework, AzureStorageDataSink and AzureStorageDataSource are initialized by DataPlaneAzureStorageExtension. AzureStorageDataSink and AzureStorageDataSource are responsible respectivelly by the consumer and provider Azure Storage API calls.

The problem here is: both beans are initialized with the same blobStoreApi - check DataPlaneAzureStorageExtension.java:64 and DataPlaneAzureStorageExtension.java:67, whereas they should get the source's or sink's respective EDC_BLOBSTORE_ENDPOINT_TEMPLATE at runtime, when the dataflow is initialized.