Large payload azure blob externalization support

[YunchuWang]

This pull request introduces support for externalizing large payloads to Azure Blob Storage in Durable Task applications. It adds a new extension package, configuration options, and a sample console app demonstrating the feature. The main changes include new abstractions for payload storage, Azure Blob Storage integration, and updates to the solution and dependencies.

Externalized Payloads Support:

• Added the IPayloadStore interface in Microsoft.DurableTask.Converters to abstract storing and retrieving large payloads out-of-band.
• Introduced LargePayloadStorageOptions for configuring payload externalization, including threshold, connection string, container name, and compression.
• Created the AzureBlobPayloads extension package (AzureBlobPayloads.csproj) to provide Azure Blob Storage-based payload storage, with dependency injection support and gRPC integration. [1] [2]
• Updated solution and dependency files to include the new extension and required packages (Azure.Storage.Blobs, System.ComponentModel.Annotations). [1] [2] [3] [4] [5]

Sample Application and Documentation:

• Added LargePayloadConsoleApp sample, including project files, code demonstrating orchestration and entity scenarios with large payloads, and a README for setup and usage. [1] [2] [3] [4]

[halspang]

Nothing opposing the core functionality, but my main concern is with the fact that this has to be setup on the client and the worker. I'm worried that users could provide different configurations and break their payload flow.

Do you think it'd be possible to instead make the client/worker extensions depend directly on inheriting an interface/abstract class that we could instantiate instead? By having the user instantiate the class itself and then pass that around, we make chances for misconfiguration go down.

[halspang]

I'm hesitant to add a new proto just for this feature. Is it possible that the middleware for this should exist elsewhere?

[AnatoliB]

I share this concern, but I don't have good understanding of what alternatives are available. Let's list our options. To start:

• Can we add support for this feature to the exising Grpc project?
• Where can we move this middleware (so that it wouldn't require this gRPC update, if this is what Hal means)?

I'm not strictly against doing it the way it's currently implemented, but let's understand and compare our options.

[YunchuWang]

• Can we add support for this feature to the exising Grpc project?
  Unfortunately not, dotnet cant generate same proto compiled types under multiple namespaces in same project assembly. we need two projects to generate proto types under two namespaces respectively.
• Where can we move this middleware (so that it wouldn't require this gRPC update, if this is what Hal means)?
  We can move the azuremanaged interceptor to azuremanaged sdk in DTMB repo . It would create coupling not make sense much to put the blob which is opt in extension logic like intereceptor into existing azuremanaged packages directly. if we move the middleware to there, we would still need a new extension package.

the other alternative is creating multiple packages:
Microsoft.DurableTask.Extensions.AzureBlobPayloads.Abstractions (common large payload apis including payloadstore, baseinterceptor,etc shared by other packages)
Microsoft.DurableTask.Extensions.AzureBlobPayloads.Sidecar (sidecar interceptorfor portable dotnet sdk connecting to sidecar grpc )
Microsoft.DurableTask.Extensions.AzureBlobPayloads.AzureManaged (azuremanaged intereceptor for df cases)

@AnatoliB

[YunchuWang]

Nothing opposing the core functionality, but my main concern is with the fact that this has to be setup on the client and the worker. I'm worried that users could provide different configurations and break their payload flow.

Do you think it'd be possible to instead make the client/worker extensions depend directly on inheriting an interface/abstract class that we could instantiate instead? By having the user instantiate the class itself and then pass that around, we make chances for misconfiguration go down.

Hi

Nothing opposing the core functionality, but my main concern is with the fact that this has to be setup on the client and the worker. I'm worried that users could provide different configurations and break their payload flow.

Do you think it'd be possible to instead make the client/worker extensions depend directly on inheriting an interface/abstract class that we could instantiate instead? By having the user instantiate the class itself and then pass that around, we make chances for misconfiguration go down.

good point, i updated to shared option and store

[AnatoliB]

Quite a lot of code duplication between this and AzureBlobPayloadsSideCarInterceptor - wondering if there is a way to extract the common code to a shared component.

[YunchuWang]

will address once we confirm on the package approach