The serialization logic for exceptions thrown by the Activity Function

[dxynnez]

Hi team,

We recently noticed that some of the exceptions being thrown by the activity function failed to be deserialized at the durable function side with the below stacktrace:

Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionFailedException: The activity function 'TestFunction' failed: "Failed to deserialize exception from TaskActivity: {"$type":"Microsoft.Rest.Azure.CloudException, Microsoft.Rest.ClientRuntime.Azure","Request":{"$type":"Microsoft.Rest.HttpRequestMessageWrapper, Microsoft.Rest.ClientRuntime","Method":{"$type":"System.Net.Http.HttpMethod, System.Net.Http","Method":"GET"}
<--rest of the stacktrace are emitted for brevity -->

I understand the problem is related to #1453 but we are more concerned about how the exception is being stored and where it's stored? The Microsoft.Rest.Azure.CloudException contains all request headers and we simply cannot have it stored in Azure table and will need to do a cleanup after the code fix. I can only find the corresponding durable function record under the task hub Instance table but not the actual serialized exception record for the activity. Could someone elaborate a little bit on the serialization behavior for the durable task framework? Thanks!

[cgillum]

Check the History table. If you filter by the PartitionId column for your orchestration's instance ID, you should see a record with EventType = TaskFailed, which includes the exception information that gets stored.

The Durable Task Framework tries to serialize the exception into a JSON object at the time the activity function fails and then deserialize it again when the orchestrator replays that particular activity function call. If you want more control over the serialization and deserialization process for exceptions, we allow you to inject your own service to do exactly this. More information here: https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-serialization-and-persistence?tabs=csharp#customizing-serialization-with-dependency-injection

BTW, if you need to purge the history, you can do so using our purge APIs rather than messing with Table Storage directly.

[dxynnez]

Thanks so much @cgillum ! I assume the purge APIs would only work for the history in the taskhub that is currently being used? If so, then probably messing around with azure table is the only way for us as we switch to a new taskhub in each deployment.

Just to double confirm, so if we delete the entire taskhub history & instances tables then there won't be any exception stored (blob, queue, table) as far as durable task's concerned.

[cgillum]

Correct, the purge APIs are specific to a particular task hub. However, I think you should be able to bind a particular function to a specific task hub using the [DurableClient(TaskHub = "XXX")] attribute. For example:

[FunctionName("PurgeInstance")]
public static async Task<HttpResponseMessage> PurgeInstance(
    [HttpTrigger(AuthorizationLevel.Function, methods: "post", Route = "purge/{instanceId}")] HttpRequestMessage req,
    [DurableClient(TaskHub = "TaskHubV1")] IDurableClient client,
    string instanceId)
{
    PurgeHistoryResult result = await client.PurgeInstanceHistoryAsync(instanceId);
    return req.CreateResponse(result);
}

The above example shows using an HTTP trigger to purge a single instance from the "TaskHubV1" task hub, but you could modify it to delete multiple instances if you wanted to, based on a time range, etc.

Just to double confirm, so if we delete the entire taskhub history & instances tables then there won't be any exception stored (blob, queue, table) as far as durable task's concerned.

The purge APIs will clean out all data from blobs and tables, including any exception info. It won't clean out queue data though since we have no way of enumerating queues to see what data qualifies for deletion. However, this seems like a corner case scenario since queue data is supposed to be transient anyways.

[dxynnez]

Thanks again @cgillum ! This probably wouldn't help for our case as we can't enumerate all the previous used taskhub names (we probably have hundreds at this point as service has been running for months and we create a new task hub with each deployment)... Is there a way to bind durable clients to each previous used task hub?

Would blobs contain any data that might be qualified for EUII or Access control data? e.g. UPN in the durable function input / output; Token in the error...

I am just trying to figure out all the places the we need to clean up in Azure storage at this point.

[cgillum]

Durable clients can only bind to one task hub at a time. I agree this may not be a practical approach if you need to clear a backlog of data from a bunch of old task hubs.

Would blobs contain any data that might be qualified for EUII or Access control data? e.g. UPN in the durable function input / output; Token in the error...

Blobs won't contain any sensitive data by default - no PII/EUII, access control, UPNs, tokens, etc. The only way it would is if your orchestrations are explicitly passing data payloads to activities or sub-orchestrations and those payloads contain sensitive data.