Distributed tracing and telemetry correlation does not work for azure durable functions (Node.js)

[Sarthak-Chokshi]

I am trying to implement distributed tracing for a feature that is implemented using the Azure durable function and a bunch of activity functions. In javascript azure durable function is implemented using generator function. I am not able to wrap a generator function the way the async function is wrapped in https://www.npmjs.com/package/applicationinsights. It seems like the azure function host assigns a new operation id each time the durable function runs.

I have an http starter function as follows:

import * as appInsights from 'applicationinsights';appInsights .setup() // assuming ikey is in env var .setAutoDependencyCorrelation(true, true) .setAutoCollectRequests(true) .setAutoCollectPerformance(true) .setAutoCollectExceptions(true) .setAutoCollectDependencies(true) .setDistributedTracingMode(appInsights.DistributedTracingModes.AI_AND_W3C) .start();
appInsights.defaultClient.context.tags['ai.cloud.role'] = 'ActivationStart';
import * as df from 'durable-functions';
import { AzureFunction, Context, HttpRequest } from '@azure/functions';

const httpStartHelper: AzureFunction = async function (
context: Context,
req: HttpRequest,
): Promise<any> {
const client = df.getClient(context);
const instanceId = await client.startNew(
context.req.params.functionName,
undefined,
context.req.body,
);
context.log('Instance id', instanceId);
return client.createCheckStatusResponse(context.req, instanceId);
};

const httpStart: AzureFunction = async function (context: Context, req: HttpRequest): Promise<any> {
const correlationContext = appInsights.startOperation(context, req);
return appInsights.wrapWithCorrelationContext(async () => {
const startTime = Date.now(); // Start trackRequest timer
const response = await httpStartHelper(context, req);
appInsights.defaultClient.trackRequest({
name: context.req.method + ' ' + context.req.url,
resultCode: context.res.status,
success: true,
url: req.url,
duration: Date.now() - startTime,
id: correlationContext.operation.parentId,
});
appInsights.defaultClient.flush();
return response;
}, correlationContext)();
};

export default httpStart;

Is it possible to use wrapWithCorrelationContext() inside the generator function, so that it logs requests with the same operation_id? or in general, any guideline to implement distributed tracing for durable functions using application insights?

[davidmrdavid]

Hi @Sarthak-Chokshi,

Thanks for reaching out. Hmm, this is an interesting one. First, you should know that we're actively working on supporting built-in support for Distributed Tracing in Durable Functions, although we're still relatively early in that journey for JavaScript. @bachuv is a good point of contact for Durable's built-in Distributed Tracing, as well as me since I'm focused in Durable for JavaScript in particular.

That aside, you're asking about integrating the standard azure functions distributed tracing tool with durable, which can be a bit hard to do and I'm also not too familiar with that API. I do have an idea though, would it be possible to manually set the operationId to be the orchestration's instanceID. That way, you could group related requests together. This is the approach I suggested in this related thread: Azure/azure-functions-durable-js#256

My recommendation would be to review the linked thread above for some examples on how to correlate durable requests for JavaScript. It's not exactly in the same way as in the npm examples, but I believe it uses the same package. Please let me know if the suggestions there work for you!

[Sarthak-Chokshi]

Hi, @davidmrdavid Thank you for the updates. For now, I have done the following workaround:

1. Set traceparent header in a client request.
2. Parse it into orchestrator starter function, update correlation context and log the request as showed below:
   `const httpStartHelper: AzureFunction = async function (
   context: Context,
   req: HttpRequest,
   ): Promise {
   const client = df.getClient(context);
   const instanceId = await client.startNew(context.req.params.functionName, undefined, context.req);
   context.log('Instance id', instanceId);
   return client.createCheckStatusResponse(context.req, instanceId);
   };

const httpStart: AzureFunction = async function (context: Context, req: HttpRequest): Promise {
const correlationContext = appInsights.startOperation(context, req);
const id = correlationContext.operation.parentId;
if (req.headers.traceparent) {
const [, traceId, spanId] = req.headers?.traceparent?.split('-');
correlationContext.operation.parentId =
correlationContext.operation.traceparent.parentId = |${traceId}.${spanId}.;
}
return appInsights.wrapWithCorrelationContext(async () => {
const startTime = Date.now(); // Start trackRequest timer
const response = await httpStartHelper(context, req);
appInsights.defaultClient.trackRequest({
name: context.req.method + ' ' + context.req.url,
resultCode: context.res.status,
success: true,
url: req.url,
duration: Date.now() - startTime,
id,
});
appInsights.defaultClient.flush();
return response;
}, correlationContext)();
};
export default httpStart;`

3. Do the same for all the activity functions. This is feasible as we can pass requests to different activity functions from the orchestrator function.

This approach works at some level but it is better than no correlation. Do you have any idea when correlation will be supported natively for Node.js/Javascript environment?

[davidmrdavid]

Great to hear you've got a partial workaround working. Unfortunately, I can't give an estimate of when the Correlation/Distributed Tracing will be fully supported in JS. However, once we get it to a stable place for C#, which @bachuv is currently working on, I expected it should then be relatively straightforward to support JS and Python and all our other languages.