feat: add w3c event correlation support (#251)

* feat: add w3c event correlation support

* Update docs/preview/02-Features/publishing-events.md

Co-authored-by: Frederik Gheysels <frederik.gheysels@telenet.be>

Co-authored-by: Frederik Gheysels <frederik.gheysels@telenet.be>

Author: stijnmoreels

docs/preview/02-Features/publishing-events.md

@@ -172,7 +172,7 @@ public void ConfigureServices(IServiceCollection services)
 
 ### Correlation tracking
 The library also provides several ways to influence the correlation tracking during the event publishing. 
-By default, each event that gets published will be enriched with two correlation properties: transaction ID and the operation parent ID. These two properties should be configured as [custom delivery properties](https://docs.microsoft.com/en-us/azure/event-grid/delivery-properties) in Azure Event Grid so that the event processor can easily access them and continue to correlate the message.
+By default, each event that gets published will be enriched with the `traceparent` correlation property. This property should be configured as a [custom delivery property](https://docs.microsoft.com/en-us/azure/event-grid/delivery-properties) in Azure Event Grid so that the event processor can easily access them and continue to correlate the message.
 
 > ⚠ The Arcus `EventGridPublisherClient` assumes that every event published is an JSON event with `data` JSON node.
 
@@ -224,17 +224,18 @@ Then the Arcus `EventGridPublisherClient` will alter the event data as follows:
         "storageDiagnostics": {
             "batchId": "681fe319-3006-00a8-0022-9e7cde000000"
         },
-        "transactionId": "<your-transaction-id>",
-        "operationParentId": "<your-dependency-id>"
+        "traceparent": "00-<your-transaction-id>-<your-parent-id>-00"
     }
 }
 ```
 
-These properties can be accessed with `data.transactionId` and `data.operationParentId` in your Azure Event Grid custom delivery properties of your event subscription.
-The example shows how these properties are transfered to application properties `Transaction-Id` and `Operation-Parent-Id`. These are the default correlation properties of the [Arcus message pump](https://messaging.arcus-azure.net/).
+This property can be accessed with `data.traceparent` in your Azure Event Grid custom delivery properties of your event subscription.
+The example shows how this property is transfered to an application property `Diagnostic-Id`. This is the default correlation property of the [Arcus message pump](https://messaging.arcus-azure.net/) using the W3C message correlation.
 Use the correct correlation properties of your event processor system to correctly correlate the event.
 
-![Azure Event Grid custom delivery properties](/media/event-grid-delivery-properties.png)
+![Azure Event Grid custom delivery properties](/media/event-grid-traceparent-delivery-properties.png)
+
+> ⚡ We also support Hierarchical event correlation, where the the event has a `data.transactionId` and `data.operationParentId` instead of a `data.traceparent` property. This can be configured using the `Format` option described below.
 
 Several other options related to correlation can be configured on the client:
 ```csharp
@@ -253,16 +254,25 @@ public void ConfigureServices(IServiceCollection services)
             "<my-authentication-secret-name>",
             options =>
             {
+                // The function to generate the dependency ID used when tracking the event publishing.
+                // This value corresponds with the operation parent ID on the receiver side, and is called the dependency ID on this side (sender).
+                options.GenerateDependencyId = () => $"custom-dependency-id-{Guid.NewGuid()}";
+
+                // W3C event correlation
+                // ---------------------
+
+                // The name of the JSON property that represents the 'traceparent' that will be added to the event data of the published event (default: traceparent).
+                options.TraceParentEventPropertyName = "customTraceParent";
+
+                // Hierarhical event correlation
+                // -----------------------------
+
                 // The name of the JSON property that represents the transaction ID that will be added to the event data of the published event (default: transactionId).
                 options.TransactionIdEventDataPropertyName = "customTransactionId";
 
                 // The name of the JSON property that represents the operation parent ID that will be added to the event data of the published event (default: operationParentId).
                 options.UpstreamServicePropertyName = "customOperationParentId";
 
-                // The function to generate the dependency ID used when tracking the event publishing.
-                // This value corresponds with the operation parent ID on the receiver side, and is called the dependency ID on this side (sender).
-                options.GenerateDependencyId = () => $"custom-dependency-id-{Guid.NewGuid()}";
-
                 // Adds a telemetry context while tracking the Azure Event Grid dependency.
                 options.AddTelemetryContext(new Dictionary<string, object>
                 {

docs/static/media/event-grid-traceparent-delivery-properties.png

@@ -0,0 +1,18 @@
+namespace Arcus.EventGrid.Core
+{
+    /// <summary>
+    /// Represents the event correlation format of the send-out events.
+    /// </summary>
+    public enum EventCorrelationFormat
+    {
+        /// <summary>
+        /// Uses the W3C event correlation system with traceparent and tracestate to represent parent-child relationship.
+        /// </summary>
+        W3C,
+
+        /// <summary>
+        /// Uses the hierarchical event correlation system with Root-Id and Request-Id to represent parent-child relationship.
+        /// </summary>
+        Hierarchical
+    }
+}

src/Arcus.EventGrid.Core/EventCorrelationFormat.cs

@@ -6,6 +6,7 @@
 using System.Text.Json.Nodes;
 using System.Threading;
 using System.Threading.Tasks;
+using Arcus.EventGrid.Core;
 using Arcus.Observability.Correlation;
 using Arcus.Observability.Telemetry.Core;
 using Arcus.Security.Core;
@@ -350,6 +351,7 @@ private async Task<Response> TrackPublishEventAsync<TEvent>(object oneOrManyEven
 
             string dependencyId = Options.GenerateDependencyId();
             string transactionId = CorrelationAccessor.GetCorrelationInfo()?.TransactionId;
+
             oneOrManyEvents = SetCorrelationPropertiesInEvent(oneOrManyEvents, dependencyId, transactionId);
 
             bool isSuccessful = false;
@@ -410,47 +412,55 @@ private static string DetermineEventType(object @event)
 
         private object SetCorrelationPropertiesInEvent(object @event, string dependencyId, string transactionId)
         {
-            if (!Options.EnableDependencyTracking)
+            if (Options.Format is EventCorrelationFormat.Hierarchical 
+                && !Options.EnableDependencyTracking)
             {
                 return @event;
             }
 
-            string upstreamServicePropertyName = Options.UpstreamServicePropertyName;
-            string transactionIdPropertyName = Options.TransactionIdEventDataPropertyName;
+            var properties = new Collection<(string propertyName, string propertyValue)>();
+            if (Options.Format is EventCorrelationFormat.Hierarchical)
+            {
+                properties.Add((Options.TransactionIdEventDataPropertyName, transactionId));
+                properties.Add((Options.UpstreamServicePropertyName, dependencyId));
+            }
 
-            switch (@event)
+            if (Options.Format is EventCorrelationFormat.W3C)
             {
-                case BinaryData data:
-                    data = SetCorrelationPropertyInCustomEvent(data, upstreamServicePropertyName, dependencyId);
-                    data = SetCorrelationPropertyInCustomEvent(data, transactionIdPropertyName, transactionId);
-                    return data;
-                case IEnumerable<BinaryData> datas:
-                    datas = SetCorrelationPropertyInCustomEvents(datas, upstreamServicePropertyName, dependencyId);
-                    datas = SetCorrelationPropertyInCustomEvents(datas, transactionIdPropertyName, transactionId);
-                    return datas;
-                case EventGridEvent ev:
-                    ev = SetCorrelationPropertyInEventGridEvent(ev, upstreamServicePropertyName, dependencyId);
-                    ev = SetCorrelationPropertyInEventGridEvent(ev, transactionIdPropertyName, transactionId);
-                    return ev;
-                case IEnumerable<EventGridEvent> events:
-                    events = SetCorrelationPropertyInEventGridEvents(events, upstreamServicePropertyName, dependencyId);
-                    events = SetCorrelationPropertyInEventGridEvents(events, transactionIdPropertyName, transactionId);
-                    return events;
-                case CloudEvent ev:
-                    ev = SetCorrelationPropertyInCloudEvent(ev, upstreamServicePropertyName, dependencyId);
-                    ev = SetCorrelationPropertyInCloudEvent(ev, transactionIdPropertyName, transactionId);
-                    return ev;
-                case IEnumerable<CloudEvent> events:
-                    events = SetCorrelationPropertyInCloudEvents(events, upstreamServicePropertyName, dependencyId);
-                    events = SetCorrelationPropertyInCloudEvents(events, transactionIdPropertyName, transactionId);
-                    return events;
-                case ReadOnlyMemory<byte> encodedCloudEvents:
-                    encodedCloudEvents = SetCorrelationPropertyInEncodedCloudEvents(encodedCloudEvents, upstreamServicePropertyName, dependencyId);
-                    encodedCloudEvents = SetCorrelationPropertyInEncodedCloudEvents(encodedCloudEvents, transactionIdPropertyName, transactionId);
-                    return encodedCloudEvents;
-                default:
-                    throw new ArgumentOutOfRangeException(nameof(@event), @event, "Unknown event type");
+                properties.Add((Options.TraceParentPropertyName, $"00-{transactionId}-{dependencyId}-00"));
             }
+
+            foreach ((string propertyName, string propertyValue) in properties)
+            {
+                switch (@event)
+                {
+                    case BinaryData data:
+                        @event = SetCorrelationPropertyInCustomEvent(data, propertyName, propertyValue);
+                        break;
+                    case IEnumerable<BinaryData> datas:
+                        @event = SetCorrelationPropertyInCustomEvents(datas, propertyName, propertyValue);
+                        break;
+                    case EventGridEvent ev:
+                        @event = SetCorrelationPropertyInEventGridEvent(ev, propertyName, propertyValue);
+                        break;
+                    case IEnumerable<EventGridEvent> events:
+                        @event = SetCorrelationPropertyInEventGridEvents(events, propertyName, propertyValue);
+                        break;
+                    case CloudEvent ev:
+                        @event = SetCorrelationPropertyInCloudEvent(ev, propertyName, propertyValue);
+                        break;
+                    case IEnumerable<CloudEvent> events:
+                        @event = SetCorrelationPropertyInCloudEvents(events, propertyName, propertyValue);
+                        break;
+                    case ReadOnlyMemory<byte> encodedCloudEvents:
+                        @event = SetCorrelationPropertyInEncodedCloudEvents(encodedCloudEvents, propertyName, propertyValue);
+                        break;
+                    default:
+                        throw new ArgumentOutOfRangeException(nameof(@event), @event, "Unknown event type");
+                }
+            }
+
+            return @event;
         }
 
         private IEnumerable<EventGridEvent> SetCorrelationPropertyInEventGridEvents(IEnumerable<EventGridEvent> eventGridEvents, string propertyName, string dependencyId)
@@ -632,7 +642,8 @@ protected virtual BinaryData SetCorrelationPropertyInCustomEvent(BinaryData data
 
         private void LogEventGridDependency(string eventType, bool isSuccessful, DurationMeasurement measurement, string dependencyId)
         {
-            if (Options.EnableDependencyTracking)
+            if (Options.Format is EventCorrelationFormat.Hierarchical 
+                && Options.EnableDependencyTracking)
             {
                 Logger.LogDependency(
                     dependencyType: "Azure Event Grid",

src/Arcus.EventGrid.Core/Publishing/EventGridPublisherClientWithTracking.cs

@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using Arcus.EventGrid.Core;
 using GuardNet;
 using Microsoft.Extensions.Azure;
 using Polly;
@@ -16,14 +17,37 @@ namespace Azure.Messaging.EventGrid
     /// </summary>
     public class EventGridPublisherClientWithTrackingOptions : EventGridPublisherClientOptions
     {
-        private string _upstreamServicePropertyName = "operationParentId", _transactionIdPropertyName = "transactionId";
+        private string _upstreamServicePropertyName = "operationParentId", _transactionIdPropertyName = "transactionId", _traceParentPropertyName = "traceparent";
         private Func<string> _generateDependencyId = () => Guid.NewGuid().ToString();
 
+        /// <summary>
+        /// Gets or sets the format of the used event correlation when publishing events to an Azure EventGrid subscription (default: W3C).
+        /// </summary>
+        public EventCorrelationFormat Format { get; set; } = EventCorrelationFormat.W3C;
+
+        /// <summary>
+        /// Gets or sets the name of the JSON property that represents the 'traceparent' that will be added to the event data of the published event (default: traceparent(.
+        /// The 'traceparent' will be available as 'data.traceparent' (if default configured) and can be added as dynamic event delivery property: <a href="https://docs.microsoft.com/en-us/azure/event-grid/delivery-properties" />.
+        /// </summary>
+        /// <remarks>
+        ///     Only used when the <see cref="Format"/> is set to <see cref="EventCorrelationFormat.W3C"/>.
+        /// </remarks>
+        public string TraceParentPropertyName
+        {
+            get => _traceParentPropertyName;
+            set
+            {
+                Guard.NotNullOrWhitespace(value, nameof(value), "Requires a non-blank JSON property name to add the 'traceparent' to the event data of the published event");
+                _traceParentPropertyName = value;
+            }
+        }
+
         /// <summary>
         /// Gets or sets the name of the JSON property that represents the transaction ID that will be added to the event data of the published event (default: transactionId).
         /// The transaction ID will be available as 'data.transactionId' (if default configured) and can be used as dynamic event delivery property: <a href="https://docs.microsoft.com/en-us/azure/event-grid/delivery-properties" />.
         /// </summary>
         /// <remarks>
+        ///     Only used when <see cref="Format"/> is set to <see cref="EventCorrelationFormat.Hierarchical"/>.
         ///     Make sure that the system processing the event (Azure Service Bus, web hook, etc.) can retrieve this property so that the correlation holds.
         ///     When using Arcus' messaging, the event header name for the dynamic event delivery property should be: 'Transaction-Id'.
         /// </remarks>
@@ -43,6 +67,7 @@ public string TransactionIdEventDataPropertyName
         /// The operation parent ID will be available as 'data.operationParentId' (if default configured) and can be used as dynamic event delivery property: <a href="https://docs.microsoft.com/en-us/azure/event-grid/delivery-properties" />.
         /// </summary>
         /// <remarks>
+        ///     Only used when <see cref="Format"/> is set to <see cref="EventCorrelationFormat.Hierarchical"/>.
         ///     Make sure that the system processing the event (Azure Service Bus, web hook, etc.) can retrieve this property so that the correlation holds.
         ///     When using Arcus' messaging, the event header name for the dynamic event delivery property should be: 'Operation-Parent-Id'.
         /// </remarks>
@@ -61,6 +86,9 @@ public string UpstreamServicePropertyName
         /// Gets or sets the function to generate the dependency ID used when tracking the event publishing.
         /// This value corresponds with the operation parent ID on the receiver side, and is called the dependency ID on this side (sender).
         /// </summary>
+        /// <remarks>
+        ///     Can only be used when the <see cref="Format"/> is set to <see cref="EventCorrelationFormat.Hierarchical"/> as Microsoft dependencies are usually automatically tracked.
+        /// </remarks>
         /// <exception cref="ArgumentNullException">Thrown when the <paramref name="value"/> is <c>null</c>.</exception>
         public Func<string> GenerateDependencyId
         {
@@ -80,6 +108,9 @@ public Func<string> GenerateDependencyId
         /// <summary>
         /// Adds a telemetry context while tracking the Azure Event Grid dependency.
         /// </summary>
+        /// <remarks>
+        ///     Can only be used when the <see cref="Format"/> is set to <see cref="EventCorrelationFormat.Hierarchical"/> as Microsoft dependencies are usually automatically tracked.
+        /// </remarks>
         /// <param name="telemetryContext">The dictionary with the contextual information about the event publishing dependency tracking.</param>
         /// <exception cref="ArgumentNullException">Thrown when the <paramref name="telemetryContext"/> is <c>null</c>.</exception>
         public void AddTelemetryContext(Dictionary<string, object> telemetryContext)
@@ -92,9 +123,13 @@ public void AddTelemetryContext(Dictionary<string, object> telemetryContext)
         }
 
         /// <summary>
-        /// <para>Gets or sets the flag indicating whether or not the <see cref="EventGridPublisherClient"/> should track the Azure Event Grid topic dependency.</para>
+        /// <para>Gets or sets the flag indicating whether or not the <see cref="EventGridPublisherClient"/> should track the Azure Event Grid topic dependency (default: <c>false</c>).</para>
         /// <para>For more information about dependency tracking <see href="https://observability.arcus-azure.net/features/writing-different-telemetry-types#dependencies"/>.</para>
         /// </summary>
+        /// <remarks>
+        ///     Note that Microsoft dependencies are automatically tracked when using the default Application Insights telemetry services.
+        ///     Consider using this only when <see cref="Format"/> is set to <see cref="EventCorrelationFormat.Hierarchical"/>.
+        /// </remarks>
         public bool EnableDependencyTracking { get; set; } = true;
 
         /// <summary>