Skip to content

Commit 763b1ed

Browse files
link04zacharycmontoya
link04
and
zacharycmontoya
authored
[OTEL] Metrics API Support (#7138)
## Summary of changes Experimental OpenTelemetry Metrics support: - Added configuration keys for enabling OTLP metrics (`DD_METRICS_OTEL_ENABLED`) and specifying enabled meters (`DD_METRICS_OTEL_METER_NAMES`). - Introduced `OtlpMetricsExporter `(with duck-typed interfaces) for initializing OTLP metrics export. - Updated tracer/test helpers to handle, deserialize, and verify OTLP metric requests. - Added metrics instrumentation to the sample OpenTelemetrySdk test app. This feature is tested across the following runtime + SDK combinations to ensure the correct set of metrics are exported for each: | .NET Runtime | OTel SDK Version | Supported Metrics | |--------------|------------------|------------------------------------------------------------------------------------| | `.NET 6` | `1.3.2` | `Counter`, `Histogram`, `ObservableCounter`, `ObservableGauge` | | `.NET 7 / 8` | `1.5.1` | All of the above + `UpDownCounter`, `ObservableUpDownCounter` | | `.NET 9` | `1.12.0` | All of the above + `Gauge<T>` (requires both .NET 9 and OTel SDK ≥ 1.10.0) | ✅ These specific versions were selected to represent the earliest SDK version that supports the full metric surface available on each runtime. We test one representative OpenTelemetry version per runtime to avoid duplication and snapshot churn. To enable OTLP metrics export via this integration, the application must: Use OpenTelemetry SDK version 1.3.2 or higher Configure an OTLP metrics exporter (e.g. to http://localhost:port) Set the following environment variables: ``` DD_METRICS_OTEL_ENABLED=true — turns on metrics export DD_TRACE_ENABLED_METERS=MyAppMeter — limits export to specific meters OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf — required format OTEL_EXPORTER_OTLP_ENDPOINT=http://<agent>:<port> — where metrics are sen ``` ## Reason for change POC to enable exporting metrics using the OpenTelemetry Metrics API and OTLP protocol for improved observability and interoperability. [APMAPI-1321](https://datadoghq.atlassian.net/browse/APMAPI-1321) ## Implementation details - Uses duck-typing to integrate with OpenTelemetry SDK and OTLP exporter without direct dependency. - Configuration-driven activation of metrics exporting and meter selection. - Test helpers extended to parse protobuf OTLP metrics for validation. - Sample application now emits various metric types (counter, histogram, up/down counter, gauge). ## Test coverage - Added end-to-end integration tests to verify OTLP metrics are emitted and received. - MockAgent updated to deserialize OTLP metric payloads and support snapshot verification, using the proto Otel files I generated locally and the adding the `Google.Protobuf` package. - Sample app exercises new metrics instrumentation for test scenarios. ## Other details The work on this PR is part of a experimental POC to demonstrate the tracer been able to setup and send the meters the customer has to the Agent using the OTLP endpoint it has, work has been scheduled to provide support and update some of the implementation details here before going GA. [APMAPI-1321]: https://datadoghq.atlassian.net/browse/APMAPI-1321?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Zach Montoya <[email protected]>
1 parent aabf016 commit 763b1ed

22 files changed

+9629
-4
lines changed

‎tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -342,6 +342,33 @@ internal static void InitializeNoNativeParts(Stopwatch sw = null)
342342
Log.Error(ex, "Error initializing activity listener");
343343
}
344344

345+
#if NET6_0_OR_GREATER
346+
try
347+
{
348+
if (Tracer.Instance.Settings.OpenTelemetryMetricsEnabled)
349+
{
350+
Log.Debug("Initializing OTel Metrics Exporter.");
351+
if (Tracer.Instance.Settings.OpenTelemetryMeterNames.Length > 0)
352+
{
353+
OTelMetrics.OtlpMetricsExporter.Initialize();
354+
}
355+
else
356+
{
357+
Log.Debug("No meters were found for DD_METRICS_OTEL_METER_NAMES, OTel Metrics Exporter won't be initialized.");
358+
}
359+
}
360+
}
361+
catch (Exception ex)
362+
{
363+
Log.Error(ex, "Error initializing OTel Metrics Exporter");
364+
}
365+
#else
366+
if (Tracer.Instance.Settings.OpenTelemetryMetricsEnabled)
367+
{
368+
Log.Information("Unable to initialize OTel Metrics collection, this is only available starting with .NET 6.0..");
369+
}
370+
#endif
371+
345372
try
346373
{
347374
if (Tracer.Instance.Settings.IsActivityListenerEnabled)

‎tracer/src/Datadog.Trace/Configuration/ConfigurationKeys.cs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -806,6 +806,18 @@ internal static class FeatureFlags
806806
/// </summary>
807807
public const string OpenTelemetryEnabled = "DD_TRACE_OTEL_ENABLED";
808808

809+
/// <summary>
810+
/// Enables experimental support for exporting OTLP metrics generated by the OpenTelemetry Metrics API.
811+
/// This feature is only available starting with .NET 6.0, as it relies on the BCL class MeterListener
812+
/// which is shipped in-box starting with .NET 6.
813+
/// </summary>
814+
public const string OpenTelemetryMetricsEnabled = "DD_METRICS_OTEL_ENABLED";
815+
816+
/// <summary>
817+
/// List of meters to add to the metrics exporter for the experimental OpenTelemetry Metrics API support.
818+
/// </summary>
819+
public const string OpenTelemetryMeterNames = "DD_METRICS_OTEL_METER_NAMES";
820+
809821
/// <summary>
810822
/// Enables generating 128-bit trace ids instead of 64-bit trace ids.
811823
/// Note that a 128-bit trace id may be received from an upstream service or from

‎tracer/src/Datadog.Trace/Configuration/TracerSettings.cs

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -761,6 +761,13 @@ _ when x.ToBoolean() is { } boolean => boolean,
761761
}
762762
}
763763

764+
OpenTelemetryMetricsEnabled = config
765+
.WithKeys(ConfigurationKeys.FeatureFlags.OpenTelemetryMetricsEnabled)
766+
.AsBool(defaultValue: false);
767+
768+
var enabledMeters = config.WithKeys(ConfigurationKeys.FeatureFlags.OpenTelemetryMeterNames).AsString();
769+
OpenTelemetryMeterNames = !string.IsNullOrEmpty(enabledMeters) ? TrimSplitString(enabledMeters, commaSeparator) : [];
770+
764771
var disabledActivitySources = config.WithKeys(ConfigurationKeys.DisabledActivitySources).AsString();
765772

766773
DisabledActivitySources = !string.IsNullOrEmpty(disabledActivitySources) ? TrimSplitString(disabledActivitySources, commaSeparator) : [];
@@ -890,6 +897,16 @@ static void RecordDisabledIntegrationsTelemetry(IntegrationSettingsCollection in
890897
/// <seealso cref="ConfigurationKeys.DisabledIntegrations"/>
891898
public HashSet<string> DisabledIntegrationNames { get; }
892899

900+
/// <summary>
901+
/// Gets a value indicating whether OpenTelemetry Metrics are enabled.
902+
/// </summary>
903+
/// <seealso cref="ConfigurationKeys.FeatureFlags.OpenTelemetryMetricsEnabled"/>
904+
internal bool OpenTelemetryMetricsEnabled { get; }
905+
906+
/// Gets the names of enabled Meters.
907+
/// <seealso cref="ConfigurationKeys.FeatureFlags.OpenTelemetryMeterNames"/>
908+
internal string[] OpenTelemetryMeterNames { get; }
909+
893910
/// <summary>
894911
/// Gets the names of disabled ActivitySources.
895912
/// </summary>

‎tracer/src/Datadog.Trace/OTelMetrics/DuckTypes/IMeterProvider.cs

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// <copyright file="IMeterProvider.cs" company="Datadog">
2+
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
4+
// </copyright>
5+
6+
#if NET6_0_OR_GREATER
7+
8+
#nullable enable
9+
10+
using System;
11+
using Datadog.Trace.DuckTyping;
12+
13+
namespace Datadog.Trace.OTelMetrics.DuckTypes
14+
{
15+
internal interface IMeterProvider : IDuckType, IDisposable
16+
{
17+
}
18+
}
19+
#endif

‎tracer/src/Datadog.Trace/OTelMetrics/DuckTypes/IMeterProviderBuilder.cs

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// <copyright file="IMeterProviderBuilder.cs" company="Datadog">
2+
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
4+
// </copyright>
5+
6+
#if NET6_0_OR_GREATER
7+
8+
#nullable enable
9+
10+
using Datadog.Trace.DuckTyping;
11+
12+
namespace Datadog.Trace.OTelMetrics.DuckTypes
13+
{
14+
internal interface IMeterProviderBuilder : IDuckType
15+
{
16+
IMeterProviderBuilder AddMeter(string[] names);
17+
18+
IMeterProvider Build();
19+
}
20+
}
21+
#endif

‎tracer/src/Datadog.Trace/OTelMetrics/DuckTypes/IOtelSdk.cs

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// <copyright file="IOtelSdk.cs" company="Datadog">
2+
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
4+
// </copyright>
5+
6+
#if NET6_0_OR_GREATER
7+
8+
#nullable enable
9+
10+
using Datadog.Trace.DuckTyping;
11+
12+
namespace Datadog.Trace.OTelMetrics.DuckTypes
13+
{
14+
internal interface IOtelSdk : IDuckType
15+
{
16+
IMeterProviderBuilder CreateMeterProviderBuilder();
17+
}
18+
}
19+
#endif

‎tracer/src/Datadog.Trace/OTelMetrics/DuckTypes/IOtlpMetricExporterExtensions.cs

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// <copyright file="IOtlpMetricExporterExtensions.cs" company="Datadog">
2+
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
4+
// </copyright>
5+
6+
#if NET6_0_OR_GREATER
7+
8+
#nullable enable
9+
10+
using Datadog.Trace.DuckTyping;
11+
12+
namespace Datadog.Trace.OTelMetrics.DuckTypes
13+
{
14+
internal interface IOtlpMetricExporterExtensions : IDuckType
15+
{
16+
IMeterProviderBuilder AddOtlpExporter(IMeterProviderBuilder builder);
17+
}
18+
}
19+
#endif

‎tracer/src/Datadog.Trace/OTelMetrics/OtlpMetricsExporter.cs

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
// <copyright file="OtlpMetricsExporter.cs" company="Datadog">
2+
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
4+
// </copyright>
5+
6+
#if NET6_0_OR_GREATER
7+
8+
using System;
9+
using Datadog.Trace.DuckTyping;
10+
using Datadog.Trace.Logging;
11+
using Datadog.Trace.OTelMetrics.DuckTypes;
12+
13+
#nullable enable
14+
15+
namespace Datadog.Trace.OTelMetrics
16+
{
17+
internal static class OtlpMetricsExporter
18+
{
19+
private static readonly IDatadogLogger Log = DatadogLogging.GetLoggerFor(typeof(OtlpMetricsExporter));
20+
21+
public static void Initialize()
22+
{
23+
var otelSdkType = Type.GetType("OpenTelemetry.Sdk, OpenTelemetry", throwOnError: false);
24+
if (otelSdkType is null)
25+
{
26+
ThrowHelper.ThrowNullReferenceException($"The OpenTelemetry SDK type is null, make sure the OpenTelemetry NuGet package is installed to collect metrics.");
27+
}
28+
29+
var otelSdkProxyResult = DuckType.GetOrCreateProxyType(typeof(IOtelSdk), otelSdkType);
30+
var otelSdkProxyResultType = otelSdkProxyResult.ProxyType;
31+
if (otelSdkProxyResultType is null)
32+
{
33+
ThrowHelper.ThrowNullReferenceException($"Resulting proxy type after Ducktyping attempt of {typeof(IOtelSdk)} is null.");
34+
}
35+
else if (otelSdkProxyResult.Success)
36+
{
37+
var otelSdkProxy = (IOtelSdk)otelSdkProxyResult.CreateInstance(null!);
38+
var meterProviderBuilder = otelSdkProxy.CreateMeterProviderBuilder();
39+
var meterProviderProxy = meterProviderBuilder.DuckCast<IMeterProviderBuilder>();
40+
meterProviderProxy.AddMeter(Tracer.Instance.Settings.OpenTelemetryMeterNames);
41+
42+
var otlpMetricExporterExtensionsType = Type.GetType("OpenTelemetry.Metrics.OtlpMetricExporterExtensions, OpenTelemetry.Exporter.OpenTelemetryProtocol", throwOnError: false);
43+
if (otlpMetricExporterExtensionsType is null)
44+
{
45+
ThrowHelper.ThrowNullReferenceException($"The OpenTelemetry Metrics Exporter Extensions type is null, make sure the OpenTelemetry NuGet package is installed to collect metrics.");
46+
}
47+
48+
var otlpMetricExporterExtensionsProxyResult = DuckType.GetOrCreateProxyType(typeof(IOtlpMetricExporterExtensions), otlpMetricExporterExtensionsType);
49+
var otlpMetricExporterExtensionsProxyResultType = otlpMetricExporterExtensionsProxyResult.ProxyType;
50+
if (otlpMetricExporterExtensionsProxyResultType is null)
51+
{
52+
ThrowHelper.ThrowNullReferenceException($"Resulting proxy type after Ducktyping attempt of {typeof(IOtlpMetricExporterExtensions)} is null.");
53+
}
54+
else if (otlpMetricExporterExtensionsProxyResult.Success)
55+
{
56+
var otlpMetricExporterExtensionsProxy = (IOtlpMetricExporterExtensions)otlpMetricExporterExtensionsProxyResult.CreateInstance(null!);
57+
otlpMetricExporterExtensionsProxy.AddOtlpExporter(meterProviderProxy);
58+
59+
var meterProvider = meterProviderProxy.Build();
60+
LifetimeManager.Instance.AddShutdownTask(_ => meterProvider.Dispose());
61+
Log.Information("Successfully Ducktyped and configured OTLP Metrics Exporter.");
62+
}
63+
}
64+
}
65+
}
66+
}
67+
#endif

‎tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/OpenTelemetrySdkTests.cs

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ public class OpenTelemetrySdkTests : TracingIntegrationTest
7272

7373
private readonly Regex _versionRegex = new(@"telemetry.sdk.version: (0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)");
7474
private readonly Regex _timeUnixNanoRegex = new(@"time_unix_nano"":([0-9]{10}[0-9]+)");
75+
private readonly Regex _timeUnixNanoRegexMetrics = new(@"TimeUnixNano: ([0-9]{10}[0-9]+)");
7576
private readonly Regex _exceptionStacktraceRegex = new(@"exception.stacktrace"":""System.ArgumentException: Example argument exception.*"",""");
7677

7778
public OpenTelemetrySdkTests(ITestOutputHelper output)
@@ -197,6 +198,59 @@ public async Task IntegrationDisabled(string packageVersion)
197198
}
198199
}
199200

201+
#if NET6_0_OR_GREATER
202+
[SkippableTheory]
203+
[Trait("Category", "EndToEnd")]
204+
[Trait("RunOnWindows", "True")]
205+
[MemberData(nameof(PackageVersions.OpenTelemetry), MemberType = typeof(PackageVersions))]
206+
public async Task SubmitsOtlpMetrics(string packageVersion)
207+
{
208+
var parsedVersion = Version.Parse(!string.IsNullOrEmpty(packageVersion) ? packageVersion : "1.12.0");
209+
var runtimeMajor = Environment.Version.Major;
210+
211+
var snapshotName = runtimeMajor switch
212+
{
213+
6 when parsedVersion >= new Version("1.3.2") && parsedVersion < new Version("1.5.0") => ".NET_6",
214+
7 or 8 when parsedVersion >= new Version("1.5.1") && parsedVersion < new Version("1.10.0") => ".NET_7_8",
215+
>= 9 when parsedVersion >= new Version("1.10.0") => string.Empty,
216+
_ => throw new SkipException($"Skipping test due to irrelevant runtime and OTel versions mix: .NET {runtimeMajor} & Otel v{parsedVersion}")
217+
};
218+
219+
var initialAgentPort = TcpPortProvider.GetOpenPort();
220+
221+
SetEnvironmentVariable("DD_METRICS_OTEL_ENABLED", "true");
222+
SetEnvironmentVariable("DD_METRICS_OTEL_METER_NAMES", "OpenTelemetryMetricsMeter");
223+
SetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT", $"http://127.0.0.1:{initialAgentPort}");
224+
SetEnvironmentVariable("OTEL_EXPORTER_OTLP_PROTOCOL", "http/protobuf");
225+
SetEnvironmentVariable("OTEL_METRIC_EXPORT_INTERVAL", "1000");
226+
SetEnvironmentVariable("OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE", "delta");
227+
228+
using var agent = EnvironmentHelper.GetMockAgent(fixedPort: initialAgentPort);
229+
using (await RunSampleAndWaitForExit(agent, packageVersion: packageVersion ?? "1.12.0"))
230+
{
231+
var metricRequests = agent.OtlpRequests
232+
.Where(r => r.PathAndQuery.StartsWith("/v1/metrics"))
233+
.ToList();
234+
235+
metricRequests.Should().NotBeEmpty("Expected OTLP metric requests were not received.");
236+
237+
var snapshotPayload = metricRequests
238+
.Select(r => r.DeserializedData)
239+
.ToList();
240+
241+
var settings = VerifyHelper.GetSpanVerifierSettings();
242+
settings.AddRegexScrubber(_timeUnixNanoRegexMetrics, @"TimeUnixNano"": <DateTimeOffset.Now>");
243+
244+
var suffix = GetSuffix(packageVersion);
245+
var fileName = $"{nameof(OpenTelemetrySdkTests)}.SubmitsOtlpMetrics{suffix}{snapshotName}";
246+
247+
await Verifier.Verify(snapshotPayload, settings)
248+
.UseFileName(fileName)
249+
.DisableRequireUniquePrefix();
250+
}
251+
}
252+
#endif
253+
200254
private static string GetSuffix(string packageVersion)
201255
{
202256
// The snapshots are only different in .NET Core 2.1 - .NET 5 with package version 1.0.1

‎tracer/test/Datadog.Trace.TestHelpers/Datadog.Trace.TestHelpers.csproj

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@
88
<PackageReference Include="HttpMultipartParser" Version="5.1.0" />
99
<PackageReference Include="PublicApiGenerator" Version="10.2.0" />
1010
<PackageReference Include="DiffPlex" Version="1.7.1" />
11+
<!-- Added this package to make use of the locally generated OTEL `.proto files by:
12+
1. Downloading protoc from: https://github.com/protocolbuffers/protobuf/releases and adding it to the PATH
13+
2. Cloning the opentelemetry-proto repository: git clone https://github.com/open-telemetry/opentelemetry-proto.git && cd into opentelemetry-proto
14+
3. Creating a folder where generated files will be placed: mkdir Generated
15+
4. Running the command to generate the files to paste where needed: protoc -I. \-\-csharp_out=./Generated opentelemetry/proto/metrics/v1/metrics.proto opentelemetry/proto/common/v1/common.proto opentelemetry/proto/resource/v1/resource.proto
16+
(Excaped the double `-` command above with `\` when running the command remove them.)-->
17+
<PackageReference Include="Google.Protobuf" Version="3.25.1" />
1118
</ItemGroup>
1219

1320
<ItemGroup>

0 commit comments

Comments
 (0)