Pooled objects in attibutes (working with Microsoft.Extensions.ComplianceRedaction)

[NatMarchand]

Bug Report

Currently working on net8.0

    <PackageReference Include="Microsoft.Extensions.Compliance.Redaction" Version="8.1.0" />
    <PackageReference Include="Microsoft.Extensions.Telemetry" Version="8.1.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.7.0" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.7.0" />

Symptom

When using redaction and batch otlp protocol, redacted values are not stored.

What is the expected behavior?

Expected: the redacted value

What is the actual behavior?

Actual: no value store (the key is even not sent)

Reproduce

Little hard to reproduce as it needs a sink to send logs to. However, I've found the issue: when using redaction, the redacted values are stored as JustInTimeRedactor (see) to avoid allocations. When using batch exporter, as it might take time to send telemetry and as JustInTimeRedactor is pooled, the value can either not be available or reused by something else.

Additional Context

I've found a kind of workaround which is adding a Processor which "freezes" pooled values.

public class FreezePooledValuesProcessor : BaseProcessor<LogRecord>
{
    public override void OnStart(LogRecord data)
    {
        base.OnStart(data);
    }

    public override void OnEnd(LogRecord data)
    {
        if (data.Attributes is not null)
        {
            data.Attributes = data.Attributes
                .Select(kvp => kvp.Value is IResettable resettable ? KeyValuePair.Create<string, object?>(kvp.Key, resettable.ToString()) : kvp)
                .ToList();
        }

        base.OnEnd(data);
    }
}

[cijothomas]

You can use a OTEL Collector and OTLP Exporter to show the repro. You may use this example app - https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/examples/Console/TestOtlpExporter.cs#L16

The SDK's ILogger integration is expected to copy all attributes in the sync path itself, so if this is not working, it is likely a bug. But need to see a minimal repro to confirm.

[NatMarchand]

Here's a repro:
https://github.com/NatMarchand/repro-dotnet-log-redactor/blob/main/ConsoleApp1/Program.cs
On my computer, I often (not every time) get a NullReferenceException in Microsoft.Extensions.Telemetry.dll!Microsoft.Extensions.Logging.JustInTimeRedactor.ToString
as the inner _redactor is null because the JustInTimeRedactor is reset when it gets back to the pool.
The issue is on a separated thread OpenTelemetry-BatchExportProcessor-OtlpLogExporter and the stack trace is the following:

Microsoft.Extensions.Telemetry.dll!Microsoft.Extensions.Logging.JustInTimeRedactor.ToString(string format, System.IFormatProvider formatProvider) Line 53
Microsoft.Extensions.Telemetry.dll!Microsoft.Extensions.Logging.JustInTimeRedactor.ToString() Line 45
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Internal.TagTransformer<OpenTelemetry.Proto.Common.V1.KeyValue>.TryTransformTag(System.Collections.Generic.KeyValuePair<string, object> tag, out OpenTelemetry.Proto.Common.V1.KeyValue result, int? maxLength) Line 73
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.OtlpLogRecordTransformer.ToOtlpLog(OpenTelemetry.Logs.LogRecord logRecord) Line 152
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.OtlpLogRecordTransformer.BuildExportRequest(OpenTelemetry.Proto.Resource.V1.Resource processResource, OpenTelemetry.Batch<OpenTelemetry.Logs.LogRecord> logRecordBatch) Line 42
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OtlpLogExporter.Export(OpenTelemetry.Batch<OpenTelemetry.Logs.LogRecord> logRecordBatch) Line 55
OpenTelemetry.dll!OpenTelemetry.BatchExportProcessor<OpenTelemetry.Logs.LogRecord>.ExporterProc()	

[cijothomas]

On my computer, I often (not every time) get a NullReferenceException in Microsoft.Extensions.Telemetry.dll!Microsoft.Extensions.Logging.JustInTimeRedactor.ToString

This is something to be reported to the owners of Microsoft.Extensions.Telemetry itself.

The issue is on a separated thread OpenTelemetry-BatchExportProcessor-OtlpLogExporter and the stack trace is the following:

Are you seeing this exception escaping or its caught by OTLPExporter and proceeds without crashing the app?

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/Shared/TagTransformer.cs#L63-L74 is the place where the exception is thrown (and caught correctly as well). I am guessing the issue is with the JustInTimeRedactor itself, that it is incorrectly assuming the state can remain after the Log call is over..... (I don't have cycles to take a deeper look to confirm this...)

@CodeBlanch who fixed similar issue in the OTel SDK itself.

[cijothomas]

I removed the bug tag. When I saw "pool", I initially assumed the LogRecord pooling in this repo, but since then, it is clear that you were referring to the pooling done inside JustInTimeRedactor.

[NatMarchand]

This is something to be reported to the owners of Microsoft.Extensions.Telemetry itself.

I'll try to report the issue on the other repo, but as you said at first glance it's ok to assume that after leaving Log in OpenTelemetryLogger the attributes are safe to collect as they should be "frozen" or projected to something safe. Like I do in my workaround.

Are you seeing this exception escaping or its caught by OTLPExporter and proceeds without crashing the app?

I catch it because for debugging purpose I configured VS to catch any CLR exception. With regular settings, it indeed doesn't break. However, in such case I don't have the attribute flowing in my log pipeline.

[joperezr]

Hey @cijothomas, thanks for looking into this. As explained by @geeknoid in this comment we believe that the issue lies in OTel here as it shouldn't try to call ToString on object references that it keeps around as it cannot be guaranteed that doing so is safe (objects might be disposed, nulled out, or simply in a state that is no longer valid).

[cijothomas]

Hey @cijothomas, thanks for looking into this. As explained by @geeknoid in this comment we believe that the issue lies in OTel here as it shouldn't try to call ToString on object references that it keeps around as it cannot be guaranteed that doing so is safe (objects might be disposed, nulled out, or simply in a state that is no longer valid).

As suggested here, OTel SDK is expected to read everything off the state in the sync path itself. If it is not working, then it is a bug in OTel .NET sdk. Would you make a minimal repro app to confirm if this is indeed an issue in OTel .NET?
OTel code section doing the buffering: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/Logs/LogRecord.cs#L475-L485

[NatMarchand]

Unless I'm missing something, I don't see how the BufferLogAttributes() method can prevent the issue.

I've read the associated issue mentioned in the method comment but we are not in the same case.
In 2905, the AttributesData property needed to be enumerated and copied because it was of type HostingRequestStartingLog and the enumeration need to be sync. However all the object in the KeyValuePair<string,object> are objects which does not have lifetime scope. Most of them are just good ol' primitive types.

Here the difference is that it is the Value property of KeyValuePair<string,object> which is scoped and the method don't freeze or copy its state.

Something like this would prevent the issue instead:

private void BufferLogAttributes()
{
    var attributes = this.AttributeData;
    if (attributes == null || attributes == this.AttributeStorage)
    {
        return;
    }

    var attributeStorage = this.AttributeStorage ??= new List<KeyValuePair<string, object?>>(attributes.Count);

    foreach(var (key, value) in attributes)
    {
        if(value is IDisposable || value is IResettable)
        {
            value = value.ToString();
        }
        attributeStorage.Add(KeyValuePair<string,object?>.Create(key, value));
    }

    this.AttributeData = attributeStorage;
}