Fix Serilog direct log submission failing when writing unknown custom properties (#8010)

## Summary of changes

This fixes a `JsonWriterException` that we were hitting when attempting
to serialize custom properties in Serilog log messages when we had
Direct Log Submission enabled. This would then result in direct log
submission to stop working.

## Reason for change

When writing a custom property in Serilog we only write the JSON when
Debug logging is enabled, when it isn't enabled we actually end up
putting the writer into an invalid state, we then hit the following
exception:


```
2025-12-23 16:04:16.209 -05:00 [ERR] An error occured sending logs to Datadog Datadog.Trace.Vendors.Newtonsoft.Json.JsonWriterException: Token PropertyName in state Property would result in an invalid JSON object. Path ''.
   at Datadog.Trace.Vendors.Newtonsoft.Json.JsonWriter.AutoComplete(JsonToken tokenBeingWritten) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\Vendors\Newtonsoft.Json\JsonWriter.cs:line 880
   at Datadog.Trace.Vendors.Newtonsoft.Json.JsonWriter.InternalWritePropertyName(String name) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\Vendors\Newtonsoft.Json\JsonWriter.cs:line 1760
   at Datadog.Trace.Vendors.Newtonsoft.Json.JsonTextWriter.WritePropertyName(String name, Boolean escape) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\Vendors\Newtonsoft.Json\JsonTextWriter.cs:line 291
   at Datadog.Trace.Logging.DirectSubmission.Formatting.LogFormatter.FormatLog[T](StringBuilder builder, T& state, DateTime timestamp, String message, Nullable`1 eventId, String logLevel, Exception exception, FormatDelegate`1 renderPropertiesDelegate) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\Logging\DirectSubmission\Formatting\LogFormatter.cs:line 335
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.Serilog.DirectSubmission.Formatting.SerilogLogFormatter.FormatLogEvent(LogFormatter logFormatter, StringBuilder sb, ILogEvent logEvent) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\ClrProfiler\AutoInstrumentation\Logging\Serilog\DirectSubmission\Formatting\SerilogLogFormatter.cs:line 26
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.Serilog.DirectSubmission.SerilogDirectSubmissionLogEvent.Format(StringBuilder sb, LogFormatter formatter) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\ClrProfiler\AutoInstrumentation\Logging\Serilog\DirectSubmission\SerilogDirectSubmissionLogEvent.cs:line 25
   at Datadog.Trace.Logging.DirectSubmission.Sink.DirectSubmissionLogSink.EmitBatch(Queue`1 events) in C:\Users\steven.bouwkamp\source\repos\dd-trace-dotnet2\tracer\src\Datadog.Trace\Logging\DirectSubmission\Sink\DirectSubmissionLogSink.cs:line 103
 { MachineName: ".", Process: "[100360 dotnet]", AppDomain: "[1 LogsInjection.Serilog]", AssemblyLoadContext: "\"\" Datadog.Trace.ClrProfiler.Managed.Loader.ManagedProfilerAssemblyLoadContext #1", TracerVersion: "3.35.0.0" }
```

From what I can tell this causes direct log submission to stop working.

I've seen this in Error Tracking and it may be related to a customer
escalation regarding issues with Serilog direct log submission to
seemingly stop working.

## Implementation details

## Test coverage

I've reproduced this, but it is strange, I guess Serilog doesn't
actually support this? I don't understand completely what was going on,
but if I added a custom property and enriched all logs with it but did
_not_ have direct log submission enabled there would be no logs found in
the tests but they did make it to the file. 😕

## Other details
<!-- Fixes #{issue} -->

I only toggled this custom property on when direct log submission is
enabled due to the above Test Coverage note.


<!--  ⚠️ Note:

Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.

MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->

Author: bouwkast

tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Logging/Serilog/DirectSubmission/Formatting/SerilogLogFormatter.cs

@@ -1,4 +1,4 @@
-// <copyright file="SerilogLogFormatter.cs" company="Datadog">
+// <copyright file="SerilogLogFormatter.cs" company="Datadog">
 // Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
@@ -107,10 +107,12 @@ private static void FormatLogEventPropertyValue(JsonTextWriter writer, object va
                 return;
             }
 
+            // Always write null for unknown types to maintain valid JSON
+            LogFormatter.WriteValue(writer, value: null);
+
             if (Log.IsEnabled(LogEventLevel.Debug))
             {
-                Log.Debug("Unknown Serilog LogEventPropertyValue '{Type}': skipping in log message", value.GetType());
-                LogFormatter.WriteValue(writer, value: null);
+                Log.Debug("Unknown Serilog LogEventPropertyValue '{Type}': writing null value", value.GetType());
             }
         }
 

tracer/test/test-applications/integrations/LogsInjection.Serilog/Program.cs

@@ -15,6 +15,42 @@
 
 namespace LogsInjection.Serilog
 {
+    /// <summary>
+    /// Custom LogEventPropertyValue that doesn't match any of the duck types
+    /// (ScalarValueDuck, SequenceValueDuck, StructureValueDuck, DictionaryValueDuck).
+    /// This reproduces the bug where FormatLogEventPropertyValue doesn't write a value
+    /// when debug logging is disabled.
+    /// </summary>
+    public class CustomPropertyValue : LogEventPropertyValue
+    {
+        private readonly object _value;
+
+        public CustomPropertyValue(object value)
+        {
+            _value = value;
+        }
+
+        public override void Render(TextWriter output, string format = null, IFormatProvider formatProvider = null)
+        {
+            output.Write(_value?.ToString() ?? "null");
+        }
+    }
+
+    /// <summary>
+    /// Custom enricher that adds a property with our custom value type.
+    /// This will trigger the bug in SerilogLogFormatter.FormatLogEventPropertyValue
+    /// when debug logging is disabled.
+    /// </summary>
+    public class CustomPropertyEnricher : ILogEventEnricher
+    {
+        public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
+        {
+            // This creates a property with our custom value type
+            var customProperty = new LogEventProperty("CustomField", new CustomPropertyValue("test-value"));
+            logEvent.AddPropertyIfAbsent(customProperty);
+        }
+    }
+
     public static class Program
     {
         public static int Main(string[] args)
@@ -81,6 +117,17 @@ public static int Main(string[] args)
                                .WriteTo.Logger(lc => lc.WriteTo.Console());
             }
 
+            // Only enable the custom enricher when direct log submission is active
+            // This tests handling of unknown Serilog property types in SerilogLogFormatter
+            // without breaking regular Serilog file sinks (which can't serialize custom types)
+            var directLogSubmissionEnabled = !string.IsNullOrEmpty(
+                Environment.GetEnvironmentVariable("DD_DIRECT_LOG_SUBMISSION_ENABLED_INTEGRATIONS"));
+
+            if (directLogSubmissionEnabled)
+            {
+                configuration = configuration.Enrich.With(new CustomPropertyEnricher());
+            }
+
             var log = configuration.CreateLogger();
 
 #if SERILOG_2_12