Duplicate trace/span fields in MDC when using micrometer-tracing with OpenTelemetry

[VladislavBakshanskij]

Describe the bug

When using micrometer-tracing, micrometer-tracing-bridge-otel and opentelemetry-spring-boot-starter together, the MDC contains duplicate trace and span fields - both in OpenTelemetry format (trace_id, span_id) and Micrometer format (traceId, spanId).

Steps to reproduce

1. Clone the example project: https://github.com/VladislavBakshanskij/opentelemetry-micrometer-duplicate-traceId
2. Run the Spring Boot application
3. Access http://localhost:8080
4. Observe the log output containing duplicate fields:

com.example.opentelemetry_micrometer.Controller - [traceId=fe7bbf4df9a89b8d622ab12a36405ecc, spanId=a8b5ecbbefddaf6c, trace_id=fe7bbf4df9a89b8d622ab12a36405ecc, trace_flags=01, span_id=a8b5ecbbefddaf6c] - Get method invoked

Expected behavior

MDC should contain trace and span fields in a single consistent format (either Micrometer or OpenTelemetry format).

Actual behavior

MDC automatically populates tracing fields in both formats simultaneously:

• Micrometer format: traceId, spanId
• OpenTelemetry format: trace_id, span_id, trace_flags

Javaagent or library instrumentation version

2.16.0

Environment

JDK: Oracle OpenJDK 21.0.2
OS: macOS sequoia 15.4.1
Spring boot: 3.4.5
micrometer-tracing: 1.4.5
micrometer-tracing-bridge-otel: 1.4.5

Additional context

No response

[laurit]

When using micrometer-tracing, micrometer-tracing-bridge-otel and opentelemetry-spring-boot-starter together, the MDC contains duplicate trace and span fields - both in OpenTelemetry format (trace_id, span_id) and Micrometer format (traceId, spanId).

Since micrometer tracing and otel are competing instrumentation solutions it is not surprising that there is overlap in the features they provide. If you are using logback then you can probably disable the otel mdc instrumentation by manually configuring the appender as described in https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-mdc-1.0/library#usage if you don't add any appenders into to otel appender then you shouldn't see the otel trace/span id values in any of your appenders.

[VladislavBakshanskij]

@laurit Yes, we're using Logback. There are no configurations from https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-mdc-1.0/library#usage in our setup.

Our Logback configuration:

<configuration>
  <springProperty scope="context" name="SpringAppName" source="spring.application.name" defaultValue="bio-project" />
  <springProperty scope="context" name="SpringAppPort" source="server.port" defaultValue="8080" />
  <property name="FILENAME" value="${SpringAppName}-${SpringAppPort}" />

  <appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="JSON-FILE">
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
      <providers>
        <pattern>
          <pattern>
            {
            "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss.nnnnnnnnXXX}",
            "level": "%level",
            "logger": "%50.50logger",
            "thread": "%15.15thread",
            "application": "${SpringAppName}",
            "port": "${SpringAppPort}",
            "message": "%d{yyyy-MM-dd HH:mm:ss.SSS} [%15.15thread] [%mdc] | %5.5level %50.50logger | - %msg %ex{full}",
            "mdc": "%mdc"
            }
          </pattern>
        </pattern>
      </providers>
    </encoder>

    <file>/var/log/app/${FILENAME}.json.log</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>
        /var/log/app/${SpringAppName}-%d{yyyy-MM-dd}-${SpringAppPort}.%i.json.log.gz
      </fileNamePattern>
      <!-- 2 days to keep -->
      <maxHistory>2</maxHistory>
      <totalSizeCap>500MB</totalSizeCap>
      <maxFileSize>250MB</maxFileSize>
    </rollingPolicy>
  </appender>

  <appender name="CMD" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>
        %d{yyyy-MM-dd HH:mm:ss.SSS} [%15.15thread] [%mdc] | %5.5level %50.50logger | - %msg%n
      </pattern>
    </encoder>
  </appender>

  <appender name="EUREKA_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>eureka.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>eureka.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>
    </rollingPolicy>
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>

  <logger name="com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver" level="INFO" additivity="false">
    <appender-ref ref="EUREKA_FILE" />
  </logger>

  <root level="info">
    <!--    <appender-ref ref="FILE"/>-->
    <appender-ref ref="JSON-FILE" />
    <appender-ref ref="CMD" />
  </root>
</configuration>

I believe it was applied through:

opentelemetry-java-instrumentation/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/internal/instrumentation/logging/LogbackAppenderApplicationListener.java

Line 58 in 310287f

LogbackAppenderInstaller.install((ApplicationEnvironmentPreparedEvent) event);

,
which is registered in:

opentelemetry-java-instrumentation/instrumentation/spring/spring-boot-autoconfigure/src/main/resources/META-INF/spring.factories

Line 19 in 310287f

io.opentelemetry.instrumentation.spring.autoconfigure.internal.instrumentation.logging.LogbackAppenderApplicationListener

[laurit]

@VladislavBakshanskij have a look at

opentelemetry-java-instrumentation/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/internal/instrumentation/logging/LogbackAppenderInstaller.java

Lines 23 to 28 in 310287f

	existingMdcAppender =
	findAppender(
	io.opentelemetry.instrumentation.logback.mdc.v1_0.OpenTelemetryAppender.class);
	if (existingMdcAppender.isPresent()) {
	initializeMdcAppenderFromProperties(
	applicationEnvironmentPreparedEvent, existingMdcAppender.get());

if configuration already contains the mdc adapter then it won't be added automatically.

[VladislavBakshanskij]

@laurit Yes, I've seen this code. The important point is that I haven't explicitly configured the opentelemetry-appender anywhere in my setup - it appears to have been added transitively and automatically registered itself in the application context (see screenshot).

I've tried excluding the opentelemetry-appender from dependencies, but this causes the application to throw numerous errors. This suggests there's some mandatory auto-configuration happening.

My key questions would be:

Is there a proper way to disable this automatic MDC injection without breaking the application?

Shouldn't this behavior be made opt-in rather than opt-out, given it's modifying core logging behavior?

Are there any configuration properties that could control this, short of completely removing the instrumentation?

The automatic injection creates problems when combining with other tracing implementations (like Micrometer Tracing in my case), and there should ideally be a clean way to manage this.

[laurit]

Yes, I've seen this code. The important point is that I haven't explicitly configured the opentelemetry-appender anywhere in my setup - it appears to have been added transitively and automatically registered itself in the application context (see screenshot).

This is intentional. The whole point of the starter is that it configures a bunch of instrumentations automatically.

Is there a proper way to disable this automatic MDC injection without breaking the application?

As described above you could add

  <appender name="OpenTelemetryMdc"
            class="io.opentelemetry.instrumentation.logback.mdc.v1_0.OpenTelemetryAppender">
  </appender>

to disable automatically adding the mdc adapter. Since the mdc adapter added this way does not wrap any other adapter it should result in the otel trace/span id not being visible to other appenders.

Shouldn't this behavior be made opt-in rather than opt-out, given it's modifying core logging behavior?

Usually users wish to see these without having to configure anything extra. As far as I know you are the first person who has an issue with trace/span id being added to mdc.

Are there any configuration properties that could control this, short of completely removing the instrumentation?

There isn't a configuration option for disabling this functionality. Most instrumentations in spring starter have a configuration option that allow disabling them, so similarly to how instrumentations are disabled with the agent you could use something like otel.instrumentation.logback-mdc.enabled=false. Unfortunately you can't disable logback-mdc this way with the spring starter because it is set up a bit differently from other instrumentations. cc @zeitlinger

The automatic injection creates problems when combining with other tracing implementations (like Micrometer Tracing in my case), and there should ideally be a clean way to manage this.

Currently we don't consider using our spring starter together with micrometer tracing a supported configuration. Since both of these libraries fill the same role some friction is not unexpected. Another strategy you could try is to use otel.instrumentation.common.logging.trace-id= traceId and otel.instrumentation.common.logging.span-id= spanId to change the name for the MDC keys. If these keys are already set in MDC by the time otel instrumentation triggers it won't do anything. See https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/logback/logback-mdc-1.0/javaagent/README.md

[zeitlinger]

Unfortunately you can't disable logback-mdc this way with the spring starter because it is set up a bit differently from other instrumentations.

This should actually work - see

opentelemetry-java-instrumentation/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/internal/instrumentation/logging/LogbackAppenderInstaller.java

Line 52 in f930598

applicationEnvironmentPreparedEvent, "otel.instrumentation.logback-mdc.enabled");

The spring starter aims to support all properties that the java agent does.

[laurit]

Unfortunately you can't disable logback-mdc this way with the spring starter because it is set up a bit differently from other instrumentations.

This should actually work - see

Nice, somehow missed it. Though with log4j disabling the context data provider won't work.

[VladislavBakshanskij]

@laurit @zeitlinger
Thank you for your assistance! The configuration properties you suggested resolved our issue completely:

otel.instrumentation.logback-appender.enabled=false
otel.instrumentation.logback-mdc.enabled=false

After applying these settings:

• The duplicate MDC fields (traceId/spanId vs trace_id/span_id) are no longer appearing
• The application maintains proper tracing functionality through Micrometer Tracing bridge
• No adverse effects or errors were observed

This solution provides the clean separation we needed between OpenTelemetry instrumentation and Micrometer Tracing's MDC management.

I'll close this issue as the configuration approach addresses the problem. Might be worth documenting this interoperability case for others combining Micrometer Tracing with OpenTelemetry auto-configuration.