OpenTelemetry version upgrade

[vidourem]

Hello,
About the PR #8304

I'm working on the migration from OpenTracing to OpenTelemetry for the Debezium project debezium/debezium#4270. OpenTelemetry doesn't yet provide intrumentation for Kafka Connect and in the meantime I've developed a Kafka interceptor to enable distributed tracability. This interceptor extends the one provided by the OpenTelemetry project, which is still in alpha. As it happens, the OpenTelemetry interceptor package has changed as of version 1.23.0
Before 1.23.0: io/opentelemetry/instrumentation/kafkaclients/TracingProducerInterceptor.java

After 1.23.0: io/opentelemetry/instrumentation/kafkaclients/v2_6/TracingProducerInterceptor.java

As a result, I've decided to use vesrion 1.23.0 in Debezium to avoid problems when OpenTelemetry versions are upgraded in Debezium.

At the same time I'm updating the example project https://github.com/debezium/debezium-examples/tree/main/outbox

This project uses a Docker Strimzi image as a Kafka distribution, however the latest version of Strimzi embeds Opentelemetry in version 1.19.0 which doesn't allow the interceptor to work for the reason mentioned above. Do you plan to upgrade Opentelemetry to version 1.23.0+ in your next release?

[ppatierno]

Hi,
so the upcoming 0.36.0 will still have OpenTelemetry 1.19.0.
The RC1 was announced today and the GA will come later this week.
Anyway, we'll updated OpenTelemetry for sure in the next future release and we could switch to one of the latest 1.23.0+.

Regarding the specific interceptors you pointed out, I see just a change in the package where they live but no changes in the code.
Is it a problem for you, by using an older version? It has the same code AFAICS ... or did I miss anything?

[scholzj]

Isn't this mainly an OpenTelemetry issue? It seems they do not provide any degree of compatibility. Because regardless how you solve it today, next week there might be a new version that will rename the package yet again and breaks it again. So if you have two different projects which expect to run together and need to reference the class directly, they might never be able to meet on a single version that works for both.

In Cruise Control, these issues with Kafka classes are handled dynamically by checking if the class is in the classpath or not. I guess you can use that in Debezium as well. But one big difference is that Cruise Control does that as a hack for classes that are not a public API. The openTelemetry classes are public API, so having to do hacks like this there is weird. (The Cruise Control code is here: https://github.com/linkedin/cruise-control/blob/fb3bae481d0f42dfed8e2bb9d674cd7bfa7d09c8/cruise-control-metrics-reporter/src/main/java/com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter.java#L223-L266)

[ppatierno]

Thanks Jakub, I totally agree with you so nothing to add on that.
Anyway @vidourem as I said we'll update OpenTelemetry at some point but a new release with such an update won't happen soon.

[vidourem]

Hello,

I agree, creating separate modules for each version is not ideal. Following your comments, I have developed and tested a dynamic solution to handle version changes in the OpenTelemetry dependency.

In brief, this solution involves using introspection to determine at runtime which class is present in the classpath. A custom "ProducerInterceptor" is used to delegate the logic to the correct instances of OpenTelemetry's interceptor, depending on their presence.

I am aware of the potential risks associated with the instability of the alpha package of OpenTelemetry's interceptor. Initially, I thought a dynamic solution might be rejected by the community due to its complexity, but since the preference has been expressed for this approach, I will proceed in this way.

Understanding that Debezium aims to provide compatibility with Strimzi, I hope the team will appreciate this dynamic solution. I plan to commit this solution to the Debezium project in the coming days.

Thank you for your valuable feedback and suggestions.

[vidourem]

Hi @ppatierno , @scholzj

I pushed the evolution for dynamic management of the OpenTelemetry interceptor dependency on the PR.

During my tests I was surprised by the way Strimzi works by default. Is automatic kafka instrumentation disabled by default in Strimzi?

The interceptor that I have developed should replace the automatic instrumentation and unlike other kafka connect distributions it is not necessary to disable the Kafka instrumentation with Strimzi.

[ppatierno]

Is automatic kafka instrumentation disabled by default in Strimzi?

Yeah Strimzi doesn't have OpenTelemetry enable by default.