[issues/14372] support ListIterator for Kafka consumer client (2)

[patpatpat123]

( I accidentaly closed #14707, my bad)

Hello team,

This PR is to address issue #14372

Problem it is trying to solve:

As of today, Spark Structured Streaming Kafka Consumer, upon consuming a message with a valid W3C header with trace, is not continuing, nor propagating the trace on the Spark side.

Root cause analysis:

It seems Spark Structured Streaming Kafka Consumer is relying on listIterator C.f. https://github.com/apache/spark/blob/master/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala#L593

As of now, Kafka instrumentation does not support listIterator.

Proposed solution

This solution will add TracingListIterator<K, V> implements ListIterator<ConsumerRecord<K, V>> (TracingListIterator.java) which will allow the handling of listIterator (instrumentation/kafka/kafka-clients/kafka-clients-common-0.11/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/common/v0_11/internal/TracingList.java )

Thanks

Many thanks Lauri, Trask, and Jay for the help provided

[otelbot-java-instrumentation]

🔧 The result from spotlessApply was committed to the PR branch.

[patpatpat123]

Thank you @laurit for the journey 🙂

[patpatpat123]

Test result using Spark

[otelbot]

Thank you for your contribution @patpatpat123! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey.