[issues/14372] support ListIterator for Kafka consumer client

[patpatpat123]

Hello team,

This PR is to address issue #14372

Problem it is trying to solve:

As of today, Spark Structured Streaming Kafka Consumer, upon consuming a message with a valid W3C header with trace, is not continuing, nor propagating the trace on the Spark side.

Root cause analysis:

It seems Spark Structured Streaming Kafka Consumer is relying on listIterator C.f. https://github.com/apache/spark/blob/master/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala#L593

As of now, Kafka instrumentation does not support listIterator.

Proposed solution

This solution will add TracingListIterator<K, V> implements ListIterator<ConsumerRecord<K, V>> (TracingListIterator.java) which will allow the handling of listIterator (instrumentation/kafka/kafka-clients/kafka-clients-common-0.11/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/common/v0_11/internal/TracingList.java )

Thanks

Many thanks Lauri, Trask, and Jay for the help provided

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: patpatpat123 / name: patpatpat123 (af9522a, 82a3f72, 2e5ca63, ed43280, da485fd, a334c45)
• ✅ login: otelbot[bot] / name: otelbot (34851ce)

[otelbot-java-instrumentation]

🔧 The result from spotlessApply was committed to the PR branch.

[patpatpat123]

Test strategy:

As of today, the traces are separated. There are two separate traces, one from the producer, one from the consumer, without a clear relationship.

To test, I will create the Java agent containing this new code locally.

From there, run the Spark Structured Streaming Kafka Consumer.

Expected result:

We will be able to see the producer-consumer relationship

[patpatpat123]

My bad...