Trace time message spent in Kafka

[arkadyb]

Is there some way to trace time message spend in Kafka? I have async application where service 1 recives requests and stores message into Kafka. Service 2 - consumer, reads messages from Kafka and handles some other activities.

So, according to the manuals i found i was able to inject span context into Kafka headers in service 1, extract them into span context on service 2 site and start span with this info there. So the resulted graph (using jaeger) shows gap at the time message was in the bus. Is there some way to mark this time with respective Kafka span?

[yurishkuro]

Why do you need an extra span? The gap between producer and consumer spans is already measurable.

[yurishkuro]

Here's a trace from https://github.com/PacktPublishing/Mastering-Distributed-Tracing/tree/master/Chapter05. You can see time difference between send and receive spans.

[arkadyb]

Thats true, but it would be nice to be able plot a span specifying what the gap is - Kafka, PG, MySQL or what ever else.

[yurishkuro]

I cannot speak for other tracing systems, but in Jaeger we made a design decision that a span "belongs" to a single process, which is reflected in the data model, e.g. in properties like span.process.serviceName. This ensures that the start/end timestamps in the span are comparable, which is not true if they come from different processes / different hosts (clock skew).

I am going to move this issue to another repo, since it's not specific to Go API.

[guilherme-souza-lima]

hi @arkadyb

how did you inject the context into the kafka header?

and how did you extract it in the second microservice?