Documentation for parent-child association limitations

[dashpole]

Problem Statement

I discussed this with @mariomac @MrAlias and a few others at KubeCon.

There are two steps to propagating context with an outgoing request:

1. Determine which parent context (if any) needs to be added to the request
2. Modify the outgoing request to inject headers/metadata containing the parent context.

Both https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/main/devdocs/features.md and https://opentelemetry.io/docs/zero-code/obi/distributed-traces/#context-propagation-at-network-level do a good job covering step (2), but i'm not aware of any documentation describing step (1).

Without digging, it is easy for users to come to the conclusion that it works for all HTTP requests, which isn't correct. From skeptical people in the industry, i've also seen this perspective: "OBI must just be using thread id to correlate! That won't work for real applications!". In reality, we use heuristics that have limitations, but they are fairly sophisticated and (I hope?) work well for most common apps and frameworks.

Proposed Solution

Document how (1) works

From my very basic digging (please don't assume this is correct):

OBI uses heuristics to correlate incoming and outgoing requests:

• For all languages, OBI will recursively look for parent threads to see if a parent thread is associated with an incoming request.
• For languages that have a lightweight thread concept, like goroutines in Go. OBI recursively looks for parent lightweight threads.

The limitations of this (from what I can tell) are that anything that does any "thread pooling", and passes data between threads, rather than forking to children won't have context correlated by OBI.

Document frameworks that OBI works with

It would be even more helpful to document the language client/server frameworks that work with the approach we've taken. This is definitely more work to keep up-to-date, but would be very helpful to users, and would help convince more skeptical types that this isn't a toy feature. A big table or something would be a good place to start.

[dashpole]

@aabmass pointed out that this might have trouble with asyncio in python. It would be nice to include that if we had a table of frameworks.

[dashpole]

I'm happy to help write this, but if @grcevski or someone can list of the language-specific concepts we use for parent-child associations, that would be helpful. The rest we can probably iron our during code review.

[grcevski]

I'll start on this @dashpole this week and we can go from there and of course happy to get contributions on this. I should have some time tomorrow or Wednesday to get started on this.

Thanks for bringing up asyncio, I think we need to make a testcase to see if it works. Any code samples would help a lot, or an OSS project we can try. We have few mitigating techniques to aid with the thread based matching, so depending on what they do it might work.

One example of this is that we see if the original handler thread is probing if the outgoing connection is still live, before they submit the work for the async client handler. The Java standard library client does this and we are able to handle that. But others like Netty reactor don't do it.

[grcevski]

Hi @dashpole,

I made some initial write-up here open-telemetry/opentelemetry.io#8512. When you get a chance, let me know if this is what you expected to see and let's iterate on it. I'm happy to expand on any part that you think it's not fully covered.

[dashpole]

Yay! Thank you @grcevski