How to recording 'waiting-for something' on telemetry

[lmolkova]

Extracted from #1192 (comment):

We're conditionally recording 'wait for connection' span to explain what's been going on in the HTTP stack.

There are other scenarios where it may be useful:

• waiting for lock
• polling queue and waiting for messages to come
• waiting for worker running CI/CD pipeline
• waiting for human response in workflows

It seems to be more of a pattern rather than specific convention. The general pattern:

• it should be recorded as an extra span. It could be verbose, but works with visualization tooling and start/stop events would be even more verbose. This span should be opt-in (out).
• kind is internal
• name should follow ~ wait_for {condition} pattern.
• span should have error.type to account for timeouts and error conditions
• it may have any additional attributes that are domain-specific.