Retrieval Span Support

[JWinermaSplunk]

Fixes #2907

Changes

Add retrieval span support to db spans

Important

Pull requests acceptance are subject to the triage process as described in Issue and PR Triage Management.
PRs that do not follow the guidance above, may be automatically rejected and closed.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• Links to the prototypes or existing instrumentations (when adding or changing conventions)

[lmolkova]

Retrieval is in many cases based on the database retrieval. e.g. postgreds or MongoDB instrumentation has no knowledge that it's used in the context of GenAI application.

So retrieval in a general case is just a DB call with semantics described in https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/README.md

There is a question of whether search engines are databases or should be covered by a separate or additional set of the conventions - #1869 - this is where OpenAI retrieval API should probably belong.

[lmolkova]

Adding more context from GenAI SIG call:

• langchain, llamaindex, haystack offer retriever abstraction.
• the question is: should corresponding implementations be instrumented and if so, which conventions they should follow

My take:

Reading through lanchain, llamaindex, haystack docs, retriever is in most cases is an thin layer on top of a database or a search client which may be instrumented using database conventions and/or hypothetical search conventions.

Retrievers could be more complicated and combine multiple source of data or perform additional logic, in these cases, the spans they emit might be significantly different than underlying DB calls and then multiple layers may be instrumented at the same time. In the case of thin layer abstraction / wrapper, having two spans does not improve observability, but increases costs and noise.

When it comes to abstractions, there is a classic problem of instrumentation layers (ORM vs database spans, lanchain LLM vs underlying model-client spans, etc): both layers could be instrumented and there are pros and cons (DB layer has more low-level info, framework layer represents caller perspective better). The duplication is a common problem. Solutions may include:

• users decide to enable/disable corresponding instrumentation (by installing instr library or by enabling/disabling retrival spans on lanchain / llamaindex / etc instrumentation)
• suppressing internal DB spans in scope of higher level retrieval span (similar to http suppression or more broadly Instrumentation layers and suppressing duplicates oteps#172)

I think path forward for this PR:

• separate retrieval from GenAI domain. Most of the attributes defined here can be generic db or search attributes. Please follow the discussion in Should search engines follow database semantic conventions? #1869
• lanchain / llamaindex / etc instrumentation would cover retrieval following that db or search conventions and would provide an option to disable their retrieval instrumentation (e.g. when user prefers to use underlying existing DB client instrumentation)

[lmolkova]

Also related #1231

[JWinermaSplunk]

Hi @lmolkova,

Here are a few examples from trace loop and our instrumentation of proposed retrieval spans, also updated in the google doc linked to the issue
https://docs.google.com/document/d/1DVE3Ht686nuxww-Z1JNyy2VInn9LeCJQYw8NffbjGtg/edit?tab=t.0.

I believe, we also discussed that we are fine moving retrieval spans from the genai to db spaces, but would the optionality to have or enable/disable genai attributes be possible as well, similar to enabling/disabling retrievals as a whole? So that retrievals could parent a db + embedding operation.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

This PR contains changes to area(s) that do not have an active SIG/project and will be auto-closed:

• database

Such changes may be rejected or put on hold until a new SIG/project is established.

Please refer to the Semantic Convention Areas
document to see the current active SIGs and also to learn how to kick start a new one.