span.record_exception when LLM calls fail

[dinmukhamedm]

Should we record exceptions on spans when an error occurs inside an LLM (or any other instrumented function) call?

This should be as simple as:

try:
    result = await wrapped_fn(*args, **kwargs)
    # other work, make sure we wrap functions in @dont_throw
except Exception as e:
    span.record_exception(e)
    # optionally span.set_status(StatusCode.ERROR, str(e))
    raise e # I might need to check whether `raise e` or  just `raise` is more correct in Python semantics
finally:
    span.end()

Reference: https://opentelemetry.io/docs/specs/otel/trace/exceptions/

We already do that in a few places:

• Langchain
• MCP
• My recent initial work on OpenAI responses

[nirga]

Yes definitely - I think it's a common practice. We can also set the error message as well in an attribute

[dinmukhamedm]

@nirga we can, but I think the OpenTelemetry way of doing things is handled in the SDKs by doing record_exception, which sends a span event (not to be confused with log events) with all the information in the event attributes.

Another tangential question: when we re-raise the exceptions, should we show in the traceback the fact that we did so in an automatic instrumentation? I've had a quick look at https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/ and can see examples of both (see below).

Here's what I mean:

def div_by_zero():
    return 1 / 0

try:
    div_by_zero()
except Exception as e:
    raise e  # this line will show up in the traceback, adding the path to the instrumentation lib
    raise  # this line will not show up in the traceback

Possible traceback with raise e:

Traceback (most recent call last):
  File "/Users/din/laminar/sandbox/py/ex.py", line 28, in <module>
    wrapper_with_raise_e()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/din/laminar/sandbox/py/ex.py", line 16, in wrapper_with_raise_e
    raise e  # This line is added to the traceback
    ^^^^^^^
  File "/Users/din/laminar/sandbox/py/ex.py", line 13, in wrapper_with_raise_e
    problematic_function()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/din/laminar/sandbox/py/ex.py", line 2, in problematic_function
    return 1 / 0  # This will raise ZeroDivisionError
ZeroDivisionError: division by zero

Similar traceback with bare raise. Notice, how here the instrumentation code that re-raises does NOT show up:

Traceback (most recent call last):
  File "/Users/din/laminar/sandbox/py/ex.py", line 20, in <module>
    wrapper_with_bare_raise()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/din/laminar/sandbox/py/ex.py", line 6, in wrapper_with_bare_raise
    problematic_function()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/din/laminar/sandbox/py/ex.py", line 2, in problematic_function
    return 1 / 0  # This will raise ZeroDivisionError
           ~~^~~
ZeroDivisionError: division by zero

More details, and an example I tried with Claude's help: https://claude.ai/share/02120653-181f-4b3e-a6bc-1c04cd4f0a5f

Bare raise examples in the wild:

• opentelemetry-instrumentation-asyncpg link
• opentelemetry-instrumentation-aiohttp-server link

Re-raise using raise e or raise e.with_traceback(e.__traceback__) in the wild:

• opentelemetry-instrumentation-aws-lambda link
• opentelemetry-instrumentation-grpc link

[nirga]

I'd go with bare raise - the wrapper should be transparant IMO.
Reg. the exception - you're right, but we should also mark the span itself as failed, right? And then there's an error.type attribute we could also use - #3009

[dinmukhamedm]

Yeah, makes sense. There goes the first one then: #3067. I'll add more gradually