Simpler events for recording (local) GenAI function tool calling as generalized function tracing events

[michaelsafyan]

Area(s)

area:gen-ai

What's missing?

Although there are existing events for tool calling (gen_ai.assistant.message, gen_ai.tool.message), these are more geared towards the Gen AI model's view of what tools should be invoked/called (rather than from the standpoint of instrumenting the tool that ultimately got invoked). Additionally, these are somewhat complicated to generate and may not always be available (if the tool calling is deeply nested inside the logic of the client).

Where the model invokes local tools, it would be useful to be able to implement recording the input/output of those tool calls by intercepting the tool registration and having a somewhat simpler interface for tracing those calls. I also think that some of this modelling could be more general than just Gen AI.

I propose that we have a more generalized representation of function tracing to events that could be used.

Describe the solution you'd like

I'd like for there to be a more general set of events associated with function tracing:

Function Start

Represents a function being invoked and provides a way to record the parameters:

• Attributes:
  • event.name: "function_start"
  • code.function.name: the name of the function that was called
  • positional_argument_count: the count of positional args
  • keyword_argument_count: the count of keyword or optional arguments
• Body:
  • positional_arguments: an array of the positional arguments
  • keyword_arguments: a dictionary of the keyword arguments

Function End

Represents the result of a function being invoked.

• Attributes:
  • event.name: "function_end"
  • code.function.name: the name of the function that was called
  • error.type: if there was an error
• Body:
  • result: JSON representation of the (succesful) result
  • error: JSON representation of the (error) result

[michaelsafyan]

There is a draft PR related to Google Gen AI SDK instrumentation that illustrates how this might be used:

• feat(google-genai): add instrumentation to supplied tool call functions opentelemetry-python-contrib#3446

[lmolkova]

@michaelsafyan agree it would be useful to define general-purpose function call semantic conventions. Any reason to define events though? We can capture params and result as arguments on the function call span.

Some relevant data points:

1. OTel Java supports something like this

   @WithSpan
   public void myMethod(@SpanAttribute("parameter1") String parameter1,
       @SpanAttribute("parameter2") long parameter2) {
       <...>
   }

   https://opentelemetry.io/docs/zero-code/java/agent/annotations/#adding-attributes-to-the-span-with-spanattribute

2. We should follow mapping from arbitrary object to OTel AnyValue for params and result

3. We added something similar in one of the Azure AI SDKs as an experiment https://github.com/Azure/azure-sdk-for-python/blob/ebf29c4252308b77788b65ccf69482e9eeef925b/sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/_trace_function.py. It records code.function.parameter.{key} and code.function.return.value attributes on the function call span. Once/if complex attributes on spans are supported, these attributes could become complex and we could do JSON in the meantime.

   Both of those attribute should be opt-in and we'll need to figure out a friendly way to let users opt-in.

   E.g. in python, I'd assume we'd define a new decorator that either allows you to opt-in into additional attributes or it's obvious from its name, like trace_with_parameters. It'd also be nice to have a decorator that can work with global tracer instead of @tracer.start_as...

/cc @trask - I recall you mentioned that something similar came up in Java, wonder if you have any thoughts on generic conventions/instrumentation for capturing function params and return value.

[lmolkova]

also cc @open-telemetry/semconv-code-attribute-approvers if they have any thoughts